Add a queued_htcondor manager on the htcondor2 bindings, shared with Galaxy

[jmchilton]

This PR description was drafted and posted by Claude (AI assistant) on jmchilton's behalf.

Summary

Adds a queued_htcondor manager built on HTCondor's version 2 Python bindings, and puts the HTCondor mechanics in pulsar.managers.util.condor.htcondor — a module shared byte-for-byte with Galaxy's htcondor job runner.

Galaxy-side counterpart branch: https://github.com/jmchilton/galaxy/tree/htcondor_pulsar (deliberately not a PR yet — it needs to land alongside this).

• Shared module (util/condor/htcondor.py + htcondor_helper.py): the htcondor2 import, schedd clients, event-log parsing, hold and failure classification, walltime/memory parsing. It knows nothing about either application's job model — callers map its vocabulary (EventLogSummary, HOLD_REASON_*, FAILURE_*) onto their own states.
• queued_htcondor manager: maps that vocabulary onto Pulsar's status vocabulary. Never shells out to condor_submit/condor_rm, and reads the event log via htcondor2.JobEventLog rather than scraping its text (contrast queued_condor).
• request_walltime is translated into a periodic_hold expression; walltime and out-of-memory holds are classified and reported as failed with a message telling the user which limit to raise.
• Two schedd client flavours: in-process, plus a helper subprocess for destinations that override CONDOR_CONFIG (htcondor2 reads its config once per process, so those cannot share a process).
• New htcondor extra in setup.py; docs for the manager and the shared package.

Stacking

Based on #485 (terminal-job-statuses), so the diff shown here is just the HTCondor work. #485 should go in first.

The dependency is real, not just convenience: the escalations below report FAILED, which only became terminal in StatefulManagerProxy in #485.

Why the code is shared rather than reimplemented

Galaxy's runner and this manager need identical answers to the same questions — is HoldReasonCode=34 an OOM hold, does TermSignal=9 mean the job was killed, has the event log gone missing. Divergence there means the same cluster event is reported two different ways depending on whether the job went through Pulsar. The module is mirrored (identical bytes in both repos) rather than depended on, because Pulsar cannot take a hard dependency on Galaxy.

Two constraints follow, both noted in the module docstring:

• It must stay Python 3.7-parseable — hence from __future__ import annotations for the X | None annotations, and no walrus operator.
• Changes have to be applied to both copies. diff between the two paths should be empty.

Behavior notes

Worth a reviewer's attention:

• Escalations report FAILED, not LOST. StatefulManagerProxy treats LOST as possibly transient — a manager also returns it for a job whose external id has not been recovered yet — so a job reported LOST is never deactivated. These escalations only fire once the grace period has elapsed, at which point the job really is over.
• Grace periods are wall-clock, not poll counts. The polling interval is configurable in both applications, and a Pulsar job is polled by both the Pulsar monitor and Galaxy, so a count of cycles has no fixed duration. 30s for repeated status-check errors, 60s for an absent event log (NFS attribute caching defaults to 60s, so a log that exists may not be visible sooner). Both are judgment calls — happy to be told the right numbers for a real pool.
• JOB_TERMINATED with TermSignal=9 is FAILED, not COMPLETE. An OOM-killed wrapper may never write its exit-code file, so COMPLETE would present a killed job as a successful one. get_status short-circuits cancelled jobs, so a SIGKILL reaching this branch was not requested by us.
• ManagerInterface.shutdown gains a concrete no-op, replacing three separate workarounds for managers that do not define it (a getattr probe, a try/except AttributeError in ManagerProxy, and a try/except Exception in base_drmaa). That last one is a small behavior change: it is now try/finally, so the DRMAA session still always closes, but a genuine error from the parent shutdown propagates instead of being swallowed.

Validation

• tox -e test-unit — 311 passed, 83 skipped
• 31 new unit tests in test/manager_htcondor_test.py, driven by a fake htcondor2 module whose JobEventType integers match the real library. The fake's event log is consume-once like the real one, so the tests exercise that constraint rather than papering over it. An injectable clock drives the escalation grace periods without sleeping.
• flake8 and mypy clean
• Galaxy side: 93 passed (the 4 remaining errors there need a real HTCondor pool)
• Shared module verified byte-identical between the two repos

Relationship to queued_condor

Pulsar already has a Condor manager, and the two are less independent than they look: they share util/condor/__init__.py — submission_params(), build_submit_description(), the submit_ prefix convention, even the job_condor.log filename. They diverge after submission, because summarize_condor_log() is a text scrape tied to the CLI's log format with no bindings equivalent.

The difference that matters when choosing between them is the status vocabulary. queued_condor reports only queued/running/complete, and several failures reach the client as success:

• no event log → complete (e.g. a log that was never written because the filesystem was full at submit time)
• JOB_ABORTED (009) → complete, indistinguishable from JOB_TERMINATED (005)
• JOB_HELD (012) is never scanned, so a job held for exceeding memory or wall time sits at queued indefinitely
• SHADOW_EXCEPTION (007) → running

docs/job_managers.rst now presents both under a single HTCondor heading with that comparison and a recommendation — prefer queued_htcondor unless htcondor2 cannot be installed — instead of leaving them as two unrelated sections.

One robustness difference is worth knowing, since it explains a design choice here: the text scrape re-derives full state from the file on every poll, so it needs no bookkeeping and survives a restart for free. JobEventLog.events() is consume-once per handle, which is why this manager carries running/held_count state and why the terminal latch from #485 matters to it.

Drive-by fixes to queued_condor

Two bugs the new manager avoids by construction, fixed in the old one rather than left behind:

• launch() did submit_params.update(self.submission_params) against a submit_params={} default argument, mutating the shared default. Two managers with different configurations would submit each other's settings on any launch that omitted explicit params. Regression test added — there was no unit coverage for this manager at all, only an integration test gated on condor_submit being present.
• open(log_path, 'w') left the handle to the garbage collector.