Skip to content

Fix DRMAA FAILED job state reported as COMPLETE - #482

Open
gkr0110 wants to merge 1 commit into
galaxyproject:masterfrom
gkr0110:fix-drmaa-failed-status
Open

Fix DRMAA FAILED job state reported as COMPLETE#482
gkr0110 wants to merge 1 commit into
galaxyproject:masterfrom
gkr0110:fix-drmaa-failed-status

Conversation

@gkr0110

@gkr0110 gkr0110 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

BaseDrmaaManager._get_status_external() maps JobState.FAILED to
status.COMPLETE, with a comment already acknowledging this is wrong:

JobState.FAILED: status.COMPLETE,  # Should be a FAILED state here as well

Any job that fails at the DRM submission or execution level (e.g. rejected
by Slurm for a missing/invalid account, or any other DRM-side failure) is
reported back to the Galaxy client as a successful completion, with no
output and no indication anything went wrong. We hit this concretely: jobs
submitted via queued_drmaa to Slurm through slurm-drmaa that got
CANCELLED for lacking --account were reported to Galaxy as complete
with an empty output directory.

status.FAILED already exists and status.is_job_done() treats FAILED
and COMPLETE identically for job-done detection, so this change only
affects the terminal status surfaced to the client, not job-done detection.

Test plan

  • Confirmed via git blame this mapping (and stale comment) has been
    unchanged since it was introduced.
  • Grepped test/ for references to _get_status_external /
    JobState.FAILED -- none found, so no existing tests assume the old
    (incorrect) behavior.
  • Reproduced against a real Slurm cluster via slurm-drmaa: a job
    submitted without an account is CANCELLED by Slurm and previously
    surfaced to the client as complete; with this change it correctly
    surfaces as failed.

BaseDrmaaManager._get_status_external() mapped JobState.FAILED to
status.COMPLETE, with a comment acknowledging it should map to FAILED
instead. This means any job that fails at the Slurm/DRMAA submission or
execution level (e.g. rejected for a missing/invalid account) is reported
back to Galaxy as a successful completion, with no output and no
indication anything went wrong.

status.FAILED already exists and is treated identically to status.COMPLETE
by status.is_job_done(), so this only affects the terminal status Galaxy
receives, not job-done detection.
@jmchilton

Copy link
Copy Markdown
Member

Amazing - thank you for such a complete description of the issue, testing, etc.. and thanks for the fix! My agents found some polish they wanted to do at the higher layers and other managers based on this fix. Merging that PR should merge this one too - thanks again so much - great work! -John

@gkr0110

gkr0110 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

@jmchilton - It’s a tiny tweak, but I think it could be helpful for others. I spent a couple of hours trying to figure out what was breaking..

We are building a new Galaxy instance for Switzerland, so testing out the Pulsar workloads heavily. So expect more PRs/Issues 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants