Fix DRMAA FAILED job state reported as COMPLETE - #482
Open
gkr0110 wants to merge 1 commit into
Open
Conversation
BaseDrmaaManager._get_status_external() mapped JobState.FAILED to status.COMPLETE, with a comment acknowledging it should map to FAILED instead. This means any job that fails at the Slurm/DRMAA submission or execution level (e.g. rejected for a missing/invalid account) is reported back to Galaxy as a successful completion, with no output and no indication anything went wrong. status.FAILED already exists and is treated identically to status.COMPLETE by status.is_job_done(), so this only affects the terminal status Galaxy receives, not job-done detection.
This was referenced Aug 18, 2026
Open
Member
|
Amazing - thank you for such a complete description of the issue, testing, etc.. and thanks for the fix! My agents found some polish they wanted to do at the higher layers and other managers based on this fix. Merging that PR should merge this one too - thanks again so much - great work! -John |
Contributor
Author
|
@jmchilton - It’s a tiny tweak, but I think it could be helpful for others. I spent a couple of hours trying to figure out what was breaking.. We are building a new Galaxy instance for Switzerland, so testing out the Pulsar workloads heavily. So expect more PRs/Issues 😄 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BaseDrmaaManager._get_status_external()mapsJobState.FAILEDtostatus.COMPLETE, with a comment already acknowledging this is wrong:Any job that fails at the DRM submission or execution level (e.g. rejected
by Slurm for a missing/invalid account, or any other DRM-side failure) is
reported back to the Galaxy client as a successful completion, with no
output and no indication anything went wrong. We hit this concretely: jobs
submitted via
queued_drmaato Slurm throughslurm-drmaathat gotCANCELLEDfor lacking--accountwere reported to Galaxy ascompletewith an empty output directory.
status.FAILEDalready exists andstatus.is_job_done()treatsFAILEDand
COMPLETEidentically for job-done detection, so this change onlyaffects the terminal status surfaced to the client, not job-done detection.
Test plan
git blamethis mapping (and stale comment) has beenunchanged since it was introduced.
test/for references to_get_status_external/JobState.FAILED-- none found, so no existing tests assume the old(incorrect) behavior.
slurm-drmaa: a jobsubmitted without an account is
CANCELLEDby Slurm and previouslysurfaced to the client as
complete; with this change it correctlysurfaces as
failed.