fix(agent): coherent job detail for array, requeued, and reused job-ids

[pramasoul]

TL;DR — The job-detail view returns incorrect, internally-contradictory data for any job whose id maps to more than one record: array jobs (each task is its own record), requeued/preempted jobs (multiple lifecycle records), and reused job-ids (slurmdbd is keyed by a recyclable id). SlurmrestdFiltered.job() takes the first element of the slurmctld and slurmdbd job lists and blindly dict.update()-merges them, so it can splice the live state of one task onto the accounting of an entirely unrelated job (and IndexError-crashes on jobs that have aged out of slurmctld). This change selects coherent records instead — the live task plus the accounting record for that same task/run — and handles the aged-out case cleanly. Single-record jobs (the common case) are unaffected.

Problem

SlurmrestdFiltered.job() builds the job-detail object by taking the first element ([0]) of the slurmdbd and slurmctld job lists and blindly merging them:

result = self._acctjob(job_id)             # _request("slurmdb", f"job/{id}", "jobs")[0]
result.update(self._ctldjob(job_id, ...))  # _request("slurm",  f"job/{id}", "jobs")[0]

This assumes GET /job/{id} returns exactly one record. It does not whenever a job-id maps to multiple records:

• array jobs — each task is a separate record (tasks also carry distinct internal job-ids);
• requeued / preempted / restarted jobs — multiple lifecycle records;
• reused job-ids — slurmdbd is keyed by job-id, which Slurm recycles.

[0] then picks an arbitrary record and dict.update() mashes two unrelated records into one. We hit this in production: GET /job/<id> showed a job displayed as RUNNING (the live array task) carrying the end-time / exit-code of a different job from a month earlier that had reused that id — a self-contradictory detail page. (For jobs aged out of slurmctld, _ctldjob's [0] can also raise IndexError rather than returning a clean not-found.)

Fix

Select coherent records instead of [0]:

• the slurmctld record (most-recently-started still-queued task) is authoritative for current state;
• the slurmdbd record merged in is the one describing the same task/run (matched by internal job-id + array task-id, latest start to skip requeue duplicates);
• accounting is optional: if the slurmdb query errors or 404s (no slurmdbd), job detail is served from slurmctld alone (the previous code seeded from slurmdbd with no fallback);
• if the job has aged out of slurmctld, fall back to the most-recent accounting record — and no IndexError.

Single-record jobs (the common case) are unaffected: one record in each list → identical result.

Tests

Adds four tests to TestSlurmrestdFiltered (mocking _request keyed on component, so no version-specific fixtures): multi-record coherence, aged-out-of-slurmctld, not-found, and no-accounting.

---

Co-authored with Claude Code (Anthropic). Opened as a draft pending CI + the CLA signature.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[pramasoul]

I have read the CLA Document and I hereby sign the CLA