fix: fence stale job recovery transitions

[obckelbley]

Summary

• add an atomic backend compare-and-update contract keyed by an execution fingerprint;
• fence every JobTransitioner write, closing both stale-sweep and old-executor overwrite races;
• conditionally release stale claimed jobs;
• treat a lost stale-sweep race as a benign no-op while preserving genuine backend errors;
• implement the operation for SQL, MySQL, MongoDB, and LazyBackend; and
• document the new custom-backend requirement.

Closes #208

Why this is needed

releaseStaleJobs() currently operates on snapshots returned by Backend.staleJobs(). Before recovery writes its transition, a live executor or another worker can complete, re-claim, or start a newer attempt of the same job. The existing update is matched only by job id, so recovery can overwrite that newer state.

A state-only condition is not sufficient: a stale snapshot and a newer attempt can both be running. This change compares the lifecycle fields that identify the execution on which the transition was based:

type JobExecutionFingerprint = Pick<
  JobData,
  "state" | "attempt" | "claimed_by" | "claimed_at" | "attempted_at"
>;

The backend performs the comparison and update atomically. If the fingerprint no longer matches, JobTransitioner raises JobTransitionConflictError. Stale recovery catches only that conflict and skips the obsolete snapshot; other backend failures continue to propagate.

Why all lifecycle transitions are fenced

Fencing only releaseStaleJobs() closes one direction of the race but leaves the inverse:

1. recovery retries a stale running job;
2. a new worker claims and starts the next execution; and
3. the original executor finishes and writes its old result over the new execution.

Using the same compare-and-update precondition in JobTransitioner prevents an old executor from completing, failing, retrying, snoozing, or canceling an execution it no longer owns.

Backend behavior

• SQLite/PostgreSQL use a single conditional UPDATE ... RETURNING.
• MySQL uses a transaction containing the conditional update and re-read.
• MongoDB uses findOneAndUpdate() with id plus the execution fingerprint in the filter.
• LazyBackend delegates the new method.

The MySQL conformance test includes a matching update that changes no values, ensuring a match is not misclassified as a conflict.

No database schema migration is required.

Compatibility note

Backend.updateJobIfCurrent() is intentionally required rather than optional. This is a source-level compatibility change for custom backend implementations, but making it optional would allow a backend to compile while silently omitting the safety guarantee. The custom-backend documentation and example are updated.

If maintainers prefer a persisted revision/fencing-token column, the same tests can validate that implementation. The submitted fingerprint approach keeps this fix focused and avoids a schema migration.

Tests added

• a completed job cannot be resurrected after a stale scan;
• a newer running attempt/owner cannot be overwritten;
• a re-claimed job cannot be released from an obsolete claimed snapshot;
• an old executor cannot complete a newer execution;
• a matching conditional update succeeds;
• a matching no-op conditional update succeeds;
• a same-state newer execution fails the compare-and-update;
• LazyBackend delegates the conditional update; and
• genuine backend failures are not swallowed.

Before the fix, the two primary stale-sweep regression tests fail deterministically on v1.16.2: a completed job becomes waiting, and a newer running attempt 2 owned by worker-b is replaced by the stale attempt 1 snapshot owned by worker-a.

Validation

• yarn build — 13/13 tasks passed
• yarn format:check — passed
• yarn lint — passed
• yarn test:all:ci — 8/8 tasks passed
• yarn test:integration:ci — 3 files, 76/76 tests passed
• MySQL backend — 170/170 tests passed
• PostgreSQL backend — 255/255 tests passed
• MongoDB backend — 103/103 tests passed
• SQLite backend — included successfully in yarn test:all:ci
• git diff --check — passed

Pull request checklist

• All tests pass (yarn test:all:ci and yarn test:integration:ci)
• Code follows the style guide and passes lint checks
• Documentation is updated
• Linked to the corresponding issue