fix(scanqueue): fence stale scan workers

[Quick104]

Problem

Related issue: N/A — audit-validated distributed correctness fix

Scan workers owned a run only by its row ID and running status. Stale-run maintenance reused that same row when requeueing, so worker A could lose its heartbeat, worker B could claim the requeued run, and A could later resume and write progress or a terminal status into B's work.

The finding was independently reproduced against fresh origin/main at 8164fd594b9fdd8c1944bb0b6251f2d00e4a24ca. No existing issue or PR matched the defect.

Approach

• Add an opaque claim token to each accepted-to-running transition. Heartbeat, progress, completion, failure, and worker cancellation now require both the run ID and current token.
• Retire an expired run as failed and create a distinct accepted successor instead of reusing the old row. The successor preserves scope, trigger, autoscan event, and original request time.
• Establish a fenced heartbeat before publishing scan.started or beginning ingestion. Local cancellation is registered first, and local lease loss cancels guarded scanner and metadata contexts synchronously.
• Run post-ingest availability work only after the queue commits a fenced completion.
• Serialize library cancellation with claim/requeue maintenance using the existing scan advisory-lock boundary.
• Add a migration guard for mixed-version rollout. The migration refuses to run while any scan is running, rejects tokenless legacy claims, and rejects legacy in-place requeues.

The protocol invariant is: a worker may mutate a run or perform scan-owned ingestion only while (run_id, claim_token) is the current durable claim. Losing that ownership cancels local work; retry creates a new run ID.

Validation

Repository gate:

make embed-stub                                      PASS
go build ./...                                       PASS
gofmt -l .                                           PASS (no output)
go vet ./...                                         PASS
golangci-lint run --new-from-merge-base=origin/main  PASS (0 issues)
make test-go                                         PASS on dev-builder
pnpm install --frozen-lockfile                       PASS
pnpm run lint                                        PASS (0 errors; 167 existing warnings)
pnpm run format:check                                PASS
pnpm run build                                       PASS
make test-web                                        PASS (294 files, 2184 tests)
make verify-settings-bindings-all                    PASS
make verify-playback-fixtures                        PASS
make verify-local-paths                              PASS
make migrate-validate                                PASS

Focused and race checks on isolated dev-builder sandbox audit-scan-lease:

SILO_TEST_DATABASE_URL=<redacted> go test -v ./internal/scanqueue -run <claim-fencing cases>
PASS: stale heartbeat, progress, success, failure, and cancellation
PASS: claim loss during ingestion and local lease expiry
PASS: A paused before process, B requeued/claimed, A resumed without starting ingestion
PASS: cancel registration before the initial heartbeat
PASS: completion-boundary lease expiry leaves the run eligible for retry
PASS: legitimate distinct-row requeue and successor completion
PASS: old tokenless claim/in-place requeue rejection and cancel/requeue serialization

go test -race ./internal/metadata -run TestProcessSeriesRootsPreservesOwnershipGuard
PASS

SILO_TEST_DATABASE_URL=<redacted> go test -race ./internal/scanqueue ./internal/scanner
PASS

Migration lifecycle against pre-existing accepted, running, and completed rows:

upgrade_with_running_row=rejected
transaction_rollback_column_present=false
upgrade_after_drain=true,trigger=true
rollback_column_present=false
final_version=20260827201717

The exact candidate was built and deployed in audit-scan-lease. Controller doctor reported the application and Redis containers healthy, the database query successful, /api/v1/health status ok, and both API and frontend reachable. The synthetic A/B lifecycle passed against the sandbox Postgres; scan claims do not store ownership state in Redis.

Two exploratory configurations were not used as gates: exporting one SILO_TEST_DATABASE_URL to every package in make test-go caused unrelated parallel database suites to mutate the same schema, and running the entire metadata package under -race exposed a pre-existing test-fake race plus unrelated schema interference. The supported full suite passed without that global opt-in; the changed Postgres packages and the new metadata guard test passed separately under -race as shown above.

No screenshots: this changes backend orchestration and schema only.

Risks

• Upgrade is intentionally fail-closed if a scan is running. Operators must drain active scans and retry the migration.
• Rollback requires all upgraded binaries to be stopped and active scans drained before dropping claim_token; upgraded code reads that column.
• A stale retry now appears as a failed attempt plus a distinct accepted successor. This is intentional and preserves an auditable ownership boundary.
• No native v1 API shape changed. Apple, Android, and Jellyfin-compatible clients require no changes.

Independent read-only review covered stale-worker bypasses, mixed-version migration safety, requeue/retry behavior, cancellation races, derived contexts, completion boundaries, and after-commit side effects. It found and drove fixes for the rollout barrier, old-maintenance bypass, cancel/requeue race, durable startup fencing, derived-context guard propagation, completion-boundary classification, and startup cancellation registration. The final review reported no actionable findings.

AI Disclosure

• Tool(s): OpenAI Codex
• Model(s): gpt-5.6-sol
• Involvement: Fully AI-generated; human verification pending
• Adversarial review: An independent read-only Codex agent reviewed the complete patch in six passes for stale-worker bypasses, migration safety, retry semantics, cancellation races, derived-context propagation, completion boundaries, side-effect ordering, and regressions. All actionable findings were confirmed, fixed, and re-tested; the final pass reported no blockers.

Checklist

• I read and can explain the complete diff.
• This pull request addresses one concern.

[coderabbitai]

Warning

Review limit reached

Next included review available in 57 minutes.

View limit details

Limit details: You’ve used the included review currently available. Your 94 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: dfef7581-89d4-4897-8402-54a85373dc5d

📥 Commits

Reviewing files that changed from the base of the PR and between 8164fd5 and 99f141c.

📒 Files selected for processing (12)

• internal/claimcontext/context.go
• internal/libraryingest/executor.go
• internal/metadata/worker.go
• internal/metadata/worker_test.go
• internal/models/scan_run.go
• internal/scanner/guard_context.go
• internal/scanner/guard_context_test.go
• internal/scanner/scanner.go
• internal/scanqueue/claim_fencing_db_test.go
• internal/scanqueue/repository.go
• internal/scanqueue/service.go
• migrations/sql/20260827201717_add_scan_run_claim_token.sql

---

_{Comment @coderabbitai help to get the list of available commands.}