feat: deploy-freshness check -- catch stale containers automatically (#200/#194)

[man4ish]

Summary

Adds a deploy-freshness check: catches the pattern hit 3x in one session -- a container running stale code relative to what's merged/available upstream, invisible until someone happens to test the exact behavior that changed.

• celery-worker -- rename regression, image rebuilt only after the fact
• workbench -- DAG-count regression, same
• toolserver -- ghcr.io image 91 days stale, no local build fallback exists at all

Design

Proposed and approved before any code was written (see #200 discussion). Two checks, one underlying question -- "is this image older than what it should reflect":

• Build-from-source (workbench, celery-worker, tes): image Created timestamp vs origin/main's latest commit, via one lightweight GitHub REST API call (GET /repos/{owner}/{repo}/commits/main) -- not git fetch, no object transfer, just commit metadata. Also reads refs/heads/main directly (plain-text file inside the mounted .git dir, no git binary needed) to distinguish "forgot to pull" (local ref also behind origin) from "forgot to rebuild" (local ref matches origin; image just predates it).
• Pull-from-registry (toolserver): no source repo to compare against by definition -- that's the actual gap. Falls back to an age threshold (default 14 days). Same threshold is also build-type's fallback if the GitHub API call fails for any reason (rate limit, network, token issue) -- never crashes, never blocks, degrades to best-available signal.

Where it runs

A one-shot deploy-verify compose service, wired via depends_on: condition: service_completed_successfully on every service it covers. Runs automatically on every docker compose up, including docker compose up -d <one-service> (the exact single-service pattern used throughout this session) -- no new command for anyone to learn or remember. This environment has no CI/CD and no wrapper deploy script (confirmed before designing this -- dev_up.sh doesn't touch Docker at all), so this had to fit inside the plain docker compose commands people already type, not assume infrastructure that doesn't exist here.

Always exits 0. This is a loud warning system, not a startup gate -- a hard block would let toolserver's already-broken upstream CI take down the entire stack for a reason that has nothing to do with today's deploy.

Output -- two redundant channels

• stdout, inside docker compose up's own combined output / docker compose logs deploy-verify
• STALE_SERVICES.txt at repo root, overwritten every run

so a finding is harder to scroll past than a single log line would be.

Per-service max_age_days override

A deploy-verify.max-age-days label in that service's labels: block (default 14 if absent) -- not a single global constant. Supports both list-style (- deploy-verify.max-age-days=30) and dict-style compose label syntax, and ${VAR:-default} interpolation inside the label value. Tested in isolation for all four shapes before committing.

Scope: 4 services today, not all ~30

workbench, celery-worker, tes, toolserver -- the ones actually hit this session. Several other build-from-source services (tes, model-registry) share a broad multi-repo build context where "which subdirectory is the real source repo" gets ambiguous to auto-derive reliably; rather than guess wrong for services nobody's hit an incident on yet, scoped this to an explicit, short list. Extending coverage is a straightforward follow-up once this is validated in practice, not a blocker.

Testing -- against the real running stack, not just docker compose config --quiet

Found and fixed 2 real bugs along the way:

1. A regex substitution intended to normalize Docker's variable fractional-second Created timestamps dropped the trailing Z in its replacement string, producing a naive datetime and crashing every comparison (TypeError: can't compare offset-naive and offset-aware datetimes). Fixed by relying on Python 3.11+'s fromisoformat, which handles Z and arbitrary fractional-second precision natively -- deleted the regex entirely rather than fixing it, simpler and correct.
2. tes has an explicit image: omnibioai-tes-local override in docker-compose.yml, not Compose's auto-derived <project>-<service> tag every other build-from-source service uses -- the script assumed the auto-derived pattern unconditionally and silently skipped tes ([SKIP] image omnibioai-studio-tes:latest not built yet) until this was caught and fixed to check for an explicit image: override first.

Final real run against the actual live stack (not synthetic test data):

==================== DEPLOY FRESHNESS CHECK ====================
[OK]     workbench        image built 2026-08-10T05:26:59+00:00, matches omnibioai@main HEAD (2026-08-10T05:23:44+00:00)
[STALE]  celery-worker    image built 2026-08-09T21:45:31+00:00; omnibioai@origin/main HEAD is 2026-08-10T05:23:44+00:00 (0d newer) -- rebuild needed (local checkout matches origin/main; image predates it)
[STALE]  tes              image built 2026-08-09T00:33:12+00:00; omnibioai-tes@origin/main HEAD is 2026-08-10T06:52:37+00:00 (1d newer) -- pull AND rebuild needed (local checkout is also behind origin/main)
[STALE]  toolserver       image created 2026-05-10T18:17:40+00:00 (91d old, threshold 14d) -- registry may have stopped publishing, see #200
===================================================================

celery-worker and tes are genuinely stale right now (this PR itself surfaced that) -- real, current findings, not test fixtures. toolserver's known 91-day staleness (see #200) is correctly caught by the age-threshold path. workbench correctly shows OK.

Also confirmed the never-blocks guarantee for real: docker compose up -d workbench ran deploy-verify automatically, it reported the stale findings above, and workbench still started normally.

Known limitations (stated in the script's own docstring too)

• Compares against origin/main fetched live, but the local ahead/behind classification only distinguishes "matches origin" vs "doesn't" via a single SHA comparison -- not a full ahead/behind commit count (would require a git fetch, deliberately avoided per the design discussion -- a lightweight GET /commits/main API call was chosen instead).
• Image Created timestamp is a reasonable but not perfectly bulletproof proxy for "when the code was actually built."
• 4-service scope, not all ~30 services in this compose file (see above).

Part of #200 (does not close it).

🤖 Generated with Claude Code