Skip to content

deploy-freshness check doesn't catch a container serving from a stale baked-in path instead of the bind mount #36

Description

@man4ish

Summary

Found while doing end-to-end validation for omnibioai-workbench#197: omnibioai-studio-workbench-1 was serving bio_agent requests from /app -- a copy of the source baked into the image at build time -- while the actual bind mount (/home/manish/Desktop/machine -> /workspace, so live source lands at /workspace/omnibioai) sat right next to it, completely unused by the running server (WorkingDir: /app, confirmed via docker inspect).

Editing files under /workspace/omnibioai and restarting the container had zero effect on served behavior -- three PRs' worth of auth-gating changes (workbench#203, #204, #205) were invisible to curl/browser testing against the live container until this was found and /app was manually synced from /workspace/omnibioai.

Why deploy-verify (#35) doesn't catch this

scripts/deploy_verify.py's check_build_service() compares the image's build timestamp against origin/main's latest commit (image_created() vs github_main_commit()/local_main_sha()). That correctly catches "image is older than what's merged" -- the pattern #35 was built for (celery-worker rename regression, workbench DAG-count regression).

It does not, and structurally can't with its current inputs, catch this case: the image itself can be arbitrarily fresh (or even irrelevant, if a bind mount is supposed to be authoritative), and the check would still pass, because it never asks "does the process actually serving traffic read its code from the path the deploy tooling assumes it does?" That's a question about the container's WorkingDir/CMD vs. its declared volume mounts, not about image age at all -- a distinct failure mode from build-staleness.

Suggested distinct check

For services where docker-compose.yml declares a source bind mount (e.g. /workspace), verify the container's actual serving path (WorkingDir, or wherever CMD/ENTRYPOINT cds to before running the app) is the mounted path itself -- not a separate baked-in directory that happens to sit alongside it unmounted. Concretely, something like: docker inspect <container> --format '{{.Config.WorkingDir}}' cross-checked against the mount Destinations from the same inspect output, flagged if WorkingDir isn't underneath any of them.

This is orthogonal to check_build_service/check_pull_service -- could be a third check function in the same script, or its own smaller check, but it answers a different question ("is dev-mode hot-reload actually wired to anything") than the existing two ("is this image current").

Scope note

Not fixing this now -- flagging per the review that surfaced it. Filed as its own issue rather than folding into #200's scan thread since it's a distinct, standalone, actionable item (see also a short pointer left on workbench#200 for discoverability from that thread).

🤖 Filed by Claude Code during workbench#197 E2E validation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions