fix(ingest-hub): survive dead conmon pipe instead of dying as a zombie

[mruwnik]

Problem

On 2026-07-02, chris-ingest-hub-1 died silently and podman kept reporting it "Up" (zombie container, dead conmon), taking down all Discord MCP tools for ~84 minutes. Investigation (incident doc included) found the death mechanism and reproduced it locally:

Supervisord 4.2.5's homegrown logger kills PID 1 when the container's stdout pipe loses its reader:

1. A log write to /dev/stdout raises BrokenPipeError; emit() catches it and calls handleError().
2. handleError() prints the traceback to sys.stderr — the same dead pipe — raising a second BrokenPipeError that escapes emit().
3. The dispatcher's handle_error() calls logger.critical(...) → third BrokenPipeError inside the main loop's except: → escapes runforever() → PID 1 exits, taking celery-beat and discord-api with it. No dying words possible by construction.

stdlib logging guards this exact path (except OSError: pass, bpo-5971); supervisord's logger doesn't, making ingest-hub the only vulnerable container in the stack.

Fix

• Dockerfile: two-phase ENTRYPOINT — validate the config via ServerOptions.realize() while stderr is live (config errors still surface at container start, exit 2), then exec supervisord with fd 2 → /dev/null so handleError() can never double-fault.
• supervisor.conf: redirect_stderr=true on both programs so child stderr reaches podman logs via the stdout channel instead of being swallowed by the new fd 2.

A dead conmon now degrades the container to logless-but-serving instead of a silent whole-container exit.

Verification

• Reproduced the death: supervisord 4.2.5 on a pipe, reader closed → exit 120 within seconds; breadcrumb instrumentation confirmed the exact cascade.
• With the fix: survived a dead pipe under ~6MB of post-death log traffic, kept draining child pipes (children never block), handled child kill/respawn cycles.
• ENTRYPOINT validation phase tested: valid conf → silent exit 0; malformed conf / bad option / missing env var → exit 2 with the parse error on real stderr.
• Config parse-validated against real supervisord 4.2.5 (redirect_stderr=True, stderr=None on both programs).

Needs an image rebuild on deploy (compose build ingest-hub), not just a restart.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U1McPhcL927v8qNdsFcLVu

[mruwnik]

Hygieia (Claude Opus 4.8): Approve. This is a correct, well-scoped, and unusually well-documented fix. I traced the mechanism and the fix end-to-end:

Root cause is right. supervisord 4.2.5's loggers.Handler.emit() catches the stdout-pipe BrokenPipeError and calls handleError(), which traceback.print_exception(..., sys.stderr). With fd 2 also on the dead conmon pipe, that write raises a second BrokenPipeError out of emit(), and the dispatcher's handle_error() → logger.critical() third fault escapes runforever() → PID 1 exits. stdlib logging guards this (except OSError: pass, bpo-5971); supervisor's homegrown logger does not. Confirmed against the 4.2.5 source.

The fix closes it at the root. Pointing fd 2 → /dev/null means handleError()'s write always succeeds, emit() returns normally (no re-raise), and every log-write path — including the logger.critical() in the dispatcher's error handler and supervisord's forwarding of child output to /dev/stdout — degrades to a silently-swallowed failure instead of a cascade. Container becomes logless-but-serving, exactly as claimed.

The two-phase ENTRYPOINT is the right call and correctly built:

• Shell parsing/redirection is correct: 2>/dev/null binds only to exec supervisord, so the validation python3 -c runs with live stderr and surfaces config errors (exit 2), while only the long-lived supervisord gets fd 2 on /dev/null.
• exec keeps supervisord as PID 1 (SIGTERM handling on podman stop preserved).
• Explicit /usr/bin/python3 is necessary and consistent: apt installs supervisor into Debian's dist-packages (reachable from /usr/bin/python3, matching /usr/bin/supervisord's shebang), not the image's /usr/local python — so validation imports the same supervisor that runs.

supervisor.conf change is consistent. redirect_stderr=true merges child stderr into the stdout channel so fd 2 → /dev/null doesn't discard uvicorn/celery stderr; it still reaches podman logs via /dev/stdout. The forwarding writes are themselves protected by the same guarded logger.

Scope is sensibly limited to "don't die," with zombie auto-remediation and dead-log-capture detection correctly deferred to follow-ups in the incident doc.

Only a non-blocking nit (readability of the inline sh -c) and a request to make the realize()-is-side-effect-free-at-uid-kb invariant explicit — see inline comment. No blocking issues; ready to merge (remember the image rebuild on deploy, compose build ingest-hub).

[mruwnik]

Hygieia (Claude Opus 4.8): Re-reviewed at ef6795f — the entry.sh extraction is clean and both of my earlier points are fully addressed. Still approve.

Checked the new docker/ingest_hub/entry.sh:

• set -eu correctly propagates phase-1's sys.exit(2): a bad config aborts the script with exit 2 and the error on live stderr, and exec is never reached. -u is safe (CONF is always set).
• 2>/dev/null binds only to exec /usr/bin/supervisord, so validation runs with live stderr and only the long-lived supervisord gets fd 2 → /dev/null (your readlink /proc/self/fd/2 check confirms it). exec keeps supervisord as PID 1 for signal delivery.
• Single CONF variable removes the duplicated conf path. All constructs are dash/POSIX-safe; sh -n clean as you noted.

Dockerfile wiring is correct: COPY + chmod +x run as root before USER kb, and the later chown -R kb /app covers /app/supervisord_entry.sh, so kb owns and can execute it. Exec-form ENTRYPOINT with a single element runs the script via its own #!/bin/sh, so the shell (then supervisord after exec) is PID 1 — no extra sh -c wrapper swallowing signals.

The realize()-is-side-effect-free-at-uid-kb invariant is now documented in the script, and your 4.2.5 source trace matches mine. No remaining issues; ready to merge (remember compose build ingest-hub on deploy).