Wait for managed supervisor before failing stopped-sandbox startup

[apurvvkumaria]

Description

After #8683 made the first managed-recovery result authoritative, ordinary stopped-sandbox startup can fail immediately after Docker reports the container running, before the managed nemoclaw-start supervisor has settled.

The focused E2E run 31376474615 failed both protected stop/start targets:

• hermes-shields-config
• shields-config

The Hermes artifact records nemoclaw e2e-hermes-shields start failing after 1.273 seconds:

Container 'openshell-default--e2e-hermes-shields-…' started.
Restoring sandbox startup state…

Error: Sandbox 'e2e-hermes-shields' started, but startup recovery failed:
supervisor not running: SUPERVISOR_NOT_RUNNING.

Onboarding, fresh-runtime verification, the first Shields cycle, and the expected locked ownership/modes all passed before the stop/start failure.

Investigation

• Ordinary Docker start waits only for the container runtime to report running, then immediately restores process state.
• Managed recovery retries exact SUPERVISOR_BUSY, but an exact SUPERVISOR_NOT_RUNNING result enters legacy relaunch classification and otherwise returns failure.
• The existing waitForManagedGatewaySupervisor helper already performs bounded authenticated read-only probes for the exact transient startup markers, but it is currently wired only into restored-clone recovery.
• Before fix(sandbox): restore protected stopped sandboxes #8683, the first recovery result was discarded and the later readiness probe provided additional settling time. fix(sandbox): restore protected stopped sandboxes #8683 did not create the supervisor-startup window; it made that first observation terminal.

The exact marker proves that no qualifying managed supervisor existed at the probe instant. It does not distinguish a delayed launch from an entrypoint that launched and exited. A supervisor-bootstrap race is the leading explanation, but persistent startup failure must remain fail-closed.

Expected behavior

• Ordinary nemoclaw <sandbox> start performs its normal recovery attempt first.
• If and only if that attempt returns the classified missing-supervisor startup condition, startup uses the existing bounded authenticated waiter.
• Waiter success triggers a second complete restoration/recovery check; it is not itself treated as startup success.
• Persistent supervisor absence, unclassified output, identity drift, integrity refusal, Shields/MCP refusal, readiness failure, or forward failure remains terminal and preserves the existing sandbox.
• Genuine legacy sleep infinity recovery continues through the existing transactional relaunch path without an unconditional settling delay.

Scope and implementation constraints

Follow the small-fix and anti-bloat guardrails established in #8720:

• Do not introduce a lifecycle state machine, journal, coordinator, recovery framework, or generalized retry framework.
• Do not add diagnostic collection, new diagnostic formats, or broad error-reporting work.
• Keep production-code changes below 200 changed lines.
• Avoid unrelated refactors and new abstractions; use the existing waiter and add at most one narrowly scoped helper if necessary.
• Add only targeted tests for the ordinary stopped-start retry branch and its fail-closed boundaries.
• Do not add new E2E targets, expand the E2E matrix, or add broad test coverage.
• Prepare a release-ready PR immediately after targeted tests pass; normal required CI still applies.

Container recreation/orphan ownership handoff remains owned by #8720. Cold-boot relaunch/rollback behavior in #7418 is related but has a different trigger and recovery path.

Acceptance criteria

• Ordinary stopped-sandbox startup performs the existing recovery check before considering any settling wait.
• An exact transient missing-supervisor result receives a bounded wait using the existing authenticated managed-control helper.
• A successful wait is followed by a second full restoration/recovery pass that revalidates Shields/MCP boundaries, managed health, OpenShell readiness, and host forwards.
• Persistent absence and all definitive or unclassified failures retain current fail-closed behavior and preserve the sandbox.
• Legacy sleep infinity transactional relaunch behavior is unchanged and does not incur a proactive settling delay.
• The existing shields-config and hermes-shields-config focused E2E targets pass.
• Targeted tests cover transient wait-to-success, persistent absence, no retry for definitive failure, and unchanged legacy relaunch behavior.
• Production-code changes remain below 200 changed lines.
• No lifecycle state machine, journal, coordinator, generalized retry/recovery framework, diagnostics expansion, or broad refactor is introduced.
• No new E2E target or broad matrix/test expansion is added.
• The implementation is submitted as a release-ready PR after targeted verification passes.

Evidence limitations

The focused artifact does not include /tmp/nemoclaw-start.log, Docker events, or immutable container inspection data. It therefore does not conclusively exclude an early entrypoint exit, container-local OOM, or recreation. The bounded wait must time out safely and preserve the current failure when the supervisor never appears.

Environment

• GitHub Actions Linux x86_64 runner
• Docker/OpenShell driver
• NemoClaw commit bf81b95b9d1801906809c7a20219105c0164a8da
• Focused workflow targets: shields-config, hermes-shields-config

Checklist

• Reproduced in the linked focused E2E run
• Reviewed the Hermes E2E artifact
• Searched existing open issues; no duplicate found