Description
Upgrading an existing prepared DGX Station from OpenShell 0.0.72 to 0.0.85 can
exit 0 and report that sandboxes were recovered even though the resumed sandbox
fails seconds later and enters Error.
This was found while creating a current generic-Ubuntu control for DGX OS
qualification. It is an upgrade-lifecycle defect, not a host-preparation defect.
Environment
- DGX Station GB300, ARM64
- Ubuntu 24.04.4, kernel
6.8.0-124-generic
- NVIDIA driver
610.43.02
- Existing NemoClaw commit
26eef0914174d752852311d2d0e5f4911d0c3b90
- Target NemoClaw commit
b47c81e0cc2b3dc22e07921167e48ad0a4ff7c1c
- Existing OpenShell
0.0.72; target OpenShell 0.0.85
- One Ready NemoClaw-managed OpenClaw sandbox and a healthy managed local vLLM
Observed Sequence
- The installer created a strict pre-upgrade backup: one backed up, zero
failed, zero skipped.
- OpenShell 0.0.85 CLI, gateway, and sandbox assets passed pinned SHA-256
verification.
- The stale 0.0.72 gateway was replaced with the 0.0.85 gateway.
- Gateway startup briefly reported the existing sandbox as
Ready.
nemoclaw upgrade-sandboxes --auto printed All sandboxes are up to date.
- The installer exited 0 and printed
Existing sandboxes were recovered and upgraded.
- About eight seconds after the temporary Ready observation, the old
supervisor stream ended with an unexpected EOF, the container exited 137,
and the sandbox moved from Provisioning to Error.
After the installer returned:
phase=Error
failureLayer=sandbox_container_stopped
openshellVersion=0.0.72
container=Exited (137)
The managed vLLM container/image/model and ECC state were unchanged.
Recovery Control
The already-created backup was valid. Explicit prepared-backup recovery found
the non-Ready sandbox, rebuilt it under OpenShell 0.0.85, restored 12
directories and one file, and passed:
- all direct sandbox GPU proofs, including
cuInit(0);
- direct and routed local inference;
- exact-response and host-evaluated tool-artifact tasks;
- final ECC
0/0.
Expected Behavior
An OpenShell gateway/supervisor upgrade must not report success based on a
transient Ready observation from the old sandbox supervisor. The installer
should wait for a stable post-upgrade supervisor/container health boundary or
rebuild from the validated backup before returning success. If the sandbox
later enters Error within that boundary, installation must return nonzero and
retain the prepared recovery guidance.
Related upgrade: #6726
Description
Upgrading an existing prepared DGX Station from OpenShell 0.0.72 to 0.0.85 can
exit 0 and report that sandboxes were recovered even though the resumed sandbox
fails seconds later and enters
Error.This was found while creating a current generic-Ubuntu control for DGX OS
qualification. It is an upgrade-lifecycle defect, not a host-preparation defect.
Environment
6.8.0-124-generic610.43.0226eef0914174d752852311d2d0e5f4911d0c3b90b47c81e0cc2b3dc22e07921167e48ad0a4ff7c1c0.0.72; target OpenShell0.0.85Observed Sequence
failed, zero skipped.
verification.
Ready.nemoclaw upgrade-sandboxes --autoprintedAll sandboxes are up to date.Existing sandboxes were recovered and upgraded.supervisor stream ended with an unexpected EOF, the container exited 137,
and the sandbox moved from
ProvisioningtoError.After the installer returned:
The managed vLLM container/image/model and ECC state were unchanged.
Recovery Control
The already-created backup was valid. Explicit prepared-backup recovery found
the non-Ready sandbox, rebuilt it under OpenShell 0.0.85, restored 12
directories and one file, and passed:
cuInit(0);0/0.Expected Behavior
An OpenShell gateway/supervisor upgrade must not report success based on a
transient Ready observation from the old sandbox supervisor. The installer
should wait for a stable post-upgrade supervisor/container health boundary or
rebuild from the validated backup before returning success. If the sandbox
later enters Error within that boundary, installation must return nonzero and
retain the prepared recovery guidance.
Related upgrade: #6726