[Ubuntu 24.04][Sandbox] Upgrading across #8677 orphans every pre-existing Docker sandbox on the first gateway restart: records are purged and baked sandbox tokens are rejected

[Hokonoken]

Investigation Summary

• fix(uninstall): isolate scoped gateway shutdown #8677 (fa2b57746) gives each gateway a state-scoped Docker sandbox_namespace and adds a hash suffix to gateway_id. Containers created before it carry openshell.ai/sandbox-namespace=default and a sandbox JWT whose iss is the unsuffixed gateway id.
• The generated TOML is rewritten when the gateway is started, so the break lands on the first gateway restart after the upgrade — the normal path after a host reboot. Commands that find a live gateway (list, <name> status, doctor) change nothing.
• Reproduced by construction: onboard on 04e28aa2a (the parent commit), upgrade to dfef83e26, restart the gateway. The resume sweep reports missing_backend=1, and 303 s later (ORPHAN_GRACE_PERIOD = 300 s) the record is removed from the store. openshell sandbox list then returns none and nemoclaw <name> status reports the sandbox as registered locally but absent from the live gateway.
• The container also crash-loops on Policy fetch failed after 5 attempts: ... invalid token: InvalidIssuer, because the JWT baked at /etc/openshell/auth/sandbox.jwt is non-expiring (exp: 0, fix(onboard): make local docker-driver gateway JWT non-expiring #6304) and still names the old issuer.
• Recovery is not guaranteed. Pinning the legacy gateway_id and sandbox_namespace back into the generated TOML brought three pre-existing sandboxes on this machine fully back (records recreated at the gateway restart, Ready, workspaces intact). On the purpose-built sandbox the same restoration only fixed the token half: InvalidIssuer became sandbox not found, and the record did not come back, through a gateway restart and a reconcile interval. I did not trace what decides between the two outcomes.

Description

What happened

On an installation whose sandboxes were created before #8677, the first gateway start on current main silently unregisters every sandbox and leaves its container unable to fetch policy.

Two independent effects of the same commit:

A — the store record is purged. The regenerated TOML sets sandbox_namespace to the new gateway id. The Docker driver's list / find paths filter on the openshell.ai/sandbox-namespace label (crates/openshell-driver-docker/src/lib.rs:1217-1257 at v0.0.101; the filter uses all(true), so container state is not the factor), and pre-existing containers are labeled default — the driver's own default when the field is absent (lib.rs:148). The sandbox becomes invisible, prune_missing_sandbox re-checks, finds nothing, and deletes the record (crates/openshell-server/src/compute/mod.rs:2348-2412).

B — the sandbox token stops validating. gatewayIdForStateDir now appends a UID+state-root hash, and the gateway mints and verifies sandbox JWTs against that id. The token written into the container at creation time still carries the old one, so every policy fetch fails with InvalidIssuer and the container crash-loops.

Effect A is what a user sees as "registered locally, but is not present in the live OpenShell gateway", with the CLI suggesting rebuild --yes or destroy — both destructive, for sandboxes whose containers and workspaces are intact on disk.

What I expected

An upgrade that changes the namespace and the gateway identity should carry existing sandboxes across: adopt containers labeled with the legacy namespace (relabel or match both during a transition), and accept the legacy issuer for tokens already minted, rather than purging records and rejecting the tokens.

Not yet released

fa2b57746 is on main and is not contained in any tag (git tag --contains returns nothing; latest tag v0.0.105), so this is reachable today only by source installs.

Related work I checked first

Several issues share the user-facing message but not this cause, so I am filing separately rather than commenting on them:

• Fix Docker orphan cleanup and restart-safe recreation handoff #8720 (closed by fix(sandbox): serialize Docker recreation handoff #8722) lists "the sandbox remains in the NemoClaw registry but is absent from the upgraded live gateway, an old exactly labeled Docker container survives" among its E2E failures. Its fix serializes the Docker recreation handoff and does not change namespace matching — I checked the diff for sandbox_namespace / sandbox-namespace and found no occurrence.
• fix(sandbox): repair legacy gateway upgrade recovery #8735 repairs legacy gateway upgrade recovery by reusing a validated installer backup for stale Ready / Running sandboxes. That is the recovery path after a sandbox has been orphaned, and it is fail-closed when no validated backup exists. It does not adopt the containers, which in this report are intact and running on disk with their workspaces.
• [Brev][Onboard] post-interrupt re-onboard succeeds with "Deployment verified" but nemoclaw status reports sandbox not in live gateway #8737 produces the same "not present in the live gateway" message from a different cause (an interrupted onboard leaves port 8080 held, the registry entry is written for the default port while the new gateway runs on 8082).
• [DGX Spark][Hermes] current LKG: gateway restart quarantines agent; rebuild can leave registry/live OpenShell divergent #8649 reaches the same divergence through an aborted hermes rebuild on an older LKG commit.

I re-ran the trigger on v0.0.105-96-gacf810d9a (current main at the time of filing) against a gateway still holding the legacy values, and it still breaks: the gateway start rewrote gateway_id and sandbox_namespace to the suffixed form, openshell sandbox list returned No sandboxes found. while the container was running, status printed the "not present in the live OpenShell gateway" message, and the container failed with invalid token: InvalidIssuer. The store record had already been purged on the earlier run, so the purge lines quoted below are from the dfef83e26 reproduction, not the re-run. No commit between the two builds touches sandbox_namespace, gateway_id, or issuer validation.

Reproduction Steps

Docker driver, dedicated gateway port so nothing else is involved.

1. Build the parent of the commit and onboard a sandbox:

   git switch --detach fa2b57746^     # 04e28aa2a
   npm run build:cli
   NEMOCLAW_GATEWAY_PORT=18900 node bin/nemoclaw.js onboard --name nsrepro --agent openclaw \
     --no-gpu --non-interactive --yes --yes-i-accept-third-party-software --fresh
   

   State after onboarding: sandbox Ready, dashboard /health returns 200, container label
   openshell.ai/sandbox-namespace=default, generated TOML has gateway_id = "nemoclaw-openshell-docker-gateway-18900"
   and no sandbox_namespace line, and the in-container JWT claims
   iss = openshell-gateway:nemoclaw-openshell-docker-gateway-18900.

2. Upgrade and confirm nothing breaks while the gateway keeps running:

   git switch --detach dfef83e26      # current main
   npm run build:cli
   NEMOCLAW_GATEWAY_PORT=18900 node bin/nemoclaw.js list
   NEMOCLAW_GATEWAY_PORT=18900 node bin/nemoclaw.js nsrepro status
   NEMOCLAW_GATEWAY_PORT=18900 node bin/nemoclaw.js doctor
   

   The TOML is unchanged and the sandbox stays Ready.

3. Restart the gateway, as a host reboot would:

   kill "$(cat <state-dir>/openshell-docker-gateway-18900/openshell-gateway.pid)"
   NEMOCLAW_GATEWAY_PORT=18900 node bin/nemoclaw.js doctor
   

Observed from there:

# TOML, rewritten by the gateway start
gateway_id       = "nemoclaw-openshell-docker-gateway-18900-1be7ec74403d"
sandbox_namespace = "nemoclaw-openshell-docker-gateway-18900-1be7ec74403d"

# gateway log
WARN  Cannot resume sandbox: backend resource is missing sandbox_name=nsrepro
INFO  Sandbox resume sweep complete resumed=0 missing_backend=1 failed=0
INFO  reconcile{backend_count=0 store_count=1}: Removing sandbox from store after it
      disappeared from the compute driver snapshot sandbox_name=nsrepro age_secs=303

# openshell sandbox list -g nemoclaw-18900
No sandboxes found.

# nemoclaw nsrepro status
Sandbox 'nsrepro' is registered locally, but is not present in the live OpenShell gateway.

# container log, after docker start
Error: Policy fetch failed after 5 attempts: code: 'The request does not have valid
       authentication credentials', message: "invalid token: InvalidIssuer"

The same sequence hit three sandboxes that predated the upgrade on this machine; there the records were gone by the end of the startup sweep, without the 300 s wait I saw on nsrepro. I did not trace what accounts for the difference.

Partial recovery, without destroying anything

Editing the generated TOML back to the legacy values and restarting the gateway re-adopted the
three pre-existing sandboxes on this machine and restored Ready, with workspaces intact. On the
sandbox built for this report the same edit only restored token validity — the purged record did
not return — so treat this as a recovery worth trying, not a reliable one:

gateway_id        = "nemoclaw-openshell-docker-gateway-<port>"
sandbox_namespace = "default"

Two caveats: it reinstates the cross-gateway coupling #8677 fixed (stopping one gateway then stops the other's containers), and any later doctor / onboard / connect rewrites the file and restarts the process, because the runtime marker check treats the restored process as legacy (src/lib/onboard/host-gateway-process.ts:241).

Environment

• OS: Ubuntu 24.04.4 LTS on WSL2 (kernel 6.6.87.2-microsoft-standard-WSL2), x86_64 workstation
• Node.js: v24.18.0
• Docker: Engine 29.2.1
• NemoClaw: v0.0.105-91-gdfef83e26 (source build) for the by-construction reproduction; pre-upgrade build 04e28aa2a; trigger re-verified on v0.0.105-96-gacf810d9a
• OpenShell: 0.0.101 (gateway, CLI, sandbox), the version the current blueprint pins
• Driver: docker

Debug Output

nemoclaw debug --quick on this host includes the machine hostname, internal DNS servers and
local paths, so I have not pasted it wholesale. The evidence specific to this bug is quoted
above: the regenerated TOML, the gateway log lines for the sweep and the purge, the
openshell sandbox list result, the CLI message, and the container's InvalidIssuer failure.
Happy to provide a redacted bundle if you want the full collection.

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[wscurran]

✨ Thanks for the detailed investigation. This upgrade regression breaks existing sandboxes on gateway restart — maintainers will prioritize a fix for the namespace/JWT migration path.

---

Related open PRs:

• #6304 fix(onboard): make local docker-driver gateway JWT non-expiring
• #8677 fix(uninstall): isolate scoped gateway shutdown

[senthilr-nv]

Claiming the v0.0.74 upgrade regression from E2E run 31895698451, job 95038810719. A literal v0.0.74 gateway-config generator followed by the current legacy-identity verifier reproduces the failure: the historical generated config uses ttl_secs = 3600, while the verifier reconstructs only the current ttl_secs = 0 form and rejects it as not generated by NemoClaw. Open-PR searches for the run, job, release lane, TTL value, and legacy-identity wording found no owner. I will prepare a small internal draft PR that recognizes only the historical generated TTL during legacy proof, keeps arbitrary or ambiguous configurations fatal, and adds a deterministic release-generated regression. The v0.0.55 failures remain separately unclaimed until their historical state is proven.