[DGX Station][Inference] inference set changes the gateway route but fails Hermes sandbox reconciliation

[mercl-lau]

Description

On a running Hermes sandbox, nemoclaw inference set updates the OpenShell gateway route to a local Nemotron model but fails to reconcile the in-sandbox Hermes configuration. The command leaves the switch incomplete and requires a disruptive sandbox rebuild.

Platform scope: Reproduced on DGX Station only; other platforms not tested.
Regression: Unknown — earlier versions not tested.

Environment

Device:        DGX Station
OS:            Ubuntu 24.04
Architecture:  aarch64
Docker:        29.2.1
OpenShell CLI: 0.0.72
NemoClaw:      v0.0.84
Hermes Agent:  v0.18.0 (2026.7.1)

Steps to Reproduce

1. Start a healthy Hermes sandbox using a local vLLM route and confirm chat works with the current model.
2. Start a second healthy local vLLM model served as nemotron-ultra.
3. Run:

   nemoclaw inference set --sandbox <sandbox> \
     --provider vllm-local \
     --model nemotron-ultra

4. Observe that the gateway route changes and the command then attempts to synchronize Hermes inside the sandbox.
5. Query the effective route and try a Hermes chat request.

Expected Result

The model switch is atomic. NemoClaw should either update the gateway route and all managed Hermes configuration successfully, or leave the previous working route unchanged and return an actionable error. A normal model switch should not require a full sandbox rebuild.

Actual Result

The OpenShell gateway route changes to vllm-local / nemotron-ultra, but Hermes synchronization fails because the compatibility hash does not match the frozen Hermes inputs. NemoClaw requires a full rebuild to continue, leaving the switch only partially applied.

Logs

Setting OpenShell inference route: vllm-local / nemotron-ultra
Syncing Hermes model route in sandbox '<sandbox>'...
[SECURITY] compat hash does not match frozen Hermes inputs during non-root reconciliation

---

NVB#6470512

[sandl99]

@mercl-lau QA re-verification update on current source (v0.0.101-69-g96b7d247e).

Scenario tested

Switch a healthy Hermes sandbox from NVIDIA Endpoints / Nemotron Super to an existing local vLLM server running Nemotron Ultra.

Steps

1. Establish a clean NVIDIA baseline by rebuilding the Hermes sandbox.

2. Verify the live route:

   node ./bin/nemohermes.js tm inference get

   Observed:

   Provider: nvidia-prod
   Model:    nvidia/nemotron-3-super-120b-a12b
   

3. Verify the registry:

   node ./bin/nemohermes.js list

   The tm entry reported nvidia-prod / nvidia/nemotron-3-super-120b-a12b.

4. Verify the in-sandbox Hermes configuration:

   node ./bin/nemohermes.js tm config get \
     --key _nemoclaw_upstream.provider --format yaml
   
   node ./bin/nemohermes.js tm config get \
     --key model.default --format yaml

   Observed:

   nvidia-prod
   nvidia/nemotron-3-super-120b-a12b
   

5. Confirm the local vLLM endpoint on port 8000 advertises the target model:

   curl -sf http://127.0.0.1:8000/v1/models |
     jq -r '.data[].id'

   Target model:

   nvidia/nemotron-3-ultra-550b-a55b
   

6. Switch the healthy sandbox to vLLM:

   node ./bin/nemohermes.js tm inference set \
     --provider vllm-local \
     --model nvidia/nemotron-3-ultra-550b-a55b
   switch_status=$?
   
   echo "exit: $switch_status"

   Observed:

   Setting OpenShell inference route: vllm-local / nvidia/nemotron-3-ultra-550b-a55b
   Syncing Hermes model route in sandbox 'tm'...
   Inference route synced for 'tm': nvidia/nemotron-3-ultra-550b-a55b
   

Result

The clean NVIDIA Endpoints -> local vLLM switch succeeded. The reported compatibility-hash failure did not reproduce in this related scenario.

An earlier retry of the already-selected vllm-local / Ultra route reached:

[SECURITY] compat hash does not match frozen Hermes inputs during non-root reconciliation

However, that was a same-route retry from a stale state, not a model switch, so we do not count it as a valid reproduction.

Exact scenario still requiring QA verification

The issue reports a healthy vLLM model A -> vLLM model B transition. To reproduce that exact scenario, please clarify and record:

1. The model ID served before the switch.
2. How the second model is exposed:
   • replacement on the registered port 8000 endpoint;
   • a second vLLM server on another port; or
   • a router in front of multiple vLLM servers.
3. The output of /v1/models before the switch.
4. The complete inference set output and exit status.
5. The route, registry, and Hermes model.default values after the switch.

Could you rerun that exact local-vLLM-A -> local-vLLM-B scenario on current source? I suggest keeping this issue open until that setup is clarified and verified.

[hunglp6d]

Live test on current lkg — DGX GB300WS Station, vllm-local A→B

Ran the reported flow on a Station box. I did not reproduce the reported message, but the same command fails the same way for a different reason, and one defect hides it from the exit code.

NemoClaw	v0.0.103 (lkg, db31c286129e)
OpenShell	0.0.85 (docker)
Hermes	v0.19.0
Host	DGX GB300WS, DGX OS 7.5.0-GB300ws-GB200ws, profile supported-colossus-baseos
Provider	vllm-local, managed nemoclaw-vllm on :8000

Steps

# 1. fresh sandbox on vllm-local, model A = nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8
nemohermes onboard      # NEMOCLAW_PROVIDER=install-vllm NEMOCLAW_VLLM_MODEL=nemotron-3-nano-4b

# 2. A -> B, same provider
nemohermes probe inference set --provider vllm-local --model Qwen/Qwen3.6-27B-FP8
# exit 0. All three layers agree, all three digests converge. Healthy.

# 3. first stop/start since onboarding
nemohermes probe stop     # exit 0
nemohermes probe start    # exit 1

# 4. re-run the exact command from step 2
nemohermes probe inference set --provider vllm-local --model Qwen/Qwen3.6-27B-FP8

Result

  Setting OpenShell inference route: vllm-local / Qwen/Qwen3.6-27B-FP8
  Syncing Hermes model route in sandbox 'probe'...
  Warning: gateway and registry now use vllm-local / Qwen/Qwen3.6-27B-FP8, but writing the
  in-sandbox config failed: Command failed: docker exec … hermes-runtime-config-guard.py write-config …
[SECURITY] Hermes runtime config guard refuses mutation under a foreign PID 1

  Run 'nemohermes probe rebuild' to finish applying the model inside the sandbox.
STEP_H4_EXIT=0

Step 4 is byte-identical to step 2. Step 2 succeeded; step 4 refuses. That is the same-route retry that was read as stale state — it is not stale state, and it is persistent: re-running it after the sandbox returned to phase Ready produced the identical refusal.

Three findings

1. inference set exits 0 after a hard in-sandbox failure. The gateway route and the registry are mutated, the in-sandbox layer is not, and the command reports success. Any script checking the exit code sees an applied model switch. inference-set.ts:1261-1272 turns the write failure into a warning; inference-set-gateway-restart.ts:115 withholds the synced line but never throws.

2. The first stop/start after onboarding can leave the sandbox running with no Hermes process. After step 3:

PID   PPID  USER     COMMAND
  1      0  root     /opt/openshell/bin/openshell-sandbox
151      1  sandbox  sleep infinity

$ ls -la /run/nemoclaw     # empty — no hermes-startup-ready marker

NemoClaw still printed Hermes Agent gateway is running in 'probe', and probe status still reported Agent: Hermes Agent v0.19.0. With PID 1 being the bare supervisor and the startup-ready marker absent, the guard's host-action gate (runtime-config-guard.py:856-869) refuses every host mutation from then on. probe recover did not fix it; a second stop/start did.

In a separate run the same first restart instead recreated the container (docker-gpu-patch-recreate.ts clones from the immutable image ID, so the writable layer is dropped), which silently reverted /sandbox/.hermes/config.yaml and the strict anchor to their onboard content while the CLI printed Workspace state is preserved and exited 0.

3. The strict anchor is already stale on a brand-new healthy sandbox. Measured on two independent onboards:

live   .env  dcb71e1e…      live   .env  3acbf401…
compat .env  dcb71e1e…  ok  compat .env  3acbf401…  ok
strict .env  15170b1a…  NO  strict .env  15170b1a…  NO

Startup mints API_SERVER_KEY as sandbox, which can advance the sandbox-writable compat anchor but not root-owned /etc/nemoclaw/hermes.config-hash. So _verify_strict_hash fails on the first host config write of every sandbox's life and _reconcile_nonroot_startup_api_key_hash is the normal path, not an edge case — and its first gate is compat == live bytes (runtime-config-guard.py:1823), which is the message in this issue. One compat drift takes out the whole host-side config surface at once, with rebuild as the only exit.

What differs from this report

• My repro needs the first stop/start between the switch and the failure; this issue does not mention a restart.
• My model B was never served by the managed vLLM — inference set moved the route in ~2 s without restarting the server, so the local backend still served only model A. probe status flags that; inference set does not. @mercl-lau — was your model B actually being served when the switch failed?
• The refusal came from the PID-1 gate, not the compat-hash gate.

Full probe log (both rounds, all digests, anchors, posture and mtimes at every step) available on request.