[DGX Spark][Hermes] current LKG: gateway restart quarantines agent; rebuild can leave registry/live OpenShell divergent

[NickOProjects]

Investigation Summary

• NVIDIA's stock lkg install path resolved to NemoClaw db31c286129e878c3356eed49f76ab259561e47e on DGX Spark; Hermes v0.19.0 and OpenShell 0.0.85 were used with no product patches, custom integrations, or custom MCP bridges.
• Family A: after supported gateway restart, the sandbox remained live but Hermes stopped; startup reported Hermes config hash does not match persisted inputs / HERMES_MCP_CONFIG_DRIFT and quarantined automatic respawn. The MCP bridge registry was empty and persisted MCP intended / applied digests matched.
• nemohermes hermes recover failed after 151.553 s and left Hermes Agent: not running; no manual gateway launch or unsupported recovery was attempted.
• Family B: supported nemoclaw hermes rebuild --yes backed up state, accepted sandbox deletion, could not confirm deletion, and aborted before registry removal/recreation. Subsequent read-only checks consistently showed hermes registered locally but absent from live OpenShell.
• Observed gateway restart record was 1 success in 3 attempts. A possible correlation with prior work execution was observed but the planned discriminator was never completed, so no causal claim is made.

Description

What happened

I ran a bounded Hermes persistence/lifecycle trial on DGX Spark using the
NVIDIA NemoClaw LKG revision:

db31c286129e878c3356eed49f76ab259561e47e

As of this filing, NVIDIA's installer still defaults to lkg, and the lkg
branch still resolves to the same commit.

Hermes itself successfully accepted, claimed, and executed durable queued work.
Task and session state also survived one earlier successful supported rebuild.

The lifecycle failures appeared in two stages:

Family A — gateway restart / config-integrity quarantine

A supported managed gateway restart left the sandbox live but the Hermes Agent
gateway stopped.

Startup reported:

Hermes config hash does not match persisted inputs

and:

HERMES_MCP_CONFIG_DRIFT: MCP intent cannot be matched to the persisted gateway state; rebuild the sandbox from its NemoClaw registry state

Automatic respawn was then quarantined.

Relevant negative controls:

• no custom MCP bridges were added
• managed MCP bridge registry was empty: "bridges": []
• persisted MCP intended and applied digests were identical
• no NemoClaw, OpenShell, or Hermes product patches were applied
• no custom integrations were present

The documented lightweight recovery command:

nemohermes hermes recover

failed after 151.553 seconds and left Hermes Agent: not running.

Family B — supported rebuild / registry-live divergence

Following the recovery failure, I ran the supported command recommended by
NemoClaw:

nemoclaw hermes rebuild --yes

Preflight checks passed and state backup completed. The rebuild then reported:

Sandbox delete was accepted, but OpenShell did not confirm that the sandbox is absent.

and:

Aborting rebuild before registry removal and sandbox recreation.

The command exited 1.

Subsequent read-only checks consistently reported that hermes remained
registered locally but was absent from the live OpenShell gateway.
nemohermes list still listed the sandbox while nemohermes hermes status
exited 1.

No manual reconciliation, restart retry, rebuild retry, onboard, sandbox
deletion, direct gateway launch, patch, or reinstall was performed afterward.

Expected behavior

A supported gateway restart should either return Hermes to a healthy state or
leave a state that the documented recover path can restore without manual
intervention.

A supported rebuild should either complete successfully or fail while leaving
the local NemoClaw registry and live OpenShell gateway in a consistent,
supported state.

Additional version context

I have not rerun this reproducer on v0.0.104.

Because v0.0.104 contains potentially relevant lifecycle changes, I would
appreciate guidance on whether those changes are expected to intersect either
failure family and whether v0.0.104 is expected to advance to lkg.

Reproducibility note

Family A reproduced on 2 of 3 supported gateway restart attempts during this
trial. Family B was observed once during the subsequent supported recovery
sequence and was intentionally not repeated after it left local registry and
live OpenShell state divergent.

Possibly related: #8614 reports Hermes gateway quarantine/rebuild behavior on
v0.0.102 following structural config writes. This report differs because no
comparable custom structural config writes or custom MCP bridges were added,
and it additionally reaches registry/live-OpenShell sandbox divergence.

I am an independent DGX Spark owner/operator and am happy to reproduce a
targeted fix or newer LKG on the same hardware and provide additional sanitized
evidence if useful.

Reproduction Steps

1. Install/onboard Hermes using stock NemoClaw at
   db31c286129e878c3356eed49f76ab259561e47e
   (the LKG revision used for the trial), with no product patches or custom MCP
   bridges.

2. Configure sandbox hermes through the supported onboarding path using the
   NVIDIA hosted inference endpoint and
   nvidia/nemotron-3-super-120b-a12b.

3. Confirm the sandbox and Hermes gateway are healthy with:

   nemohermes hermes status

4. Use Hermes normally. In this trial it autonomously claimed and completed
   disposable queued/Kanban work.

5. Perform a supported managed gateway restart:

   nemohermes hermes gateway restart

   Observed restart record across the trial was 1 success in 3 attempts.
   The two failures occurred after normal Hermes work, but a planned clean
   work/no-work discriminator was not completed, so this is not claimed as a
   causal requirement for reproduction.

6. On failure, check:

   nemohermes hermes status

   Observed: sandbox remained live/Ready but Hermes Agent: not running, with
   startup logging HERMES_MCP_CONFIG_DRIFT and automatic respawn quarantine.

7. Attempt the documented lightweight recovery:

   nemohermes hermes recover

   Observed: exit 1 after 151.553 s; Hermes remained not running. The command
   instructed rebuilding the sandbox.

8. Run the supported rebuild:

   nemoclaw hermes rebuild --yes

   Observed: state backup succeeded; sandbox deletion was accepted but could
   not be confirmed; rebuild aborted before registry removal/recreation.

9. Check the resulting state:

   nemohermes hermes status
nemohermes list

   Observed:

   • status exits 1 and says hermes is registered locally but absent from the
     live OpenShell gateway
   • list exits 0 and still lists hermes

10. Wait and repeat only the read-only status/list checks.

    The same registry/live-gateway divergence remained present approximately
    ten minutes after the failed rebuild. No repair action was attempted.

Environment

• OS: Ubuntu 24.04.4 LTS
• Kernel: 6.17.0-1026-nvidia
• Architecture: aarch64
• Hardware: NVIDIA DGX Spark / GB10
• Node.js: v22.23.2
• Docker: 29.2.1 (build a5c7197)
• NemoClaw: v0.0.103
• NemoClaw source revision: db31c286129e878c3356eed49f76ab259561e47e
• OpenShell: 0.0.85
• Hermes Agent: v0.19.0
• Inference: NVIDIA hosted endpoint, nvidia/nemotron-3-super-120b-a12b

Debug Output

Sanitized output from the issue form's requested command:

nemoclaw debug --quick

Personal hostname/username, process IDs, unrelated Docker container inventory,
local paths, and other machine-specific details have been omitted. Relevant
error text has not been altered.

Warning: default sandbox 'hermes' exists in the local registry but not in OpenShell.
  Use --sandbox NAME to target a specific sandbox, or run nemoclaw onboard again.

[debug] Collecting diagnostics for sandbox 'hermes'...
[debug] Quick mode: true

═══ System ═══

OS: Ubuntu 24.04.4 LTS
Kernel: 6.17.0-1026-nvidia
Architecture: aarch64

═══ GPU ═══

NVIDIA GB10
NVIDIA driver: 580.159.03
CUDA: 13.0

═══ OpenShell ═══

Server Status

  Gateway: nemoclaw
  Status: Connected
  Version: 0.0.85

No sandboxes found.

Error: × code: 'Some requested entity was not found', message: "sandbox not found"
  (command exited with non-zero status)

Error: × code: 'Some requested entity was not found', message: "sandbox not found"
  (command exited with non-zero status)

═══ Onboard Session ═══

status: complete
resumable: false
mode: non-interactive
sandboxName: hermes
provider: nvidia-prod
model: nvidia/nemotron-3-super-120b-a12b
failure: null

gatewayAuthority:
  gatewayName: nemoclaw
  mode: nemoclaw-managed

machine:
  state: complete

[debug] Done.

debug_quick_exit_code=0

Additional read-only consistency checks:

nemoclaw list

• exit code: 0
• lists sandbox hermes

nemohermes list

• exit code: 0
• lists sandbox hermes

nemohermes debug --quick --sandbox hermes

• exit code: 1
• reports: Error: Sandbox 'hermes' is not registered.
• then instructs: Run nemoclaw list to see available sandboxes.

No repair or state-changing operation was performed between these commands.

Logs

Relevant preserved lifecycle excerpts follow. Credential values, personal
identifiers, absolute local paths, and unrelated configuration/log content are
intentionally omitted.

### Family A — gateway restart / integrity quarantine

After a supported:

`nemohermes hermes gateway restart`

the preserved startup log contained:


[config] Config integrity check skipped for mutable default (/sandbox/.hermes/.config-hash is not locked)
[SECURITY] Hermes config hash does not match persisted inputs
[SECURITY] HERMES_MCP_CONFIG_DRIFT: MCP intent cannot be matched to the persisted gateway state; rebuild the sandbox from its NemoClaw registry state
[SECURITY] Hermes automatic respawn is quarantined until MCP integrity is restored by rebuilding the sandbox


The persisted MCP marker at that time showed matching digests:


intended=44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
applied=44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a


Managed MCP bridge registry:


"bridges": []


### Documented recovery attempt

Command:

`nemohermes hermes recover`

Relevant result:


Probe failed: Hermes Agent gateway is not running in 'hermes' and automatic recovery failed.
The in-sandbox supervisor quarantined gateway relaunch after a startup refusal. Retrying the restart cannot clear it.
Restore the registered configuration and refresh its integrity metadata with `nemoclaw hermes rebuild --yes`.


Observed:

- exit code: 1
- elapsed: 151.553 s
- post-recovery: `Hermes Agent: not running`

No manual gateway launch or unsupported repair was attempted.

### Family B — supported rebuild

Command:

`nemoclaw hermes rebuild --yes`

Relevant output:


Backing up sandbox state...
✓ State backed up (15 directories, 5 files)

Deleting old sandbox...

Sandbox delete was accepted, but OpenShell did not confirm that the sandbox is absent.
Aborting rebuild before registry removal and sandbox recreation.
State backup is preserved at: [local path omitted]
Sandbox deletion could not be confirmed.


Observed:

- exit code: 1
- elapsed: 40.336 s

Subsequent status reported:


Sandbox 'hermes' is registered locally, but is not present in the live OpenShell gateway.


Later read-only checks continued to show:

- `nemoclaw list` -> exit 0, lists `hermes`
- `nemohermes list` -> exit 0, lists `hermes`
- live OpenShell -> no sandboxes
- targeted debug -> exit 1, reports `Sandbox 'hermes' is not registered.`

No subsequent recovery, rebuild retry, onboard, deletion, manual registry
reconciliation, OpenShell repair, patch, or reinstall was attempted.

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[wscurran]

✨ Thanks for the detailed investigation summary. This reports a Hermes quarantine and recovery failure on DGX Spark after gateway restart. Maintainers will review the sandbox gateway interaction and Hermes recovery path.

[cmesser1]

We hit Family A again on an unmodified config path and can add the missing piece: what actually causes the drift. It isn't tampering or operator error — Hermes itself writes its own config at runtime, and the startup ordering turns every such write into a time bomb.

The two runtime writers

1. Hermes onboarding flags (recurring, fires by itself). The agent's onboarding.py mark_seen() writes onboarding.seen.<flag> into config.yaml the first time each behaviour fork occurs (e.g. busy_input_prompt fires the first time a user messages the agent while it is busy). We observed it twice in 24h on one install — the second time hours after a rebuild, because the rebuild reset the flag and ordinary use re-tripped it. Four such flags exist, so every rebuild re-arms up to four chances for ordinary use to invalidate the anchor.

2. /sethome (deliberate, one-way). The gateway's Slack slash-command handler appends SLACK_HOME_CHANNEL* to .env. (v0.0.101+ preserves these across rebuilds via #7817, which removes the re-arming on rebuild — but the original runtime append still invalidates the anchor on the install where it first runs.)

Why boot can never self-heal

.config-hash is written only at gateway start, but in prepare_hermes_nonroot_runtime, inspect_hermes_mcp_integrity runs before refresh_hermes_runtime_config_hashes under || return 1 (still true on current main, agents/hermes/start.sh ~3104 vs ~3111). So after any legitimate self-write, every subsequent boot dies at the gate before reaching the refresh, the supervisor burns its crash budget, and the sandbox lands in quarantine / a docker restart loop. recover then polls a container that can never come up.

Since the guard hard-fails on any byte of drift (runtime-config-guard.py: "Hermes config hash does not match persisted inputs"), the agent's own supported behaviour is indistinguishable from tampering.

Reproduction without any restart involved

1. Fresh rebuild (clean anchor).
2. Message the agent while it is busy once (onboarding.seen.busy_input_prompt gets written).
3. sha256sum config.yaml .env vs .config-hash → already divergent.
4. Any restart — planned reboot, docker restart, gateway restart — crash-loops with HERMES_MCP_CONFIG_DRIFT.

We confirmed step 4 on a planned host reboot (v0.0.95, Ubuntu 24.04, Hermes Agent 0.18.0): the sandbox container crash-looped at Restarting (1) with exactly the two log lines quoted in this issue.

Suggested fixes

• Either refresh/re-anchor before the integrity gate for the mutable-default (non-shields) posture, or
• route the agent's own runtime writes (onboarding flags, /sethome) through the guarded write-config transaction so config and hashes update together — which is already the documented contract for config set / inference set, or
• pre-seed the onboarding.seen.* flags at image build/rebuild time so the agent has no reason to self-write config.yaml.

Until then, every mutable-default Hermes install drifts within hours of normal use and is one restart away from a rebuild.

Addendum: what the recovery rebuild itself surfaced (v0.0.103)

Recovering via the documented rebuild --yes on v0.0.103 hit four more config-lifecycle/preflight issues before succeeding on the sixth attempt:

1. glibc preflight probe race. After a ~30-minute cold base-image build, the docker run --entrypoint /usr/bin/ldd <image> --version probe (20s timeout, errors swallowed) timed out while the daemon was still committing layers → "has glibc unknown; expected >= 2.39" → abort. The same probe against the same image passes in <1s afterwards. Suggest probing with retry/backoff, and distinguishing "probe failed" from "version too old".
2. Backup sanitizer caps reject real-world state. The snapshot sanitizer fully reads every .json/.yaml/.yml/.env under the backup root with 16MB/file and 32MB/total caps. A year-old Hermes here had a 26.4MB skills/.hub/index-cache/clawhub_catalog_v1.json and ~38MB of sessions/*.json, so every backup failed closed — reported only as "Credential sanitization failed; removed the incomplete backup" with the cause swallowed. Any long-lived install will hit this; the caps need to scale, skip known-bulk non-credential paths (session logs, index caches), or at minimum report the offending file.
3. Cron preflight hard-blocks on a missing script with no supported bulk remedy surfaced. An active job whose script: file disappeared (itself apparently lost by a previous v0.0.95 rebuild's backup/restore) fails the whole rebuild; error names the job index but not the pause command that unblocks it.
4. config set permanently broken after rebuild on non-root sandboxes. Rebuild writes the root-owned frozen anchor (/etc/nemoclaw/hermes.config-hash) before the MCP-bridge restore mutates config.yaml; non-root startup refreshes only the sandbox-side .config-hash, so the frozen copy stays stale indefinitely and every subsequent config set fails with "[SECURITY] compat hash does not match frozen Hermes inputs during non-root reconciliation". Root-cause family of config set: undocumented gateway restarts; a partial structural write can quarantine the sandbox (rebuild required); rebuild-then-recover ordering is load-bearing but undocumented #8614. Manual fix: copy the sandbox-side anchor over the frozen file as root inside the container.

Also reproduced Family A verbatim during the rebuild's "preserve MCP bridges" step: GATEWAY_HEALTH_TIMEOUT → "managed supervisor is quarantined without another launch". A full stop + start cleared it and the retry succeeded with a freshly-started gateway.

[NickOProjects]

Update from a bounded retest on the same physical DGX Spark, now using NemoClaw/NemoHermes v0.0.113, OpenShell 0.0.106, and Hermes Agent v0.19.0.

I used a clean isolated canary on a separate gateway (8990) and left the original failed/default-port environment untouched.

What improved

The original failure families from this issue did not reproduce in this retest:

• clean onboarding: PASS
• normal Hermes file-tool workflow: PASS
• session persistence: PASS
• Kanban persistence: PASS
• cron persistence and automatic execution: PASS
• supported managed gateway restart: PASS
• registry/live sandbox identity after restart: PASS

In particular, I did not reproduce the earlier config-drift quarantine or registry/live OpenShell divergence.

New bounded failure: supported stop → recover

I then exercised only the documented lifecycle path:

1. supported stop: PASS
2. supported recover: FAIL

Exact failure:

Hermes cron prepare-recover privileged transport was unavailable

Post-failure state:

• same sandbox identity remained
• Docker container still existed but was stopped
• sandbox phase became Error
• API was unhealthy
• owner repair interventions: 0

I stopped there intentionally: no retry, rebuild, registry reconciliation, manual container start, Docker repair, or lifecycle surgery.

Possible ordering issue

The current v0.0.113 documentation describes recover as able to recover/start an existing stopped Docker-driver sandbox.

Looking at the current Hermes recovery path, it appears to order the relevant work approximately as:

prepareHermesCronRestoreRecovery()
→ connectSandbox() / container recovery
→ recoverHermesCronRestore()

prepareHermesCronRestoreRecovery() invokes the root-owned Hermes cron controller through the privileged direct-container transport.

That transport appears to resolve a running OpenShell-managed container. After a normal supported stop, however, the target container is intentionally stopped.

So there appears to be a circular dependency at this boundary: recovery attempts the Hermes cron prepare-recover operation before restoring the container, while the privileged transport used for that operation appears to require the container to already be running.

I am not claiming this is proven root cause; it is the strongest code-level hypothesis I have for the observed failure. Current main also appears to retain the prepare → connect → recover ordering, while the unit test mocks the control calls and therefore does not exercise the real stopped-container transport boundary.

Is this ordering intentional?

If useful, I can run one targeted commit / PR / discriminator on the same DGX Spark and return sanitized evidence. I would prefer to keep that bounded rather than start another open-ended recovery/debug campaign.

[apurvvkumaria]

Current-main verification\n\nVerified commit 5d4ac392 with focused E2E run 34326954683, attempt 1. The run used Ubuntu 24.04 x86_64, Docker, OpenShell 0.0.106, Hermes 0.20.6, mock inference, and the revision-bound managed-image cohort from run 34321731082.\n\nThe original failures remain fixed:\n\n- Hermes gateway restart passed. The sandbox remained listed and healthy, both inference routes returned PONG, configuration integrity passed, and Hermes did not enter quarantine or a crash loop.\n- Legacy Hermes rebuild passed from the v0.0.80 Hermes 0.18.0 fixture to Hermes 0.20.6. The replacement published a matching lifecycle identity, preserved the registry session and state marker, restored inference with exit status 0, found no credential leak in the backup, and removed all test resources.\n\nThe reporter’s earlier same-DGX-Spark retest on v0.0.113 also did not reproduce these two failures. This current run used x86_64, so it supplements that hardware result.\n\nThe later whole-sandbox stop → recover report is a separate boundary and was not exercised here. It should receive a focused issue only if it reproduces on current main.\n\nClosing #8649 as completed.