[DGX Station GB300][Security] After rebuild, OpenClaw sandbox shields report unlocked but skills dir + credentials stay locked — skill install fails and agent turns EACCES

[wangericnv]

Description

After nemoclaw {sandbox} rebuild --yes, an OpenClaw sandbox ends up in a contradictory shields state: shields status reports the config UNLOCKED (permissive, "auto-unlock for rebuild"), but the agent skills directory and the credentials directory remain LOCKED. As a result, skill install fails claiming shields are up, and OpenClaw agent turns fail with EACCES on the credentials file. shields down cannot recover the state because it believes the config is already unlocked.

Platform scope: Reproduced on DGX Station GB300 (aarch64). Other platforms not tested.
Regression: Unknown — earlier versions not tested.

Likely the same shields-lock family as #7538 (there: shields down fails from a valid mutable-default posture and rollback flips it to locked). This report is the rebuild trigger and additionally surfaces the skill-install block and the agent-turn EACCES.

Environment

Device:        DGX Station GB300
OS:            Ubuntu 24.04.4 LTS
Architecture:  aarch64
NemoClaw:      v0.0.98
Agent:         OpenClaw

Steps to Reproduce

1. Onboard an OpenClaw sandbox and exercise shields (e.g. nemoclaw {sandbox} shields up then shields down).
2. Rebuild the sandbox: nemoclaw {sandbox} rebuild --yes.
3. Check shields: nemoclaw {sandbox} shields status.
4. Try to install a skill: nemoclaw {sandbox} skill install /tmp/some-skill.
5. Run an agent turn: nemoclaw {sandbox} exec -- openclaw agent --agent main --json -m "Reply OK".
6. Try to lower shields: nemoclaw {sandbox} shields down.

Expected Result

After rebuild with shields reporting unlocked, the agent skills directory and credentials are actually accessible: skill install succeeds and agent turns run normally. If any lock remains, shields down clears it.

Actual Result

Step 3 shields status:  "Reason: auto-unlock for rebuild | Policy: permissive"  (reports UNLOCKED)
Step 4 skill install:   RC=1 -> "Skill uploads write to the agent skills directory, which is locked
                        while shields are up. If shields are up, run `nemoclaw {sandbox} shields down`
                        before installing skills."
Step 5 agent turn:      "GatewayClientRequestError: Error: EACCES: permission denied, lstat
                        '/sandbox/.openclaw/credentials/oauth.json'" (reply text becomes
                        "Agent couldn't generate a response").
                        An `exec -- ls -ld /sandbox/.openclaw/credentials` also prints
                        "OpenClaw permission cleanup failed".
Step 6 shields down:    "Config is already unlocked for {sandbox} (since ...). Run
                        `nemoclaw shields up` first, or use --extend (not yet implemented)." -> cannot recover.

So shields status and the actual filesystem lock state disagree: status says unlocked, but skills dir + credentials stay locked; installs and agent turns are broken; shields down refuses because it thinks the config is already unlocked. Only a full re-onboard clears it.

Logs

Not captured beyond the Actual Result output above.

[hunglp6d]

Reproduction notes

TL;DR: The reproduction steps as written (shields up → rebuild --yes → observe) do not trigger this on a fast x86 host — the closing rebuild relock succeeds and the host shields state + in-container state-dir posture stay consistent (shields UP). The split only appears when the closing shields relock loses a race to an in-sandbox privileged reconciler that re-permissions /sandbox/.openclaw/.config-hash back to mutable during the relock's re-confirm window. That reconciler runs natively on DGX Station (slower + gateway perm-normalization active); on a fast idle host it doesn't land in the default 750 ms window. Forcing that race reproduces every symptom. This is a cross-platform defect (reproduced on x86) — DGX Station just makes the race natural, it is not an aarch64/Station-specific bug.

Environment

NemoClaw:  v0.0.99   OpenClaw: 2026.7.1   OpenShell: 0.0.85
Host:      Ubuntu x86_64, docker, no GPU
Agent:     OpenClaw (NVIDIA Endpoints, gateway inference route)

1. Issue steps alone — not reproduced (x86)

shields up → rebuild --yes ends cleanly; both layers agree:

Re-applying shields lockdown...
Restoring restrictive policy from snapshot...
Lockdown active for my-assistant
✓ Shields restored to UP

shields status → UP; stat on skills/credentials stays consistent with UP. No split.

2. What actually triggers it

The closing relock does a lock → wait (settle window) → re-lock/re-confirm cycle (the #4663 mitigation). It fails only if something reverts the just-locked .config-hash permissions during that wait. To force the race deterministically I (a) widened the settle window, and (b) reproduced the in-sandbox reconciler's exact action — reverting .config-hash to 660 sandbox:sandbox — during the window. Nothing here hand-places the buggy state; the split is produced entirely by the real relock code failing.

Terminal 1:

nemoclaw my-assistant shields up
NEMOCLAW_SHIELDS_SETTLE_MS=60000 nemoclaw my-assistant rebuild --yes

Terminal 2, started when Restoring workspace state... prints (simulates the DGX-Station gateway reconciler):

for i in $(seq 1 90); do
  C=$(docker ps --filter label=openshell.ai/sandbox-name=my-assistant --format '{{.Names}}' | head -1)
  [ -n "$C" ] && docker exec -u 0 "$C" sh -c 'chattr -i /sandbox/.openclaw/.config-hash 2>/dev/null; chmod 660 /sandbox/.openclaw/.config-hash; chown sandbox:sandbox /sandbox/.openclaw/.config-hash' 2>/dev/null
  sleep 1
done

3. Reproduced — the relock fails, leaving the split

Rebuild tail (trimmed):

Re-applying shields lockdown...
Restoring restrictive policy from snapshot...
CRITICAL: OpenClaw lock rollback could not restore the trusted posture. Restore from a trusted
  backup and recreate the sandbox. top-level config rollback failed: Config not locked: OpenClaw
  config guard lock [config-not-locked] /sandbox/.openclaw/.config-hash: restart seal requires the
  exact shields-locked file posture
ERROR: Config not locked: /sandbox/.openclaw/.config-hash mode=660 (expected 444),
  /sandbox/.openclaw/.config-hash owner=sandbox:sandbox (expected root:root)
Config remains unlocked — manual intervention required.
Failed to re-apply shields lockdown.

Resulting split state — host state reads UNLOCKED, filesystem stays LOCKED:

$ nemoclaw my-assistant shields status
  Shields: DOWN (temporarily unlocked)
  Auto-lockdown in: 26m 25s
  Reason:  auto-unlock for rebuild
  Policy:  permissive

$ nemoclaw my-assistant exec -- stat -c '%U:%G %a %n' /sandbox/.openclaw/skills /sandbox/.openclaw/credentials
  root:sandbox 755 /sandbox/.openclaw/skills
  root:root 700 /sandbox/.openclaw/credentials
  OpenClaw permission cleanup failed (command exit 0; cleanup exit 1): ... (before repair:
    /sandbox/.openclaw mode 755 (expected 2770 setgid+group-writable); openclaw.json mode 444
    (expected 660 group-writable); owner root:root (expected sandbox:sandbox))

$ nemoclaw my-assistant skill install /tmp/sandbox-smoke-check
  ✓ Validated SKILL.md (name: sandbox-smoke-check, 1 file)
  Failed to upload 1 file(s): SKILL.md
  Skill uploads write to the agent skills directory, which is locked while shields are up.
  If shields are up, run `nemoclaw my-assistant shields down` before installing skills.

shields status says unlocked, but skills/credentials remain root-locked → skill install fails, agent turns hit EACCES on credentials, and shields down refuses (Config is already unlocked …), so there is no in-band recovery.

4. Why the relock can't self-heal

Once the first lock lands, .openclaw is in a locked directory posture, so the re-confirm routes into the guard's verify-only branch — it re-checks the files but never re-applies the lock. A single .config-hash revert therefore dooms every retry (restart seal requires the exact shields-locked file posture). Meanwhile the recursive state-dir chown (skills→root:sandbox, credentials→root:root) from the first pass is never rolled back, and the host shields state never advances to shieldsDown:false — leaving unlocked-host / locked-filesystem.

(Note: on this x86 host the split self-heals when the deferred auto-restore timer fires later, because the injected reconciler has stopped; on DGX Station the reconciler is persistent, so it stays stuck — matching the original report.)

Fixing on it — PR to follow.

[hunglp6d]

Resolved in v0.0.103 (and later). This report combined two independent lockdown defects; both are now fixed and released.

1. Config-lock drift split — fixed by #8086, shipped in v0.0.102.
On the rebuild's closing relock, an in-sandbox reconciler could re-permission .config-hash (permissions only, contents unchanged) after the lock. The config guard's already-locked path treated that as a hard config-not-locked reject, so the relock never re-applied: host shields state was left unlocked while the in-container state directories stayed root-locked, with no clean recovery. The guard now re-seals a permissions-only drift in place instead of failing closed (content or structural drift still fails closed), so the relock converges to a consistent state. This resolves the shields status split (step 3), the skill install block (step 4), and the shields down refusal (step 6).

2. Agent-turn EACCES on credentials/oauth.json — fixed by #7603 (closes #7545), shipped in v0.0.103.
With shields up, the state-dir guard locked the confidentiality root to root:root 700, so the gateway's step-down user could not traverse it and the startup lstat of the optional legacy oauth.json returned EACCES instead of ENOENT. Lock now sets the confidentiality root to root:sandbox 710 (group traverse, no read), so the file is reported absent rather than access-denied. This resolves the agent-turn EACCES (step 5).

Residual scope note: the config re-seal ends the deterministic reject loop and heals a reconciler that reverts once after the lock. A reconciler that reverts continuously can still lose a relock to a narrow settle-window race; that pre-existing race is tracked in #4663 (the re-seal narrows it; a durable immutable flag would close it, but that is unavailable on overlayfs). It is not the defect reported here.

Both fixes are present from v0.0.103 onward (latest v0.0.105). Closing as resolved.