fix(e2e): repair recovered runtime qualification

ci/platform-matrix.json

@@ -203,7 +203,7 @@
     {
       "name": "Sandboxed execution",
       "status": "caveated",
-      "notes": "Landlock, seccomp, network namespace isolation, no-new-privileges, privilege dropping, and process limits (ulimit -u 512 at `scripts/lib/sandbox-init.sh:237`) are tested and on by default. The `DANGEROUS_CAPS` list at `scripts/lib/sandbox-init.sh:288-299` drops cap_sys_admin, cap_sys_ptrace, cap_net_raw, cap_dac_override, cap_sys_chroot, cap_fsetid, cap_setfcap, cap_mknod, cap_audit_write, cap_net_bind_service with `capsh --drop` when CAP_SETPCAP is present. Limitation (active issue #3280): the fail-closed bounding-set gate is opt-in via `NEMOCLAW_REQUIRE_CAP_DROP=1`; the default is warn-and-continue so hosts without CAP_SETPCAP still boot, which means dangerous caps can remain in the bounding set on some hosts even though the effective set is empty."
+      "notes": "Landlock, seccomp, network namespace isolation, no-new-privileges, privilege dropping, and process limits (nproc 512 at `scripts/lib/sandbox-rlimits.sh:8`) are tested and on by default. The `DANGEROUS_CAPS` list at `scripts/lib/sandbox-init.sh:275-286` drops cap_sys_admin, cap_sys_ptrace, cap_net_raw, cap_dac_override, cap_sys_chroot, cap_fsetid, cap_setfcap, cap_mknod, cap_audit_write, cap_net_bind_service with `capsh --drop` when CAP_SETPCAP is present. Limitation (active issue #3280): the fail-closed bounding-set gate is opt-in via `NEMOCLAW_REQUIRE_CAP_DROP=1`; the default is warn-and-continue so hosts without CAP_SETPCAP still boot, which means dangerous caps can remain in the bounding set on some hosts even though the effective set is empty."
     },
     {
       "name": "Routed inference",

docs/reference/commands.mdx

@@ -1869,7 +1869,9 @@ The route probe treats any final HTTP status from `200` through `499` as reachab
 The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable.
 When the live provider and model match the recorded route, the request also uses the sandbox's recorded API family, including `openai-responses`.
 During route drift, NemoClaw does not carry the sandbox's recorded API family to the live provider and model.
-Each run sends one 16-token request through the stored provider credential, so `status` waits up to 30 seconds for it and consumes provider tokens on a hosted route.
+An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route.
+When the same `status` run recovers a managed gateway, it retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts.
+Each attempt can consume another 16 tokens on a hosted route.
 When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered.
 `inferenceHealth.failureLabel` reports why the inference request failed:
 

docs/reference/platform-support.mdx

@@ -134,7 +134,7 @@ Each row below is a launch-facing capability claim that NemoClaw makes in docs,
 | Capability | Status | Notes |
 |------------|--------|-------|
 | Guided onboarding | Tested | Single-command interactive wizard (`$$nemoclaw onboard`) that walks the user through inference provider selection, credential setup, and sandbox creation or update. It opens the agent dashboard when the selected runtime provides one. Non-interactive mode is supported with `--non-interactive` and `NEMOCLAW_*` environment variables for CI and scripted installs. |
-| Sandboxed execution | Tested with limitations | Landlock, seccomp, network namespace isolation, no-new-privileges, privilege dropping, and process limits (ulimit -u 512 at `scripts/lib/sandbox-init.sh:237`) are tested and on by default. The `DANGEROUS_CAPS` list at `scripts/lib/sandbox-init.sh:288-299` drops cap_sys_admin, cap_sys_ptrace, cap_net_raw, cap_dac_override, cap_sys_chroot, cap_fsetid, cap_setfcap, cap_mknod, cap_audit_write, cap_net_bind_service with `capsh --drop` when CAP_SETPCAP is present. Limitation (active issue #3280): the fail-closed bounding-set gate is opt-in via `NEMOCLAW_REQUIRE_CAP_DROP=1`; the default is warn-and-continue so hosts without CAP_SETPCAP still boot, which means dangerous caps can remain in the bounding set on some hosts even though the effective set is empty. |
+| Sandboxed execution | Tested with limitations | Landlock, seccomp, network namespace isolation, no-new-privileges, privilege dropping, and process limits (nproc 512 at `scripts/lib/sandbox-rlimits.sh:8`) are tested and on by default. The `DANGEROUS_CAPS` list at `scripts/lib/sandbox-init.sh:275-286` drops cap_sys_admin, cap_sys_ptrace, cap_net_raw, cap_dac_override, cap_sys_chroot, cap_fsetid, cap_setfcap, cap_mknod, cap_audit_write, cap_net_bind_service with `capsh --drop` when CAP_SETPCAP is present. Limitation (active issue #3280): the fail-closed bounding-set gate is opt-in via `NEMOCLAW_REQUIRE_CAP_DROP=1`; the default is warn-and-continue so hosts without CAP_SETPCAP still boot, which means dangerous caps can remain in the bounding set on some hosts even though the effective set is empty. |
 | Routed inference | Tested | Provider-routed model calls through the OpenShell gateway, transparent to the agent. The agent uses `inference.local` inside the sandbox; provider credentials stay on the host. Supports every entry in the Providers table. |
 | Declarative network policy | Tested | YAML-defined egress with policy presets. Presets include `slack`, `discord`, `telegram`, `weather`, `openclaw-pricing`, `huggingface`, `npm`, `pypi`, `brew`, and others. Hot-reloadable at runtime with `$$nemoclaw <name> policy add`. |
 | Snapshot and restore | Tested | Create, list, and restore named snapshots of sandbox state with the `$$nemoclaw <name> snapshot` subcommands (`create`, `list`, `restore`). Credential stripping is enforced on capture. Unsafe symlinks are rejected on restore. |

scripts/lib/sandbox-init.sh

@@ -38,13 +38,17 @@ source "${_SANDBOX_INIT_DIR}/sandbox-rlimits.sh"
 # File                         Owner      Mode  Writer   Reader    Sourced?
 # /tmp/nemoclaw-proxy-env.sh   root       444   root     sandbox   YES (/etc shell hooks)
 # /tmp/gateway.log             gateway    644   gateway  all       no (world-readable for diagnostics)
-# /tmp/auto-pair.log           sandbox    600   sandbox  sandbox   no
+# /tmp/auto-pair.log           root*      600   root*    inherited no
 # /tmp/nemoclaw-plugin-refresh.log sandbox 600   sandbox  sandbox   no (OpenClaw refresh output)
 # /tmp/.npm-cache/             sandbox    755   sandbox  sandbox   no (tool data)
 # /tmp/.cache/                 sandbox    755   sandbox  sandbox   no (tool data)
 # /tmp/.config/                sandbox    755   sandbox  sandbox   no (tool data)
 # /tmp/.gnupg/                 sandbox    700   sandbox  sandbox   no (key data)
 #
+# * In non-root mode the sandbox user owns and opens auto-pair.log. In root
+# mode PID 1 owns and opens it before the stepped-down watcher inherits the
+# descriptor; PID 1 has already dropped CAP_DAC_OVERRIDE at that boundary.
+#
 # In non-root mode privilege separation is disabled — all files are
 # owned by sandbox. chmod 444 is best-effort (owner can chmod back).
 # This is an accepted limitation documented in the OpenShell security model.

scripts/nemoclaw-start.sh

@@ -3167,6 +3167,16 @@ PYAUTOPAIR
   echo "[gateway] auto-pair watcher launched (pid $AUTO_PAIR_PID)" >&2
 }
 
+prepare_auto_pair_log() {
+  if [ "$(id -u)" -eq 0 ]; then
+    # PID 1 opens the redirection after CAP_DAC_OVERRIDE is gone, then passes
+    # the already-open descriptor to the stepped-down watcher.
+    _nemoclaw_safe_create_tmp_file /tmp/auto-pair.log 600 root:root
+  else
+    _nemoclaw_safe_create_tmp_file /tmp/auto-pair.log 600
+  fi
+}
+
 # ── Proxy environment ────────────────────────────────────────────
 # OpenShell injects HTTP_PROXY/HTTPS_PROXY/NO_PROXY into the sandbox, but its
 # NO_PROXY is limited to 127.0.0.1,localhost,::1 — missing the gateway IP.
@@ -5894,8 +5904,7 @@ if [ "$(id -u)" -ne 0 ]; then
   write_auth_profile
   harden_auth_profiles
 
-  # Separate log for auto-pair in non-root mode as well.
-  _nemoclaw_safe_create_tmp_file /tmp/auto-pair.log 600
+  prepare_auto_pair_log
 
   prepare_plugin_refresh_log || exit 1
 
@@ -6028,8 +6037,7 @@ if [ ${#NEMOCLAW_CMD[@]} -gt 0 ]; then
   exit "$_nemoclaw_cmd_rc"
 fi
 
-# Separate log for auto-pair so sandbox user can write to it
-_nemoclaw_safe_create_tmp_file /tmp/auto-pair.log 600 sandbox:sandbox
+prepare_auto_pair_log
 
 prepare_plugin_refresh_log || exit 1
 

src/lib/actions/sandbox/forward-recovery.ts

@@ -11,13 +11,16 @@ import {
 } from "../../adapters/openshell/timeouts";
 import * as agentRuntime from "../../agent/runtime";
 import { DASHBOARD_PORT, HERMES_OPENAI_API_PORT } from "../../core/ports";
-import { waitUntil } from "../../core/wait";
 import { getActiveMessagingHostForward } from "../../messaging/host-forward";
 import { hydrateDerivedSandboxMessagingPlanFields } from "../../messaging/hydration";
 import type { SandboxMessagingHostForwardPlan } from "../../messaging/manifest";
 import { parseSandboxMessagingPlan } from "../../messaging/plan-validation";
 import { isRemoteDashboardBindRequested } from "../../onboard/dockerfile-remote-dashboard-bind-contract";
 import { resolveSandboxGatewayName } from "../../onboard/gateway-binding";
+import {
+  waitForForwardRecoveryState,
+  waitForStoppedForwardPortRelease,
+} from "../../onboard/forward-cleanup";
 import {
   resolveSandboxHermesApiPort,
   retargetHermesApiPortInUrl,
@@ -56,9 +59,6 @@ type DashboardForwardStopRunner = (
   options: { ignoreError: true; stdio: "ignore"; timeout: number },
 ) => { status?: number | null };
 
-const FORWARD_RELEASE_TIMEOUT_MS = 5_000;
-const FORWARD_RELEASE_POLL_MS = 250;
-
 function isValidPort(value: unknown): value is number {
   return typeof value === "number" && Number.isInteger(value) && value >= 1 && value <= 65535;
 }
@@ -83,20 +83,6 @@ function runDashboardForwardStopBestEffort(
   }
 }
 
-function confirmDashboardForwardReleased(
-  port: number,
-  isForwardReachable: (port: number) => boolean,
-): boolean {
-  const now = Date.now;
-  return waitUntil(() => !isForwardReachable(port), {
-    deadlineMs: now() + FORWARD_RELEASE_TIMEOUT_MS,
-    initialIntervalMs: FORWARD_RELEASE_POLL_MS,
-    maxIntervalMs: FORWARD_RELEASE_POLL_MS,
-    backoffFactor: 1,
-    now,
-  });
-}
-
 export function resolveSandboxDashboardPort(
   sandboxName: string,
   deps: SandboxPortDeps = {},
@@ -170,7 +156,10 @@ export function teardownSandboxDashboardForward(
       timeout: OPENSHELL_OPERATION_TIMEOUT_MS,
     });
     if (result.status !== 0) return;
-    confirmDashboardForwardReleased(port, deps.isLocalForwardReachable ?? isLocalForwardReachable);
+    waitForStoppedForwardPortRelease(
+      port,
+      deps.isLocalForwardReachable ?? isLocalForwardReachable,
+    );
   } catch {
     // Defense in depth for injected or future runners: teardown is best-effort.
   }
@@ -328,7 +317,7 @@ export function ensureSandboxPortForwardForPort(
       health: forwardHealth,
       portReleased: false,
     };
-    waitUntil(
+    waitForForwardRecoveryState(
       () => {
         stopState.health = isSandboxPortForwardHealthy(sandboxName, port, expectedBind);
         stopState.portReleased = !isLocalForwardReachable(port);
@@ -338,12 +327,7 @@ export function ensureSandboxPortForwardForPort(
           stopState.portReleased
         );
       },
-      {
-        deadlineMs: Date.now() + waitMs,
-        initialIntervalMs: 100,
-        maxIntervalMs: 500,
-        backoffFactor: 1.5,
-      },
+      waitMs,
     );
     if (stopState.health === true && !forceRestart) return acceptSuccessfulForward();
     if (stopState.health === "occupied") return false;
@@ -382,7 +366,7 @@ export function ensureSandboxPortForwardForPort(
   if (waitMs === 0) return false;
 
   let occupied = false;
-  const settled = waitUntil(
+  const settled = waitForForwardRecoveryState(
     () => {
       health = isSandboxPortForwardHealthy(sandboxName, port, expectedBind);
       if (health === "occupied") {
@@ -391,12 +375,7 @@ export function ensureSandboxPortForwardForPort(
       }
       return health === true;
     },
-    {
-      deadlineMs: Date.now() + waitMs,
-      initialIntervalMs: 100,
-      maxIntervalMs: 500,
-      backoffFactor: 1.5,
-    },
+    waitMs,
   );
   return settled && !occupied && acceptSuccessfulForward();
 }

src/lib/actions/sandbox/start.test.ts

@@ -232,7 +232,7 @@ describe("startSandbox", () => {
 
   it(
     "retries startup after a structured recovery failure (#8662)",
-    testTimeoutOptions(15_000),
+    testTimeoutOptions(30_000),
     async () => {
       const h = harness();
       h.restoreStartupState.mockReturnValueOnce(FAILED_RECOVERY);

src/lib/actions/sandbox/status-snapshot-inference-health.test.ts

@@ -135,6 +135,101 @@ describe("collectSandboxStatusSnapshot inference route health", () => {
     expect(snapshot.inferenceHealth).toMatchObject({ ok: true, probed: true });
   });
 
+  it("waits for the authoritative invocation after recovering the agent gateway", async () => {
+    const healthy: SandboxInferenceRouteHealth = {
+      ok: true,
+      endpoint: "https://inference.local/v1/models",
+      httpStatus: 200,
+      detail: "reachable",
+    };
+    const options = snapshotDeps(healthy);
+    options.deps.reconcile = async () => ({
+      state: "present",
+      output: "Phase: Ready",
+      recoveredSandbox: true,
+      recoverySandboxVia: "started-stopped-original",
+    });
+    const probeSandboxInferenceGatewayHealthImpl = vi.fn(async () => healthy);
+    const probeSandboxInferenceInvocationImpl = vi
+      .fn()
+      .mockReturnValueOnce({
+        ok: false,
+        detail: "sandbox inference invocation failed with status 6",
+        httpStatus: null,
+      })
+      .mockReturnValueOnce({ ok: true });
+    const delayInferenceRecoveryProbe = vi.fn(async () => undefined);
+    const recoverSandboxProcesses = vi.fn(() => ({
+      checked: true,
+      wasRunning: false,
+      recovered: true,
+      forwardRecovered: true,
+    }));
+
+    const snapshot = await collectSandboxStatusSnapshot("alpha", {
+      ...options,
+      deps: {
+        ...options.deps,
+        delayInferenceRecoveryProbe,
+        probeSandboxInferenceGatewayHealthImpl,
+        probeSandboxInferenceInvocationImpl,
+        recoverSandboxProcesses,
+      },
+    });
+
+    expect(probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalledTimes(2);
+    expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(2);
+    expect(delayInferenceRecoveryProbe).toHaveBeenCalledOnce();
+    expect(snapshot.inferenceHealth).toMatchObject({ ok: true, probed: true });
+  });
+
+  it("reports unhealthy after every recovered inference request fails", async () => {
+    const healthy: SandboxInferenceRouteHealth = {
+      ok: true,
+      endpoint: "https://inference.local/v1/models",
+      httpStatus: 200,
+      detail: "reachable",
+    };
+    const failedInvocation: SandboxInferenceInvocationResult = {
+      ok: false,
+      detail: "sandbox inference invocation failed with status 503",
+      httpStatus: 503,
+    };
+    const options = snapshotDeps(healthy);
+    options.deps.reconcile = async () => ({
+      state: "present",
+      output: "Phase: Ready",
+      recoveredSandbox: true,
+      recoverySandboxVia: "started-stopped-original",
+    });
+    const probeSandboxInferenceGatewayHealthImpl = vi.fn(async () => healthy);
+    const probeSandboxInferenceInvocationImpl = vi.fn(() => failedInvocation);
+    const delayInferenceRecoveryProbe = vi.fn(async () => undefined);
+    const recoverSandboxProcesses = vi.fn(() => ({
+      checked: true,
+      wasRunning: false,
+      recovered: true,
+      forwardRecovered: true,
+    }));
+
+    const snapshot = await collectSandboxStatusSnapshot("alpha", {
+      ...options,
+      deps: {
+        ...options.deps,
+        delayInferenceRecoveryProbe,
+        probeSandboxInferenceGatewayHealthImpl,
+        probeSandboxInferenceInvocationImpl,
+        recoverSandboxProcesses,
+      },
+    });
+
+    expect(probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalledTimes(3);
+    expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(3);
+    expect(delayInferenceRecoveryProbe).toHaveBeenCalledTimes(2);
+    expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000);
+    expect(snapshot.inferenceHealth).toMatchObject({ ok: false, failureLabel: "unhealthy" });
+  });
+
   it("reports the inference route as unreachable after all post-recovery probes", async () => {
     const unreachable: SandboxInferenceRouteHealth = {
       ok: false,