Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -771,8 +771,15 @@ For Linux Docker-driver gateways, onboarding also checks that a helper container
If a host firewall blocks that sandbox path, onboarding exits with a `sudo ufw allow from <subnet> to <gateway-ip> port <gateway-port> proto tcp` command before it reports the gateway healthy.
Set `NEMOCLAW_AUTO_FIX_FIREWALL=1` to opt in to automatic UFW remediation for this specific failure: NemoClaw uses `sudo -n` only, validates the Docker bridge subnet/gateway/port, applies the narrow UFW rule only after a proven TCP reachability failure, and re-probes before continuing.
If passwordless sudo, UFW, or active UFW is unavailable, NemoClaw falls back to the manual guidance path without prompting for a password.
Tune the wait via `NEMOCLAW_REUSE_HEALTH_POLL_COUNT` (default `6`) and `NEMOCLAW_REUSE_HEALTH_POLL_INTERVAL` (default `5` seconds).
The poll count is clamped to a minimum of `1` so the probe always runs at least once, and the interval is clamped to a minimum of `0` (no sleep between attempts).

For the portable experimental profile, the helper maps `host.openshell.internal` to the OpenShell Podman host gateway instead of the inspected network gateway.
This path does not use Docker bridge UFW remediation.
After all portable TCP probe attempts fail, onboarding prints commands for the user-scoped Podman service and socket.

Onboarding prints the same commands when the portable probe cannot reach the user-scoped Podman service.
The printed rerun command keeps the portable experimental profile selected.
To tune the existing-gateway HTTP health poll, use `NEMOCLAW_REUSE_HEALTH_POLL_COUNT` (default `6`) and `NEMOCLAW_REUSE_HEALTH_POLL_INTERVAL` (default `5` seconds).
The poll count is clamped to a minimum of `1` so the health probe always runs at least once, and the interval is clamped to a minimum of `0` (no sleep between attempts).

#### `--from <Dockerfile>`

Expand Down
47 changes: 47 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3054,6 +3054,53 @@ Then rerun portable onboarding:
$$nemoclaw onboard --experimental-profile portable
```

### Portable Host Gateway Is Unreachable

The portable experimental profile maps `host.openshell.internal` to the OpenShell Podman host gateway.
Do not apply the Docker bridge UFW command when portable onboarding reports this route as unreachable.

Use the same procedure when onboarding reports that the Podman service is unreachable for the portable gateway probe.

Portable onboarding reports output like this:

```text
✗ Sandbox containers cannot reach the gateway at host.openshell.internal:8080.
The probe mapped host.openshell.internal to the OpenShell Podman host gateway.
```

If `podman.service` is active, restart it.
Then enable and start the user-scoped Podman socket:

```bash
systemctl --user try-restart podman.service
systemctl --user enable --now podman.socket
```

The first command does not start an inactive service.
The second command enables and starts the current user's Podman API socket.
These commands affect only the current user's Podman units.
They do not read or write credentials.

Verify that the socket is active:

```bash
systemctl --user is-active podman.socket
```

Expected output:

```text
active
```

Then rerun portable onboarding:

```bash
$$nemoclaw onboard --experimental-profile portable
```

Continue only when onboarding no longer reports that the Podman service or OpenShell Podman host gateway is unreachable.

<AgentOnly variant="hermes">

## Hermes
Expand Down
27 changes: 9 additions & 18 deletions src/lib/onboard/docker-driver-platform.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,15 @@ import { resolveCurrentOpenShellComputePlan, usesManagedDockerGateway } from "./

export { resolveCurrentOpenShellComputePlan } from "./compute/plan";

export const EXPERIMENTAL_PROFILE_ENV = "NEMOCLAW_EXPERIMENTAL_PROFILE";
export const PORTABLE_EXPERIMENTAL_PROFILE = "portable";
export const PORTABLE_HOST_GATEWAY_IP = "169.254.1.2";
export const PORTABLE_LOCAL_REGISTRY = "localhost:5000";

export type ExperimentalOnboardProfile = typeof PORTABLE_EXPERIMENTAL_PROFILE;

export function resolveExperimentalOnboardProfile(
env: NodeJS.ProcessEnv = process.env,
): ExperimentalOnboardProfile | null {
return env[EXPERIMENTAL_PROFILE_ENV] === PORTABLE_EXPERIMENTAL_PROFILE
? PORTABLE_EXPERIMENTAL_PROFILE
: null;
}

export function isPortableExperimentalProfile(env: NodeJS.ProcessEnv = process.env): boolean {
return resolveExperimentalOnboardProfile(env) === PORTABLE_EXPERIMENTAL_PROFILE;
}
export {
type ExperimentalOnboardProfile,
EXPERIMENTAL_PROFILE_ENV,
isPortableExperimentalProfile,
PORTABLE_EXPERIMENTAL_PROFILE,
PORTABLE_HOST_GATEWAY_IP,
PORTABLE_LOCAL_REGISTRY,
resolveExperimentalOnboardProfile,
} from "./experimental/portable-profile";

export function isLinuxDockerDriverGatewayEnabled(
platform: NodeJS.Platform = process.platform,
Expand Down
21 changes: 21 additions & 0 deletions src/lib/onboard/experimental/portable-profile.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

export const EXPERIMENTAL_PROFILE_ENV = "NEMOCLAW_EXPERIMENTAL_PROFILE";
export const PORTABLE_EXPERIMENTAL_PROFILE = "portable";
export const PORTABLE_HOST_GATEWAY_IP = "169.254.1.2";
export const PORTABLE_LOCAL_REGISTRY = "localhost:5000";

export type ExperimentalOnboardProfile = typeof PORTABLE_EXPERIMENTAL_PROFILE;

export function resolveExperimentalOnboardProfile(
env: NodeJS.ProcessEnv = process.env,
): ExperimentalOnboardProfile | null {
return env[EXPERIMENTAL_PROFILE_ENV] === PORTABLE_EXPERIMENTAL_PROFILE
? PORTABLE_EXPERIMENTAL_PROFILE
: null;
}

export function isPortableExperimentalProfile(env: NodeJS.ProcessEnv = process.env): boolean {
return resolveExperimentalOnboardProfile(env) === PORTABLE_EXPERIMENTAL_PROFILE;
}
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ describe("sandbox bridge reachability severity (#6004)", () => {
gatewayIp: "172.18.0.1",
}),
}),
).rejects.toThrow("sandbox-bridge unreachable");
).rejects.toThrow("cannot reach the OpenShell gateway");
expect(warn.mock.calls[0]?.[0]).toMatch(
/^ \x1b\[33m⚠ NEMOCLAW_AUTO_FIX_FIREWALL=1 set but could not auto-apply UFW rule/,
);
Expand Down
111 changes: 106 additions & 5 deletions src/lib/onboard/gateway-sandbox-reachability.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,29 @@ describe("isSandboxBridgeGatewayReachable", () => {
expect(seen.args.join(" ")).toContain("nc -zw7 host.openshell.internal 9090");
});

it("routes probes for the portable experimental profile through the OpenShell Podman host gateway", async () => {
vi.stubEnv("NEMOCLAW_EXPERIMENTAL_PROFILE", "portable");
const seen: { args: readonly string[] } = { args: [] };

const result = await isSandboxBridgeGatewayReachable({
inspectNetworkImpl: () => ({ subnet: "10.89.0.0/24", gatewayIp: "10.89.0.1" }),
usesHostGatewayRouteImpl: () => false,
runImpl: (args) => {
seen.args = args;
return { status: 0 };
},
});

expect(result).toMatchObject({
ok: true,
gatewayIp: "169.254.1.2",
routeKind: "portable_host_gateway",
});

expect(seen.args).toContain("host.openshell.internal:169.254.1.2");
expect(seen.args).not.toContain("host.openshell.internal:10.89.0.1");
});

it("does not call a missing Docker network a firewall failure", async () => {
const result = await isSandboxBridgeGatewayReachable({
inspectNetworkImpl: () => undefined,
Expand All @@ -100,6 +123,28 @@ describe("isSandboxBridgeGatewayReachable", () => {
expect(result.detail).toContain("not found");
});

it("classifies an unavailable portable daemon before route inspection completes", async () => {
vi.stubEnv("NEMOCLAW_EXPERIMENTAL_PROFILE", "portable");
const runtimeProbeImpl = vi.fn(() => ({
status: 1,
stderr: "Cannot connect to Podman. Verify the user service and socket.",
}));

const result = await isSandboxBridgeGatewayReachable({
inspectNetworkImpl: () => undefined,
runtimeProbeImpl,
usesHostGatewayRouteImpl: () => false,
});

expect(result).toMatchObject({
ok: false,
reason: "docker_daemon_unreachable",
networkName: "openshell-docker",
});
expect(result.detail).toContain("Cannot connect to Podman");
expect(runtimeProbeImpl).toHaveBeenCalledOnce();
});

it("does not call helper DNS failures firewall failures", async () => {
const result = await isSandboxBridgeGatewayReachable({
inspectNetworkImpl: () => ({ subnet: "172.19.0.0/16", gatewayIp: "172.19.0.1" }),
Expand Down Expand Up @@ -467,6 +512,37 @@ describe("formatSandboxBridgeUnreachableMessage", () => {
expect(msg).toContain("host-gateway");
expect(msg).not.toContain("ufw allow");
});

it("reports Podman recovery for portable host-gateway failures", () => {
const msg = formatSandboxBridgeUnreachableMessage({
ok: false,
reason: "tcp_failed",
routeKind: "portable_host_gateway",
networkName: "openshell-docker",
subnet: "10.89.0.0/24",
gatewayIp: "169.254.1.2",
});
expect(msg).toContain("OpenShell Podman host gateway");
expect(msg).toContain("systemctl --user try-restart podman.service");
expect(msg).toContain("systemctl --user enable --now podman.socket");
expect(msg).toContain("nemoclaw onboard --experimental-profile portable");
expect(msg).not.toContain("Restart Docker");
expect(msg).not.toContain("ufw allow");
});

it("reports Podman recovery when the portable profile cannot reach its daemon", () => {
vi.stubEnv("NEMOCLAW_EXPERIMENTAL_PROFILE", "portable");
const msg = formatSandboxBridgeUnreachableMessage({
ok: false,
reason: "docker_daemon_unreachable",
detail: "Cannot connect to the container runtime",
});
expect(msg).toContain("Podman service is not reachable");
expect(msg).toContain("systemctl --user try-restart podman.service");
expect(msg).toContain("systemctl --user enable --now podman.socket");
expect(msg).toContain("nemoclaw onboard --experimental-profile portable");
expect(msg).not.toContain("Restart the Docker daemon");
});
});

describe("tryAutoApplyUfwRule (#4265)", () => {
Expand Down Expand Up @@ -645,6 +721,31 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => {
}
});

it("retries a transient portable host-gateway TCP failure", async () => {
const portableFailure = {
...hostGatewayTcpFailure,
routeKind: "portable_host_gateway" as const,
gatewayIp: "169.254.1.2",
};
const reachabilityImpl = vi
.fn()
.mockResolvedValueOnce(portableFailure)
.mockResolvedValueOnce({ ...portableFailure, ok: true as const, reason: "ok" as const });
const sleepMsImpl = vi.fn().mockResolvedValue(undefined);
vi.spyOn(console, "log").mockImplementation(() => undefined);

await verifySandboxBridgeGatewayReachableOrExit(true, {
reachabilityImpl,
retryAttempts: 3,
retryDelayMs: 25,
sleepMsImpl,
});

expect(reachabilityImpl).toHaveBeenCalledTimes(2);
expect(sleepMsImpl).toHaveBeenCalledOnce();
expect(sleepMsImpl).toHaveBeenCalledWith(25);
});

it("fails after exhausting persistent host-gateway tcp failures", async () => {
const reachabilityImpl = vi.fn().mockResolvedValue(hostGatewayTcpFailure);
const sleepMsImpl = vi.fn().mockResolvedValue(undefined);
Expand All @@ -658,7 +759,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => {
retryDelayMs: 25,
sleepMsImpl,
}),
).rejects.toThrow("sandbox-bridge unreachable");
).rejects.toThrow("cannot reach the OpenShell gateway");
expect(reachabilityImpl).toHaveBeenCalledTimes(3);
expect(sleepMsImpl).toHaveBeenCalledTimes(2);
expect(sleepMsImpl).toHaveBeenNthCalledWith(1, 25);
Expand Down Expand Up @@ -689,7 +790,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => {
reachabilityImpl,
sleepMsImpl,
}),
).rejects.toThrow("sandbox-bridge unreachable");
).rejects.toThrow("cannot reach the OpenShell gateway");
expect(reachabilityImpl).toHaveBeenCalledTimes(10);
expect(sleepMsImpl).toHaveBeenCalledTimes(9);
expect(sleepMsImpl).toHaveBeenCalledWith(1000);
Expand Down Expand Up @@ -721,7 +822,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => {
retryDelayMs: 25,
sleepMsImpl,
}),
).rejects.toThrow("sandbox-bridge unreachable");
).rejects.toThrow("cannot reach the OpenShell gateway");
expect(reachabilityImpl).toHaveBeenCalledTimes(1);
expect(sleepMsImpl).not.toHaveBeenCalled();
expect(error).toHaveBeenCalledWith(expect.stringContaining("ufw allow"));
Expand Down Expand Up @@ -787,7 +888,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit UFW auto-apply (#4265)", ()
autoApplyOptedInImpl: () => true,
reachabilityImpl,
}),
).rejects.toThrow("sandbox-bridge unreachable");
).rejects.toThrow("cannot reach the OpenShell gateway");
expect(reachabilityImpl).toHaveBeenCalledTimes(2);
expect(error).toHaveBeenCalledWith(expect.stringContaining("ufw allow"));
log.mockRestore();
Expand All @@ -803,7 +904,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit UFW auto-apply (#4265)", ()
autoApplyOptedInImpl: () => true,
reachabilityImpl: () => tcpFailure,
}),
).rejects.toThrow("sandbox-bridge unreachable");
).rejects.toThrow("cannot reach the OpenShell gateway");
expect(warn).not.toHaveBeenCalled();
expect(error).toHaveBeenCalledWith(expect.stringContaining("ufw allow"));
warn.mockRestore();
Expand Down
Loading
Loading