Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions docs/inference/switch-providers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,15 @@
# SPDX-License-Identifier: Apache-2.0
title: "Switch Inference Providers"
sidebar-title: "Switch Providers"
description: "Move a NemoClaw-managed sandbox to another registered inference provider."
description: "Move a NemoClaw-managed sandbox to another inference provider."
description-agent: "Switches inference provider families. Use when moving a sandbox between hosted, local, or compatible provider routes."
keywords: ["switch nemoclaw provider", "change inference provider", "nemoclaw inference set provider"]
content:
type: "how_to"
---
Move a sandbox to another provider family while keeping the OpenShell route, agent configuration, and host registry aligned.
Use onboarding first when the target provider is not registered.
Use onboarding first when the target is neither `compatible-endpoint` nor `compatible-anthropic-endpoint` and is not registered.
The compatible endpoint workflow below can register an absent custom provider from complete route metadata.

## Find a Registered Provider

Expand All @@ -25,8 +26,9 @@ $$nemoclaw credentials list
`inference set` also accepts installer-facing provider names such as `anthropicCompatible`, `build`, and `custom`.
NemoClaw normalizes an accepted alias to its canonical OpenShell provider ID and records the canonical name in the sandbox registry.

If OpenShell cannot find the requested provider, NemoClaw leaves the route and sandbox state unchanged.
If OpenShell cannot find a requested provider other than `compatible-endpoint` or `compatible-anthropic-endpoint`, NemoClaw leaves the route and sandbox state unchanged.
Run `$$nemoclaw onboard` to register the provider, then retry the switch.
For a compatible custom endpoint, pass the complete route metadata described below so NemoClaw can register the provider.

</AgentOnly>

Expand Down Expand Up @@ -80,17 +82,26 @@ $$nemoclaw onboard --fresh --name <sandbox-name> --recreate-sandbox
<AgentOnly variant="openclaw,hermes">

When moving from another provider family to `compatible-endpoint` or `compatible-anthropic-endpoint`, provide the trusted endpoint URL and enough API metadata to record the complete route identity.
Export the canonical credential environment variable for the target provider.
Run this command within the shields-down window shown in the runtime switching workflow.

```bash
export COMPATIBLE_API_KEY="<api-key>"
$$nemoclaw inference set \
--provider compatible-endpoint \
--model <model-name> \
--endpoint-url <trusted-url> \
--credential-env COMPATIBLE_API_KEY \
--inference-api openai-completions \
--sandbox <name>
```

NemoClaw validates the endpoint before it changes the route.
For an HTTPS IP-literal or DNS-pinned HTTP endpoint, NemoClaw registers and verifies an absent provider before selecting the route.
If OpenShell still reports the provider as absent, NemoClaw retries the switch once.
If route selection fails, NemoClaw removes a newly created provider and leaves the inference selection unchanged.
For a DNS-backed HTTPS endpoint, NemoClaw registers the HTTPS Pin Runtime provider before the first route attempt.

Supported API-family values are `openai-completions`, `anthropic-messages`, and `openai-responses`.
For a Hermes `compatible-anthropic-endpoint` target, omit `--inference-api` because NemoClaw selects `openai-completions`.
An explicit different API family is rejected for that route.
Expand Down
227 changes: 225 additions & 2 deletions src/lib/actions/inference-set-compatible-provider.test.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { describe, expect, it, vi } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { ensureHttpsPinRuntimeAdapter as realEnsureHttpsPinRuntimeAdapter } from "../inference/https-pin-runtime-adapter";
import type { ConfigObject } from "../security/credential-filter";
import { runInferenceSet } from "./inference-set";
import { baseSession, createDeps } from "./inference-set.test-support";
import {
baseSession,
createDeps,
createExistingCompatibleProviderCapture,
} from "./inference-set.test-support";

describe("runInferenceSet compatible providers", () => {
afterEach(() => vi.unstubAllEnvs());

it("reuses durable endpoint metadata for same-provider model switches", async () => {
const config: ConfigObject = {
agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } },
Expand Down Expand Up @@ -177,6 +183,216 @@ describe("runInferenceSet compatible providers", () => {
expect(deps.calls.updateSandbox).not.toHaveBeenCalled();
});

it.each([
["an HTTPS IP-literal", "https://198.51.100.10/v1", "https://198.51.100.10/v1"],
["a DNS-pinned HTTP", "http://compatible.example/v1", "http://198.51.100.10/v1"],
])("creates an absent direct compatible provider for %s endpoint (#7725)", async (_kind, endpointUrl, validatedEndpointUrl) => {
let providerCreated = false;
const captureOpenshell = vi.fn((args: string[]) => {
switch (`${args[0]}:${args[1]}`) {
case "inference:set":
return providerCreated
? { status: 0, output: "", stdout: "", stderr: "" }
: {
status: 1,
output: "Error: provider 'compatible-endpoint' not found",
stdout: "",
stderr: "Error: provider 'compatible-endpoint' not found",
};
case "provider:get": {
const output = [
"Name: compatible-endpoint",
"Id: 11111111-2222-4333-8444-555555555555",
"Type: openai",
"Resource version: 1",
"Credential keys: COMPATIBLE_API_KEY",
"Config keys: OPENAI_BASE_URL",
].join("\n");
return providerCreated
? { status: 0, output, stdout: output, stderr: "" }
: {
status: 1,
output:
"Error: code: 'Some requested entity was not found', message: \"provider not found\"",
stdout: "",
stderr:
"Error: code: 'Some requested entity was not found', message: \"provider not found\"",
};
}
case "provider:create":
providerCreated = true;
return { status: 0, output: "", stdout: "", stderr: "" };
default:
return { status: 0, output: "", stdout: "", stderr: "" };
}
});
const deps = createDeps({
config: { agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } } },
entry: {
name: "alpha",
agent: "openclaw",
provider: "nvidia-prod",
model: "nvidia/model-a",
},
session: baseSession({
provider: "nvidia-prod",
model: "nvidia/model-a",
}),
captureOpenshell,
rewriteConfigUrlsWithDnsPinning: async () => validatedEndpointUrl,
resolveCredentialValue: () => "real-upstream-secret",
});

await expect(
runInferenceSet(
{
provider: "compatible-endpoint",
model: "mock-model",
noVerify: true,
endpointUrl,
credentialEnv: "COMPATIBLE_API_KEY",
inferenceApi: "openai-completions",
},
deps,
),
).resolves.toMatchObject({
sandboxName: "alpha",
provider: "compatible-endpoint",
model: "mock-model",
});

const providerCreateIndex = captureOpenshell.mock.calls.findIndex(
([args]) => args[0] === "provider" && args[1] === "create",
);
const successfulSetIndex = captureOpenshell.mock.calls.findIndex(
([args], index) =>
index > providerCreateIndex && args[0] === "inference" && args[1] === "set",
);
expect(providerCreateIndex).toBeGreaterThanOrEqual(0);
expect(successfulSetIndex).toBeGreaterThan(providerCreateIndex);
expect(captureOpenshell.mock.calls[providerCreateIndex]).toEqual([
[
"provider",
"create",
"-g",
"nemoclaw",
"--name",
"compatible-endpoint",
"--type",
"openai",
"--credential",
"COMPATIBLE_API_KEY",
"--config",
`OPENAI_BASE_URL=${validatedEndpointUrl}`,
],
expect.objectContaining({
env: { COMPATIBLE_API_KEY: "real-upstream-secret" },
}),
]);
expect(deps.calls.updateSandbox.mock.calls.at(-1)).toEqual([
"alpha",
expect.objectContaining({
provider: "compatible-endpoint",
endpointUrl: validatedEndpointUrl,
}),
]);
});

it("updates an existing direct compatible provider when its endpoint changes (#7725)", async () => {
let providerVersion = 4;
const captureOpenshell = vi.fn((args: string[]) => {
switch (`${args[0]}:${args[1]}`) {
case "provider:get": {
const output = [
"Name: compatible-endpoint",
"Id: 11111111-2222-4333-8444-555555555555",
"Type: openai",
`Resource version: ${providerVersion}`,
"Credential keys: COMPATIBLE_API_KEY",
"Config keys: OPENAI_BASE_URL",
].join("\n");
return { status: 0, output, stdout: output, stderr: "" };
}
case "provider:update":
providerVersion += 1;
return { status: 0, output: "", stdout: "", stderr: "" };
default:
return { status: 0, output: "", stdout: "", stderr: "" };
}
});
const deps = createDeps({
config: { agents: { defaults: { model: { primary: "inference/old-model" } } } },
entry: {
name: "alpha",
agent: "openclaw",
provider: "compatible-endpoint",
model: "old-model",
endpointUrl: "http://198.51.100.9/v1",
endpointSource: "inference-set",
credentialEnv: "COMPATIBLE_API_KEY",
preferredInferenceApi: "openai-completions",
},
session: baseSession({
provider: "compatible-endpoint",
model: "old-model",
endpointUrl: "http://198.51.100.9/v1",
credentialEnv: "COMPATIBLE_API_KEY",
preferredInferenceApi: "openai-completions",
}),
captureOpenshell,
rewriteConfigUrlsWithDnsPinning: async () => "http://198.51.100.10/v1",
resolveCredentialValue: () => "replacement-upstream-secret",
});

await runInferenceSet(
{
provider: "compatible-endpoint",
model: "new-model",
noVerify: true,
endpointUrl: "http://compatible.example/v1",
credentialEnv: "COMPATIBLE_API_KEY",
inferenceApi: "openai-completions",
},
deps,
);

const providerGetIndex = captureOpenshell.mock.calls.findIndex(
([args]) => args[0] === "provider" && args[1] === "get",
);
const inferenceSetIndex = captureOpenshell.mock.calls.findIndex(
([args]) => args[0] === "inference" && args[1] === "set",
);
const providerUpdateIndex = captureOpenshell.mock.calls.findIndex(
([args]) => args[0] === "provider" && args[1] === "update",
);
expect(providerGetIndex).toBeLessThan(inferenceSetIndex);
expect(inferenceSetIndex).toBeLessThan(providerUpdateIndex);
expect(captureOpenshell.mock.calls[providerUpdateIndex]).toEqual([
[
"provider",
"update",
"-g",
"nemoclaw",
"compatible-endpoint",
"--credential",
"COMPATIBLE_API_KEY",
"--config",
"OPENAI_BASE_URL=http://198.51.100.10/v1",
],
expect.objectContaining({
env: { COMPATIBLE_API_KEY: "replacement-upstream-secret" },
}),
]);
expect(deps.calls.updateSandbox.mock.calls.at(-1)).toEqual([
"alpha",
expect.objectContaining({
provider: "compatible-endpoint",
model: "new-model",
endpointUrl: "http://198.51.100.10/v1",
}),
]);
});

it("preserves explicit inference API through the final registry and session sync", async () => {
let providerVersion = 1;
const captureOpenshell = vi.fn((args: string[]) => {
Expand Down Expand Up @@ -275,6 +491,12 @@ describe("runInferenceSet compatible providers", () => {
agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } },
models: { providers: { inference: { api: "openai-completions", models: [] } } },
};
const captureOpenshell = createExistingCompatibleProviderCapture({
name: "compatible-anthropic-endpoint",
type: "anthropic",
credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY",
configKey: "ANTHROPIC_BASE_URL",
});
const deps = createDeps({
config,
entry: {
Expand All @@ -289,6 +511,7 @@ describe("runInferenceSet compatible providers", () => {
endpointUrl: "https://integrate.api.nvidia.com/v1",
credentialEnv: "NVIDIA_INFERENCE_API_KEY",
}),
captureOpenshell,
});

await runInferenceSet(
Expand Down
12 changes: 11 additions & 1 deletion src/lib/actions/inference-set-degraded-state.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@ import { describe, expect, it } from "vitest";
import { SandboxConfigError } from "../sandbox/config";
import type { ConfigObject } from "../security/credential-filter";
import { InferenceSetError, runInferenceSet } from "./inference-set";
import { baseSession, createDeps } from "./inference-set.test-support";
import {
baseSession,
createDeps,
createExistingCompatibleProviderCapture,
} from "./inference-set.test-support";

describe("runInferenceSet degraded state handling", () => {
it("aborts before mutating any layer when the sandbox config read fails (#6997)", async () => {
Expand Down Expand Up @@ -130,6 +134,12 @@ describe("runInferenceSet degraded state handling", () => {
provider: "nvidia-prod",
model: "nvidia/nemotron-3-super-120b-a12b",
}),
captureOpenshell: createExistingCompatibleProviderCapture({
name: "compatible-endpoint",
type: "openai",
credentialEnv: "COMPATIBLE_API_KEY",
configKey: "OPENAI_BASE_URL",
}),
});
deps.calls.readSandboxConfig.mockImplementation(() => structuredClone(persistedConfig));
deps.calls.updateSandbox.mockImplementation((_name, updates) => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,7 @@ describe("runtime shared gateway route containment", () => {
onboardEndpointUrl: null,
getSandboxes: () => [alpha, peer],
rewriteUrlWithDnsPinning,
resolveCredentialValue: () => "",
ensureHttpsPinRuntimeAdapter,
}),
).rejects.toThrow("custom-peer");
Expand Down Expand Up @@ -371,6 +372,7 @@ describe("runtime shared gateway route containment", () => {
onboardEndpointUrl: null,
getSandboxes: () => [alpha, peer],
rewriteUrlWithDnsPinning: vi.fn(async (value: unknown) => value as string),
resolveCredentialValue: () => "upstream-token",
ensureHttpsPinRuntimeAdapter: vi.fn(async () => ({
baseUrl: adapterBaseUrl as string,
credentialEnv: HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV,
Expand Down Expand Up @@ -459,6 +461,7 @@ describe("runtime shared gateway route containment", () => {
onboardEndpointUrl: null,
getSandboxes: () => [alpha, beta],
rewriteUrlWithDnsPinning,
resolveCredentialValue: () => "upstream-token",
ensureHttpsPinRuntimeAdapter,
});

Expand Down
Loading
Loading