Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/nemoclaw-maintainer-e2e/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ After a failure, inspect the artifacts and remove resources that target cleanup
These credentials remain valid until they expire or an administrator revokes them in their issuing services. If cleanup fails, remove the recorded Brev workspace. Rotate or revoke each credential to remove later access.
This Brev credential boundary applies to trusted `main` pushes and Launchable or full manual runs. It does not apply to manual PR runs, which keep `include_staging_brev_launchable=false`.

For `managed-image-protected-runtime`, the workflow supplies the long-lived `NVIDIA_API_KEY` repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as `NGC_API_KEY` and `NIM_NGC_API_KEY` to the temporary NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. The live fixture attempts to stop and remove `nemoclaw-managed-image-nim-e2e`, but Docker stop or removal errors do not fail the test. A surviving container can retain the API key until runner teardown. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Rotate or revoke it in the issuing NVIDIA service to remove later access.
For `managed-image-protected-runtime`, the workflow supplies the long-lived `NVIDIA_API_KEY` repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as `NGC_API_KEY` and `NIM_NGC_API_KEY` to the temporary NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. The live fixture removes the temporary NIM container only if its exact ID, name, requested image, immutable image ID, cohort owner, and provider kind match the recorded authority. The test fails if evidence is missing or ambiguous, a name is reused, authority drifts, removal is indeterminate, or the exact ID or name remains. A cleanup refusal can leave the container and its API key in place until runner teardown. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Revoke it, or rotate it and disable the old value, in the issuing NVIDIA service. Verify that the exposed key is no longer valid.

Resolve the current PR and trusted workflow identities:

Expand Down
9 changes: 7 additions & 2 deletions scripts/checks/run-managed-image-openshell-e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -659,11 +659,15 @@ function assertProtectedLocalInference(
}
}

function failureInjectingAdapter(onboard: OnboardModule): ManagedBootstrapAdapter {
export function failureInjectingAdapter(
onboard: OnboardModule,
stateRoot: string,
): ManagedBootstrapAdapter {
const adapter = createDockerManagedBootstrapAdapter({
runCaptureOpenshell: onboard.runCaptureOpenshell,
runOpenshell: onboard.runOpenshell,
sleep: onboard.sleepSeconds,
stateRoot,
});
return {
...adapter,
Expand Down Expand Up @@ -1002,7 +1006,8 @@ async function run<T extends ManagedImageOpenShellE2eLocalInferenceEvidence = ne
verifyDirectSandboxGpu,
...(input.failureInjection
? {
createManagedBootstrapAdapter: () => failureInjectingAdapter(onboard!),
createManagedBootstrapAdapter: (stateRoot: string) =>
failureInjectingAdapter(onboard!, stateRoot),
}
: {}),
},
Expand Down
10 changes: 9 additions & 1 deletion src/lib/onboard/sandbox-gpu-create-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,8 @@ describe("runSandboxGpuCreateFlow provider-owned managed create", () => {
expectedSupervisorArgv: ["/mxc/supervisor"],
};
const deps = createDeps();
const adapterOverride = {} as never;
deps.createManagedBootstrapAdapter = vi.fn(() => adapterOverride);
vi.mocked(deps.runCaptureOpenshell).mockImplementation((args) =>
args[1] === "get" ? "ID: mxc-alpha\n" : "alpha Ready",
);
Expand All @@ -340,8 +342,14 @@ describe("runSandboxGpuCreateFlow provider-owned managed create", () => {

expect(result).toMatchObject({ route: "none", runtimePatch: patch });
expect(createLifecycle).toHaveBeenCalledWith(
expect.objectContaining({ providerId: "mxc", route: "none" }),
expect.objectContaining({
providerId: "mxc",
route: "none",
stateRoot: "/tmp/nemoclaw-mxc-bootstrap",
adapterOverride,
}),
);
expect(deps.createManagedBootstrapAdapter).toHaveBeenCalledWith("/tmp/nemoclaw-mxc-bootstrap");
expect(mocks.streamSandboxCreate).toHaveBeenCalledWith(
"mxc-launch",
input.createArgv.slice(1),
Expand Down
2 changes: 1 addition & 1 deletion src/lib/onboard/sandbox-gpu-create-flow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ export interface SandboxGpuCreateFlowDeps {
/** Production callers configure the hidden portable lifecycle through the default implementation. */
installPortableDemoLifecycle?: typeof installPortableDemoSandboxLifecycle;
/** Production callers omit this factory and use the runtime provider's adapter. */
createManagedBootstrapAdapter?: () => ManagedBootstrapAdapter;
createManagedBootstrapAdapter?: (stateRoot: string) => ManagedBootstrapAdapter;
}

export interface SandboxGpuCreateFlowResult {
Expand Down
2 changes: 1 addition & 1 deletion src/lib/onboard/sandbox-gpu-create-run-attempt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ export function createSandboxGpuCreateAttemptRunner(
heldWorkloadArgv: input.sandboxStartupCommand,
authorityStore: managedBootstrap.authorityStore,
...(deps.createManagedBootstrapAdapter
? { adapterOverride: deps.createManagedBootstrapAdapter() }
? { adapterOverride: deps.createManagedBootstrapAdapter(managedBootstrap.stateRoot) }
: {}),
route,
persistStartupCommand: input.persistStartupCommand === true,
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -837,7 +837,7 @@ The workflow does not rotate or revoke these API keys or messaging credentials.
Live targets can create external resources.
After a failure, inspect the workflow artifacts and remove resources that target cleanup did not remove.

For `managed-image-protected-runtime`, the workflow supplies the long-lived `NVIDIA_API_KEY` repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as `NGC_API_KEY` and `NIM_NGC_API_KEY` to the temporary NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. The live fixture attempts to stop and remove `nemoclaw-managed-image-nim-e2e`, but Docker stop or removal errors do not fail the test. A surviving container can retain the API key until runner teardown. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Rotate or revoke it in the issuing NVIDIA service to remove later access.
For `managed-image-protected-runtime`, the workflow supplies the long-lived `NVIDIA_API_KEY` repository secret only to the trusted qualification step. Trusted host code uses it for NGC login and passes it as `NGC_API_KEY` and `NIM_NGC_API_KEY` to the temporary, cohort-owned NIM container. Candidate managed sandboxes receive generated local route tokens instead of this key. Before starting NIM or vLLM, the live fixture rejects a pre-existing cohort container name. It records the full container ID, requested image, immutable image ID, cohort owner, and provider label, then removes only that exact container after revalidating every field. Missing, ambiguous, name-reused, drifted, or indeterminate cleanup evidence fails the test, as does any retained exact ID or name. A fail-closed refusal can leave the secret-bearing NIM container alive until runner teardown; inspect the redacted artifacts and remove only the verified container. The final workflow step removes the job's isolated Docker credential directory and fails if that removal does not complete. The workflow does not revoke the NVIDIA API key. Revoke it, or rotate it and disable the old value, in the issuing NVIDIA service. Verify that the exposed key is no longer valid.

For a manual PR run, provide the current PR number, lowercase 40-character head SHA, head repository, lowercase 40-character base SHA, trusted `main` workflow SHA, and a review reason containing 10 to 500 printable characters.
Leave `jobs` and `targets` empty and keep `include_staging_brev_launchable=false` to use this PR revision selection.
Expand Down
Loading
Loading