[Pi 6/8] Add deterministic tests and live E2E qualification

[prekshivyas]

Summary

Qualify the supported Pi runtime with deterministic tests and exact-candidate live E2E evidence across the accepted compute-runtime matrix.

Parent epic: #7923

Coordination epic: #7744

Landed dependency baseline: #7756, #7772, #8234, #8143, and #7718.

Depends on #7926, #7925, #7927, #7930, and #7924. #7929 consumes this evidence, and #8818 owns final activation.

Target completion: 2026-08-25.

Problem Statement

Existing Pi contributor and CI uses do not test the supported user onboarding and sandbox lifecycle.

NemoClaw needs release evidence bound to the exact candidate commit, managed image, OpenShell version, inference route, compute runtime, and policy.

Pi may ship before native Podman activates. The test plan must not block Pi on unfinished Podman work unless #7926 includes Podman at launch, but Podman must include Pi before #7744 claims all-shipped-agent support.

Desired Behavior

Deterministic projects cover Pi manifests, managed-image contracts, configuration, onboarding, lifecycle, inference, state, security boundaries, and compute-runtime neutrality.

A fresh live environment on each supported launch platform completes one interactive and one non-interactive tool-using Pi task through the public NemoClaw path.

If Podman is launch scope, the same exact Pi image and agent contracts pass under native Podman. If Podman is deferred, #7744 tracks Pi as required activation evidence.

Qualification contract

Before implementation begins, #7926 must record the platform, architecture, compute-runtime, provider and API-family matrix, model-validation rules, and exact release qualification model.

The qualification plan must define one versioned deterministic tool-using task, an independent success oracle, timeout and retry limits, and a repository-owned evidence manifest location. Agent self-reporting is not a sufficient oracle.

Live Qualification Flow

1. Install the exact NemoClaw candidate through the supported path.
2. Onboard Pi through public non-interactive inputs.
3. Verify recorded agent, compute runtime, managed-image digest, inference route, and policy identities.
4. Prove that stock onboarding did not build a repository Dockerfile.
5. Run a deterministic tool-using task through managed inference.
6. Run an interactive terminal flow.
7. Rebuild the sandbox and repeat the task.
8. Verify declared persistent state and denied access.
9. Repeat under each compute runtime accepted for Pi launch.
10. Capture logs and bounded evidence without credentials.

Constraints and Non-goals

• Do not qualify source patches or manual sandbox-state edits.
• Do not use an unpinned runtime, image, or mutable alias.
• Do not treat an agent assertion as the only success oracle.
• Do not expose credentials in artifacts or logs.
• Do not claim Podman support before [Epic] Support native Podman with buildless managed onboarding #7744's activation gate passes.
• Do not require unfinished Podman activation when [Pi 1/8] Accept product scope, ownership, threat model, and compatibility matrix #7926 records a Docker-only Pi launch.
• Do not activate Pi in the supported inventory or atomic release cohort; [Pi 8/8] Activate Pi in supported inventory and release cohort #8818 owns that final change.

Acceptance Criteria

• Unit and integration tests cover Pi configuration and lifecycle behavior.
• Package-contract tests cover compiled artifacts, the shipped manifest, and managed-image contracts.
• Publication tests prove that the candidate Pi cohort is complete and can be activated atomically without activating it before [Pi 8/8] Activate Pi in supported inventory and release cohort #8818.
• Tests prove stock onboarding selects an exact digest without a host Dockerfile build.
• Tests prove Pi code contains no Docker or Podman lifecycle branch.
• Security tests cover credential, filesystem, network, tool, state, and container-runtime authority boundaries.
• A fresh supported live environment completes the interactive flow.
• A fresh supported live environment completes the versioned non-interactive task and an independent oracle verifies the exact result.
• Rebuild and recovery repeat the accepted task without manual repair.
• Each compute runtime accepted at Pi launch passes the same agent task and state assertions.
• If Podman is deferred, [Epic] Support native Podman with buildless managed onboarding #7744 records Pi in its all-shipped-agent activation matrix.
• Evidence identifies the exact commit, runtime package, managed-image digest, OpenShell version, compute runtime, provider, model, policy digest, task version, oracle result, and evidence-manifest location.
• CI and live evidence contain no credentials.

Category

Testing

Checklist

• I searched existing issues and this is not a duplicate.
• I described the problem and desired behavior.