feat(nvca): persist durable model cache storage decisions

[balajinvda]

TL;DR

Wire the storage capability catalog introduced by #1334 into NVCA model-cache request creation and runtime. New requests persist the exact nvcf-sc and catalog decision, create an immutable ModelCacheBinding before storage side effects, and fail closed on drift or ownership mismatch.

Additional Details

Why this is needed:

• Provider selection previously depended on live feature gates and StorageClass presence at several runtime call sites.
• Retries could not prove that they were using the same provider decision or the same cache data identity.
• Same-name Kubernetes objects needed typed ownership checks before adoption or deletion.

What changed:

• Resolve the exact nvcf-sc provisioner, require Retain, and persist the selected catalog transition on each new model-cache request.
• Create a namespaced ModelCacheBinding with immutable provider, StorageClass, resource-intent, and request-UID state before starting regular or Helm cache work.
• Revalidate binding UID, request UID, immutable object intent, PVC-to-PV identity, and delete preconditions on retry and cleanup.
• Make regular failure cleanup atomically move a sole-user binding from Active to Retiring before destructive work. Concurrent users prevent retirement; the same request can resume interrupted cleanup.
• Make Helm writer adoption and cleanup validate the exact StorageRequest, PVC, Job, Secret, Lease holder, and reader PV/PVC generation.
• Force the PVC volume source and every model-cache mount path to read-only for durable readers.
• Install the binding CRD and RBAC, mirror catalog changes into agent namespaces, and fail operator uninstall closed if binding cleanup fails.
• Preserve annotation-free compatibility behavior. Only the registered NVMesh transitions execute. Weka and OCI transitions remain disabled.

This PR is stacked on #1334. General zero-reference retirement and retained-data garbage collection remain follow-up work. Container cache and live-provider functional or performance qualification are outside this PR.

No third-party dependency changed. License review found no new violations, and NOTICE does not change.

For the Reviewer

Please focus on:

• selection immutability and binding-reference ordering in pkg/nvca;
• regular Active to Retiring race handling and destructive cleanup guards;
• Helm typed adoption, Lease revalidation, and reader cleanup in pkg/storage;
• MiniService exact StorageRequest handoff and webhook read-only enforcement;
• operator CRD, RBAC, catalog event, and uninstall behavior.

The design and supported limitations are documented in docs/dev/sdd-storage-agnostic-cache-architecture.md.

For QA

QA is recommended for one NVMesh regular request and one NVMesh Helm request in a test cluster. Local validation completed:

• code generation and OpenAPI generation twice with identical output;
• Go tests for 8 impacted packages with Kubernetes 1.34.1 envtest assets: pass;
• go vet for the same packages: pass;
• 8 impacted Bazel test targets: pass;
• golangci-lint for the impacted packages: 0 issues;
• NVCA Helm lint and catalog/chart parity checks: pass;
• license/SPDX check: pass;
• diff, TOC, ASCII, public-hygiene, and targeted secret checks: pass.

./tools/ci/check-docs was not completed locally because the installed Fern CLI requires a newer Node runtime (crypto is unavailable under Node 18), and its advisory version-sync helper did not locate the repository root. Targeted checks for the changed design document passed.

Issues

Relates to #1326

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1563568c-fa85-4bc5-88f9-ff5f8554f375

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-feat-nvca-storage-runtime-selection.docs.buildwithfern.com/nvcf