createContainer hook `disable-device-node-modification` fails: /proc/driver/nvidia/params not available in containerRootFs (gvisor#13034 follow-up)

[a7i]

Summary

After #13034, three of four NVIDIA CDI createContainer hooks emitted by k8s-device-plugin (DEVICE_LIST_STRATEGY=cdi-annotations) now run successfully inside the gofer (create-symlinks, enable-cuda-compat, update-ldcache). The fourth — disable-device-node-modification — still fails because /proc/driver/nvidia/params does not exist inside the prepared containerRootFs at hook time:

nvidia-ctk hook disable-device-node-modification
stderr: failed to mount modified params file: open o_path procfd:
  open /run/containerd/io.containerd.runtime.v2.task/k8s.io/<id>/rootfs/proc/driver/nvidia/params:
  no such file or directory
FATAL ERROR: error executing CreateContainer hooks

The hook is part of every spec the NVIDIA k8s-device-plugin generates today (no opt-out yet — NVIDIA/k8s-device-plugin#1245), so any cluster using gVisor + GPU via the plugin currently can't start a gVisor sandbox without patching the spec.

Environment

Component	Value
runsc --version	release-20260520.0 (includes #13034)
containerd	2.2.2
Kubernetes	1.35.2
OS	Ubuntu 22.04, kernel 6.8
GPU	Tesla T4 (Turing)
NVIDIA Container Toolkit	1.19.1-1
Driver	580.159.04 host; nvproxy-driver-version = 580.126.20 (latest in runsc nvproxy list-supported-drivers for 580 series)
k8s-device-plugin	DEVICE_LIST_STRATEGY=cdi-annotations

Sequence

sequenceDiagram
  participant Gofer
  participant CRoot as containerRootFs
  participant Hook as nvidia-cdi-hook
  participant Sentry

  Note over Gofer,CRoot: After #13034
  Gofer->>CRoot: SetupMounts + SetupDev<br/>(libs, /dev/nvidia*, /dev/nvidiactl, ...)
  Note over CRoot: /proc is an empty dir<br/>(procfs not mounted)
  Gofer->>Hook: create-symlinks (libcuda etc.)
  Hook-->>Gofer: success
  Gofer->>Hook: enable-cuda-compat --host-driver-version=580.159.04
  Hook-->>Gofer: success
  Gofer->>Hook: update-ldcache --folder /usr/lib/...
  Hook-->>Gofer: success
  Gofer->>Hook: disable-device-node-modification
  Hook--xCRoot: open o_path procfd:<br/>open /...rootfs/proc/driver/nvidia/params → ENOENT
  Note over Gofer: FATAL — sandbox never reaches Sentry

Loading

Live evidence from /tmp/runsc-debug/runsc.<ts>.gofer.log:

hooks.go:63] Executing hook nvidia-ctk hook create-symlinks ...
hooks.go:118] Execute hook success!
hooks.go:63] Executing hook nvidia-ctk hook enable-cuda-compat --host-driver-version=580.159.04
hooks.go:118] Execute hook success!
hooks.go:63] Executing hook nvidia-ctk hook update-ldcache --folder ...
hooks.go:118] Execute hook success!
hooks.go:63] Executing hook nvidia-ctk hook disable-device-node-modification
util.go:107] FATAL ERROR: error executing CreateContainer hooks: failure executing hook "/usr/bin/nvidia-ctk", err: exit status 1
stderr: failed to mount modified params file: open o_path procfd:
  open /run/containerd/.../rootfs/proc/driver/nvidia/params: no such file or directory

Re-running with the failing hook removed from the device-plugin spec → sandbox starts cleanly, torch.cuda.is_available() returns True, T4 visible.

Why the hook fails on gVisor

The hook's implementation (NVIDIA Container Toolkit, PR #927) opens <containerRootFs>/proc/driver/nvidia/params, writes a tmpfs copy with ModifyDeviceFiles: 0, and bind-mounts that tmpfs copy back over the original path. Under runc, /proc has already been mounted into the container's mount namespace before createContainer hooks run (per OCI spec), so the kernel-provided /proc/driver/nvidia/params is visible. Under gVisor with #13034, the containerRootFs has device cdevs and library bind-mounts but /proc is left empty until the sentry boots its synthetic procfs later.

The hook's semantic purpose (prevent in-container libnvidia-ml from auto-creating extra /dev/nvidiaN nodes) is moot under gVisor anyway — nvproxy mediates all device access, and the sentry owns /dev. The hook just needs to be able to complete so sandbox setup proceeds.

Proposed fix

In runsc/cmd/sandboxsetup/gofer_mount.go, when nvproxy is enabled, bind-mount the host's /proc/driver/nvidia directory into <containerRootFs>/proc/driver/nvidia before running createContainer hooks (i.e. right alongside the SetupDev step that already exposes /dev/nvidia*). This:

1. Makes <containerRootFs>/proc/driver/nvidia/params available for the hook's open(o_path).
2. Lets the hook complete (its overmount lives in the gofer-side view of containerRootFs; the sentry's procfs is unaffected, which is fine — the hook's effect doesn't apply under gVisor regardless).
3. Has no observable effect when the hook isn't present in the spec.

Sketch:

// In SetupDev (or a sibling), when specutils.NVProxyEnabled(spec, conf):
nvidiaProcSrc := "/proc/driver/nvidia"
if _, err := os.Stat(nvidiaProcSrc); err == nil {
    nvidiaProcDst := filepath.Join(root, "proc/driver/nvidia")
    if err := os.MkdirAll(nvidiaProcDst, 0755); err != nil {
        return fmt.Errorf("creating nvidia procfs mount target: %w", err)
    }
    if err := specutils.SafeSetupAndMount(nvidiaProcSrc, nvidiaProcDst,
        "bind", unix.MS_BIND|unix.MS_RDONLY, procPath); err != nil {
        return fmt.Errorf("bind-mounting %q to %q: %w", nvidiaProcSrc, nvidiaProcDst, err)
    }
}

Happy to send a PR for this — I'll prepare against master with unit-test coverage for the helper.

Workaround today

For anyone hitting this on release-20260520.0+, on each GPU node (re-run when the device-plugin pod restarts):

sudo python3 -c 'import json
p="/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json"
d=json.load(open(p))
d["containerEdits"]["hooks"]=[h for h in d["containerEdits"]["hooks"]
                              if h["args"][2] != "disable-device-node-modification"]
json.dump(d,open(p,"w"),indent=2)'

Related

• #13034 — fixed the other three hooks; container cgroupRoot setup is now done before hooks
• NVIDIA/nvidia-container-toolkit#927 — introduced the failing hook
• NVIDIA/k8s-device-plugin#1245 — cluster-side opt-out (frozen since Jan)
• NVIDIA/nvidia-container-toolkit#1833 — adds a jit-cdi feature flag to skip the hook (different failure mode, same hook)