feat: add nvidia gpu support to coco pattern

.gitignore

@@ -20,4 +20,5 @@ openshift-install*
 node_modules
 .envrc
 .ansible/
-__pycache__/
+__pycache__/
+LAB.md

README.md

@@ -2,21 +2,23 @@
 
 Validated pattern for deploying confidential containers on OpenShift using the [Validated Patterns](https://validatedpatterns.io/) framework.
 
-Confidential containers use hardware-backed Trusted Execution Environments (TEEs) to isolate workloads from cluster and hypervisor administrators. This pattern deploys and configures the Red Hat CoCo stack — including the sandboxed containers operator, Trustee (Key Broker Service), and peer-pod infrastructure — on Azure.
+Confidential containers use hardware-backed Trusted Execution Environments (TEEs) to isolate workloads from cluster and hypervisor administrators. This pattern deploys and configures the Red Hat CoCo stack — including the sandboxed containers operator, Trustee (Key Broker Service), and peer-pod infrastructure — on Azure and bare metal.
 
 ## Topologies
 
-The pattern provides two deployment topologies:
+The pattern provides three deployment topologies:
 
-1. **Single cluster** (`simple` clusterGroup) — deploys all components (Trustee, Vault, ACM, sandboxed containers, workloads) in one cluster. This breaks the RACI separation expected in a remote attestation architecture but simplifies testing and demonstrations.
+1. **Single cluster** (`simple` clusterGroup) — deploys all components (Trustee, Vault, ACM, sandboxed containers, workloads) in one cluster on Azure. This breaks the RACI separation expected in a remote attestation architecture but simplifies testing and demonstrations.
 
 2. **Multi-cluster** (`trusted-hub` + `spoke` clusterGroups) — separates the trusted zone from the untrusted workload zone:
    - **Hub** (`trusted-hub`): Runs Trustee (KBS + attestation service), HashiCorp Vault, ACM, and cert-manager. This cluster is the trust anchor.
    - **Spoke** (`spoke`): Runs the sandboxed containers operator and confidential workloads. The spoke is imported into ACM and managed from the hub.
 
+3. **Bare metal** (`baremetal` clusterGroup) — deploys all components on bare metal hardware with Intel TDX or AMD SEV-SNP support. NFD (Node Feature Discovery) auto-detects the CPU architecture and configures the appropriate runtime. Supports SNO (Single Node OpenShift) and multi-node clusters.
+
 The topology is controlled by the `main.clusterGroupName` field in `values-global.yaml`.
 
-Currently supports Azure via peer-pods. Peer-pods provision confidential VMs (`Standard_DCas_v5` family) directly on the Azure hypervisor rather than nesting VMs inside worker nodes.
+Azure deployments use peer-pods, which provision confidential VMs (`Standard_DCas_v5` family) directly on the Azure hypervisor. Bare metal deployments use layered images and hardware TEE features directly.
 
 ## Current version (4.*)
 
@@ -42,9 +44,21 @@ All previous versions used pre-GA (Technology Preview) releases of Trustee:
 
 ### Prerequisites
 
+**Azure deployments:**
+
 - OpenShift 4.17+ cluster on Azure (self-managed via `openshift-install` or ARO)
 - Azure `Standard_DCas_v5` VM quota in your target region (these are confidential computing VMs and are not available in all regions). See the note below for more details.
 - Azure DNS hosting the cluster's DNS zone
+
+**Bare metal deployments:**
+
+- OpenShift 4.17+ cluster on bare metal with Intel TDX or AMD SEV-SNP hardware
+- BIOS/firmware configured to enable TDX or SEV-SNP
+- Available block devices for LVMS storage (auto-discovered)
+- For Intel TDX: an Intel PCS API key from [api.portal.trustedservices.intel.com](https://api.portal.trustedservices.intel.com/)
+
+**Common:**
+
 - Tools on your workstation: `podman`, `yq`, `jq`, `skopeo`
 - OpenShift pull secret saved at `~/pull-secret.json` (download from [console.redhat.com](https://console.redhat.com/openshift/downloads))
 - Fork the repository — ArgoCD reconciles cluster state against your fork, so changes must be pushed to your remote
@@ -53,20 +67,20 @@ All previous versions used pre-GA (Technology Preview) releases of Trustee:
 
 These scripts generate the cryptographic material and attestation measurements needed by Trustee and the peer-pod VMs. Run them once before your first deployment.
 
-1. `bash scripts/gen-secrets.sh` — generates KBS key pairs, attestation policy seeds, and copies `values-secret.yaml.template` to `~/values-secret-coco-pattern.yaml`
-2. `bash scripts/get-pcr.sh` — retrieves PCR measurements from the peer-pod VM image and stores them at `~/.coco-pattern/measurements.json` (requires `podman`, `skopeo`, and `~/pull-secret.json`)
-3. Review and customise `~/values-secret-coco-pattern.yaml` — this file is loaded into Vault and provides secrets to the pattern
+1. `bash scripts/gen-secrets.sh` — generates KBS key pairs, PCCS certificates/tokens (for bare metal), and copies `values-secret.yaml.template` to `~/values-secret-coco-pattern.yaml`
+2. `bash scripts/get-pcr.sh` — retrieves PCR measurements from the peer-pod VM image and stores them at `~/.coco-pattern/measurements.json` (requires `podman`, `skopeo`, and `~/pull-secret.json`). **Not required for bare metal deployments.**
+3. Review and customise `~/values-secret-coco-pattern.yaml` — this file is loaded into Vault and provides secrets to the pattern. For bare metal, uncomment the PCCS secrets section and provide your Intel PCS API key.
 
 > **Note:** `gen-secrets.sh` will not overwrite existing secrets. Delete `~/.coco-pattern/` if you need to regenerate.
 
-### Single cluster deployment
+### Single cluster deployment (Azure)
 
 1. Set `main.clusterGroupName: simple` in `values-global.yaml`
 2. Ensure your Azure configuration is populated in `values-global.yaml` (see `global.azure.*` fields)
 3. `./pattern.sh make install`
 4. Wait for the cluster to reboot all nodes (the sandboxed containers operator triggers a MachineConfig update). Monitor progress in the ArgoCD UI.
 
-### Multi-cluster deployment
+### Multi-cluster deployment (Azure)
 
 1. Set `main.clusterGroupName: trusted-hub` in `values-global.yaml`
 2. Deploy the hub cluster: `./pattern.sh make install`
@@ -76,6 +90,25 @@ These scripts generate the cryptographic material and attestation measurements n
    (see [importing a cluster](https://validatedpatterns.io/learn/importing-a-cluster/))
 6. ACM will automatically deploy the `spoke` clusterGroup applications (sandboxed containers, workloads) to the imported cluster
 
+### Bare metal deployment
+
+1. Set `main.clusterGroupName: baremetal` in `values-global.yaml`
+2. Run `bash scripts/gen-secrets.sh` to generate KBS keys and PCCS secrets
+3. For Intel TDX: uncomment the PCCS secrets in `~/values-secret-coco-pattern.yaml` and provide your Intel PCS API key
+4. `./pattern.sh make install`
+5. Wait for the cluster to reboot nodes (MachineConfig updates for TDX kernel parameters and vsock)
+
+The system auto-detects your hardware:
+
+- **NFD** discovers Intel TDX or AMD SEV-SNP capabilities and labels nodes
+- **LVMS** auto-discovers available block devices for storage
+- **RuntimeClass** `kata-cc` is created automatically pointing to the correct handler (`kata-tdx` or `kata-snp`)
+- Both `kata-tdx` and `kata-snp` RuntimeClasses are deployed; only the one matching your hardware has schedulable nodes
+- MachineConfigs are deployed for both `master` and `worker` roles (safe on SNO where only master exists)
+- PCCS and QGS services deploy unconditionally; DaemonSets only schedule on Intel nodes via NFD labels
+
+Optional: pin PCCS to a specific node with `bash scripts/get-pccs-node.sh` and set `baremetal.pccs.nodeSelector` in the baremetal chart values.
+
 ## Sample applications
 
 Two sample applications are deployed on the cluster running confidential workloads (the single cluster in `simple` mode, or the spoke in multi-cluster mode):

ansible/init-data-gzipper.yaml

@@ -114,21 +114,33 @@
       register: debug_initdata_encoded
       changed_when: false
 
+    - name: Compute raw SHA-256 hash of default initdata
+      ansible.builtin.shell: |
+        set -o pipefail
+        sha256sum "{{ rendered_path }}" | cut -d' ' -f1
+      register: raw_hash
+      changed_when: false
+
+    - name: Compute raw SHA-256 hash of debug initdata
+      ansible.builtin.shell: |
+        set -o pipefail
+        sha256sum "{{ debug_rendered_path }}" | cut -d' ' -f1
+      register: debug_raw_hash
+      changed_when: false
+
     - name: Register init data pcr into a var
       ansible.builtin.shell: |
         set -o pipefail
-        hash=$(sha256sum "{{ rendered_path }}" | cut -d' ' -f1)
         initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
-        PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
+        PCR8_HASH=$(echo -n "${initial_pcr}{{ raw_hash.stdout }}" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
       register: pcr8_hash
       changed_when: false
 
     - name: Register debug init data pcr into a var
       ansible.builtin.shell: |
         set -o pipefail
-        hash=$(sha256sum "{{ debug_rendered_path }}" | cut -d' ' -f1)
         initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
-        PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
+        PCR8_HASH=$(echo -n "${initial_pcr}{{ debug_raw_hash.stdout }}" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
       register: debug_pcr8_hash
       changed_when: false
 
@@ -147,6 +159,7 @@
           data:
             INITDATA: "{{ initdata_encoded.stdout }}"
             PCR8_HASH: "{{ pcr8_hash.stdout }}"
+            RAW_HASH: "{{ raw_hash.stdout }}"
             version: "0.1.0"
             algorithm: "sha256"
             aa.toml: "{{ raw_aa_toml.stdout }}"
@@ -168,6 +181,7 @@
           data:
             INITDATA: "{{ debug_initdata_encoded.stdout }}"
             PCR8_HASH: "{{ debug_pcr8_hash.stdout }}"
+            RAW_HASH: "{{ debug_raw_hash.stdout }}"
             version: "0.1.0"
             algorithm: "sha256"
             aa.toml: "{{ raw_aa_toml.stdout }}"

ansible/reconcile-kataconfig-gpu.yaml

@@ -0,0 +1,42 @@
+---
+- name: Reconcile KataConfig for GPU RuntimeClass
+  hosts: localhost
+  connection: local
+  become: false
+  gather_facts: true
+  tasks:
+    - name: Check for nodes with NVIDIA GPU labels
+      kubernetes.core.k8s_info:
+        api_version: v1
+        kind: Node
+        label_selectors:
+          - "nvidia.com/gpu.present=true"
+      register: gpu_nodes
+
+    - name: Check if kata-cc-nvidia-gpu RuntimeClass exists
+      kubernetes.core.k8s_info:
+        api_version: node.k8s.io/v1
+        kind: RuntimeClass
+        name: kata-cc-nvidia-gpu
+      register: gpu_runtimeclass
+
+    - name: Trigger KataConfig re-reconciliation
+      kubernetes.core.k8s:
+        state: patched
+        api_version: kataconfiguration.openshift.io/v1
+        kind: KataConfig
+        name: default-kata-config
+        definition:
+          metadata:
+            annotations:
+              kata-reconcile: "{{ ansible_date_time.epoch }}"
+      when:
+        - gpu_nodes.resources | length > 0
+        - gpu_runtimeclass.resources | length == 0
+
+    - name: Report status
+      ansible.builtin.debug:
+        msg: >-
+          GPU nodes: {{ gpu_nodes.resources | length }},
+          RuntimeClass exists: {{ gpu_runtimeclass.resources | length > 0 }},
+          Action: {{ 'triggered re-reconciliation' if (gpu_nodes.resources | length > 0 and gpu_runtimeclass.resources | length == 0) else 'no action needed' }}

charts/all/baremetal/Chart.yaml

@@ -0,0 +1,9 @@
+apiVersion: v2
+description: Bare metal platform configuration (NFD rules, MachineConfigs, RuntimeClasses, Intel device plugin).
+keywords:
+- pattern
+- upstream
+- sandbox
+- baremetal
+name: baremetal
+version: 0.0.1

charts/all/baremetal/templates/kata-nfd.yaml

@@ -0,0 +1,80 @@
+apiVersion: nfd.openshift.io/v1alpha1
+kind: NodeFeatureRule
+metadata:
+  name: consolidated-hardware-features
+  namespace: openshift-nfd
+spec:
+  rules:
+    - name: "runtime.kata"
+      labels:
+        feature.node.kubernetes.io/runtime.kata: "true"
+      matchAny:
+        - matchFeatures:
+            - feature: cpu.cpuid
+              matchExpressions:
+                SSE42: { op: Exists }
+                VMX: { op: Exists }
+            - feature: kernel.loadedmodule
+              matchExpressions:
+                kvm: { op: Exists }
+                kvm_intel: { op: Exists }
+        - matchFeatures:
+            - feature: cpu.cpuid
+              matchExpressions:
+                SSE42: { op: Exists }
+                SVM: { op: Exists }
+            - feature: kernel.loadedmodule
+              matchExpressions:
+                kvm: { op: Exists }
+                kvm_amd: { op: Exists }
+
+    - name: "amd.sev-snp"
+      labels:
+        amd.feature.node.kubernetes.io/snp: "true"
+      extendedResources:
+        sev-snp.amd.com/esids: "@cpu.security.sev.encrypted_state_ids"
+      matchFeatures:
+        - feature: cpu.cpuid
+          matchExpressions:
+            SVM: { op: Exists }
+        - feature: cpu.security
+          matchExpressions:
+            sev.snp.enabled: { op: Exists }
+
+    - name: "intel.sgx"
+      labels:
+        intel.feature.node.kubernetes.io/sgx: "true"
+      extendedResources:
+        sgx.intel.com/epc: "@cpu.security.sgx.epc"
+      matchFeatures:
+        - feature: cpu.cpuid
+          matchExpressions:
+            SGX: { op: Exists }
+            SGXLC: { op: Exists }
+        - feature: cpu.security
+          matchExpressions:
+            sgx.enabled: { op: IsTrue }
+        - feature: kernel.config
+          matchExpressions:
+            X86_SGX: { op: Exists }
+
+    - name: "intel.tdx"
+      labels:
+        intel.feature.node.kubernetes.io/tdx: "true"
+      extendedResources:
+        tdx.intel.com/keys: "@cpu.security.tdx.total_keys"
+      matchFeatures:
+        - feature: cpu.cpuid
+          matchExpressions:
+            VMX: { op: Exists }
+        - feature: cpu.security
+          matchExpressions:
+            tdx.enabled: { op: Exists }
+
+    - name: "ibm.se.enabled"
+      labels:
+        ibm.feature.node.kubernetes.io/se: "true"
+      matchFeatures:
+        - feature: cpu.security
+          matchExpressions:
+            se.enabled: { op: IsTrue }

charts/all/baremetal/templates/nfd-instance.yaml

@@ -0,0 +1,12 @@
+apiVersion: nfd.openshift.io/v1
+kind: NodeFeatureDiscovery
+metadata:
+  name: nfd-instance
+  namespace: openshift-nfd
+spec:
+  operand:
+    image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.20
+    imagePullPolicy: Always
+    servicePort: 12000
+  workerConfig:
+    configData: |

charts/all/baremetal/templates/vsock-mco.yaml

@@ -0,0 +1,24 @@
+{{- range list "master" "worker" }}
+---
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: {{ . }}
+  name: 99-enable-coco-{{ . }}
+spec:
+  kernelArguments:
+  - nohibernate
+{{- if $.Values.tdx.enabled }}
+  - kvm_intel.tdx=1
+{{- end }}
+  config:
+    ignition:
+      version: 3.2.0
+    storage:
+      files:
+        - path: /etc/modules-load.d/vsock.conf
+          mode: 0644
+          contents:
+            source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK
+{{- end }}

charts/all/baremetal/values.yaml

@@ -0,0 +1,2 @@
+tdx:
+  enabled: true

charts/all/coco-kyverno-policies/templates/inject-coco-initdata.yaml

@@ -6,14 +6,14 @@ metadata:
     policies.kyverno.io/title: Inject CoCo InitData
     policies.kyverno.io/category: Confidential Computing
     policies.kyverno.io/severity: medium
-    policies.kyverno.io/subject: Pod,Deployment
+    policies.kyverno.io/subject: Pod
     policies.kyverno.io/description: >-
       Injects cc_init_data annotation into pods with a kata runtime class
       by reading from a ConfigMap specified via the coco.io/initdata-configmap
-      annotation. Kyverno autogen extends this to Deployments, StatefulSets,
-      DaemonSets, and Jobs automatically.
+      annotation. Targets Pods only (not Deployments) so that Deployments
+      remain stable and a rollout restart resolves the latest initdata.
     argocd.argoproj.io/sync-wave: "1"
-    pod-policies.kyverno.io/autogen-controllers: Deployment,StatefulSet,DaemonSet,Job
+    pod-policies.kyverno.io/autogen-controllers: none
 spec:
   rules:
     - name: inject-initdata
@@ -28,7 +28,7 @@ spec:
         all:
           - key: "{{ "{{" }}request.object.spec.runtimeClassName || '' {{ "}}" }}"
             operator: AnyIn
-            value: ["kata", "kata-cc", "kata-remote"]
+            value: ["kata", "kata-cc", "kata-remote", "kata-cc-nvidia-gpu"]
           - key: "{{ "{{" }}request.object.metadata.annotations.\"coco.io/initdata-configmap\" || '' {{ "}}" }}"
             operator: NotEquals
             value: ""

charts/all/coco-kyverno-policies/values.yaml

@@ -1,5 +1,6 @@
 workloadNamespaces:
   - hello-openshift
   - kbs-access
+  - gpu-workload
 
 initdataSourceNamespace: imperative

charts/all/intel-dcap/Chart.yaml

@@ -0,0 +1,10 @@
+apiVersion: v2
+description: Intel DCAP services (PCCS and QGS) for TDX remote attestation.
+keywords:
+- pattern
+- intel
+- tdx
+- pccs
+- qgs
+name: intel-dcap
+version: 0.0.1

charts/all/intel-dcap/templates/intel-dpo-sgx.yaml

@@ -0,0 +1,11 @@
+apiVersion: deviceplugin.intel.com/v1
+kind: SgxDevicePlugin
+metadata:
+  name: sgxdeviceplugin-sample
+spec:
+  image: registry.connect.redhat.com/intel/intel-sgx-plugin@sha256:f2c77521c6dae6b4db1896a5784ba8b06a5ebb2a01684184fc90143cfcca7bf4
+  enclaveLimit: 110
+  provisionLimit: 110
+  logLevel: 4
+  nodeSelector:
+    intel.feature.node.kubernetes.io/sgx: "true"