feat(self-managed): secure LLM gRPC ingress

deploy/helm/gateway-routes/Makefile

@@ -25,4 +25,4 @@ test-vanity-gateway-route:
 	@sh ./scripts/test-vanity-gateway-route.sh
 
 check-llm-worker-routes:
-	@sh ./scripts/check-llm-worker-routes.sh
+	@bash ./scripts/check-llm-worker-routes.sh

deploy/helm/gateway-routes/README.md

@@ -7,11 +7,13 @@ This repository contains the Helm chart for deploying NVCF ingress routes via th
 The chart deploys `HTTPRoute`, `GRPCRoute`, `TCPRoute`, `UDPRoute`, and
 `ReferenceGrant` resources that attach to an existing Gateway provisioned
 separately by the cluster operator, such as Envoy Gateway, Istio, Traefik, or
-Kong. It also includes optional `PodMonitor` resources for scraping Envoy
+Kong. Secure LLM worker routing also renders an optional cert-manager
+`Certificate` and an Envoy Gateway `BackendTrafficPolicy` for long-lived gRPC
+streams. The chart includes optional `PodMonitor` resources for scraping Envoy
 Gateway proxy metrics with Prometheus.
 
-The chart deploys routing configuration only. It does not include any
-container images. Backend services referenced by the routes (`api`,
+The chart does not include any container images or create Gateways. Backend
+services referenced by the routes (`api`,
 `nvct-api`, `api-keys`, `invocation`, `llm-api-gateway`,
 `llm-request-router-backend-router`, `vanity-gateway`, `reval`, `sis`, `grpc`,
 `nats`) must already be deployed separately.
@@ -23,7 +25,11 @@ container images. Backend services referenced by the routes (`api`,
 - `kubectl`
 - A Gateway API compatible controller installed in the cluster
 - Existing `Gateway` resources with the listeners required by each enabled route
-- A Gateway controller with `UDPRoute` support when LLM worker routing is enabled
+- A Gateway controller with `GRPCRoute` and `UDPRoute` support when secure LLM
+  worker routing is enabled
+- cert-manager when `llmRequestRouter.grpcTls.mode=certManager`
+- Envoy Gateway's `BackendTrafficPolicy` CRD when secure LLM worker routing is
+  enabled
 - The backend services that the routes target, deployed in their respective namespaces
 
 ## Getting Started
@@ -66,8 +72,10 @@ Important settings to review before deployment:
 - `nvcfGatewayRoutes.gateways.shared.*` for the HTTP Gateway name, namespace, and listener
 - `nvcfGatewayRoutes.gateways.grpc.*` for the TCP Gateway name, namespace, and listener
 - `nvcfGatewayRoutes.gateways.nats.*` for the NATS TCP Gateway name, namespace, and listener
-- `nvcfGatewayRoutes.gateways.llmGrpc.*` for the LLM worker gRPC TCP listener
+- `nvcfGatewayRoutes.gateways.llmGrpc.*` for the LLM worker gRPC HTTPS listener
 - `nvcfGatewayRoutes.gateways.llmQuic.*` for the LLM reverse-tunnel UDP listener
+- `llmRequestRouter.grpcTls.*` for the dedicated gRPC listener identity,
+  explicit plaintext opt-in, and certificate ownership mode
 - `nvcfGatewayRoutes.routes.<route>.enabled` to toggle individual routes
 - `nvcfGatewayRoutes.routes.nvcfApi.grpc.enabled` and
   `nvcfGatewayRoutes.routes.nvctApi.grpc.enabled` to expose API gRPC routes
@@ -98,13 +106,14 @@ Enabled `HTTPRoute` entries must not share a resolved hostname because each `HTT
 | `grpc` | TCPRoute | Not rendered | `grpc.nvcf:10081` |
 | `grpcWorker` | TCPRoute (disabled by default) | Not rendered | `grpc.nvcf:10086` |
 | `nats` | TCPRoute (disabled by default) | Not rendered | `nats.nats-system:4222` |
-| `llmWorker` | TCPRoute and UDPRoute (disabled by default) | Not rendered | `llm-request-router-backend-router.<backend namespace>:50071/TCP,50072/UDP` |
+| `llmWorker` | GRPCRoute (secure) or TCPRoute (explicit development), plus UDPRoute; disabled by default | Not rendered | `llm-request-router-backend-router.<backend namespace>:50071/h2c,50072/UDP` |
 
 Cross-namespace routing is supported via `ReferenceGrant` resources rendered into each backend namespace.
 
 ## Notes
 
-- The chart assumes the Gateway is reachable at the resolved hostnames. DNS records and TLS termination are out of scope and must be configured in the surrounding infrastructure.
+- The chart assumes the Gateway is reachable at the resolved hostnames. DNS
+  records and Gateway creation remain infrastructure responsibilities.
 - The `nats` TCPRoute is plain TCP and does not render hostnames. Configure DNS or TCP load balancer routing outside this chart.
 - The `grpc` TCPRoute does not enforce HTTP hostname matching at the Gateway layer. Configure DNS or TCP load balancer routing outside this chart.
 - The `grpcWorker` TCPRoute is beta support for split or multi-cluster gRPC worker callbacks. It carries HTTP/1 CONNECT callback traffic only. Enable it only when the control-plane grpc-proxy runs one replica with HPA disabled. Multi-replica grpc-proxy requires pod-specific callback routing and is not supported by this shared TCPRoute.
@@ -116,5 +125,18 @@ Cross-namespace routing is supported via `ReferenceGrant` resources rendered int
   Use `nvcfGatewayRoutes.routes.llmWorker.backend.grpcPort` for registration
   traffic and `nvcfGatewayRoutes.routes.llmWorker.backend.quicPort` for reverse
   tunnels; this route does not use the generic `backend.port` setting.
-  Keep the TCP and UDP Gateways separate when the infrastructure requires
+  Secure mode renders a `GRPCRoute` with no `hostnames` and requires a
+  dedicated HTTPS listener with no `hostname`. The listener serves the
+  configured certificate for normal external SNI verification, while the
+  backend router receives the advertised Stargate identity as HTTP/2
+  `:authority`. Setting the listener hostname to the public dial name would
+  incorrectly require that same value in `:authority`.
+  `grpcTls.mode=certManager` creates the named Secret through a dedicated
+  `Certificate` in the gRPC Gateway namespace. `mode=existingSecret` expects
+  the operator to create that Secret. The HTTPS listener must reference the
+  same Secret. The Envoy `BackendTrafficPolicy` sets the request timeout to
+  `0s` for Watch and Register streams.
+  Plaintext is intended only for development and requires
+  `grpcTls.allowInsecureHttp=true`; it renders the legacy `TCPRoute`.
+  Keep the HTTPS and UDP Gateways separate when the infrastructure requires
   separate load balancers for each protocol.

deploy/helm/gateway-routes/chart/templates/_helpers.tpl

@@ -57,6 +57,36 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 {{- required "nvcfGatewayRoutes.routes.llmWorker.backend.namespace is required when llmWorker.enabled is true" .Values.nvcfGatewayRoutes.routes.llmWorker.backend.namespace -}}
 {{- end }}
 
+{{/* Validate worker-facing gRPC TLS identity ownership and plaintext policy. */}}
+{{- define "nvcf-gateway.validateLLMWorkerGrpcTls" -}}
+{{- $routeEnabled := .Values.nvcfGatewayRoutes.routes.llmWorker.enabled -}}
+{{- $grpcTls := .Values.llmRequestRouter.grpcTls | default dict -}}
+{{- $tlsEnabled := dig "enabled" false $grpcTls -}}
+{{- $allowInsecure := dig "allowInsecureHttp" false $grpcTls -}}
+{{- $mode := dig "mode" "certManager" $grpcTls | toString -}}
+{{- if and $tlsEnabled $allowInsecure -}}
+{{- fail "llmRequestRouter.grpcTls.enabled and llmRequestRouter.grpcTls.allowInsecureHttp cannot both be true" -}}
+{{- end -}}
+{{- if and $routeEnabled (not $tlsEnabled) (not $allowInsecure) -}}
+{{- fail "llmRequestRouter.grpcTls.allowInsecureHttp must be true when LLM worker routing is plaintext" -}}
+{{- end -}}
+{{- if $tlsEnabled -}}
+{{- if not $routeEnabled -}}
+{{- fail "nvcfGatewayRoutes.routes.llmWorker.enabled must be true when llmRequestRouter.grpcTls.enabled is true" -}}
+{{- end -}}
+{{- if not (has $mode (list "certManager" "existingSecret")) -}}
+{{- fail (printf "llmRequestRouter.grpcTls.mode must be certManager or existingSecret, got %q" $mode) -}}
+{{- end -}}
+{{- required "llmRequestRouter.grpcTls.secretName is required when grpcTls.enabled is true" (dig "secretName" "" $grpcTls) -}}
+{{- if eq $mode "certManager" -}}
+{{- if empty (dig "dnsNames" (list) $grpcTls) -}}
+{{- fail "llmRequestRouter.grpcTls.dnsNames is required when grpcTls.mode is certManager" -}}
+{{- end -}}
+{{- required "llmRequestRouter.grpcTls.issuerRef.name is required when grpcTls.mode is certManager" (dig "issuerRef" "name" "" $grpcTls) -}}
+{{- end -}}
+{{- end -}}
+{{- end }}
+
 {{/*
 Validate that enabled HTTPRoutes do not compete for the same hostname and
 root PathPrefix match on the shared Gateway. All HTTPRoute templates in this

deploy/helm/gateway-routes/chart/templates/backendtrafficpolicy-llm-worker-grpc.yaml

@@ -0,0 +1,22 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled .Values.llmRequestRouter.grpcTls.enabled }}
+{{- include "nvcf-gateway.validateLLMWorkerGrpcTls" . }}
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: BackendTrafficPolicy
+metadata:
+  name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-grpc-streams
+  namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+    app.kubernetes.io/component: llm-worker-grpc-stream-policy
+spec:
+  targetRefs:
+    - group: gateway.networking.k8s.io
+      kind: GRPCRoute
+      name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-grpc
+  timeout:
+    http:
+      requestTimeout: 0s
+{{- end }}

deploy/helm/gateway-routes/chart/templates/certificate-llm-worker-grpc.yaml

@@ -0,0 +1,29 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+{{- $grpcTls := .Values.llmRequestRouter.grpcTls | default dict -}}
+{{- if and .Values.nvcfGatewayRoutes.enabled (dig "enabled" false $grpcTls) }}
+{{- include "nvcf-gateway.validateLLMWorkerGrpcTls" . }}
+{{- if eq (dig "mode" "certManager" $grpcTls) "certManager" }}
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: {{ dig "secretName" "" $grpcTls | quote }}
+  namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
+  labels:
+    {{- include "nvcf-gateway.labels" . | nindent 4 }}
+    app.kubernetes.io/component: llm-worker-grpc-certificate
+spec:
+  secretName: {{ dig "secretName" "" $grpcTls | quote }}
+  dnsNames:
+    {{- dig "dnsNames" (list) $grpcTls | toYaml | nindent 4 }}
+  issuerRef:
+    kind: {{ dig "issuerRef" "kind" "ClusterIssuer" $grpcTls | quote }}
+    name: {{ dig "issuerRef" "name" "" $grpcTls | quote }}
+    {{- with dig "issuerRef" "group" "" $grpcTls }}
+    group: {{ . | quote }}
+    {{- end }}
+  usages:
+    - server auth
+{{- end }}
+{{- end }}

deploy/helm/gateway-routes/chart/templates/referencegrant-llm-worker.yaml

@@ -13,7 +13,7 @@ metadata:
 spec:
   from:
     - group: gateway.networking.k8s.io
-      kind: TCPRoute
+      kind: {{ if .Values.llmRequestRouter.grpcTls.enabled }}GRPCRoute{{ else }}TCPRoute{{ end }}
       namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}
     - group: gateway.networking.k8s.io
       kind: UDPRoute

deploy/helm/gateway-routes/chart/templates/tcproute-llm-worker.yaml

@@ -2,9 +2,11 @@
 # SPDX-License-Identifier: Apache-2.0
 
 {{- if and .Values.nvcfGatewayRoutes.enabled .Values.nvcfGatewayRoutes.routes.llmWorker.enabled }}
+{{- include "nvcf-gateway.validateLLMWorkerGrpcTls" . }}
 {{- $backendNamespace := include "nvcf-gateway.llmWorkerBackendNamespace" . }}
-apiVersion: gateway.networking.k8s.io/v1alpha2
-kind: TCPRoute
+{{- $tlsEnabled := .Values.llmRequestRouter.grpcTls.enabled }}
+apiVersion: gateway.networking.k8s.io/{{ if $tlsEnabled }}v1{{ else }}v1alpha2{{ end }}
+kind: {{ if $tlsEnabled }}GRPCRoute{{ else }}TCPRoute{{ end }}
 metadata:
   name: {{ .Values.nvcfGatewayRoutes.routes.llmWorker.name }}-grpc
   namespace: {{ .Values.nvcfGatewayRoutes.gateways.llmGrpc.namespace }}

deploy/helm/gateway-routes/chart/values.yaml

@@ -283,3 +283,24 @@ nvcfGatewayRoutes:
   # PodMonitors for Envoy Gateway proxy pods.
   podMonitors:
     enabled: false
+
+# Worker-facing gRPC TLS is configured with the request router because the
+# certificate and advertised dial URI form one transport contract. The
+# Gateway itself remains cluster-operator owned; its dedicated HTTPS listener
+# must reference this Secret without constraining the HTTP/2 authority.
+llmRequestRouter:
+  grpcTls:
+    enabled: false
+    # Plaintext is retained only for an explicitly selected development path.
+    allowInsecureHttp: false
+    # certManager renders a Certificate. existingSecret uses an operator-owned
+    # Secret and renders no issuance resource.
+    mode: certManager
+    secretName: ""
+    # Required only in certManager mode. In existingSecret mode, the operator
+    # owns and verifies the certificate SANs.
+    dnsNames: []
+    issuerRef:
+      kind: ClusterIssuer
+      name: ""
+      group: ""