feat(self-managed): secure Pylon gRPC registration on port 50071

[mikeyrcamp]

Description

Provide a supported, certificate-validated gRPC registration endpoint for Pylon workers on port 50071.

The self-managed stack currently exposes the request-router gRPC service as plaintext h2c and advertises a scheme-less address that Pylon normalizes to HTTP. Existing request-router certificate settings configure reverse QUIC TLS on port 50072; mounting that certificate does not enable gRPC TLS on port 50071.

For a secure multi-cluster deployment, workers need this explicit path:

Pylon -- TLS/h2 --> worker-reachable endpoint :50071
                         `-- plaintext h2c --> stargate-k8s-router :50071
                                                    `-- gRPC --> selected Stargate pod

Certificates identify DNS names rather than ports. Reuse an existing certificate when its SAN covers the worker dial hostname and the terminator can consume it; otherwise issue a certificate for that hostname.

Definition of Done

• LLM/PKI-enabled self-managed profiles configure TLS/h2 termination for worker gRPC registration on port 50071.
• The configured certificate chains to the worker trust contract and covers the external dial hostname.
• ALPN negotiates h2; the backend connection remains supported plaintext h2c.
• The NVCF API worker bootstrap address and addons.llm.requestRouter.backendRouter.pylonGrpcDialAddress both use the same explicit HTTPS endpoint.
• The self-managed configuration schema and validation accept explicit URI schemes and reject incomplete TLS/certificate combinations.
• Secure profiles do not silently fall back to plaintext. Plaintext remains available only through an explicit development-mode choice.
• Listener routing selects the certificate by public SNI while permitting the internal per-pod HTTP/2 authorities used by stargate-k8s-router.
• Streaming Watch and Register RPCs are not terminated by an HTTP request timeout.
• openssl s_client verifies the chain and negotiates h2 without insecure flags.
• grpcurl receives WatchStargates, and every returned grpcPylonDialAddr is HTTPS.
• A multi-cluster end-to-end test proves worker registration, authentication, reverse QUIC, and invocation.
• Documentation clearly separates gRPC TLS/50071 from reverse QUIC TLS/50072.

Resources

• Tracked by epic(llm-router): secure and reliable Pylon registration across clusters #1292.
• Related network exposure work: Expose the LLM request router to split-cluster workers #689.
• Related QUIC PKI epic: Complete PKI support for multi-cluster QUIC transport #19.
• NVCF API authorization gRPC on port 9090 is separate: feat(self-managed): support TLS on NVCF API gRPC listener #1207.

---

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow the project code of conduct and contributing guidelines.

[mikeyrcamp]

Implementation is now in public review as
#1300, stacked on the explicit URI
contract in #1299.

The PR adds:

• a dedicated HTTPS listener identity with cert-manager and existing-Secret modes
• TLS/h2 termination through GRPCRoute, forwarding h2c to the backend router
• exact HTTPS bootstrap/Pylon endpoint matching and explicit development-only HTTP
• distinct external TLS SNI and canonical Stargate HTTP/2 :authority
• an Envoy route policy that disables request and maximum stream durations
• separate, unchanged UDP/50072 QUIC routing and certificate ownership
• single- and multi-cluster BDD contracts plus operator documentation

Gateway route render/lint/tests, request-router chart tests, the complete
self-managed stack suite, local multi-cluster dry tests, all short BDD packages,
new-code Go lint, formatting, syntax, diff, independent review, and the
exact-commit secret scan are green. The scan reported zero findings and the
second review reported no remaining findings.

The fresh Deployment/StatefulSet rollout and multi-cluster TLS/PKI matrix is
still pending, so this issue should remain open. The stack also needs the
gateway-routes release containing #1300 before its released chart pin can be
advanced; source validation uses ingress.gatewayApi.chartPath meanwhile.

[mikeyrcamp]

Final PR-head validation for #1300 at dcccef6d passed for both the default Deployment and explicitly pinned StatefulSet paths.

Each fresh split-cluster run verified the dedicated gRPC listener certificate and ALPN h2, HTTPS Watch, h2c backend forwarding, forwarded Register streams, worker authentication, three pylon_registration_stream_connected gauges at 1, three reverse-QUIC gauges at 1, stability beyond 15 seconds, distinct dial SNI and per-pod authority, and successful LLM invocation. Hostname mismatch, wrong issuer, and plaintext HTTP were rejected; the worker/listener trust roots and canonical fingerprint matched. The focused LLM scenario passed 1/1 scenario and 11/11 steps, including the unauthenticated 401 check. Running images matched public IDs 39c11f1a8cd4 and deb065861afd.

All currently reported #1300 checks pass or are intentionally skipped. #1300 remains stacked on #1299, and review/merge plus latest-main repetition remain. No merge or issue closure was performed.