epic(llm-router): secure and reliable Pylon registration across clusters

[mikeyrcamp]

Description

Make Pylon discovery and registration secure and reliable across self-managed and managed multi-cluster NVCF deployments.

The current failure chain contains independent client, ingress, discovery, and configuration defects. They need separate implementation and validation so one failure does not hide the next:

• HTTPS Stargate gRPC dial endpoints are constructed without tonic client TLS support.
• Pylon gRPC needs to reuse the existing system-plus-custom trust bundle contract.
• Self-managed worker registration on port 50071 has no supported TLS/h2 termination and certificate contract by default.
• Deployment-backed SRV discovery can produce an unroutable dashed-IP authority alongside the real pod identity.
• Scheme-less cross-region watch addresses normalize to plaintext HTTP even when the listener requires TLS.

Keep TLS dial identity separate from routing authority: certificate verification and SNI use the external dial hostname, while HTTP/2 :authority continues to carry the concrete Stargate identity selected by stargate-k8s-router.

Definition of Done

• HTTPS Pylon watch and registration channels use verified TLS and preserve plaintext behavior only for explicit HTTP endpoints.
• Pylon supports public/system roots and the existing --tls-cert-path / STARGATE_TLS_CERT_PATH merged trust bundle without an insecure fallback.
• LLM/PKI-enabled self-managed profiles expose worker-reachable TLS/h2 on port 50071 with a certificate covering the dial hostname and h2c forwarding to the backend router.
• The NVCF API bootstrap address and all WatchStargates dial advertisements use explicit secure URIs in secure profiles.
• StatefulSet- and Deployment-backed discovery are both supported first-class and emit only
  router-addressable Stargate identities; requiring one workload kind is not an acceptable fix.
• Multi-cluster tests independently cover TLS, authority routing, Deployment discovery, cross-region watch, registration, authenticated reverse QUIC, and invocation.
• Negative tests prove wrong-root, wrong-hostname, missing-trust, plaintext-to-TLS, and invalid authority cases fail closed.
• Documentation distinguishes gRPC TLS on port 50071 from reverse QUIC TLS on port 50072.

Workstreams

• fix(pylon): enable TLS for HTTPS Stargate gRPC endpoints #1293 — Enable verified TLS for HTTPS Pylon Watch and Register channels.
• Support custom CA bundles for Pylon Stargate gRPC #1281 — Reuse the existing merged system-plus-custom trust bundle for Stargate gRPC.
• feat(self-managed): secure Pylon gRPC registration on port 50071 #1294 — Provide supported TLS/h2 termination and certificate wiring for port 50071.
• fix(stargate): support StatefulSet and Deployment discovery identities #1295 — Support canonical router-addressable identities for both StatefulSets and Deployments.
• fix(llm-router): require explicit schemes for cross-region watch URLs #1296 — Require explicit schemes for secure cross-region watch URLs.

These are also attached as native GitHub sub-issues. Existing issue #689 remains related to split-cluster request-router exposure rather than duplicating that network-plumbing scope here.

Resources

• Related: Complete PKI support for multi-cluster QUIC transport #19
• Related: Expose the LLM request router to split-cluster workers #689
• Related but separate NVCF API authorization listener work: feat(self-managed): support TLS on NVCF API gRPC listener #1207

---

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow the project code of conduct and contributing guidelines.

[mikeyrcamp]

Workstream update for Pylon HTTPS and trust:

• PR #1282 implements verified HTTPS Watch/Register channels and optional custom gRPC CA support.
• Dependent PR #1297 adds reverse-mode fallback to the existing transport trust path, keeps custom roots additive to enabled roots, isolates direct-mode server identities, and rejects partially malformed bundles.

The public Pylon range passed 66 Pylon unit tests, 1 CLI test, 385 pylon-lib unit tests, 2 public API tests, doc tests, formatting, clippy with warnings denied, and documentation checks. Sensitive-content scans reported 0 verified and 0 unknown findings. Independent review reported no findings.

Dependency order is now explicit: #1297 depends on #1282. The epic remains open. Next implementation is #1295 for Deployment-first dual workload support and canonical EndpointSlice identities, followed by #1296 explicit remote Watch URI schemes and #1294/#689 secure worker-reachable gRPC ingress. Final closure still requires the latest-main Deployment and StatefulSet multi-cluster TLS matrix.