fix(stargate): support StatefulSet and Deployment discovery identities

[mikeyrcamp]

Describe the bug

HeadlessDnsDiscovery assumes SRV records contain stable pod hostnames suitable for use as Stargate identities and HTTP/2 authorities. With a Deployment-backed request router, Kubernetes can return both a real pod hostname and a dashed-IP target for the same physical pod.

Pylon creates registration tasks for both identities. stargate-k8s-router can route the real pod identity, but rejects the dashed-IP identity because it does not match a ready Stargate target. Once HTTPS registration works, the invalid registration stream retries indefinitely and obscures the healthy stream.

Steps or code to reproduce bug

1. Run the request-router workload as a Deployment behind its headless discovery Service.
2. Query WatchStargates and inspect the returned Stargate identities.
3. Observe the same pod advertised once with its real pod name and once with a dashed-IP identity.
4. Register using each authority. The real pod identity forwards; the dashed-IP identity is rejected as not ready.

Keep this reproduction separate from HTTPS client tests so transport and discovery failures are not conflated.

Expected behavior

• llm-request-router discovery and backend routing support both StatefulSet and Deployment
  workloads as first-class configurations. Requiring a migration from one workload kind to the
  other is not an acceptable fix.
• Discovery emits only stable identities that stargate-k8s-router can resolve to ready targets.
• One physical Stargate pod does not produce duplicate logical registration targets.
• StatefulSets retain their stable pod-DNS identity contract.
• Deployments derive a canonical, router-addressable identity from Kubernetes pod/EndpointSlice
  metadata instead of treating a dashed-IP SRV target as a separate Stargate.
• Preserve the separation between external TLS dial hostname and internal routing authority.
• EndpointSlice updates, pod replacement, and scale events do not leave stale or duplicate identities.

Acceptance criteria

• A focused discovery test reproduces the dashed-IP plus pod-name result before the fix.
• The fixed result contains only router-addressable identities.
• Registration through the backend router succeeds for every advertised target.
• Deployment rollout and scaling tests do not create duplicate or permanently retrying registration streams.
• StatefulSet rollout and scaling tests retain stable discovery and successful registration.
• A shared conformance matrix proves both workload kinds produce exactly one logical identity per
  ready pod and route that identity to the correct current pod IP and ports.
• Chart or runtime configuration can select either workload kind without changing Pylon's
  registration contract.
• A dedicated integration case runs after, and independently from, the HTTPS registration test.

Additional context

Tracked by #1292.

---

By submitting this issue, you agree to follow the project code of conduct and contributing guidelines.

[mikeyrcamp]

Implementation is now in public review: #1298 (381a4f50).

The PR makes the public chart Deployment-first while retaining an explicit StatefulSet pin, requires the EndpointSlice backend router for multi-replica Deployments, and automatically makes a direct single-replica Deployment self-only. Backend Watch advertisements and registration/QUIC forwarding now use the same EndpointSlice snapshot, with Pod targetRef.name as the canonical identity. It also preserves unchanged Watch streams with a configurable heartbeat and ensures the latest observed overlapping slice wins over a stale address.

Sanitized validation results:

• 82 stargate-k8s-router tests passed; 3 benchmarks were intentionally ignored.
• Strict Clippy and Rust formatting checks passed.
• Helm workload, PKI, backend-router render checks and Helm lint passed.
• The complete self-managed render suite passed.
• Exact-commit sensitive-content scan found 0 verified and 0 unverified findings.
• Independent review found no remaining Critical or Important issues.

The self-managed stack keeps its separately released OCI chart pin in this PR. Its existing release-driven pin-bump workflow should advance the stack only after the chart containing #1298 publishes; forwarding the option sooner would be ineffective.

Live rollout/scaling and final multi-cluster TLS evidence remains gated on #1294 and #1296. This issue and parent #1292 should remain open until the PR merges and the latest-main Deployment/StatefulSet matrix passes.

[mikeyrcamp]

Validation update for #1298 (8570cd0e) and downstream integration #1300 (dcccef6d):

• The default rendered workload is a Deployment; explicit StatefulSet pinning renders only the StatefulSet path.
• Both kinds passed fresh installs, 1→3→1 scale transitions, and rolling replacement.
• Both fresh split-cluster runs produced exactly one router-advertised target per ready pod, using EndpointSlice targetRef.name as the canonical identity for discovery and forwarding. No dashed-IP SRV aliases appeared.
• The three-replica Deployment and three-replica StatefulSet each completed secure registration, reverse QUIC, and the 11-step LLM invocation/authentication scenario with the exact sideloaded images.

All currently reported #1298 CI checks pass or are intentionally skipped. Review/branch-protection approval remains; the issue should stay open until merge and latest-main repetition.

[balajinvda]

🎉 This issue has been resolved in version stargate-v0.13.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[balajinvda]

🎉 This issue has been resolved in version helm-nvcf-llm-request-router-v1.11.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀