fix(llm-router): require explicit schemes for cross-region watch URLs

[mikeyrcamp]

Describe the bug

Cross-region watchStargateUrls can be advertised as a raw hostname and port without a URI scheme. Pylon normalizes scheme-less gRPC addresses to http://.

When the remote endpoint is a TLS listener, Pylon therefore sends plaintext HTTP/2 to a TLS port. The remote watch fails before it can discover or register with Stargates in that region.

Steps or code to reproduce bug

1. Configure a remote watchStargateUrls entry as <host>:50071 while the endpoint requires TLS.
2. Start Pylon with otherwise working local HTTPS registration.
3. Observe the remote address normalize to HTTP and the remote watch fail at the TLS boundary.

Test this independently from the Pylon HTTPS client and Deployment/SRV identity defects.

Expected behavior

• Secure cross-region watch configuration always carries an explicit https:// URI.
• Configuration rendering/validation rejects scheme-less remote watch addresses in secure profiles rather than silently downgrading them to HTTP.
• Explicit http:// remains possible only where a development/plaintext profile deliberately permits it.
• TLS SNI and certificate verification use the remote dial hostname.
• Returned per-pod Stargate authorities remain independent of the remote dial hostname.

Acceptance criteria

• Unit and render tests cover HTTPS, explicit development HTTP, missing scheme, and unsupported scheme values.
• A multi-region test proves the remote Watch reaches stargate-k8s-router over verified TLS.
• Every returned grpcPylonDialAddr is an explicit URI consistent with its listener.
• Wrong-root, wrong-hostname, and plaintext-to-TLS cases fail closed.
• Cross-region registration, authenticated reverse QUIC, and invocation succeed after discovery.

Additional context

Tracked by #1292.

---

By submitting this issue, you agree to follow the project code of conduct and contributing guidelines.

[mikeyrcamp]

Implementation is open in #1299.

Dependency order:

• feat(pylon): support custom CA for Stargate gRPC #1282: Pylon custom gRPC CA support
• fix(pylon): preserve shared gRPC trust precedence #1297: shared gRPC trust precedence
• feat(llm-request-router): support Deployment workloads #1298: Deployment-first backend-router discovery
• fix(llm-request-router): require explicit remote Watch URIs #1299: explicit remote Watch and Pylon dial URIs

#1299 is intentionally stacked on rebased, patch-equivalent dependency commits
and must be rebased before merge.

Sanitized validation completed for #1299:

• Helm render checks and lint passed
• self-managed stack aggregate render/wiring tests passed
• 1,073 Rust tests passed; 3 benchmarks remained intentionally ignored
• strict Rust formatting and Clippy passed for all changed packages
• exact-commit secret scan reported zero verified or unverified findings
• independent review found no remaining Critical, Important, or Minor findings

The final multi-region verified-TLS, failure-mode, authenticated reverse-QUIC,
and invocation evidence remains in the dependent #1294/#689 integration phase.
This issue should remain open until that latest-main live matrix passes.

[mikeyrcamp]

Dependent secure-ingress review
#1300 now consumes #1299's explicit
URI contract.

It requires the API worker bootstrap and every backend-router Pylon gRPC dial
address to use the same explicit https:// endpoint for secure remote-worker
routing. Scheme-less secure routing, secure http://, mismatched endpoints,
and implicit plaintext are rejected; development HTTP remains an explicit
opt-in only.

The non-live integration suites and exact-commit scan are green. The final
multi-cluster TLS/PKI matrix remains pending, so #1296 should remain open until
the dependency chain merges and the latest-main validation passes.

[mikeyrcamp]

Validation update for #1299 (aafd5ba9) and its downstream secure-ingress integration #1300: explicit https:// dial URIs are accepted, development-only http:// is gated, and scheme-less or unsupported endpoints are rejected instead of silently becoming plaintext. Tests also preserve the dial hostname for TLS SNI while keeping the advertised Stargate authority as host:port.

The secure split-cluster Deployment and StatefulSet runs both completed HTTPS Watch/Register, distinct SNI/authority, worker authentication, reverse QUIC, and LLM invocation; hostname mismatch, wrong issuer, and plaintext HTTP were rejected. All currently reported #1299 CI checks pass, but review is still required and #1300 remains stacked on #1299. Validation in an actual multi-region environment and a post-merge latest-main run remain, so this issue stays open.

[mikeyrcamp]

Implemented by PR #1299, merged into main (merge commit 6beaa6b), with the secure-ingress integration subsequently merged in PR #1300. The merged validation rejects scheme-less/unsupported secure endpoints, gates explicit development HTTP, preserves TLS SNI versus per-pod routing authority, and covers secure Watch/Register integration. Closing as completed.