Skip to content

design(pylon): define a scalable multi-region WatchStargates topology #1328

Description

@mikeyrcamp

Problem

WatchStargatesResponse can advertise additional watch_stargate_urls, and Pylon recursively watches newly discovered endpoints, deduplicates them, and publishes the union of the returned registration routers.

A reciprocal peer configuration is straightforward for two regions, but the supported and recommended topology for three or more regions is not clearly defined. Operators need to know whether to configure a full mesh, a directed ring, redundant neighbors, or another peer layout, and what failure and recovery behavior each layout provides.

Without an explicit contract, a topology can be reachable under normal conditions but become partitioned when a peer or link is unavailable. Membership changes can also create temporarily disconnected subgraphs.

Desired outcome

Define and validate a scalable multi-region discovery model in which a Pylon bootstrapping from any supported regional endpoint converges on every reachable registration router without creating duplicate watch activity or unbounded traversal.

Acceptance criteria

  • Document the graph and reachability requirements for watch_stargate_urls, including whether cycles are supported.
  • Define a recommended production topology for more than two regions and describe its configuration and failure-tolerance tradeoffs.
  • Clarify how the repeatable remote Watch URL configuration represents peer routes and how regions are safely added or removed.
  • Add tests covering at least three regional Watch endpoints with transitive discovery.
  • Add tests covering cycles, duplicate URLs, an unreachable peer, recovery after reconnection, and membership removal.
  • Verify that a failed redundant seed does not suppress registration routers discovered through healthy endpoints.
  • Document observable signals that allow operators to determine which Watch endpoints and registration routers are currently active.

Notes

A full mesh should not be assumed to be required. A topology with a bounded number of peer routes per region may scale linearly, but it must have explicit reachability and redundancy guarantees.

Follow-up to #1292.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions