feat: peer clustering over the AoI grid with a NATS feed

[mikhail-dcl]

Pulse becomes the sole author of peer clusters, derived from the same grids and SnapshotBoard that drive area-of-interest, and feeds NATS with what archipelago-core publishes today. No client protocol change, no worker involvement, nothing added to the per-tick or per-packet path.

Design and rationale: docs/clustering-on-aoi.md. Plan of record: Notion "Archipelago ⇒ Pulse migration plan", iteration 1.

What ships

• ClusterTracker — BackgroundService on its own thread, one pass per second: weighted union-find with path halving over occupied grid cells, 8-neighbour adjacency, run one realm at a time. Sticky IDs plus a dwell debounce for stability. Buffers are fields cleared between passes; no LINQ.
• ClusterBoard — the immutable ClusterPass swapped in with one Volatile.Write; readers lock-free.
• NatsPublisher — sole connection owner, publish-only, fail-soft, coalescing outbox.
• Config — Clusters and Nats. Clusters:Enabled ships true, Nats:Url ships empty, so the default is shadow mode: the tracker runs and reports metrics but publishes nothing until a URL is injected. Clearing the URL is the rollback.

"Island" now means archipelago's concept and the wire contracts carrying it only — engine.islands, IslandStatusMessage, IslandChangedMessage, the /islands stats paths. Clusters are uncapped, realm-partitioned, and carry no transport details.

Realm isolation

Realm used to be a read-time filter: a per-candidate string.Equals in both AoI implementations, plus per-pass realm interning and a per-cell member partition in the tracker. It is now structural — RealmSpatialGrids holds one SpatialGrid per realm and routes each peer into exactly one, so an observer resolves its own realm's grid and every candidate it finds is already same-realm. Seven realm comparisons leave the read paths and NodeKey drops its realm.

• Per-peer bookkeeping stays global (one array pair, not one per grid): realm names are client-supplied and only length-validated, so a per-grid array indexed by PeerIndex would let one peer mint unbounded full-size arrays by teleporting to fresh names. A grid is dropped with its last occupant, bounding live grids by connected peers.
• One write lock shared by all realms — the same contention profile as the previous global grid. Set adds to the new cell before vacating the old, so the peer is never momentarily in neither, and a solo peer changing cells cannot evict the grid it is moving within.
• A cross-realm teleport vacates the old grid before publishing the snapshot. Publishing first leaves a peer whose snapshot names the new realm sitting in the old realm's grid, which an observer there reads as a cross-realm subject and holds until the stale-view sweep.
• SnapshotBoard.Publish now returns the ledger-resolved snapshot (public API change), so the publisher picks a grid without a second seqlock read. SpatialAreaOfInterest and its options are deleted — never DI-registered, no tests, config section bound to nothing.

NATS output

Subject	Payload	Cadence
peer.{addr}.cluster_change	decentraland.pulse.PeerClusterChange { cluster_id, realm }	per published assignment change
engine.islands	kernel.comms.v3.IslandStatusMessage	per pass
engine.discovery	kernel.comms.v3.ServiceDiscoveryMessage	timer, default 10 s

Gatekeeper subscribes to cluster_change, mints the LiveKit conn-string and re-emits the existing island_changed, so WS Connector and clients are untouched. PeerClusterChange is deliberately not IslandChangedMessage: that carries conn_str (a signed LiveKit JWT) and its producer runs a ban check, both of which stay with gatekeeper.

The broker URL is read from either Nats__Url or the flat NATS_URL archipelago's services use, so one injected secret serves both; Nats__Url wins.

The outbox is not one queue. engine.islands is a whole-world snapshot, so a newer one fully replaces an undelivered one — one latest-wins slot. cluster_change supersedes only per peer, so those are held one entry per peer. A shared FIFO with oldest-first eviction, the original design, could discard peer A's assignment to admit peer B's, leaving A on a stale cluster until a reassignment that may never come if A stops moving. Loss now needs more than Nats:ChannelCapacity distinct peers pending at once, and is counted separately (dropped) from benign superseding (superseded).

Topology is emitted before the events referencing it, but best-effort only: gatekeeper and stats are separate subscribers, so consumers must tolerate an unknown cluster id. Per peer, its own events are ordered.

Reconnection. Client defaults suit a fail-soft feed (unlimited retries, 2–5 s jittered backoff), plus two additions: IgnoreAuthErrorAbort = true, since by default the client stops reconnecting permanently after the same auth error twice — turning a rotated credential into a feed only a restart recovers; and a supervision loop rebuilding after 5 s, since broker loss is handled inside the client and the pipeline exiting therefore means it faulted.

Benchmarks

src/DCLPulseBenchmarks, -c Release. BenchmarkDotNet 0.15.8, .NET 10.0.10, Ryzen 9 9955HX, X64 RyuJIT x86-64-v4.

Clustering pass — ClusterTrackerBenchmarks. Cold is Pass + churn − Churn only, a difference of means, so its error is both rows' summed. Quote cold: Pass repeats over an unchanging grid and stays cached, which a second of production traffic does not.

Scenario	Peers	Warm	Cold	Alloc/pass	Clusters
Sporadic	100	7.10 µs	7.55 µs	7.6 KB	sparse + singletons
Chained	1 000	27.78 µs	30.82 µs	56 KB	1 (transitive chain)
DenseAndSparse	1 000	31.41 µs	34.55 µs	57 KB	9
CeilingUniform	4 095	320.9 µs	394.8 µs	230 KB	2

At the 4 095-peer ceiling that is ~0.04 % of one core at 1 Hz. Allocation is dominated by the immutable ClusterPass, which readers hold by reference and so cannot be pooled.

CeilingUniform documents a limit, not a win: cell-adjacency clustering is site percolation, and 4 095 peers over Genesis City at 100 u cells occupy ≈ 0.83 of cells against a Moore threshold of ≈ 0.407, so the partition collapses to 2 clusters with 4 091 peers in one. Downstream room sharding becomes load-bearing rather than an overflow path — see §3.2.

Feed encoding — NatsEncodeBenchmarks. Justifies publishing straight into the client's writer rather than ToByteArray(), which allocated, walked the message twice and memcpy'd:

Payload	ToByteArray + copy	Straight into the writer
cluster_change (13 B)	51.00 ns / 104 B	29.83 ns / 0 B
topology (44 KB)	28.18 µs / 44.3 KB	19.51 µs / 0 B

An intermediate pooled-bytes shape removed the allocation but not the double walk (36.12 ns / 28.10 µs) — only a benchmark separates it from zero-copy.

Metrics

Nine series on the existing /metrics, types as emitted by PrometheusFormatter.

Metric	Type	Notes
dcl_pulse_clusters	gauge	Zero with peers connected ⇒ no peer has a realm, or clustering is off
dcl_pulse_cluster_passes_total	counter	~1/s expected; below that the tracker is stalling
dcl_pulse_cluster_pass_duration_us_total	counter	Sum half of a sum/count pair — divide by passes for the mean
dcl_pulse_cluster_reassignments_total	counter	Post-debounce. High with a stable cluster count ⇒ flapping; raise DwellPasses
dcl_pulse_nats_published_total	counter	Delivered to the broker
dcl_pulse_nats_dropped_total	counter	Genuinely lost — the actionable one; a peer may be on a stale cluster
dcl_pulse_nats_superseded_total	counter	Replaced pre-delivery. Expected under load; freshness degrades, nothing lost
dcl_pulse_nats_reconnects_total	counter	Steady growth ⇒ flapping broker or network path
dcl_pulse_nats_connected	gauge	0 is correct in shadow mode, the shipped default

Grafana. Scrape /metrics on port 5000; supply authorization.credentials when MetricsBearerToken is set. Add a "Clusters" row to pulse-server-dashboard.json: dcl_pulse_clusters; mean pass duration as rate(…pass_duration_us_total[5m]) / rate(…passes_total[5m]); rate() of passes, reassignments and published; dropped and superseded on one panel, since the pair is only meaningful read together; dcl_pulse_nats_connected as a stat with 0 → Down. Use rate() on every _total, not irate(), at a 1 Hz production rate.

Alerts worth having: rate(dropped[5m]) > 0 for 5m; passes < 0.5/s for 5m; mean pass duration > 500 000 µs for 10m. A connected == 0 alert must be scoped to deployments where Nats:Url is set — Prometheus cannot see whether a URL is configured, so ungated it fires everywhere by design.

Decommissioned with archipelago-core (gone after cutover)

Metric	Replacement
dcl_archipelago_islands_count	dcl_pulse_clusters (gauge, clusters derived by the last pass)
dcl_archipelago_peers_count	dcl_pulse_cluster_peers (gauge; counts clustered peers — excludes peers with no realm yet or no wallet in IdentityBoard, so it can read lower than connected peers)

Dashboards using dcl_archipelago_islands_count and/or dcl_archipelago_peers_count:

1. Comms — Folder: Core Team: Catalysts
   → Open
   • Peers Over Time (timeseries) — sum(dcl_archipelago_peers_count{instance="$instance"}) by (instance)
   • Islands Over Time (timeseries) — sum(dcl_archipelago_islands_count{instance="$instance"}) by (instance)

2. Main realm comms — Folder: Core Team: Comms Service
   → Open
   • sum(dcl_archipelago_peers_count{catalyst=""})
   • sum(dcl_archipelago_islands_count{catalyst=""})

3. Online users — Folder: Core Team: Services
   → Open
   • Online users (timeseries) — sum(dcl_archipelago_peers_count{catalyst=~"peer-.*"}) by (catalyst)
   • Online users (delta) (timeseries) — sum(delta(dcl_archipelago_peers_count[$__interval:])) by (catalyst)
   • Online users (timeseries) — sum(dcl_archipelago_peers_count)

4. Performance — Folder: World team
   → Open
   • Connected users all clients (timeseries) — sum(max_over_time(dcl_archipelago_peers_count[$__interval:]))
   • Connected users E@ (timeseries) — sum(max_over_time(dcl_archipelago_peers_count{service="archipelago-ea-core"}[$__interval:]))

Verification

• dotnet build clean; 527 tests pass, 0 failures.
• All three Docker images build — required by the NATS.Client.Core package addition.
• Live broker: engine.islands at 1/s, engine.discovery on its timer, connected 1, dropped 0.
• Live reconnection: stopping and restarting the broker gave connected 1 → 0 → 1, reconnects 1, publishing resumed, dropped 0 — the outbox retained each peer's latest assignment across the outage.
• Live shadow mode: with no URL the tracker runs, the publisher exits with NATS feed disabled, connected 0, no errors.

peer.{addr}.cluster_change is unit-tested but not live-verified — it needs an authenticated peer.

Three defects came from tests rather than review: sticky-ID inheritance read the published assignment instead of the previous pass's computed one, starving the debounce so a mid-debounce fragment was minted a fresh ID every pass and could never be reassigned; the shared-outbox eviction above; and the teleport ordering under Realm isolation, which the deleted per-candidate realm filter had been masking.

Dependency

Requires protocol#453. ServiceStatus/ServiceDiscoveryMessage were never merged to @dcl/protocol main — archipelago reaches them only via a pinned CDN branch build whose source commit no longer exists. Proto regeneration here fails without it.

pulse_clusters.proto (carrying PeerClusterChange) still needs its own protocol PR; gatekeeper also needs the generated TS types to subscribe.

Deployment order

1. Deploy definitions
2. Deploy archipelago-workers (new ws-connector + stats image). Inert with respect to the cutover — WS Connector behavior is unchanged and stats still consumes heartbeats — but it lands the protocol re-pin, the subject-address normalization fix, and the contract tests before anything starts publishing the new feeds.
3. Deploy comms-gatekeeper (with the definitions PR merged, CLUSTER_SUBSCRIBER_ENABLED=true + NATS_URL arrive at deploy). It connects and idles — nobody publishes cluster_change yet. This must precede Pulse because NATS is at-most-once: events published with no subscriber simply vanish, and clients would silently get no conn-strings. Verify: NATS connected, zero received, no errors.
4. Stop archipelago-core (infra scales the archipelago-ea-core task to zero — the repo's deploy jobs are gone, but the runbook notes the running task keeps its last image until infra retires it). This must happen before Pulse's feed goes live, or you get dueling publishers: core and gatekeeper both emitting engine.peer.{addr}.island_changed and two conflicting engine.islands topologies — clients flapping between I{n} and island-C{n} rooms. The cost of this ordering is a short gap where new sessions get no island assignment (existing sessions keep their rooms — tokens are already delivered), so plan the window; don't do it mid-event.
5. Deploy Pulse (Clusters__Enabled=true + NATS_URL). This is the cutover moment — the feed starts publishing and the chain closes: cluster_change → gatekeeper mints → island_changed → WS Connector → client. Verify immediately: dcl_pulse_nats_connected 1, published_total climbing, publish_failed_total and dropped_total at 0.
6. Run the runbook verification end to end: stats /islands serving C{n} with a non-empty peer total (the join check, not just island count), /core-status healthy, heartbeat-fed endpoints unchanged, and a real client joining an island-C{n} LiveKit room.

Known gaps

• Cluster IDs are not unique beyond one process. C{n} counts from zero, so it resets on restart and collides across instances: after a restart C1 names a different crowd and gatekeeper maps it onto the previous C1's LiveKit room. Scope to server_id or a boot epoch. Live-voice-room correctness, not cosmetics.
• The §3.6 periodic re-publish sweep is unimplemented; delivery is at-most-once.
• Teleport bypass reads IsTeleport on the latest snapshot only, so a teleport followed by movement inside one pass goes through the debounce. Cross-realm teleports are always immediate.
• A subject moving mid-scan can be missed for one tick: AoI walks up to 25 cells lock-free, so the multi-cell read is not atomic. Pre-existing, unrelated to the write ordering above.
• engine.islands reports max_peers = 0 since clusters are uncapped, so GET /islands reads 0 where it read 100.
• Stats HTTP endpoints (§3.5) and engine.parcel_changes are iteration 2.

---

QA instructions — verifying this change from the client

This change is what ultimately assigns a player their island, so it is verified in unity-explorer through the Island room. Run it against zone, where the server side of this feature is deployed, on a client build that carries the room indicator described in Part 3 — see Step 1. The steps are self-contained; they also live in unity-explorer/docs/qa-archipelago-island-room.md.

Three things to hold on to before starting. The client is told nothing unprompted after its handshake, so an unassigned island is indistinguishable from a client hang — Part 5 separates them. The Island room deliberately overlaps with gatekeeper's Scene room, so an avatar carried by both is correct, not a duplicate. And which tag is healthy depends on the client's transport: with Pulse active no client announces its profile over LiveKit at all, so the LiveKit rooms report presence without an announcement. Part 3 is the legend for all three.

Archipelago governs exactly one thing in the client: the Island room, the global comms room that carries remote avatars. Gatekeeper's Scene room is a separate path and appears here only where the two interact — which they do constantly, so it cannot be ignored.

Every step states what a pass looks like (✅) and what to treat as a failure (❌). Report any ❌ with the values from Part 2 and the realm details from Part 5.

---

Part 1 — Build, launch, and set up the debug menu

Step 1. Use a unity-explorer build that has the room indicator

The per-room state glyphs every test below is read from come from unity-explorer#9980. Use its branch (feat/room-indicator-livekit-presence), or dev once that has merged.

• ✅ Room: Info (Part 1, Step 7) has an Avatars on LiveKit row. It only exists with the change.
• ❌ It does not. On a build without it, a Pulse-carried avatar reports Pulse and nothing else, and Tests 2, 3 and 4 cannot be executed at all.

Step 2. Launch against zone, with debug enabled

decentraland.exe --dclenv zone --debug

• --dclenv zone points every backend at decentraland.zone. The server side of this feature is deployed to zone only — on org there is nothing to test.

• --debug builds the room widgets. Without it they do not exist.

• ✅ The client reaches world and the debug panel is visible.

• ❌ No panel (see --debug above), or you land on an org realm — zone is a separate account space on Sepolia, so log in with a zone/test wallet, not a mainnet one.

Step 3. Confirm both flags actually took

Type /app-args in chat.

• ✅ The listing shows dclenv set to zone and debug present.
• ❌ Either is missing. The client is not testing what you think it is — fix the launch command and restart. Do not proceed.

Step 4. Learn the panel toggle

Type /debug in chat.

• ✅ The panel hides; typing /debug again shows it.
• ❌ Unknown command — the debug flag did not apply. Return to Step 2.

Step 5. Confirm the room widgets exist

Type /debug help in chat.

• ✅ The listing includes Room: Island, Room: Scene and Room: Info.
• ❌ Any of the three is missing. Stop — the rest of this document cannot be executed. Report the build and branch.

Step 6. Open the Island widget

Expand Room: Island.

• ✅ It shows the rows listed in Part 2, plus a button reading Deactivate.
• ❌ The widget is present but empty, or the button reads Activate at startup — the room should start activated.

Step 7. Turn on the room indicators

Expand Room: Info and enable Show Room Indicator.

• ✅ Each remote avatar's nametag gains a debug line naming the rooms that account for it, each prefixed by a state glyph — normally 🔗Gatekeeper 🔗Island ⚡Pulse. Part 3 is the legend. Your own avatar is not tagged.
• ❌ No debug line appears while other avatars are visible, or a tag reads None for an avatar that is on screen.

Leave this toggle on for the whole session. Every test below is read from it.

---

Part 2 — Reading "Room: Island"

Row	Healthy value	What it means
Room State	Running	The room's lifecycle
Connecting State	ConnConnected	The LiveKit session — the field that says comms actually works
Attempt to Connect	not stuck retrying	A connection attempt in progress
Connection Loop	healthy	The loop that sends position heartbeats
Connection Quality	Excellent / Good	LiveKit's own quality signal
Remote Participants	tracks nearby players	Peers in your island
Room Sid	RM_…	The island you are in. Changes on reassignment
Self Sid	PA_…	Your participant id

Not connected in the Sid rows only means Room State is not Running yet — read Room State first.

---

Part 3 — Reading the room indicator

Needed to judge Tests 2–4.

The tag above a remote avatar names every room that accounts for it, each prefixed by a glyph. The glyph separates two facts that are not the same thing:

Glyph	Means
🟢	LiveKit's participant roster lists the wallet in that room and that room's data channel delivered its profile announcement
🔗	LiveKit lists the wallet in that room, but it never announced over it
👻	an announcement arrived from a wallet the roster no longer lists. Only reachable for a peer that is not on Pulse, and only for the instant between it leaving the room and the client dropping it
⚡	the profile arrived over Pulse. Pulse publishes no roster, so it has this one state only

So 🔗Gatekeeper 🔗Island ⚡Pulse reads: in both LiveKit rooms, announcing over neither, and carried by Pulse. That is what almost every avatar reads in these steps. A peer outside your scene reads 🔗Island ⚡Pulse.

⚡ never shares a tag with 🟢 or 👻. Announcing over LiveKit and being carried by Pulse are mutually exclusive per peer: a peer on Pulse addresses its LiveKit announcement only to peers that announced to it first, and your client, being on Pulse itself, never does. So an avatar either reads 🔗 on its rooms with ⚡Pulse, or reads 🟢 on its rooms without ⚡Pulse because that peer is not on Pulse. A tag carrying both is a bug — report it.

What normal looks like. These steps assume Pulse is on, which is the default whenever the pulse feature flag is enabled. With Pulse carrying profiles no client announces over LiveKit, so the expected reading is 🔗 on each LiveKit room plus ⚡Pulse. An avatar reading 🟢 and no ⚡Pulse is not a failure either — that peer is not on Pulse, which is what a client whose Pulse connection fell back at start-up looks like. Report a 👻 that sticks, and any tag mixing ⚡ with 🟢 or 👻.

The glyph is what makes this document executable under Pulse. Before it, a Pulse-carried avatar reported Pulse and nothing else, so the tag could not say whether the avatar was on LiveKit at all — which is what this PR is being tested through.

Who owns the avatar. The rooms overlap on purpose. A player in your scene is normally seen by both. The client keeps one avatar per wallet and records the sources as a set of flags:

• The first source to see a wallet creates the avatar.
• A second source seeing the same wallet only adds its flag — no duplicate avatar.
• A source dropping the wallet removes its flag. The avatar is destroyed only when the last flag goes.

Under Pulse, Pulse is normally that last flag: it creates the avatar and it is what keeps it alive. Archipelago is still the safety net at scene borders in the sense that matters here — the avatar must not blink out when the Scene room drops the player.

Counting. Room: Info reconciles the avatar roster against LiveKit's:

Row	Means
Active Avatars	avatars the client is showing
Avatars on LiveKit	of those, how many LiveKit also lists in the Island or Scene room
Avatars off LiveKit	the remainder — an avatar with no LiveKit session behind it
LiveKit w/o Avatar	participants in the Island or Scene room with no avatar

The last two are not failures on their own: the Island room covers a wider area than the transport that creates avatars, so it legitimately lists people you cannot see, and the reverse happens briefly during hand-offs. The load-bearing reading is Avatars on LiveKit — it must be non-zero whenever both rooms are Running and other players are nearby. Zero there while Remote Participants is non-zero means none of the avatars on screen has a LiveKit session behind it.

---

Part 4 — Tests

Test 1 — Connects on login

1. Log in and stand still.
2. Read Room: Island.

• ✅ Within a few seconds: Room State: Running, Connecting State: ConnConnected, Room Sid populated.
• ❌ Connecting State never reaches ConnConnected; Room Sid stays empty beyond ~30 s; or Attempt to Connect cycles forever. Do Part 5 before filing — an unassigned island looks exactly like this and is a backend state, not a client bug.

Test 2 — Avatars arrive tagged, both rooms on

1. Go where other players are (Genesis Plaza).
2. Read the nametag tags and both widgets.

• ✅ Players in your scene carry both a Gatekeeper and an Island entry; players outside your scene carry Island only. Nearly all read 🔗 on those entries plus ⚡Pulse; a peer that is not on Pulse reads 🟢 with no ⚡Pulse instead. Remote Participants is non-zero and Avatars on LiveKit is non-zero.
• ❌ Two avatars for one player; an on-screen avatar tagged None; Avatars on LiveKit is 0 while Remote Participants is not; nobody carries an Island entry while players are clearly outside your scene; or any tag mixes ⚡ with 🟢 or 👻.

Test 3 — Scene-border handoff

1. With another player, walk together across a scene border.
2. Watch their nametag tag and their avatar continuously.

• ✅ The Gatekeeper entry appears and disappears as the Scene room picks them up or drops them, while the Island entry stays, and the avatar never disappears, reloads or flickers.
• ❌ The avatar vanishes and respawns at the border; the nametag blanks; wearables reload; the tag drops to None while the player is still visible; or a 👻Gatekeeper appears at all — a peer on Pulse announces over no LiveKit room, so a hand-off cannot produce one.

Test 4 — Prove each room's contribution with Deactivate

Do this while a player in your scene is visible and carries both a Gatekeeper and an Island entry.

Pulse owns the avatars, so this test reads the tag, not the screen. Deactivating a LiveKit room removes that room's entry from every tag, but the avatars stay — Pulse created them and Pulse keeps them alive. An avatar vanishing here is a failure. The one exception is an avatar reading 🟢 with no ⚡Pulse: that peer is not on Pulse, so LiveKit is the only thing holding it and it is expected to go when its rooms do.

1. In Room: Scene, press Deactivate. Confirm Room State: Stopped, Connection Loop: Stopped, Attempt to Connect: None.
   • ✅ Their Gatekeeper entry disappears and the avatar stays.
   • ❌ The Gatekeeper entry survives a stopped room, or the avatar disappears.
2. Press Activate on Room: Scene and wait for the Gatekeeper entry to come back.
3. Press Deactivate on Room: Island.
   • ✅ Their Island entry disappears and the avatar stays. Players outside your scene stay too, now with no Island entry.
   • ❌ The Island entry survives a stopped room, or the avatar disappears.
4. Deactivate both.
   • ✅ Every tag reads ⚡Pulse alone, Avatars on LiveKit drops to 0, and every Pulse-carried avatar is still on screen.
   • ❌ Any LiveKit entry remains on any tag; Avatars on LiveKit stays above 0; or an avatar that was carrying ⚡Pulse disappears.
5. Re-activate both before continuing.

While a room is deactivated, do not file bugs about what it carries — with Room: Scene off, in-scene voice and scene streams are expected to be dead.

Test 5 — Island reassignment while moving

1. Deactivate Room: Scene and leave it off, so avatar churn is Archipelago's alone. Deactivation is sticky: it survives scene changes, teleports and realm changes.
2. Walk a long distance in a straight line, watching Room Sid.

• ✅ Room Sid changes at least once. After each change Connecting State returns to ConnConnected and the set of avatars carrying an Island entry turns over — old ones lose it, new ones gain it. A brief flicker during the switch is normal. Avatars from the old island may stay on screen without an Island entry; that is Pulse's own area of interest, not a stale island.
• ❌ Room Sid never changes over a long traverse; Connecting State does not return to ConnConnected after a change; or avatars keep an Island entry long after the sid changed.

Test 6 — Realm change and teleport

1. Keep Room: Scene deactivated.
2. Teleport to another realm or a world, then back.

• ✅ The room restarts: Room State leaves Running and returns to it with a new Room Sid. No avatar from the previous realm still carries an Island entry.
• ❌ Room State never returns to Running; the old Room Sid persists into the new realm; or an avatar from the previous realm keeps its Island entry.

Test 7 — Reconnect after network loss

1. Disable the network for ~20 s.
2. Re-enable it and watch Room: Island without touching anything.

• ✅ Connecting State leaves ConnConnected, then recovers on its own. Recovery can take ~15–20 s: the client retries on a 5 s backoff and forces a fresh handshake after 3 attempts.
• ❌ It never recovers without a client restart; it recovers but avatars never come back; or Connection Loop stays stopped.

Test 8 — Same wallet elsewhere

1. Log in with the same wallet on a second machine.
2. Watch the first client.

• ✅ The first session is kicked and Room State leaves Running.
• ❌ The first client hangs silently in Running with no participants; or both sessions stay connected.

---

Part 5 — Triage before filing

An unassigned island is indistinguishable from a client hang. The server sends nothing unprompted after the handshake, so with no island assigned the client sits on a healthy connection receiving nothing. Rule that out first.

Step 1. Check the realm the explorer actually uses (zone)

curl -s https://realm-provider-ea.decentraland.zone/main/about

Read healthy, acceptingUsers and the comms block.

• If healthy: false or acceptingUsers: false → the realm's comms are down. Not a client bug.
• ⚠️ This is a different host from peer.decentraland.zone, which serves a different realm. The explorer resolves Genesis from realm-provider-ea; checking the wrong host gives a healthy answer about a realm nobody is on.

Step 2. Decide what you have

• Backend or environment: no island was ever assigned — Room Sid stayed empty all session.
• Client bug worth filing: an island was assigned (Room Sid populated at least once) and the client still failed to reach ConnConnected, or failed to recover in Tests 3–7.

Step 3. Include in the report

• All rows from Room: Island (Part 2), plus the same from Room: Scene if the test involved both.
• The realm name and comms block from Step 1.
• Whether the room indicator was on, and the exact tag text on the affected avatar, glyphs included.
• The four Room: Info counters.
• Build number, and whether --debug or a Debug build was used.

[decentraland-bot]

Code Review — PR #34: Peer Clustering over AoI Grid with NATS Feed

Thoroughly reviewed this large PR (~6,500 new lines across 88 files). The design is well-documented, the architecture is sound, and the test coverage is strong. One correctness bug needs fixing before merge.

Findings Summary

• P1 Critical: 1 — blocks merge
• P2 Minor: 3 — suggestions

---

P1 — Blocks Merge

1. PruneVanishedClusters modifies Dictionary during foreach enumeration

ClusterTracker.PruneVanishedClusters() iterates clusterRecords (a Dictionary<string, ClusterRecord>) and calls clusterRecords.Remove(clusterId) inside the loop. In .NET, this throws InvalidOperationException on the first removal.

The try/catch in RunPassLoop catches the exception and logs it, so the BackgroundService recovers on the next pass — but:

• Pruning never succeeds: every attempt to remove a vanished cluster throws before reaching the second entry.
• clusterRecords grows unbounded with stale cluster IDs, bounded only by the rate of cluster creation.
• An error is logged on every pass that has vanished clusters, creating log noise.

Fix: collect keys to prune into a temporary list, then remove outside the enumeration:

private void PruneVanishedClusters()
{
    if (clusterRecords.Count == components.Count) return;

    List<string>? toRemove = null;
    foreach ((string clusterId, ClusterRecord record) in clusterRecords)
    {
        if (record.LastLivePass != passNumber)
            (toRemove ??= []).Add(clusterId);
    }

    if (toRemove is null) return;
    foreach (string id in toRemove)
        clusterRecords.Remove(id);
}

---

P2 — Suggestions

2. No length validation on client-supplied realm names (RealmSpatialGrids.cs)

Realm names arrive from clients and are used as ConcurrentDictionary keys. The grid lifecycle is correctly bounded (one grid per peer, dropped when last occupant leaves), so the count is safe. However, there is no length cap on the realm string itself — a malicious client could send multi-megabyte realm names that persist as dictionary keys until the peer leaves. Consider capping realm name length at the intake boundary (e.g., TeleportHandler).

3. dropped counter may miscount on defensive eviction path (NatsPublisher.QueueChange)

In the eviction branch, dropped = true is set unconditionally after the changeOrder.TryDequeue, even when pendingChangeBySubject.Remove(evicted) returns false (i.e., the evicted subject was stale). Under normal operation, the lock invariant ensures that every subject in changeOrder is in pendingChangeBySubject, so the Remove should always succeed. The TryDequeueNext drain loop has the same defensive continue pattern, suggesting the author considered the possibility. In practice this appears correct, but the unconditional dropped = true means if the invariant ever breaks, the metric would overcount without an actual eviction.

4. Consider a contention metric for the shared RealmSpatialGrids write lock

All realms share a single Lock for writes. At sub-microsecond hold times and MaxPeers=4095, this is likely fine, but adding a contention metric would provide visibility under production load.

---

Security Review

• ✅ SanitizeBrokerUrl properly strips userinfo (credentials) from NATS URLs before logging, using Uri.TryCreate to extract only host and port.
• ✅ ConnStringRedaction redacts access_token= values from LiveKit connection strings via regex.
• ✅ No secrets or credentials in docker-compose.debug.yml or docker-compose.e2e.yml. E2E_NATS_URL is read from host env with a safe local default.
• ✅ AddNatsUrlAlias precedence is correct — the alias cannot override an explicit Nats__Url.
• ✅ SnapshotBoard.Publish return-type change (void → PeerSnapshot) is additive and backward-compatible for callers.

Architecture Notes

• The weighted union-find with path halving is a good fit — O(N+C) per pass, measured at ~395μs at the 4095-peer ceiling.
• Structural realm isolation via per-realm SpatialGrid instances eliminates per-candidate realm comparisons in the hot path.
• The coalescing outbox with per-feed supersede semantics (latest-wins for topology, per-peer for cluster_change) is well-designed.
• Fail-soft NATS integration: empty URL → stats-only mode; broker outage → outbox retention; no blocking on the hot path.
• The known gaps (cluster ID uniqueness, missing re-publish sweep, percolation at high density) are well-documented and appropriate to defer.

CI Status

All checks passing ✅ (Build linux-x64, Build osx-arm64, Build win-x64, Run tests).

---

Reviewed by Jarvis 🤖 · Requested by mikhail-dcl via GitHub

[decentraland-bot]

[P1] Bug: Dictionary modification during foreach enumeration.

This will throw InvalidOperationException in .NET whenever a cluster vanishes. The try/catch in RunPassLoop catches it, but pruning never completes — clusterRecords grows unbounded with stale entries.

Suggested fix: collect keys first, then remove:

private void PruneVanishedClusters()
{
    if (clusterRecords.Count == components.Count) return;

    List<string>? toRemove = null;
    foreach ((string clusterId, ClusterRecord record) in clusterRecords)
    {
        if (record.LastLivePass != passNumber)
            (toRemove ??= []).Add(clusterId);
    }

    if (toRemove is null) return;
    foreach (string id in toRemove)
        clusterRecords.Remove(id);
}

[mikhail-dcl]

Not a bug on this runtime — Dictionary<TKey,TValue>.Remove has not invalidated an active enumerator since .NET Core 3.0, and this repo targets .NET 10. The loop completes, clusterRecords stays bounded, and nothing reaches the try/catch in RunPassLoop.

Verified on the actual runtime rather than from memory:

.NET 10.0.11
Remove during foreach: OK - removed 5, 5 left
Add during foreach: THREW (expected - proves the guard is live)

The second line is the control. An Add during enumeration does still throw, so the version check is live — the Remove result isn't a case of the guard being absent or optimised away.

Since this is easy to re-flag, 999a649 records the guarantee at the loop itself so the next reader doesn't reach for a second pass:

// Removing during enumeration is deliberate and supported: since .NET Core 3.0
// Dictionary.Remove does not invalidate an active enumerator, so this needs no second pass
// and no key list. Adding still would invalidate it — only removal is exempt.

Happy to take the two-pass rewrite anyway if the team would rather not depend on that guarantee, but it would be defensive rather than a fix — and it would add a List<string> allocation to a method that currently has none.

The other three findings are answered in #34 (comment): #3 fixed in c5e55f2, #2 refuted (realm length is already capped at 255 by FieldValidator on all three intake paths), #4 declined with measurements.

[mikhail-dcl]

Thanks — thorough review. Branch is now merged up to main (6e499f9) and rebuilt on it. Taking the findings in turn: #3 fixed, #1 and #2 refuted with evidence, #4 declined with reasoning.

P1 — PruneVanishedClusters — not a bug on this runtime

Dictionary<TKey,TValue>.Remove during foreach has been explicitly supported since .NET Core 3.0; it does not invalidate the enumerator. This repo targets .NET 10, so the loop is correct as written.

Verified on the actual runtime rather than from memory:

.NET 10.0.11
Remove during foreach: OK - removed 5, 5 left
Add during foreach: THREW (expected - proves the guard is live)

The second line is the control: an Add does still throw, so the enumerator's version check is live and the Remove result isn't a case of the guard being absent. Pruning completes, clusterRecords stays bounded, and no exception reaches RunPassLoop.

Happy to take the suggested rewrite anyway if the team prefers not to rely on that guarantee, but it would be defensive rather than a fix.

P2 #2 — realm names are already length-capped

Every path that can set a realm goes through FieldValidator, which rejects an over-long one at intake before it can become a dictionary key:

• ValidateTeleport — FieldValidator.cs:102
• ValidateHandshakeInitialState — FieldValidator.cs:91
• ValidateSceneListenerHandshake — FieldValidator.cs:129

All three test Realm.Length > maxRealmLength, with MaxRealmLength = 255 (FieldValidatorOptions, set in appsettings.json). A multi-megabyte realm name is refused before RealmSpatialGrids ever sees it.

P2 #3 — fixed in c5e55f2

Correct, and worth tightening even though the invariant holds today. dropped = true now sits inside the successful Remove, so the counter can only report an assignment that was really lost. The drain loop already tolerates an order entry with no message; the counter now agrees with it. This matters because dcl_pulse_nats_dropped_total is documented as the actionable signal — an overcount sends an operator after capacity for a non-event.

P2 #4 — lock contention metric: declining, with numbers

Measured on this machine (Ryzen 9 9955HX, .NET 10, Release):

Uncontended Lock enter+exit	18.2 ns
One peer move (grid write)	262–617 ns by cell occupancy
of which copy-on-write HashSet clone	~90%, and all the allocation (232 B → 14 KB at occupancy 256)

The lock is ~3–7% of a write even uncontended, and the dominant cost is the occupant-set clone. A production counter on that path would add cost to the thing being measured for a signal that is not the bottleneck. SpatialInterestBenchmarks already exercises the 4-worker contended write path, which is the right place to answer this — and if it ever does become load-bearing, the fix named in BucketHistogram's own comment applies here too: shard per worker rather than measure.

One note on the merge

main brought BucketHistogram + WriteHistogramHeader/WriteHistogramSeries with the latency SLIs, which duplicated the cluster-size histogram this PR had added. Folded onto yours in bae3102 — ClusterSizeHistogram and the second WriteHistogram are both gone.

The merge also resolved scene-listener against per-realm grids: SceneListenerCellMapper now takes RealmSpatialGrids, and CollectSceneListenerSubjects resolves the listener's realm grid once — which let its per-subject realm comparison go, the same way the observer path lost one.

755/755 tests pass on the merged branch; no new compiler warnings.

[mikhail-dcl]

E2E validation on zone — 2026-09-02

Deployed this branch to dev via Deploy Dev (Debug) (run 33658198567), then ran the numbered scenarios from docs/tasks/e2e-livekit-connstring.md by hand — the scenario runner doesn't exist yet, so each is a manual run read from the output.

Bots: DCLPulseTestClient → pulse-server.decentraland.zone:7777.
Observation: archipelago-ea-stats.decentraland.zone/islands and /core-status — after the core decommission these are fed by Pulse's engine.islands / engine.discovery, so they are a direct read-out of this branch's feed.

Headline: the feed works, the last hop does not

/core-status flipped healthy:false → healthy:true on deploy, and /islands began serving live C{n} topology. So Pulse clusters and publishes correctly.

No island_changed ever reaches a client, on any scenario. comms-gatekeeper is not turning peer.*.cluster_change into engine.peer.{addr}.island_changed — its subscriber needs CLUSTER_SUBSCRIBER_ENABLED=true on the same broker Pulse publishes to. Every assertion phrased on island_id is blocked behind that, and none of it is a defect in this branch.

Scenario results

#	Scenario	Result	Evidence
1	Single bot, stationary	⚠️ Blocked	Cluster published at the bot's spawn (C2 center=(-104.2, 4.9)). No conn string — gatekeeper.
2	Two bots co-located	✅ Pass	Both welcomed by ws-connector; exactly one cluster C2.
3	Two bots > 2 cells apart	✅ Pass	(0,0) and (500,500), same realm → two clusters C6, C5; stable over 10 polls.
4	Bot walks A → B	⛔ Not runnable	Locomotion is a bounded drift around spawn — measured ~7 units in 4 minutes, so it cannot cross a 100 u cell. Needs a directed walk input.
5	Bot idles inside a cluster	✅ Pass	Id stayed C2 for ~90 s with no reassignment while the centre drifted (-108.0,6.6 → -113.6,6.4). Debounce + sticky id holding.
6	Cross-realm teleport	⛔ Not runnable	TeleportRequest is sent once at startup (Program.cs:200); no mid-session teleport exists.
7	Same coords, different realms	✅ Pass	Two bots both at (2000, 2000), realms realm-alpha / realm-beta → two clusters C3, C4 (both r=0.0), stable over 12 polls. A shared grid would have merged them.
8	Bot disconnects	✅ Pass	Killing all bots cleared /islands to [] immediately.
9	Broker restart	⛔ Out of scope	Restarting shared zone infrastructure.
10	Pulse restart	⛔ Out of scope	Same.

Scenario 7 is the one the task doc says to write first — the regression guard for RealmSpatialGrids. It passes against a deployed server.

Scale

10 concurrent bots spread over a 400-unit spawn radius collapsed into a single cluster of radius 266 — the percolation behaviour documented in docs/clustering-on-aoi.md §3.2, reproduced live rather than in a benchmark. 40 bots was attempted but blocked by half-created MetaForge accounts (wallet present, profile never deployed), which is harness state, not a server result.

Harness defects found while running this

1. account create crashes on any re-run. metaforge exits 2 for an existing account. Program.cs:85 passes throwOnNonZeroExit: false with a comment explaining exactly that, but ProcessOrchestrator.cs:23 and the scene-listener path Program.cs:237 did not — so each was usable exactly once per account prefix. Fixed in this branch.
2. The orchestrator cannot relaunch itself when started as dotnet X.dll. Children spawn as dotnet ---account=… (no assembly). Use dotnet run --project … for any run above BotsPerProcess.
3. BuildChildArgs silently drops options. It forwards account/count/offset/ip/port/pos/radius/rotate only — --realm, --transport, --comms-enabled, --comms-url, --join-livekit and --scene-listener-parcels are all lost, so any run above 20 bots quietly ignores them. Not fixed here.

Also added on this branch: --join-livekit, which joins the room a token was minted for and asserts ConnectionState == ConnConnected — the same criterion unity-explorer uses — so a run can distinguish "a token arrived" from "a token works".

[mikhail-dcl]

Addendum — provenance of the /islands data

The table above rested on island count and id prefix. That is the check core-decommission-runbook.md explicitly warns is insufficient ("every island reporting peers: [] means the join is failing … an island-count-only check would pass"), and the earlier runs did report peers: []. Re-validated properly:

Wiring. /islands is served from stats.getIslands(), populated solely by nats.subscribe('engine.islands', …) (stats/src/logic/subscriptions.ts, whose own comment reads "peer.* come from this repo's WS Connector; engine.* from Pulse"). On the deployed stats build 081ac634 core is absent from the tree, so Pulse is the only publisher of that subject.

Format markers.

Field	Observed	Pulse	Old core
id	C2 … C14	Clusters:IdPrefix = "C"	ROOM_PREFIX — I default, peer-zone on artemis
maxPeers	0	uncapped → reports 0	LIVEKIT_ISLAND_SIZE = 100

Cross-source join. With a heartbeating bot (--comms-enabled), one island resolves across two different subjects and two different services:

{"id":"C14","maxPeers":0,"center":[1499.98,0,1499.98],"radius":0,
 "peers":[{"address":"0x3f4c…afdb","parcel":[93,93],"position":[1500,0,1500]}]}

Island id/centre/radius come from Pulse's engine.islands; the peer entry comes from ws-connector's peer.*.heartbeat. They agree on wallet and position — a correlation stale or cached data cannot produce. The earlier peers: [] was simply bots run without --comms-enabled, which therefore never heartbeat and cannot be resolved by stats; not a broken join.

Liveness. Centres tracked the coordinates each run chose, cleared within one poll of killing the bots, and the cluster counter advanced C2 → C14 across the session.

(lastPing appears frozen within a poll window — expected: HeartbeatPump's default interval is 30 s. It is not a liveness signal at this timescale.)

[decentraland-bot]

Code Review — PR #34: Peer Clustering over AoI Grid with NATS Feed

Independent review of this PR after the prior review's findings were addressed. Reviewed the full diff (~3,600 lines production code, ~5,800 lines tests/benchmarks, across 95 files), the author's responses, and the fix commits.

Prior findings — resolved

#	Prior finding	Resolution	Status
P1	PruneVanishedClusters dictionary modification during foreach	Not a bug — Dictionary.Remove during enumeration is supported since .NET Core 3.0; repo targets .NET 10. Author verified on actual runtime and added an explanatory comment (999a649).	✅ Correctly refuted
P2	No realm-name length cap	Already capped at 255 by FieldValidator on all three intake paths.	✅ Already handled
P2	dropped counter could miscount on stale changeOrder entry	Fixed in c5e55f2 — dropped = true now sits inside the if (Remove(...)) branch.	✅ Fixed
P2	Lock contention metric	Declined — sub-µs hold times at ceiling.	✅ Acceptable

Independent analysis

Architecture — Clean separation of concerns:

• ClusterTracker (derivation on dedicated thread) → ClusterBoard (immutable pass, lock-free read) → NatsPublisher (fail-soft outbox, sole connection owner)
• RealmSpatialGrids (structural realm isolation, per-realm grids) replaces per-candidate realm predicates in AoI and clustering
• The two BackgroundService instances (tracker, publisher) have distinct failure domains linked only through the outbox interface

Threading model — Correctly reasoned:

• Single-writer-per-peer-slot invariant upheld: RealmSpatialGrids.Set is called only by the owning worker, Remove only by the disconnect cleanup path
• Volatile.Read/Write in RealmSpatialGrids and ClusterBoard are correct: the Volatile operations guard the array/reference stores against reordering, and single-writer ensures no TOCTOU on the per-slot reads outside the lock
• The tracker's dedicated LongRunning thread avoids thread-pool starvation; Task.Delay rather than Thread.Sleep keeps the cancellation responsive

Outbox design — Well-considered:

• Per-subject coalescing for cluster_change (one slot per peer, not a shared FIFO) prevents the cross-peer eviction that a shared queue would allow
• Separate latest-wins slot for topology prevents topology from competing with peer assignments for capacity
• Object pooling (rent/return) with correct lifecycle: the rented = null idiom on hand-off ensures exactly-once return on every exit path (success, failure, cancellation)
• The c5e55f2 fix ensures dropped counts only genuine evictions, not stale order entries

Union-find — Correct weighted union-find with path halving. Forward-half neighborhood probing (4 directions instead of 8) is sound because adjacency is symmetric. NodeKey custom hash avoids the entropy-destroying long.GetHashCode fold (x ^ z) that would collapse cells on anti-diagonals.

Sticky ID inheritance — The two-notion design (computed vs. published assignment per peer) correctly prevents the debounce starvation bug the author documented: a fragment mid-debounce inherits its own computed ID for overlap measurement, while the published assignment drives the feed change-detection.

Teleport ordering — The cross-realm teleport sequence (remove from old grid → publish snapshot → place in new grid) correctly avoids the stale-view problem where an observer of the old realm sees a subject whose snapshot names the new realm. The brief invisibility window (peer in no grid) is documented and preferable to the alternative.

NATS reconnection — Sound:

• Client defaults (unlimited retries, 2–5s jittered backoff) suit a fail-soft feed
• IgnoreAuthErrorAbort = true prevents a rotated credential from permanently killing the feed
• Supervision loop rebuilds after a faulted pipeline, with 5s backoff to prevent spin
• Linked CancellationTokenSource ensures drain and heartbeat fail together
• MarkDisconnected() in the finally block prevents the connected gauge from sticking at 1 after a rebuild

Discovery heartbeat — The single ServiceDiscoveryMessage instance is safely reused: it's rewritten before each publish, and PublishAsync completes serialization before its task returns (verified against NATS.Client.Core 3.0.1's CommandWriter). The heartbeat is sent directly rather than through the outbox, correctly avoiding backlog delays.

Security review

• ✅ Credential handling: SanitizeBrokerUrl strips userinfo from NATS URLs before logging via Uri.TryCreate, extracting only host:port. Unparseable URLs get a placeholder, never echoed raw.
• ✅ No secrets committed: NATS URL comes from environment variables; appsettings.json ships "Url": "" (shadow mode).
• ✅ Config precedence: AddNatsUrlAlias fills Nats:Url only when not already set, preventing the flat NATS_URL from silently overriding an explicit Nats__Url.
• ✅ Client-supplied realm names: Capped at 255 by FieldValidator; grid instances are bounded by connected peers and dropped when empty.
• ✅ Wallet addresses in NATS subjects: Lower-cased for consistent keying. Ethereum addresses are hex-safe, so no NATS subject injection risk from the auth-validated wallet format.
• ✅ NATS client logging: Floored at Warning in appsettings.json so a Debug-level Default can't pull it down to per-publish verbosity.
• ✅ SnapshotBoard.Publish API change: void → PeerSnapshot return is backward-compatible (callers can ignore).

Consumer impact

New NATS subjects (peer.{addr}.cluster_change, engine.islands, engine.discovery) are consumed by gatekeeper (PR #283) and archipelago-workers (PR #128) — separate deployment PRs. No existing consumers broken. SpatialGrid's API changes are internal to Pulse.

Observations (no action required)

• Known gap — cluster ID uniqueness: C{n} resets on restart and collides across instances. Documented in the PR and tracked for iteration 2. The mitigation (scope to server_id or boot epoch) is straightforward when needed.
• Scene listener cost: CollectSceneListenerSubjects now iterates realms × cells instead of just cells. For the typical single-realm case this is zero overhead; multi-realm listeners pay proportionally but the inner grid lookup is a ConcurrentDictionary.TryGetValue.
• ClusterPass.clusterIdByPeer allocation: Full maxPeers-sized array per pass (~32KB at 4095). Documented as the dominant allocation, intentionally not pooled because readers hold by reference.

Verdict

All prior findings are resolved. No P0 or P1 issues found in independent review. The code is exceptionally well-documented, the threading model is carefully reasoned, error handling is robust, test coverage is thorough (527 tests, including the three defects caught by tests that the PR description documents), and the benchmarks validate the performance claims. The shadow-mode default (tracker runs, feed off until URL injected) provides a safe rollout path.

Approved.

---

Reviewed by Jarvis 🤖 · Requested by mikhail-dcl via GitHub

[mikhail-dcl]

Cluster compositions from ClusterScenario (docs/clustering-on-aoi.md §3.2) — live on zone

The four documented population shapes, run against the deployed branch and read from /islands. Scaled down hard: the originals are 100 / 1 000 / 1 000 / 4 095 peers, these are ≤ 8 per run. That tests the topology rules — bridging, splitting, chaining — not the density-dependent behaviour. Peer counts and cluster counts below are therefore not comparable to the documented figures.

#	Composition	Result	Evidence
1	Sporadic — areas close enough to bridge, a second pair far enough to stay split	✅ Pass	8 bots, realm csB. Bots at x=0,10 and x=150,160 merged into C17 @ 80, r=80; bots at x=1000,1010 and x=1400,1410 stayed split as C18 @ 1005, r=5 and C19 @ 1405, r=5. Stable over 12 polls. Exactly the documented semantic: adjacent cells bridge, a 400 u gap does not.
2	DenseAndSparse — a crowd plus sparser regions, some merging	⚠️ Partial	Not run as a shaped composition (blocked, below). Related evidence from the scale run: 10 bots over a 400 u spread collapsed to one cluster of radius 266 — crowd-merging behaviour, but without the separate sparse regions that make this case distinct.
3	Chained — areas in a line, each bridging into the next	⛔ Blocked	Launched as 4 pairs at x=0/150/300/450 (realm csC); every bot failed before connecting — see below. The bridging mechanism is nonetheless demonstrated by case 1, which is one link of the same chain.
4	CeilingUniform — uniform Genesis fill at Transport.MaxPeers	⛔ Not reproducible at this scale	The case is defined at 4 095 peers, and its documented result (percolation collapse into ~2 clusters) is a property of density. At ≤ 50 bots the realm is far below the percolation threshold, so any run here would confirm nothing about it. ClusterTrackerBenchmarks remains the right instrument.

Blocker. Catalyst began returning 403 Forbidden on every profile fetch (CatalystProfileGateway.GetAsync), after earlier returning No profile found on Catalyst for most freshly bulk-created wallets. Both are harness/environment limits reached by the volume of runs in this session — not Pulse behaviour. Cases 2 and 3 should be re-run once the limit clears; the layouts are:

• Chained: pairs at x = 0, 150, 300, 450 (150 u spacing, one cell apart) → expect a single cluster spanning ~450 u.
• DenseAndSparse: a crowd at (0,0), sparse regions at x = 600 and 750 (should merge with each other) plus x = 1500 and 2200 (should not) → expect 4 clusters.

Two harness notes for whoever re-runs this:

• Bulk account creation is ~15–20 s per account and the profile is not immediately fetchable afterwards; pre-create the pool well ahead of the run, and expect some wallets to need a retry.
• --bot-offset with --total-bot-count puts a process in worker mode, which reuses pre-created accounts instead of creating them. That is the way to run several shaped groups concurrently without racing MetaForge's account store.

[mikhail-dcl]

Cluster compositions, second pass — with the IP whitelist in place

40 bots now connect cleanly (IP_CONNECTION_LIMIT_EXCEEDED gone), so the shaped runs actually ran. Two findings, one of them unresolved.

Measured adjacency threshold — matches the design

Three separations, three realms, 18 bots, stable over 12 polls:

Separation	Cells at CellSize=100	Observed
100 u	0 ↔ 1, adjacent	merged — C30 @ 50, r=50
200 u	0 ↔ 2	split
300 u	0 ↔ 3	split

Exactly ±1-cell adjacency at a 100 u cell. Clusters are also confirmed uncapped: 25 co-located bots formed one cluster (C27), so the 20/20 split below is not a size limit.

Unresolved: an eight-cell chain splits at one specific link

Chained, 8 areas of 5 bots at an exact 100 u pitch (cells 0–7), all 40 connected:

C33 @ 150, r=150   ← cells 0,1,2,3
C34 @ 550, r=150   ← cells 4,5,6,7

Deterministic across two runs, and reproducible with just the two failing groups in isolation (x=300 and x=400 → C35@300, C36@400). Every other 100 u link in the same run merged — (0,1), (1,2), (2,3), (4,5), (5,6), (6,7) — only 3↔4 failed. Distance and cell adjacency are identical to the pair that does merge at the origin, so the behaviour is position-dependent, which nothing in the algorithm should be.

The tracker is not the cause. Four unit tests now assert exactly these shapes and all pass (794 total, 0 failed):

• PeersOneCellApart_FormOneCluster — parameterised at x=0/100 and x=300/400
• EightCellsInALine_FormOneCluster — the chain at a uniform one-cell pitch
• ClusterFormedBeforeNeighbourArrives_MergesOnNextPass — the wave-arrival case, since production groups connect at different times

Union uses the forward half-ring (+1,−1), (+1,0), (+1,+1), (0,+1), which visits each adjacent pair once; float precision in CellCoord is clean at these coordinates (checked: x*1/100f floors to 0…7 exactly). So the split is not explainable by this branch's clustering code, and no cell size fits the observations either — 300/400 splitting requires C < 100, while 0/100 merging requires C ≥ 50, and C = 50 would split 0/100.

What that leaves: the deployed artifact or its configuration differs from what these tests cover. Resolving it needs the running server's commit and its SpatialHashAreaOfInterest:CellSize, neither of which is reachable from outside — Pulse's /metrics answers on localhost only and is not exposed on pulse-server.decentraland.zone (probed: 5000 and 443 both refuse). Whoever can read the dev task's logs or env can close this in one look.

I deliberately have not pushed a fix: the validation exonerates the logic rather than convicting it, so there is nothing verified to change.

Case status

#	Composition	Status
1	Sporadic	✅ passed earlier (bridge + split + duo)
2	DenseAndSparse	⛔ not run — superseded by the chain finding, which is the same adjacency question in a cleaner shape
3	Chained	⚠️ ran; split at one link, unexplained (above)
4	CeilingUniform	⛔ not reproducible under 50 bots — its result is a density property at 4 095 peers

[mikhail-dcl]

Root cause of the chain split — found, and it is not a clustering bug

Recorded here because the earlier comment left it unexplained.

What happens. The in-parcel offset is quantized to 8 bits over [0, 16] (step ≈ 0.0627). An offset of 12 encodes to code 191, not 191.25, and decodes to 11.984 — so a peer published at global x = 300 is stored at 299.984 and indexes into cell 2, not cell 3.

Harmless in itself: the two cells are adjacent, so a peer's clustering relative to its neighbours is unchanged. It only decides anything for a position sitting exactly on a cell boundary — which is what every group in my layout was doing.

It predicts the observation exactly. Eight groups at multiples of 100:

requested x	parcel	offset	code	stored x	cell
0	0	0	0	0.000	0
100	6	4	64	100.016	1
200	12	8	128	200.031	2
300	18	12	191	299.984	2
400	25	0	0	400.000	4
500	31	4	64	500.016	5
600	37	8	128	600.031	6
700	43	12	191	699.984	6

Occupied cells are {0,1,2,4,5,6} — cell 3 is empty, so the components are {0,1,2} and {4,5,6}. That is precisely the split observed: C33 @ 150, r=150 (the groups at 0–300) and C34 @ 550, r=150 (400–700). Same mechanism for the isolated pair: x=300 → cell 2, x=400 → cell 4, gap at 3, split.

Confirmed by moving off the boundaries. The same eight groups at 350, 450 … 1050 (cells 3–10) give one cluster: C37 @ 700, r=350, all 40 bots. So the chaining behaviour is correct and my layout was the fault.

No production impact. The error is bounded by half a quantization step (≈ 0.031 u), so it can only reclassify a peer within 0.031 u of a boundary, and always into an adjacent cell. Real populations are continuously distributed; a cell gap requires a genuinely empty 100 u band. Nothing to fix, so nothing was changed in the tracker.

Covered by tests (797 passing). The earlier tests set the grid directly and so skipped the decode entirely — that is why they passed while a live run failed:

• PeersPublishedOneCellApart_FormOneCluster — goes through PeerSnapshotPublisher, parameterised near the origin and away from it, coordinates kept off boundaries
• PublishingOnACellBoundary_CanLandInTheLowerCell — asserts the stored position is below the input by less than one step, and that it indexes cell 2; documents the whole trap for the next person writing a fixture

Case status — all four now resolved

#	Composition	Result
1	Sporadic	✅ 8 bots: x=0/10 + 150/160 bridged (C17 @ 80, r=80); 1000s and 1400s stayed split
2	DenseAndSparse	✅ 40 bots: 4 clusters — crowd C38 @ 350, the adjacent sparse pair merged into C39 @ 1300, r=50 (1250+1350), and C40 @ 2050 separate
3	Chained	✅ 40 bots mid-cell: one cluster C37 @ 700, r=350 spanning 350–1050
4	CeilingUniform	⛔ not reproducible under 50 bots — its documented result is a density property at 4 095 peers; ClusterTrackerBenchmarks owns it

Also confirmed along the way: clusters are uncapped (25 co-located bots → one cluster), and the adjacency band is ±1 cell at CellSize = 100 (100 u merges, 200 u and 300 u split).

One incidental finding: my fourth sparse region was requested at x=2750, beyond the encodable world (MaxParcelX 163 + padding 2 → ≈ 2640 u). The client encoded an out-of-range parcel, the index aliased into another row, and the group materialised at x ≈ −2338. The server cannot detect this — FieldValidator checks IsValidIndex, and the aliased index is in range — which is exactly what ParcelEncoder.IsValidCoordinate's comment warns about. Client-side error, no server change warranted, but worth knowing when placing fixtures.

[mikhail-dcl]

Conclusions

Clustering works. Validated on zone against the deployed branch, read from Pulse's own engine.islands feed:

• Adjacency is ±1 cell at CellSize = 100 — 100 u merges, 200 u and 300 u split
• Clusters are uncapped — 25 co-located bots form one cluster
• Realm isolation holds — two bots at identical coordinates in different realms stay in separate clusters (the RealmSpatialGrids regression guard)
• All three runnable compositions pass: Sporadic, DenseAndSparse, Chained. CeilingUniform needs 4 095 peers and stays with the benchmark

The one anomaly was mine, not the server's. An eight-group chain split in half because every group sat exactly on a cell boundary, where the in-parcel offset's 8-bit quantization (step ≈ 0.0627) moves x = 300 to 299.984 — cell 2, not 3 — leaving cell 3 empty. Moving the same layout mid-cell gives one cluster. Bounded by half a step and always into an adjacent cell, so real populations are unaffected. No tracker change made.

Still blocking end-to-end. No island_changed reaches any client: comms-gatekeeper is not converting peer.*.cluster_change into engine.peer.{addr}.island_changed. Pulse's half is confirmed live (/core-status healthy, /islands serving C{n}), so the remaining work is gatekeeper's CLUSTER_SUBSCRIBER_ENABLED=true on the same broker.

Changes on this branch from the exercise: --join-livekit (asserts ConnConnected, the explorer's own criterion), a non-fatal Catalyst profile fetch, account create no longer crashing on re-run, and tests that go through PeerSnapshotPublisher instead of setting the grid directly — which is why they now catch what a live run catches. 797 passing.