feat(registry): paginate discovery/listDevices to bound NATS payloads

[atsyplikhin]

Summary

This PR addresses a class of fleet-scale failures we hit at ~1400 phones
on a single registry. The headline fix is server-side pagination of
discovery/listDevices, but the same outage also exposed several
adjacent bottlenecks (a registration thundering-herd, an undersized
etcd HTTP pool, an unbounded portal NATS reconnect path, a dashboard
that fetched every device's detail block on every poll). Each is fixed
in its own commit; they're shipped together because they were the same
incident response and they interact (e.g. pagination is much less
valuable if the registry still hangs on a registration herd).

Splitting was considered and rejected as churn — each change is small
and they share tests/fixtures. Reviewers should still feel free to
critique any commit individually.

---

Discovery pagination (the headline)

Problem

At ~1400 devices the JSON-encoded fleet snapshot exceeds NATS's default
1 MB max_payload and any agent-tools call that resolves a
selector (discover_labels, discover('device(category:phone)'),
broadcast, even narrow selectors) fails with
nats: maximum payload exceeded. The registry loads the entire tenant
from etcd before any filter is applied, so even fine-grained selectors
hit the wall.

Wire contract (additive, backward compatible)

• Request params for discovery/listDevices gain optional
  offset (int, default 0) and limit (int).
• Reply gains next_offset (int | null) and total_matched (int)
  alongside the existing devices array — only when the caller
  opted in by passing limit.

Compatibility matrix

Client behavior	Server behavior
Old client, no limit	Server takes the legacy unpaged path — returns the full filtered fleet in one reply. No next_offset/total_matched fields. Small fleets keep working untouched; large fleets fail loudly with the original max_payload exceeded error rather than silently truncating. Operators hitting the ceiling must upgrade clients.
New client, passes limit	Server paginates, silently clamps the page to _LIST_DEVICES_MAX_LIMIT (env DC_LIST_DEVICES_MAX_LIMIT, default 200), returns next_offset/total_matched. A client asking for limit=10000 gets the first 200 plus a forward-pointing next_offset — no error, no truncation; the client just paginates more aggressively than it expected. Client loops until next_offset is None.
New client, old server	Server ignores unknown params and returns the legacy single-reply shape. Client sees next_offset absent, exits the loop after one request — forward-compatible.
Malformed offset / limit	Server returns a clean JSON-RPC -32602 error instead of a 500.

Server

• DeviceRegistry.list_devices_page(tenant, *, device_type, location, offset, limit) returns (devices_page, next_offset, total_matched). The existing list_devices(...) is untouched and remains the legacy path. Stability and ACL caveats documented in the docstring.
• discovery/listDevices handler routes to the paged variant only when limit is in params; otherwise keeps the legacy reply shape. Both offset and limit are validated up front.

Clients (edge + server RegistryClient)

• RegistryClient.list_devices() loops 100-device pages internally. External signature unchanged; callers keep getting a flat List[Dict]. Page size tunable via DEVICE_CONNECT_LIST_PAGE_SIZE.
• New RegistryClient.list_devices_page(...) returns (devices, next_offset, total_matched) for callers that want explicit paging (e.g. a UI). Docstrings call out the ACL caveat: total_matched is pre-ACL and pages may be shorter than limit for restricted callers.

Defense in depth

• security_infra/setup_deployment.sh bakes max_payload: 8MB into the regenerated NATS config so operators rerunning the script raise the broker ceiling alongside the protocol-level fix.

---

Other bundled fixes (same incident)

1. Registration thundering herd (device.py)

Problem: 1000+ phones spinning up in lockstep hit a 2s request
timeout, every late reply triggered a retry, and the queue collapsed
into a sustained storm.
Fix: registration request timeout now DEVICE_CONNECT_REGISTER_TIMEOUT
(default 15s, was 2s) and a one-shot pre-registration jitter window
DEVICE_CONNECT_REGISTER_JITTER (default 2s, set to 0 to disable;
bump to 10s+ for fleet scale) decorrelates the initial herd. 2s still
spreads ~1000 lockstep devices into roughly 500/sec while staying
tolerable for single-device dev. Subsequent retries keep their existing
exponential backoff.

2. etcd HTTP pool exhaustion (registry/service/registry.py)

Problem: etcd3gw's default urllib3 pool of 10 sockets bottlenecked
the registry under the herd — every lease+put is two sequential HTTP
calls, so thousands of requests queued behind 10 sockets.
Fix: new _enlarge_etcd_pool() mounts a larger HTTPAdapter
(DC_ETCD_POOL_SIZE, default 64) onto the etcd3gw client's
underlying requests.Session. Implemented via post-construction mount
(not etcd3gw.client(session=...)) so it works on 2.5.x and 2.6+.
Logs a warning if etcd3gw ever stops exposing client.session — the
fix would otherwise silently regress.

3. Portal NATS reconnect storm (portal/services/nats_rpc.py)

Problem: every "Run" click on the dashboard opened a fresh NATS
connection (TCP + JWT-auth handshake) per RPC.
Fix: long-lived cached client reused across invoke() calls. The
NATS client is concurrent-safe (nc.request allocates its own inbox
sub) so one shared client serves the whole portal. On hard transport
failure the cached client is dropped and best-effort closed before
the next call reconnects. The module-level asyncio.Lock is created
lazily inside the first awaiter so it binds to the live event loop,
not import-time.

4. Dashboard fan-out (portal/views/dashboard.py, templates)

Problem: the live-devices fragment rendered every device's full
detail panel (functions, events, raw JSON) on every 3s htmx poll,
turning the dashboard into a fleet-sized data fan-out.
Fix:

• Detail markup moved to a new lazy endpoint
  /api/devices/{device_id}/live-detail, fetched on row expand. The
  main poll now returns only summary rows.
• etcd get_prefix and credential listing moved off the event loop
  via asyncio.to_thread so other portal requests don't queue behind
  the poll.
• Header counters (creds/online/registered) refresh in place via
  hx-swap-oob. Previously they froze at first-render values and
  showed "1401 online" long after the swarm shut down.
• Poll cadence eased from 3s -> 10s.
• revealed-based lazy loading was rejected: htmx's revealed
  implementation calls getBoundingClientRect, and display:none
  rows return a zero-rect at (0,0) which falsely registers as
  in-viewport — every hidden row would fetch on page load.

5. @on handler collection (drivers/base.py)

Problem: _collect_event_subscriptions skipped every single-
underscore-prefixed method, so drivers that wrote @on async def _on_phone_state(...) (Python's conventional "this is private") had
their event handlers silently dropped.
Fix: scan everything except dunders. The _on_meta attribute
gate filters out unrelated private attributes; new test covers it.

6. @on(device_type=...) in registry/portal mode (drivers/base.py)

Problem: device_type filtering relies on the D2D peer cache. In
portal/registry mode there is no peer cache, so the filter silently
passed every device's matching event through.
Fix: warn once at subscription setup so subscribers don't silently
get cross-device events. Strict filtering can be added in-handler.

7. Portal RPC timing logs (portal/views/dashboard.py)

Added structured timing logs around backend.rpc_invoke so the next
slow-dashboard incident has data on whether the latency is in the
portal handler, the pre-RPC setup, or the device round-trip.

---

Tests

New

• test_registry_service.py — 6 tests in TestListDevicesPage:
  slice + next_offset, final page sets next_offset=None, offset
  past end returns empty, limit=None returns the remainder, a full
  1400-device round trip, and filter-before-pagination.
• test_rpc_handlers.py — handler-level coverage rewritten around the
  dual-path behavior. New regression tests:
  • test_list_devices_legacy_no_limit_returns_unbounded — old-client
    path returns full fleet and emits no pagination metadata.
  • test_list_devices_with_limit_paginates — new-client path emits
    next_offset/total_matched.
  • test_list_devices_paged_acl_total_is_unfiltered — documents and
    locks in the ACL × pagination caveat (page shorter than limit,
    total_matched reflects pre-ACL count).
  • test_list_devices_invalid_offset_returns_error /
    test_list_devices_negative_offset_returns_error /
    test_list_devices_invalid_limit_returns_error — malformed
    pagination params surface as JSON-RPC -32602, not 500s.
• test_registry_client.py (edge) — 5 tests in
  TestListDevicesPagination: full-fleet paging across 14 round
  trips, wire-level offset/limit round-trip, legacy-server
  single-reply fallback, explicit list_devices_page() metadata
  exposure, filter forwarding.
• test_drivers.py — test_underscore_prefixed_handler_is_still_collected
  for the _collect_event_subscriptions fix.

Regressions

All existing registry / edge / agent-tools tests remain green; the only
test-file edits are in test_rpc_handlers.py to reflect the new
dual-path behavior of the handler.

Out of scope (deliberately)

• Selector pushdown to the registry / etcd indexes. That would let
  the server filter by category:phone etc. before paginating and
  reduce wire traffic further, but is a much bigger change.
  Pagination alone is enough to keep individual replies under
  max_payload regardless of fleet size.
• Keyset pagination. Offset pagination has known stability gaps
  under concurrent registrations (documented in the
  list_devices_page docstring); a device_id-cursor version is the
  next iteration.
• Streaming replies.

Test plan

• pytest packages/device-connect-server/tests/device_connect_server/test_registry_service.py -v — 43 passed
• pytest packages/device-connect-server/tests/device_connect_server/test_rpc_handlers.py -v — 41 passed
• pytest packages/device-connect-edge/tests/test_registry_client.py -v — passes
• pytest packages/device-connect-edge/tests/test_drivers.py -v — passes
• pytest packages/device-connect-agent-tools/tests/test_connection_unit.py tests/test_tools_unit.py -v — 65 passed (from previous run; no semantic change to that surface)
• Integration test (Docker) on the 1400-device deployment that originally hit the cliff: re-run discover('device(category:phone)'), discover_labels(), and broadcast(...) end-to-end.

🤖 Generated with Claude Code