feat(multi_layer): cross-node layer coherence via pg-tracked coordinators

[MikaAK]

Problem

MultiLayer's node-local fast layers (e.g. Cache.ETS) go stale on every node except the writer: put writes all layers on the calling node only, and delete removes from all layers including the shared one — so there was no way to invalidate another node's L1 without destroying the shared L2 entry. Any multi-node deployment reading through an ETS L1 serves stale data for up to backfill_ttl after every remote write, including the pathological case where an update-triggered consumer re-reads immediately and re-caches the previous value.

Design

Each MultiLayer cache now runs a per-node Cache.MultiLayer.Coordinator (replacing the placeholder Agent in child_spec) that joins a :pg group named after the cache — group membership doubles as the registry of which nodes hold the cache. Two new (optional, default-off) cache opts:

• broadcast_mode: :invalidate — after a successful put/delete, other members drop the key from their broadcast_layers and lazily re-read through the shared layer. Key-sized messages; the right default for large values.
• broadcast_mode: :replicate — members write the new value into their local layers immediately. Full value copy per member; for small values.

broadcast_layers names the node-local layer modules the remote change applies to (never the shared layer).

Guarantees (stated in docs)

Best-effort delivery — send/2 to pg members, no acks. backfill_ttl/layer TTLs remain the correctness floor for members that miss a message. The pg scope is started unlinked (:pg.start/1, not start_link — linking would tie the scope to whichever coordinator started it first); every coordinator monitors the scope and re-joins on :DOWN because a restarted scope has empty membership.

Tests

8 new tests: pg membership, invalidate/replicate message handling scoped to the listed layers only, broadcast excluding the writing node's own coordinator (by pid, so it's testable single-node), and put/delete integration for both modes. Existing MultiLayer suite untouched and green (20/20 combined); full suite green except the pre-existing RedisJSON env failures (module not installed in local Redis). mix credo --strict clean.

Motivating use case

CheddarFlow's OpenInterestCache (325KB surfaces, Redis-only today) moves to {Cache.MultiLayer, [ETS, Redis]} with broadcast_mode: :invalidate — the OI update broadcast currently races the update itself, and remote L1s would otherwise re-cache the previous surface for hours.

[codecov]

Codecov Report

❌ Patch coverage is 92.10526% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.23%. Comparing base (043ddbe) to head (70e4145).

Files with missing lines	Patch %	Lines
lib/cache/multi_layer.ex	88.23%	2 Missing ⚠️
lib/cache/multi_layer/coordinator.ex	95.23%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
+ Coverage   83.81%   84.23%   +0.41%     
==========================================
  Files          23       24       +1     
  Lines         655      685      +30     
==========================================
+ Hits          549      577      +28     
- Misses        106      108       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.