[Feature] Nothing bounds concurrent recoveries, so a shard region recovering many entities at once has no back pressure against the journal

[pathosDev]

Use case

There is no limit anywhere on how many persistent actors may recover at the same time. grep -rniE "recoveryPermit|maxConcurrentRecover|recoveryConcurren|concurrentRecover" over src/ returns nothing.

This is a feature gap before it is a test gap, which is why this is filed as [Feature]: there is no bound to write a test for.

The scenario that makes it matter is ordinary. A shard region with rememberEntities enabled restores its entities after a rebalance or a node restart. Each entity is a PersistentActor and each starts by replaying its stream. With a thousand remembered entities, a thousand replays are issued against the journal at once, on a connection pool sized for steady-state traffic.

The likely outcomes are all bad and none is a clean failure: pool exhaustion surfacing as timeouts attributed to whichever query lost the race; a database rejecting connections so recovery fails for a subset while the rest succeed, leaving the region partially live; or recovery succeeding but taking long enough that the node is marked unreachable and downed, which starts the whole sequence again elsewhere.

Startup is also exactly when the system has the least headroom, so this is a thundering herd at the worst possible moment.

Proposed shape

• A recovery permit mechanism: a bounded number of concurrent replays, with the rest queued rather than rejected. Queued is important — rejecting a recovery means the entity is unavailable, whereas queuing only makes it slower.
• Configured through the established precedence (explicit options over HOCON over default), with the key reachable from src/config/ConfigKeys.ts and read by something in src/, per the dead-key rule.
• A sensible default rather than unlimited. The number should come from a measurement, not from taste — worth stating on the issue what was measured and on what.
• Observability: how many recoveries are running and how many are queued. Without it the symptom is indistinguishable from a slow database, which is how this class of problem usually gets misdiagnosed.

Then the tests that are currently unwritable:

• N entities recover with the bound set to k; at most k replays are ever in flight.
• All N eventually complete — the bound queues, it does not drop.
• A recovery that fails does not consume its permit permanently.

Acceptance

• A configurable bound on concurrent recoveries exists, with a measured default.
• Exceeding it queues rather than rejects.
• A metric or log exposes in-flight and queued counts.
• Tests assert the bound holds and that all recoveries complete.
• EN + DE persistence docs describe it.

Verification status

Confirmed by search. grep -rniE "recoveryPermit|maxConcurrentRecover|recoveryConcurren|concurrentRecover" over src/ returns nothing. src/persistence/ was surveyed via the knowledge graph for a recovery-throttling component; none exists. Recovery is initiated per actor in the PersistentActor start path with no coordination point between instances.