[Meta] Test-methodology review 2026-08-28 — comparative gap catalogue (#1368–#1386)

[pathosDev]

Catalogue for the test-methodology review of 2026-08-28, run against v0.15 on develop. Filed as #1368–#1386.

What this review asked

Not "which test cases are missing" but "which methods does this suite not use". The suite was compared against how mature actor-model implementations in other ecosystems verify their own runtimes, and against the general distributed-systems testing literature.

The headline result is that the gaps are almost all methodological. There is no large body of missing test cases — core semantics are covered to a reasonable depth, and several areas are genuinely strong. What is absent is a set of techniques that find defects nobody thought to write a case for.

Where this suite is already ahead — do not "improve" these

Recorded so a later reader does not mistake the list below for a general verdict, and so none of this is accidentally traded away:

• The persistence contract suite — 37 scenarios (19 journal / 10 snapshot / 8 durable-state) across 11 in-process harnesses and 10 live databases, with genuine capability negotiation so a backend can decline a scenario rather than pass it vacuously. tests/integration/brokers/lib/persistence-contract/.
• MailboxProperties.test.ts — a real model-based test, driving 10 variants against a deliberately differently-implemented reference model (the priority model re-sorts rather than binary-searching, so a shared off-by-one cannot hide), comparing drain order and drop-reason logs.
• The timing-discipline meta-tests — SleepRatchet.test.ts (three downward-only ledgers, its own TypeScript scanner, a sanity floor so a broken regex cannot satisfy every ceiling by reading nothing), AwaitConditionBudgets.test.ts, TimerTolerance.ts. Nothing in the comparison set has an equivalent.
• Backpressure depth — five dedicated mailbox files including the hard interactions: a Terminated surviving a full mailbox ([Security] Death-watch Terminated is re-enqueued onto the bounded USER queue, so the default drop-head policy silently evicts it (oldest-first) and the watcher is permanently blinded, with no dead letter #729), the prepend/unstash path holding the bound ([Security] BoundedMailbox does not override prependUser, so unstashAll() unshifts up to 1024 envelopes straight past the capacity check, the overflow policy and the drop accounting #772), drop attribution before the actor instance exists ([Feature] A withMailbox-supplied BoundedMailbox reports no drops — actor_mailbox_dropped_total needs a chaining observer #1149).
• ParallelMultiNodeSpec — real OS threads behind nearly the same API as the in-process variant.
• Protocol.test.ts — asserts decode cost stays linear in bytes received, i.e. an algorithmic-complexity attack test.

The catalogue

Methods the suite does not use

Issue	Gap
#1368	Nothing measures whether an assertion discriminates — thirteen open issues are hand-found surviving mutants
#1369	No generative byte input reaches any decoder
#1370	Promise-resolution order is never controlled
#1371	Property tests stop at leaf data structures
#1372	No seed is pinned and no counterexample corpus exists
#1373	The single-active-instance claim is checked by scenarios, never as a property over histories

Testkit capability our users do not get

Issue	Gap
#1374	awaitCondition and the recording logger are internal only
#1375	No dead-letter assertion
#1376	Every deadline is a hard millisecond literal — no scaling factor
#1377	TestKit.within is a post-hoc deadline only
#1378	No synchronous actor driver, so absence assertions must be fixed waits

Scenarios with no coverage

Issue	Gap
#1379	Every multi-node scenario injects one fault episode; none runs a second
#1380	A reused path is never tested against a Terminated for the previous incarnation
#1381	Become.test.ts has four cases and none crosses a restart
#1382	No node restarts under the same address; no seed restart; no rejoin after unreachable
#1383	Nothing bounds concurrent recoveries (a feature gap before a test gap)
#1384	Supervision is never stressed with failures in preRestart / postRestart / constructor
#1385	No test asserts a large message does not delay unrelated traffic

Boundary semantics

Issue	Gap
#1386	A message can lose its prototype on a worker hop and fail on the wire, undetected

Ordering

Cheapest first, because in three cases the work is already written and merely on the wrong side of src/, and in two more the tool is already a dependency:

1. [Feature] awaitCondition and the recording logger are internal test helpers, so a user testing an actor has neither a retrying assertion nor a log assertion #1374 — a move, not a build.
2. [Feature] A message that works in-process can silently lose its prototype on a worker hop and fail on the wire, and nothing catches either #1386 — small, and the only item that catches a silent-degradation class.
3. [Test] No generative byte input reaches any decoder — every parser is guarded by hand-picked fixtures alone #1369 and [Test] Promise-resolution order is never controlled, so the ordering-bug family the stress harness documents as unfindable stays unfindable #1370 — fast-check is already installed; both need an import, not a package.
4. [Feature] Every testkit deadline is a hard millisecond literal — no global factor scales them for slow machines #1376 — one factor, applied where deadlines are consumed.
5. [Test] Nothing measures whether an assertion discriminates, and thirteen open issues are hand-found surviving mutants #1368 — highest value of anything here, but needs a runner decision and a noise filter first.

Everything else follows. #1378 is the largest single item and should start with a prototype establishing whether a controllable dispatcher is possible without changing production scheduling.

Runtime

Only #1374, #1376, #1377 and #1372 belong on the per-commit path. Byte fuzzing, interleaving search, mutation runs and the repeated-fault suites go to the nightly slot — .github/workflows/nightly-flakes.yml already has the shape for long jobs, including a watchdog and per-run aggregation.

Related open issues this review touched but did not duplicate

#200 (deterministic simulation — #1370 is its cheap prerequisite), #536 (public persistence testkit — closer to a move than the issue currently implies), #671 (probe assertions), #818 (soak and leak), #914 (#1384 would reproduce it), #1017 (#1369 and #1371 are the constructive successors), #1023 (fault primitives that #1379 and #1373 both want), #1176 (#1382 covers the seed-restart direction), #1225, #1291.

Method note

Every "missing" claim in the linked issues was checked by search against src/, package.json, the workflow files and the tracker — open and closed — rather than asserted from reading alone. Where a claim rests on reading, the file and line are quoted in that issue's verification section.

One correction worth recording, because it nearly went the other way: an intermediate research step reported that fast-check was not present in the project. It is — ^4.9.0, in seven files, landed by #543. Two issues in this catalogue (#1369, #1370) depend on that being true, and had the error survived they would have been written as "adopt a new dependency" rather than "use the one we have".