ci(d0): run Tests/eval non-vacuously, and fix what that surfaced - #46
Merged
Conversation
`Tests/eval` has seven modules. Exactly one ran in CI — `test_adr_references.py`,
via D0-R's direct step. The other six executed nowhere, on any branch, ever.
Turning them on required fixing four things they were hiding.
**The harness could not have worked.** Six of seven modules are bare `def test_*`
with no `unittest.TestCase`, so `python -m unittest` collects zero from them, and
`unittest discover` fails outright because `Tests/eval` has no `__init__.py`.
Either path reports success while running nothing. pytest is forced, not
stylistic.
**An import-time crash.** `Scripts/eeg_spectral.py` had
_trapz = getattr(np, "trapezoid", getattr(np, "trapz"))
Python evaluates that default *before* the lookup it is a fallback for, so on
numpy 2.x — which removed `trapz` — it raised `AttributeError` at import, on
precisely the version the fallback existed to support. Resolved lazily now.
Three more modules failed collection for a missing scipy, which is a real
transitive dependency and is now installed rather than assumed.
With collection fixed, three assertions ran for the first time and failed.
**`cohens_d` returned 0.0 for an unbounded effect.** Zero pooled SD means both
samples are constant; `cohens_d([1,1,1,1], [5,5,5,5])` reported "no effect" for
the most extreme separation representable. The limit is signed infinity, and
identical constants are the only genuinely zero case. Fixed in all four copies
— consolidating them belongs to the lane that owns that structure, not here.
**The Mann-Whitney test was impossible.** At n=3 vs 3 the exact two-sided
p-value cannot go below 2/C(6,3) = 0.1, so `p < 0.05` was unreachable by
construction and no implementation could have passed it. It now asserts the
floor — which also pins that the implementation is exact rather than
normal-approximating — with a companion at n=4, where 2/C(8,4) ≈ 0.0286 clears
0.05.
**The CKA test described a different estimator.** Biased linear CKA inflates
badly when d >> n: 50 samples against 128 and 256 features scores ~0.78 on
independent data. `< 0.3` describes an unbiased estimator this code does not
implement. Now bounded above with the reason recorded, plus a companion showing
the inflation shrinks as n grows — so a later switch to an unbiased estimator
fails here and forces the expectation to be revisited deliberately.
Two guards, because a job that runs nothing still exits 0:
collection errors == 0 a module that stops collecting fails loudly
collected >= 55 floor below the current 63
Proven non-vacuous: reintroducing the trapz bug yields 1 collection error and
drops collection to 51, tripping both.
`NeuralComposeEEG/` is absent on main until lane B. The guard says so explicitly
and exits 0; once present, zero collected tests is an error.
Tests/eval 30 collected + 4 errors → 63 passed, 0 failures
ADR gate (D0-R) 16 tests OK, PASS (1 historical warning)
swift build unaffected
git diff --check clean
`test_joint_embedding.py` is the one exclusion, by name and reason: it imports
mlx (Apple Silicon only) and needs a converted model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two statistical-contract defects found in review of the D0 CI work. Both are in
functions the new job now executes for the first time.
**Cohen's d pooled the wrong way for unequal groups.**
sqrt((var_a + var_b) / 2)
equals the conventional pooled estimator only when the two samples are the same
size, and this function accepts arbitrary lengths — so every unequal-n caller
was dividing by the wrong denominator. It survived because all the tests used
equal groups. Replaced with degrees-of-freedom weighting in all four copies. The
difference is not cosmetic: on n=3 against n=9 the reported effect moves 2.540 →
2.033, about 25%.
The new regression test is built so the two formulas disagree, and asserts
against the weighted value while asserting *away* from the unweighted one, so
reverting the fix fails rather than merely changing a number.
**The Mann-Whitney p-value was inherited, not chosen.** The call omitted
`method=`, taking SciPy's `auto` — a selection rule that has changed across
releases and would move a reported p-value with no commit here. Method is now
explicit: exact when there are no ties and the groups are small enough to
enumerate, asymptotic otherwise, with the size cap stated as policy rather than
borrowed from a library default. Ties make exact invalid, and a test now covers
that path.
The previous commit's claim that the test "pins an exact p-value" was true of
the value but not of the mechanism; it is true of both now.
n=3 exact floor 0.100000 = 2/C(6,3)
n=4 exact 0.028571 = 2/C(8,4)
with ties 0.103754 asymptotic, as required
**±inf is documented as project policy**, not as the definition of Cohen's d,
along with the serialisation trap it creates: JSON has no Infinity literal and
CSV no convention, so callers that persist an effect size must convert at the
boundary.
CI now bounds the majors this gate's semantics depend on — numpy>=2,<3 and
scipy>=1.14,<2 in particular, since the trapz fix is numpy-2 specific and the
Mann-Whitney contract is a SciPy behaviour. Unconstrained installs would let a
major bump change a p-value silently.
Tests/eval 65 passed, 0 failures (was 63)
trapz mutation 1 collection error, 53 collected — both guards fire
ADR gate 16 tests OK, PASS
swift build unaffected
git diff --check clean
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The finding
Tests/evalhas seven modules. Exactly one ran in CI —test_adr_references.py, via D0-R's direct step. The other six executed nowhere, on any branch, ever.Turning them on required fixing four things they were hiding.
1 · The harness could not have worked
Six of seven modules are bare
def test_*with nounittest.TestCase, sopython -m unittestcollects zero from them — andunittest discoverfails outright, sinceTests/evalhas no__init__.py. Either path reports success while running nothing. pytest is forced here, not stylistic.2 · An import-time crash
Python evaluates that default before the lookup it is a fallback for. On numpy 2.x — which removed
trapz— it raisedAttributeErrorat import, on precisely the version the fallback existed to support. Three further modules failed collection on a missing scipy, a real transitive dependency now installed rather than assumed.3 · Three assertions that had never run
cohens_d([1,1,1,1],[5,5,5,5])→0.0mann_whitney_u([1,2,3],[10,11,12]),p < 0.05cka(randn(50,128), randn(50,256)),< 0.3< 0.3describes an unbiased estimator this code does not implement. Now bounded with the reason recorded, plus a companion showing the inflation shrinks as n grows.Consolidating the four
cohens_dcopies is deliberately not done here — that structural change belongs to the lane that owns it.4 · Guards, because a job that runs nothing still exits 0
Proven non-vacuous: reintroducing the trapz bug yields 1 collection error and drops collection to 51 — both gates fire.
NeuralComposeEEG/is absent on main until lane B. The guard says so explicitly and exits 0; once present, zero collected tests is an error.Evidence
5 · Statistical-contract fixes (second commit)
Review of the first commit found two defects in functions this job now executes
for the first time. Both shared a shape: the code was more general than its
tests, and the untested region was exactly where it was wrong.
Cohen's d pooled variance is df-weighted for unequal sample sizes.
sqrt((var_a + var_b) / 2)matches the conventional estimator only at equaln, and the function accepts arbitrary lengths — every earlier test used
equal groups. On n=3 against n=9 the reported effect moves 2.540 → 2.033.
The regression test asserts toward the weighted value and away from the
unweighted one, so reverting fails rather than silently changing a number.
±Infinity for unequal constant groups is an explicit project policy, not
the definition of Cohen's d. Persistence callers must map non-finite values at
JSON/CSV boundaries: JSON has no Infinity literal (
json.dumpsemits a bareInfinitythat strict parsers reject) and CSV has no convention.Mann-Whitney uses exact for tie-free samples within a stated size cap,
asymptotic otherwise. Previously it omitted
method=and inherited SciPy'sauto, whose rule has changed across releases and would move a reportedp-value with no commit here. The earlier claim that the test pinned an exact
p-value was true of the value but not the mechanism; it is true of both now.
CI bounds
numpy>=2,<3andscipy>=1.14,<2. The trapz fix isnumpy-2-specific and the Mann-Whitney contract is SciPy behaviour, so an
unconstrained install could change a p-value with no commit here.
test_joint_embedding.pyis the one exclusion, by name and reason: it imports mlx (Apple Silicon only) and needs a converted model.Independent of #44, which stays parked for review.