Skip to content

ci(d0): run Tests/eval non-vacuously, and fix what that surfaced - #46

Merged
aurascoper merged 2 commits into
mainfrom
d0/non-vacuous-ci-substrate
Jul 28, 2026
Merged

ci(d0): run Tests/eval non-vacuously, and fix what that surfaced#46
aurascoper merged 2 commits into
mainfrom
d0/non-vacuous-ci-substrate

Conversation

@aurascoper

@aurascoper aurascoper commented Jul 28, 2026

Copy link
Copy Markdown
Owner
head:    b8c57af56b8cfa426097bf555f323101c64f9eb0
base:    main@859feaa        # the D0-R merge commit
lane:    D0  (CI substrate; precedes A0/A → B)
files:   8
commits: 2

The finding

Tests/eval has seven modules. Exactly one ran in CI — test_adr_references.py, via D0-R's direct step. The other six executed nowhere, on any branch, ever.

Turning them on required fixing four things they were hiding.

1 · The harness could not have worked

Six of seven modules are bare def test_* with no unittest.TestCase, so python -m unittest collects zero from them — and unittest discover fails outright, since Tests/eval has no __init__.py. Either path reports success while running nothing. pytest is forced here, not stylistic.

2 · An import-time crash

_trapz = getattr(np, "trapezoid", getattr(np, "trapz"))     # Scripts/eeg_spectral.py

Python evaluates that default before the lookup it is a fallback for. On numpy 2.x — which removed trapz — it raised AttributeError at import, on precisely the version the fallback existed to support. Three further modules failed collection on a missing scipy, a real transitive dependency now installed rather than assumed.

3 · Three assertions that had never run

test verdict
cohens_d([1,1,1,1],[5,5,5,5])0.0 code defect. Zero pooled SD means both samples are constant; this reported no effect for the most extreme separation representable. The limit is signed infinity; identical constants are the only genuinely zero case. Fixed in all four copies.
mann_whitney_u([1,2,3],[10,11,12]), p < 0.05 impossible test. At n=3 vs 3 the exact two-sided floor is 2/C(6,3) = 0.1. No implementation could have passed. Now asserts the floor — which also pins that the implementation is exact rather than normal-approximating — plus a companion at n=4 where 2/C(8,4) ≈ 0.0286 clears 0.05.
cka(randn(50,128), randn(50,256)), < 0.3 wrong estimator. Biased linear CKA inflates when d ≫ n; independent data scores ~0.78. < 0.3 describes an unbiased estimator this code does not implement. Now bounded with the reason recorded, plus a companion showing the inflation shrinks as n grows.

Consolidating the four cohens_d copies is deliberately not done here — that structural change belongs to the lane that owns it.

4 · Guards, because a job that runs nothing still exits 0

collection errors == 0     a module that stops collecting fails loudly
collected >= 55            floor below the current 63

Proven non-vacuous: reintroducing the trapz bug yields 1 collection error and drops collection to 51 — both gates fire.

NeuralComposeEEG/ is absent on main until lane B. The guard says so explicitly and exits 0; once present, zero collected tests is an error.

Evidence

Tests/eval          30 collected + 4 errors  →  65 passed, 0 failures
trapz mutation      1 collection error, 53 collected — both guards fire
ADR gate (D0-R)     16 tests OK, PASS
swift build         unaffected
git diff --check    clean
CI                  build-and-test + python-eval SUCCESS

5 · Statistical-contract fixes (second commit)

Review of the first commit found two defects in functions this job now executes
for the first time. Both shared a shape: the code was more general than its
tests
, and the untested region was exactly where it was wrong.

  • Cohen's d pooled variance is df-weighted for unequal sample sizes.
    sqrt((var_a + var_b) / 2) matches the conventional estimator only at equal
    n, and the function accepts arbitrary lengths — every earlier test used
    equal groups. On n=3 against n=9 the reported effect moves 2.540 → 2.033.
    The regression test asserts toward the weighted value and away from the
    unweighted one, so reverting fails rather than silently changing a number.

  • ±Infinity for unequal constant groups is an explicit project policy, not
    the definition of Cohen's d. Persistence callers must map non-finite values at
    JSON/CSV boundaries: JSON has no Infinity literal (json.dumps emits a bare
    Infinity that strict parsers reject) and CSV has no convention.

  • Mann-Whitney uses exact for tie-free samples within a stated size cap,
    asymptotic otherwise. Previously it omitted method= and inherited SciPy's
    auto, whose rule has changed across releases and would move a reported
    p-value with no commit here. The earlier claim that the test pinned an exact
    p-value was true of the value but not the mechanism; it is true of both now.

    n=3 exact floor   0.100000   = 2/C(6,3)
    n=4 exact         0.028571   = 2/C(8,4)
    with ties         0.103754   asymptotic, as required
    
  • CI bounds numpy>=2,<3 and scipy>=1.14,<2. The trapz fix is
    numpy-2-specific and the Mann-Whitney contract is SciPy behaviour, so an
    unconstrained install could change a p-value with no commit here.

test_joint_embedding.py is the one exclusion, by name and reason: it imports mlx (Apple Silicon only) and needs a converted model.

Independent of #44, which stays parked for review.

aurascoper and others added 2 commits July 28, 2026 13:29
`Tests/eval` has seven modules. Exactly one ran in CI — `test_adr_references.py`,
via D0-R's direct step. The other six executed nowhere, on any branch, ever.

Turning them on required fixing four things they were hiding.

**The harness could not have worked.** Six of seven modules are bare `def test_*`
with no `unittest.TestCase`, so `python -m unittest` collects zero from them, and
`unittest discover` fails outright because `Tests/eval` has no `__init__.py`.
Either path reports success while running nothing. pytest is forced, not
stylistic.

**An import-time crash.** `Scripts/eeg_spectral.py` had

    _trapz = getattr(np, "trapezoid", getattr(np, "trapz"))

Python evaluates that default *before* the lookup it is a fallback for, so on
numpy 2.x — which removed `trapz` — it raised `AttributeError` at import, on
precisely the version the fallback existed to support. Resolved lazily now.
Three more modules failed collection for a missing scipy, which is a real
transitive dependency and is now installed rather than assumed.

With collection fixed, three assertions ran for the first time and failed.

**`cohens_d` returned 0.0 for an unbounded effect.** Zero pooled SD means both
samples are constant; `cohens_d([1,1,1,1], [5,5,5,5])` reported "no effect" for
the most extreme separation representable. The limit is signed infinity, and
identical constants are the only genuinely zero case. Fixed in all four copies
— consolidating them belongs to the lane that owns that structure, not here.

**The Mann-Whitney test was impossible.** At n=3 vs 3 the exact two-sided
p-value cannot go below 2/C(6,3) = 0.1, so `p < 0.05` was unreachable by
construction and no implementation could have passed it. It now asserts the
floor — which also pins that the implementation is exact rather than
normal-approximating — with a companion at n=4, where 2/C(8,4) ≈ 0.0286 clears
0.05.

**The CKA test described a different estimator.** Biased linear CKA inflates
badly when d >> n: 50 samples against 128 and 256 features scores ~0.78 on
independent data. `< 0.3` describes an unbiased estimator this code does not
implement. Now bounded above with the reason recorded, plus a companion showing
the inflation shrinks as n grows — so a later switch to an unbiased estimator
fails here and forces the expectation to be revisited deliberately.

Two guards, because a job that runs nothing still exits 0:

    collection errors == 0        a module that stops collecting fails loudly
    collected >= 55               floor below the current 63

Proven non-vacuous: reintroducing the trapz bug yields 1 collection error and
drops collection to 51, tripping both.

`NeuralComposeEEG/` is absent on main until lane B. The guard says so explicitly
and exits 0; once present, zero collected tests is an error.

    Tests/eval          30 collected + 4 errors  →  63 passed, 0 failures
    ADR gate (D0-R)     16 tests OK, PASS (1 historical warning)
    swift build         unaffected
    git diff --check    clean

`test_joint_embedding.py` is the one exclusion, by name and reason: it imports
mlx (Apple Silicon only) and needs a converted model.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two statistical-contract defects found in review of the D0 CI work. Both are in
functions the new job now executes for the first time.

**Cohen's d pooled the wrong way for unequal groups.**

    sqrt((var_a + var_b) / 2)

equals the conventional pooled estimator only when the two samples are the same
size, and this function accepts arbitrary lengths — so every unequal-n caller
was dividing by the wrong denominator. It survived because all the tests used
equal groups. Replaced with degrees-of-freedom weighting in all four copies. The
difference is not cosmetic: on n=3 against n=9 the reported effect moves 2.540 →
2.033, about 25%.

The new regression test is built so the two formulas disagree, and asserts
against the weighted value while asserting *away* from the unweighted one, so
reverting the fix fails rather than merely changing a number.

**The Mann-Whitney p-value was inherited, not chosen.** The call omitted
`method=`, taking SciPy's `auto` — a selection rule that has changed across
releases and would move a reported p-value with no commit here. Method is now
explicit: exact when there are no ties and the groups are small enough to
enumerate, asymptotic otherwise, with the size cap stated as policy rather than
borrowed from a library default. Ties make exact invalid, and a test now covers
that path.

The previous commit's claim that the test "pins an exact p-value" was true of
the value but not of the mechanism; it is true of both now.

    n=3 exact floor   0.100000   = 2/C(6,3)
    n=4 exact         0.028571   = 2/C(8,4)
    with ties         0.103754   asymptotic, as required

**±inf is documented as project policy**, not as the definition of Cohen's d,
along with the serialisation trap it creates: JSON has no Infinity literal and
CSV no convention, so callers that persist an effect size must convert at the
boundary.

CI now bounds the majors this gate's semantics depend on — numpy>=2,<3 and
scipy>=1.14,<2 in particular, since the trapz fix is numpy-2 specific and the
Mann-Whitney contract is a SciPy behaviour. Unconstrained installs would let a
major bump change a p-value silently.

    Tests/eval          65 passed, 0 failures   (was 63)
    trapz mutation      1 collection error, 53 collected — both guards fire
    ADR gate            16 tests OK, PASS
    swift build         unaffected
    git diff --check    clean

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aurascoper
aurascoper merged commit bc55d6c into main Jul 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant