Skip to content

Detect benchmark coverage gaps and state a verdict (#206) - #211

Merged
deverman merged 1 commit into
masterfrom
issue-206-benchmark-coverage
Aug 2, 2026
Merged

Detect benchmark coverage gaps and state a verdict (#206)#211
deverman merged 1 commit into
masterfrom
issue-206-benchmark-coverage

Conversation

@deverman

@deverman deverman commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Addresses the core of #206.

The problem, restated from evidence

Validating #88 spent four ten-minute smoke suites on a gate that never benchmarks list_projects — so none of them measured a line of the changed path. The question was ultimately answered by a ten-minute interleaved A/B. Two things were missing: knowing the suite could not measure the change, and a rule for turning numbers into a decision.

What this adds

BenchmarkCoverage maps changed files to the public tools they affect, and reports which of those the suite can actually measure. Only 3 of 9 public tools are benchmarked today, so the gap is the common case rather than an exotic one. classify now prints the assessment:

Benchmark coverage:
  ⚠️  list_projects — no benchmark in the suite
  ⚠️  list_tags — no benchmark in the suite

Benchmark coverage gap: list_projects, list_tags has no benchmark in the suite.
A passing benchmark run does not clear this change, because the suite never
exercises the path it modifies. Measure the changed tool directly with an
interleaved A/B against the baseline ref before drawing a performance conclusion.

That output is a replay against #88's real file list — the warning that would have saved four suites and about two hours.

BenchmarkVerdict turns two sample sets into an outcome plus a merge recommendation. A direction is claimed only when the change clears both a 5% relevance floor and three standard errors, so:

  • a statistically clean but tiny difference reads as INDISTINGUISHABLE, not a win;
  • a large difference on a noisy host reads as INCONCLUSIVE, not a regression.

Two outcomes exist specifically because of what happened during this work:

Tests replay the real cases

historical case expected result
#171 measured numbers (2402ms → 432ms) IMPROVEMENT
#88 unchanged path (385ms vs 390ms) INDISTINGUISHABLE
observed 3-4x host swings INCONCLUSIVE
pre-bugfix #171 (arms disagree) NOT COMPARABLE
2% change with near-zero variance INDISTINGUISHABLE

334 tests pass (17 new). validate --impact performance passes with all semantic gates green.

Not included

The remaining #206 scope, left for follow-up so this lands reviewable: adding actual benchmarks for the six uncovered tools, a --baseline <ref> flag that builds both refs and interleaves samples automatically, and deriving the request deadline from observed p99. This PR makes the gap visible and the verdict stated; automating the comparison is the next slice.

Validating #88 spent four ten-minute smoke suites on a gate that never
benchmarks list_projects, so none of them measured the changed path. The
question was finally answered by a ten-minute interleaved A/B. This adds
the two pieces that were missing.

BenchmarkCoverage maps changed files to the public tools they affect and
reports which of those the suite can actually measure. Only three of nine
public tools are benchmarked today, so the gap is common rather than
exotic. classify now prints the assessment and explains why a green run
does not clear a change to an unmeasured tool.

BenchmarkVerdict turns two sets of samples into a stated outcome and a
merge recommendation. A direction is claimed only when the change clears
both a relevance floor of 5% and three standard errors, so a clean but
tiny difference reads as indistinguishable and a large difference on a
noisy host reads as inconclusive rather than as a win. Two outcomes exist
specifically because of what happened here: INCONCLUSIVE for a host whose
variance swamps the effect, and NOT COMPARABLE for arms that did not
return equivalent results -- the state #171 was in before a field bug was
found, where the fast arm was fast because it returned nothing.

Tests replay the real cases: #171's measured numbers verdict IMPROVEMENT,
#88's unchanged path INDISTINGUISHABLE, the observed 3-4x host swings
INCONCLUSIVE, and pre-bugfix #171 NOT COMPARABLE. Replaying #88's file
list through classify now surfaces the list_projects gap that four smoke
suites missed.

Validation impact: performance.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@deverman
deverman merged commit 5b4e94e into master Aug 2, 2026
4 checks passed
@deverman
deverman deleted the issue-206-benchmark-coverage branch August 2, 2026 17:24
deverman added a commit that referenced this pull request Aug 2, 2026
)

BenchmarkVerdict shipped in #211 with no caller: 130 lines of library plus
ten tests that nothing in the product invoked. The part of #206 that earns
its place is the coverage warning wired into classify, which catches the
gap that cost four smoke suites on #88. Deleting the rest.

#207 is closed and drops out of the delivery order. Its fix changed 233
lines while fixing nothing reachable -- the rootOnly gap already has a
regression test from #88, and the batch path added in #171 does not touch
the cache at all.

#206 stays, rescoped to the work that is actually left: benchmarks for the
six tools the suite cannot measure, to be added when a change to one of
them needs measuring rather than in advance.

Adds two standing decisions drawn from both mistakes: fix defects that
have a reproduction rather than refactoring latent risk, and do not build
tooling ahead of a caller.

Validation impact: docs.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant