Detect benchmark coverage gaps and state a verdict (#206) - #211
Merged
Conversation
Validating #88 spent four ten-minute smoke suites on a gate that never benchmarks list_projects, so none of them measured the changed path. The question was finally answered by a ten-minute interleaved A/B. This adds the two pieces that were missing. BenchmarkCoverage maps changed files to the public tools they affect and reports which of those the suite can actually measure. Only three of nine public tools are benchmarked today, so the gap is common rather than exotic. classify now prints the assessment and explains why a green run does not clear a change to an unmeasured tool. BenchmarkVerdict turns two sets of samples into a stated outcome and a merge recommendation. A direction is claimed only when the change clears both a relevance floor of 5% and three standard errors, so a clean but tiny difference reads as indistinguishable and a large difference on a noisy host reads as inconclusive rather than as a win. Two outcomes exist specifically because of what happened here: INCONCLUSIVE for a host whose variance swamps the effect, and NOT COMPARABLE for arms that did not return equivalent results -- the state #171 was in before a field bug was found, where the fast arm was fast because it returned nothing. Tests replay the real cases: #171's measured numbers verdict IMPROVEMENT, #88's unchanged path INDISTINGUISHABLE, the observed 3-4x host swings INCONCLUSIVE, and pre-bugfix #171 NOT COMPARABLE. Replaying #88's file list through classify now surfaces the list_projects gap that four smoke suites missed. Validation impact: performance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deverman
added a commit
that referenced
this pull request
Aug 2, 2026
) BenchmarkVerdict shipped in #211 with no caller: 130 lines of library plus ten tests that nothing in the product invoked. The part of #206 that earns its place is the coverage warning wired into classify, which catches the gap that cost four smoke suites on #88. Deleting the rest. #207 is closed and drops out of the delivery order. Its fix changed 233 lines while fixing nothing reachable -- the rootOnly gap already has a regression test from #88, and the batch path added in #171 does not touch the cache at all. #206 stays, rescoped to the work that is actually left: benchmarks for the six tools the suite cannot measure, to be added when a change to one of them needs measuring rather than in advance. Adds two standing decisions drawn from both mistakes: fix defects that have a reproduction rather than refactoring latent risk, and do not build tooling ahead of a caller. Validation impact: docs. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the core of #206.
The problem, restated from evidence
Validating #88 spent four ten-minute smoke suites on a gate that never benchmarks
list_projects— so none of them measured a line of the changed path. The question was ultimately answered by a ten-minute interleaved A/B. Two things were missing: knowing the suite could not measure the change, and a rule for turning numbers into a decision.What this adds
BenchmarkCoveragemaps changed files to the public tools they affect, and reports which of those the suite can actually measure. Only 3 of 9 public tools are benchmarked today, so the gap is the common case rather than an exotic one.classifynow prints the assessment:That output is a replay against #88's real file list — the warning that would have saved four suites and about two hours.
BenchmarkVerdictturns two sample sets into an outcome plus a merge recommendation. A direction is claimed only when the change clears both a 5% relevance floor and three standard errors, so:INDISTINGUISHABLE, not a win;INCONCLUSIVE, not a regression.Two outcomes exist specifically because of what happened during this work:
INCONCLUSIVE— host variance swamps the effect. The 3-4x swings seen while validating Expose project folder membership and root filtering in list_projects #88 land here instead of producing a confident verdict.NOT COMPARABLE— the arms did not return equivalent results, so timings are meaningless. This is exactly the state Resolve multiple project or tag names in one bounded query #171 was in before its field bug was found: the fast arm was fast because it returned nothing. Without this outcome, benchmarking a broken implementation yields a beautiful, false improvement.Tests replay the real cases
334 tests pass (17 new).
validate --impact performancepasses with all semantic gates green.Not included
The remaining #206 scope, left for follow-up so this lands reviewable: adding actual benchmarks for the six uncovered tools, a
--baseline <ref>flag that builds both refs and interleaves samples automatically, and deriving the request deadline from observed p99. This PR makes the gap visible and the verdict stated; automating the comparison is the next slice.