feat(phase-55): Performance Truth — split setup vs query, honest percentiles - #33
Merged
Merged
Conversation
The 64.6s "TOC navigation" p50 was build_toc_segment (ingest-time MockSummarizer rollup) plus two RocksDB lookups. Query navigation is now timed separately: medium/warm single.toc p50 = 0.13ms (n=30). Vector model load, index build, and query are three steps. p90 requires n>=10; p99 requires n>=30; otherwise min/median/max. Warm = one setup + N queries. Schema 2 baseline regenerated from a real medium/warm/30 run.
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
RichardHightower
temporarily deployed
to
e2e-cli
August 30, 2026 17:35 — with
GitHub Actions
Inactive
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements v3.1 Phase 55 (Performance Truth). No new retrieval capabilities — the recorded numbers now match what the steps actually do.
Phase 54 (#32) and the v3.1 spec (#31) are already on main.
Root cause of the 64.6s TOC p50
perf_benchtimedbuild_toc_segment(MockSummarizer rollup of 240 events) under the step nametoc. Twoget_toc_nodelookups were in the same timer. p90/p99 were interpolated from 3 samples.What this makes true
*.toc= threeget_toc_nodelookups. Rollup is*.toc_build.single.tocwarm p50 = 0.13 ms (n=30, 240 events).single.toc_build= 76.7 s — that is the retired number, honestly labeled.vector_model_load(156 ms cached),vector_index(12.6 s),vector(query embed + HNSW).Execution evidence
Committed
crates/e2e-tests/benchmarks/latest.json(schema 2, 2026-08-30, linux/x86_64, debugcargo run, 240 synthetic events, 30 warm query samples).docs/benchmarks.mdrecords corpus, hardware, profile, and caveats.Unit tests in
perf_bench.rscover percentile gating, including that p99 of 3 samples is an interpolation artifact.Not in this PR
toc_build(ingest-time rollup). It is now labeled, not mis-sold as query.