Skip to content

feat(phase-55): Performance Truth — split setup vs query, honest percentiles - #33

Merged
RichardHightower merged 1 commit into
mainfrom
feature/phase-55-performance-truth
Aug 30, 2026
Merged

feat(phase-55): Performance Truth — split setup vs query, honest percentiles#33
RichardHightower merged 1 commit into
mainfrom
feature/phase-55-performance-truth

Conversation

@RichardHightower

Copy link
Copy Markdown
Contributor

Summary

Implements v3.1 Phase 55 (Performance Truth). No new retrieval capabilities — the recorded numbers now match what the steps actually do.

Phase 54 (#32) and the v3.1 spec (#31) are already on main.

Root cause of the 64.6s TOC p50

perf_bench timed build_toc_segment (MockSummarizer rollup of 240 events) under the step name toc. Two get_toc_node lookups were in the same timer. p90/p99 were interpolated from 3 samples.

What this makes true

Plan Claim Now
55-01 TOC query is navigation, not rollup *.toc = three get_toc_node lookups. Rollup is *.toc_build. single.toc warm p50 = 0.13 ms (n=30, 240 events). single.toc_build = 76.7 s — that is the retired number, honestly labeled.
55-01 Vector init vs query vector_model_load (156 ms cached), vector_index (12.6 s), vector (query embed + HNSW).
55-02 Percentiles survive scrutiny Default 30 query iterations. p90 omitted unless n≥10; p99 unless n≥30. Setup steps report min/median/max only.
55-02 Cold vs warm are structural Warm: one setup, one discarded warmup, N query samples. Cold: new store per iteration.

Execution evidence

Committed crates/e2e-tests/benchmarks/latest.json (schema 2, 2026-08-30, linux/x86_64, debug cargo run, 240 synthetic events, 30 warm query samples). docs/benchmarks.md records corpus, hardware, profile, and caveats.

Unit tests in perf_bench.rs cover percentile gating, including that p99 of 3 samples is an interpolation artifact.

Not in this PR

  • Release-profile vector numbers (debug Candle query-embed is ~4 s; do not quote as production HNSW).
  • Speeding up toc_build (ingest-time rollup). It is now labeled, not mis-sold as query.
  • Phases 56–58.

The 64.6s "TOC navigation" p50 was build_toc_segment (ingest-time
MockSummarizer rollup) plus two RocksDB lookups. Query navigation is now
timed separately: medium/warm single.toc p50 = 0.13ms (n=30).

Vector model load, index build, and query are three steps. p90 requires
n>=10; p99 requires n>=30; otherwise min/median/max. Warm = one setup +
N queries. Schema 2 baseline regenerated from a real medium/warm/30 run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants