docs: regenerate benchmark charts with all 7 set implementations #none - #45
Open
freeformz wants to merge 1 commit into
Open
docs: regenerate benchmark charts with all 7 set implementations #none#45freeformz wants to merge 1 commit into
freeformz wants to merge 1 commit into
Conversation
The committed bench_stats.csv and charts predated SortedSet and BitSet, and bench_plot.py silently dropped any implementation missing from its hardcoded IMPL_ORDER list, so the README graphs only showed 5 of the 7 set types. - add benchmarks/regenerate.sh: one-step CSV + chart regeneration (with --plot-only mode and a uv fallback when matplotlib isn't installed) - bench_plot.py: cover all 7 implementations, warn-and-plot unknown ones instead of dropping them, switch to a colorblind-safe palette with per-series markers, and use a log y-axis so the fast-path implementations are visible alongside the element-wise ones - regenerate bench_stats.csv and all 11 charts with all implementations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VjUKQx2QYDuetR8z8GkGyU
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the benchmark documentation and artifacts so the README charts reflect all 7 set implementations (including SortedSet and BitSet) and can be regenerated reliably going forward.
Changes:
- Add
benchmarks/regenerate.shto regeneratebench_stats.csvand re-render charts in one step (with a--plot-onlymode). - Update
benchmarks/bench_plot.pyto include all implementations, avoid silently dropping unknown series, add markers + a colorblind-safe palette, and switch the y-axis to log scale. - Regenerate
benchmarks/bench_stats.csvwith fresh benchmark results including all implementations.
Reviewed changes
Copilot reviewed 3 out of 15 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| README.MD | Adds a quick “regenerate benchmarks” section pointing to the new script. |
| benchmarks/regenerate.sh | New helper script to (re)generate the benchmark CSV and charts. |
| benchmarks/bench_stats.csv | Updated benchmark dataset used to produce the README charts. |
| benchmarks/bench_plot.py | Updates plotting to handle all implementations and improve chart readability. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
88
to
92
| type_data = [r for r in op_data if r['type'] == typ] | ||
|
|
||
| for impl in IMPL_ORDER: | ||
| for impl in IMPL_ORDER + extra_impls: | ||
| impl_data = sorted( | ||
| [r for r in type_data if r['impl'] == impl], |
Comment on lines
+111
to
+113
| To regenerate `benchmarks/bench_stats.csv` and the charts below in one step (requires | ||
| python3 + matplotlib; the benchmark step takes on the order of an hour): | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The README's performance graphs only showed 5 of the 7 set implementations: the committed
bench_stats.csvpredatedSortedSetandBitSet, andbench_plot.pysilently dropped any implementation missing from its hardcodedIMPL_ORDERlist.benchmarks/regenerate.sh(new): one-step regeneration of the CSV and all charts. RunsTestBenchStatsacross every implementation registered inbenchImpls, so future set types are picked up automatically. Supports--plot-onlyto re-plot from the existing CSV, and falls back touv run --with matplotlibwhen matplotlib isn't installed.bench_plot.py: covers all 7 implementations; warns and plots (in black) any implementation found in the CSV but missing fromIMPL_ORDERinstead of silently dropping it; switches to a colorblind-safe categorical palette (validated: adjacent-pair CVD ΔE ≥ 8) with distinct per-series markers; y-axis is now log scale — the series span ~4 orders of magnitude (word-wiseBitSetops vs element-wise iteration), which a linear axis flattened into the x-axis.bench_stats.csvand all 11 charts from a fresh full run (sizes 10 → 1,000,000, 30 samples/op).BitSetappears in theintpanels only (integer-only implementation, as documented).Verification
./benchmarks/regenerate.sh --plot-onlyreproduces the committed charts from the committed CSV.implcolumn contains all 7 implementations; charts eyeballed — every series visible and distinguishable in both int and string panels.🤖 Generated with Claude Code
https://claude.ai/code/session_01VjUKQx2QYDuetR8z8GkGyU