Skip to content

tools: add one-command datapoint runner - #952

Open
gouravkargwal wants to merge 1 commit into
JustVugg:devfrom
gouravkargwal:tools/datapoint-runner
Open

tools: add one-command datapoint runner#952
gouravkargwal wants to merge 1 commit into
JustVugg:devfrom
gouravkargwal:tools/datapoint-runner

Conversation

@gouravkargwal

Copy link
Copy Markdown
Contributor

Adds c/tools/datapoint.py — a stdlib-only, one-command runner for the standard machine datapoint that docs/benchmarks.md asks for (engine load, capped greedy decode tok/s, iobench disk figures), printing a ready-to-paste markdown block.

What it measures, and how:

  • Exact decode tok/s — generation is capped at --max-new tokens, so the token count is known without re-tokenizing output. TEMP=0 greedy.
  • True cold vs warm — before measuring, the page cache is evicted (macOS purge if permitted, otherwise a portable temp-file write larger than RAM). iobench cold runs right after eviction, before decode re-warms the container. --no-evict skips it.
  • Disk — iobench 19 MB × 64, 8 threads, cold + buffered, with the macOS F_NOCACHE caveat [Bug]: iobench test command output misleading #86 called out in the emitted block.
  • Machine info auto-detected (CPU, RAM, cores, OS) per platform.

docs/benchmarks.md Quick Start now points at it as step 3 (steps renumbered).

The M3 datapoint row it produced is in a separate PR (docs: add Apple M3 16 GB OLMoE datapoint row, #951).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant