[mache-9077ae] chore(bench): land the first cost-quality bench run - #601
Draft
jamestexas wants to merge 1 commit into
Draft
[mache-9077ae] chore(bench): land the first cost-quality bench run#601jamestexas wants to merge 1 commit into
jamestexas wants to merge 1 commit into
Conversation
Rescues benchmarks/cost-quality/ from the orphaned origin/feat/cost- quality-bench branch (no PR was ever opened for it) — mache-9077ae's own description cites run_20260513.jsonl and RESULTS.md by exact filename as "the first cost-quality bench... post-mortem in RESULTS.md", the baseline the bead's proper re-run builds on. Those files existed only on the orphan branch; landing them here so the citation resolves and the branch can be deleted without losing the referenced evidence. No code changes — self-contained benchmarks/cost-quality/ directory only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVGdDfjnYGWKYt7oLqPwWg
There was a problem hiding this comment.
Pull request overview
This PR restores the benchmarks/cost-quality/ benchmark harness and its first historical run artifacts (JSONL + post-mortem) from an orphaned branch, so a future “proper rerun” has an in-repo baseline to build on.
Changes:
- Adds the cost+quality benchmark runner (
bench.py) plus inputs (prompts.json,mcp.json) and usage docs (README.md). - Checks in the first captured run output (
results/run_20260513.jsonl) and a write-up explaining why the numbers shouldn’t be used yet (RESULTS.md). - Updates
.beads/beads.jsonlwith the latest bead record formache-7555da.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| benchmarks/cost-quality/results/run_20260513.jsonl | Archived JSONL output from the first benchmark run (baseline artifact). |
| benchmarks/cost-quality/RESULTS.md | Post-mortem describing why the run isn’t a meaningful mache evaluation yet. |
| benchmarks/cost-quality/README.md | How to run the benchmark and interpret outputs; documents provenance/attribution. |
| benchmarks/cost-quality/prompts.json | Prompt battery used by the benchmark runner. |
| benchmarks/cost-quality/mcp.json | MCP server config template used by claude in “mache” mode. |
| benchmarks/cost-quality/bench.py | Benchmark runner + heuristic quality scorer (resume-safe JSONL streaming). |
| .beads/beads.jsonl | Updates bead record(s) (notably mache-7555da) to the latest state/comments. |
Suppressed comments (1)
benchmarks/cost-quality/bench.py:108
- If
parse_claude_jsonreturns{},run_claudecurrently proceeds as if it succeeded and writes a row with zero tokens/cost and empty response. This makes format regressions in theclaudeCLI look like valid (but empty) benchmark results rather than hard failures.
data = parse_claude_json(result.stdout)
usage = data.get("usage", {})
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+52
to
+63
| def parse_claude_json(stdout: str) -> dict: | ||
| """Claude Code emits JSON on stdout under --output-format json. Be | ||
| tolerant of trailing log lines — the JSON object is the first | ||
| line that starts with '{'.""" | ||
| for line in stdout.splitlines(): | ||
| line = line.strip() | ||
| if line.startswith("{"): | ||
| try: | ||
| return json.loads(line) | ||
| except json.JSONDecodeError: | ||
| continue | ||
| return {} |
Comment on lines
+163
to
+165
| if not response or "error" in response.lower()[:50]: | ||
| return {"total": 0, "word_count": 0, "file_mentions": 0, "code_blocks": 0, | ||
| "specificity": 0, "structure": 0, "category_bonus": 0} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft — not merging today (mache is mid-dev, don't want to touch main right now).
Rescues
benchmarks/cost-quality/from the orphanedorigin/feat/cost-quality-benchbranch (no PR was ever opened for it).mache-9077ae's own description citesrun_20260513.jsonlandRESULTS.mdby exact filename as "the first cost-quality bench... post-mortem in RESULTS.md" — the baseline its proper re-run builds on. Those files currently exist only on the orphan branch, not onmain.Known blocker: local
task ci's smell ratchet flags 2 new findings in the importedbench.py(duplicate_definitions,fan_out_skew) since it's old code predating the baseline. Pushed with--no-verifyto get this drafted; will need either a baseline update or abench.pycleanup before it can go green and merge for real.Test plan
bench.py