Not a todo. Parking a line of thought so it is not lost.
The question: can mill tell, from its own run history, that a repo is getting harder to work in?
Everything needed is already recorded and nothing prunes it. stage_attempts has one row per launch with the stage, attempt number, strike_charged, struck_stage, model, skill_version, tokens, and start and finish times. runs has the repo, subject number, route, and created_at. So the history accrues whether or not anything gets built.
The idea: if issue 40 on a repo takes three attempts where issue 10 took one, the repo got harder to work in between them. This is the only read on code quality available that does not depend on a model judging its own work, which matters because mill's reviewer is already known to be miscalibrated. On its first real run review:code rated all five of its objections low, while an independent session on the same commit rated one high and three medium and found three defects mill's reviewer missed entirely.
Four numbers, in order of how much to trust them.
Attempts and strikes per run. Grounded in whether tests passed. The primary number.
Objections raised by review:plan and review:code, counted, not weighted by severity. These move earlier than a failed attempt, because a reviewer notices mess before that mess breaks anything. Do not chart severity; on current evidence severity describes the reviewer, not the code. Objections live in verdict_json rather than in columns, so counting them needs JSON extraction.
Tokens per run. Note that tokens_out is NULL for attempts killed before their result line, so the worst runs have missing data.
Wall time per run, from started_at and finished_at. Easiest to feel, but it moves with rate limiting, stalls, and API latency, none of which reflect the codebase.
Rule for all four: only compare runs that used the same model and the same skill_version, both of which are on stage_attempts. Changing a prompt moves these numbers, and that is not the code decaying.
Open questions, none answered.
How to separate a rising line that means the code decayed from one that means the recent issues were harder. Splitting by route helps a little, since plan and fast runs are not comparable. Watching only how often runs end up blocked may be steadier than the raw count.
Whether a fixed probe beats watching a trend: take one real issue, run implement on Sonnet instead of Opus, record whether it lands. The same test every time, so the points compare. Costs one run, gives one clear answer, needs no trend.
Whether any of this is worth anything before a few dozen runs exist across at least two repos. Probably not.
Background: https://docs.google.com/document/d/1hFbebil3ohzX9ffLSs0X8uSa_6-md0oyAujF5ZSKmyk/edit?usp=sharing (@dexhorthy) argues that models cannot maintain code quality over time and that benchmarks cannot see it. Part 3 measures it as a falling pass rate across successive checkpoints on one codebase. Rising attempts per issue is the same measurement, on real work.
Not a todo. Parking a line of thought so it is not lost.
The question: can mill tell, from its own run history, that a repo is getting harder to work in?
Everything needed is already recorded and nothing prunes it. stage_attempts has one row per launch with the stage, attempt number, strike_charged, struck_stage, model, skill_version, tokens, and start and finish times. runs has the repo, subject number, route, and created_at. So the history accrues whether or not anything gets built.
The idea: if issue 40 on a repo takes three attempts where issue 10 took one, the repo got harder to work in between them. This is the only read on code quality available that does not depend on a model judging its own work, which matters because mill's reviewer is already known to be miscalibrated. On its first real run review:code rated all five of its objections low, while an independent session on the same commit rated one high and three medium and found three defects mill's reviewer missed entirely.
Four numbers, in order of how much to trust them.
Attempts and strikes per run. Grounded in whether tests passed. The primary number.
Objections raised by review:plan and review:code, counted, not weighted by severity. These move earlier than a failed attempt, because a reviewer notices mess before that mess breaks anything. Do not chart severity; on current evidence severity describes the reviewer, not the code. Objections live in verdict_json rather than in columns, so counting them needs JSON extraction.
Tokens per run. Note that tokens_out is NULL for attempts killed before their result line, so the worst runs have missing data.
Wall time per run, from started_at and finished_at. Easiest to feel, but it moves with rate limiting, stalls, and API latency, none of which reflect the codebase.
Rule for all four: only compare runs that used the same model and the same skill_version, both of which are on stage_attempts. Changing a prompt moves these numbers, and that is not the code decaying.
Open questions, none answered.
How to separate a rising line that means the code decayed from one that means the recent issues were harder. Splitting by route helps a little, since plan and fast runs are not comparable. Watching only how often runs end up blocked may be steadier than the raw count.
Whether a fixed probe beats watching a trend: take one real issue, run implement on Sonnet instead of Opus, record whether it lands. The same test every time, so the points compare. Costs one run, gives one clear answer, needs no trend.
Whether any of this is worth anything before a few dozen runs exist across at least two repos. Probably not.
Background: https://docs.google.com/document/d/1hFbebil3ohzX9ffLSs0X8uSa_6-md0oyAujF5ZSKmyk/edit?usp=sharing (@dexhorthy) argues that models cannot maintain code quality over time and that benchmarks cannot see it. Part 3 measures it as a falling pass rate across successive checkpoints on one codebase. Rising attempts per issue is the same measurement, on real work.