Plan 3a — autonomy: the poller, the supervisor, and board writes - #1
Open
slowernet wants to merge 38 commits into
Open
Plan 3a — autonomy: the poller, the supervisor, and board writes#1slowernet wants to merge 38 commits into
slowernet wants to merge 38 commits into
Conversation
The design put pid, pid_started_at and host_boot_at on stage_attempts, but Mill::Ledger writes that row when the attempt ends, in one insert. While a stage is actually running there is no row, so a supervisor reaping a live process would have nothing to identify it against. runs.pgid was already there for this reason; the other three now sit beside it, along with board_item_id, which the poller learns and the finishing run needs. Mill::Spawn reports the identity through on_spawn the moment the group exists, rather than only in its return value. A callback that raises — a locked database being the likely way — would otherwise leave a running process group that nothing has recorded, so the group is reaped before the exception is allowed out. Settings are parsed rather than coerced. MILL_CONCURRENCY=lots through to_i is 0, which makes at_cap? true forever: mill claims nothing, with every check green and nothing in the log. Mill.setting_int and setting_float range-check and fall back with a warning naming the value. 320 runs, 1270 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fresh worktree holds tracked files only, so .env and config/master.key are absent and a repo whose suite needs them fails identically on both attempts — which reads as the stage being wrong and is not. Mill::Rules.env_for was the hook the design left for this and carried one variable; it now carries the repo's own environment, and GH_TOKEN for pr and push alone. Setting GH_TOKEN is enough to re-point both gh and git push at the scoped credential, because the helper the runbook configures asks gh, and gh prefers GH_TOKEN over its stored login. Values shorter than sixteen characters are injected but never redacted. The scrubber gsubs literally over every log line, and the log is stream-json mill parses back: an env file carrying DEBUG=true would turn "success":true into "success":[redacted], which stops being JSON, and the stage would then read as having produced no verdict and be charged a strike for mill's own scrubber. No real credential is that short. A secrets file whose mode has drifted off 600 is refused rather than read. 333 runs, 1294 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scanning ~/code for a matching origin is a laptop assumption: a server keeps no working copies, so the clone has to come from somewhere. MILL_CLONES lists directories of clones you keep — defaulting to ~/code on Darwin and empty on Linux — and mill clones into ~/.mill/clones when nothing matches. Two matches block the item rather than resolving. Choosing silently commits the whole run to a checkout the operator did not pick, and the run then works somewhere they are not looking. git clone and git init have no repository to run inside, so they cannot go through Git.run, which passes -C. They live in Mill::Git anyway: that module being the only place mill runs git is what makes the rules about forcing a checkout enforceable rather than aspirational. 345 runs, 1327 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…issing Lazy and per-item: resolve or make the clone, set gc.auto and maintenance.auto to 0 so a stage's commit cannot trigger a gc that rewrites refs other runs are holding, read .mill.yml, and check the secrets it names are present. .mill.yml is read with git show against the base branch, never from a checkout. An agent can edit that file in its own worktree, and that edit must not weaken the next run. Nothing here raises at the caller. A repo with a malformed .mill.yml, an unquoted date in it, or a missing secret blocks that one item and names what is wrong; left to raise, one badly configured repo would wedge the poller in a retry cycle and stop every other item too. 357 runs, 1365 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mill has never written a Status. Every write is a network call that can fail, and nothing else re-drives one: the poller only ever asks which items are Ready. So a run that blocks while the network is down would show Running forever, and because a comment's meaning depends on Status, the answer to its questions would never be read as an answer. Board records the decision first and confirms it second, because a crash between the two must leave something redrive can act on. confirm re-reads the label rather than taking it as an argument, and stamps board_status_at only if the decision has not changed underneath it — redrive runs in the poller thread while run threads decide, and stamping a stale label is worse than not writing at all, since that stamp is the only thing that would have caused a retry. Only unreachability is swallowed. A board missing a Status option is a configuration error, raises, and is checked once when the ids resolve rather than at the moment mill first needs the option it lacks. 371 runs, 1402 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four things stand between an item and a worktree and each has a different answer. The cap means try later. A branch a live run holds means skip, and say so once — a blocked run holds its branch indefinitely by design, so an item queued behind one waits forever and silence reads as mill ignoring you. A branch checked out in your own clone blocks the item and names it: mill does not switch your clone and does not force the worktree, because two live checkouts of one branch can diverge the ref without either side noticing. A stale lock or worktree admin entry is cleared and claiming carries on. The row and the worktree are inserted together. A row committed before a worktree that then fails to appear is a running run with no process and no thread: nothing reaps it, because there is nothing to identify, and it holds a concurrency slot for as long as the database survives. Two of those and mill claims nothing ever again with every check green. The worktree is not transactional, so a partial one is removed by hand before the error is re-raised. checked_out? does not rescue. A git failure means mill does not know whether the branch is checked out, and answering "it is not" is the rescue-into-a-pass that produces the two-checkout case above. 385 runs, 1434 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A route walk takes tens of minutes. A supervisor that walked one would claim a single item and then stop reconciling, so each claimed run gets its own thread and the cap is what bounds them. Until now nothing told GitHub anything. Mill::Run returned a blocked run's questions and only rake mill:run printed them, which is no use to anyone who has walked away — so the supervisor posts them on the subject, names the pull request when a run finishes, and says plainly when one fails. A thread that dies marks its run failed rather than leaving it running forever, because a run stuck in running is a concurrency slot nothing else releases. Mill::Run.adopt builds a Run from an existing row, and restores prior verdicts only when that row is blocked — a fresh run has nothing to restore. resume is now adopt plus call. 393 runs, 1450 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mill::Runner wrote current_stage only when it halted, so for the whole time a stage was actually running the column was nil. Everything in this task depends on knowing which stage a live run is in, so the runner now records it before launching — without that, interrupt has nothing to charge and the reaper silently does nothing, which is exactly the shape of failure that looks like the feature working. Three branches, in order, and nothing is signalled on the boot time alone: kern.boottime moves when NTP corrects the clock, which it does routinely on waking, so the live process is what settles it. :ours means a thread is walking the run right now, not merely that no process is recorded. pid and pgid are nil for the whole gap between two stages — five times over on the plan route — so reading nil as "mill has this in hand" would strand any run mill was restarted during, and each one holds a concurrency slot nothing releases. Interrupting is half the job. A run interrupted and not re-entered stays running with no thread forever, and the poller skips its item because it has an active run. reap now restarts it, unless interrupt just blocked it for hitting the interruption cap, in which case it is waiting for a person. A running row with no current_stage raises rather than quietly charging nothing, because it means something above lost track of the run. 407 runs, 1473 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One idempotent question: which items are Ready with no active run. It needs no dedupe key and heals itself when mill crashes mid-transition. The label design that preceded it consumed change events, and four bugs came from that shape — a relabelled issue deduped permanently, an item Ready and Running at once, nothing clearing Running when a run was killed, and no label change reaching a terminal state. A single-select cannot express any of them. Mill::Poller takes its supervisor as a required keyword and never builds one. There must be exactly one per process: it alone knows which process groups mill spawned and which runs have a live thread, and a second instance answers "none" to both — a reaper holding that belief kills every healthy stage it finds. :no_branch and :no_spec carry no questions, because the answer is a branch or a file rather than a decision. The generic block comment would post a heading over an empty list and read as a bug in mill, so those two are told plainly instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comments are genuinely events, unlike board state, so they are consumed rather than reconciled. Two rules keep that honest and each has a specific failure behind it. The cursor advances inside the same transaction as the inserts, so a fetch that stops partway writes no cursor and loses nothing. And the cursor is actually sent to GitHub as `since`, which is the point of keeping one: without it a run blocked for a week on a busy issue re-fetches its entire comment history every tick and ends in a secondary rate limit that wedges the poller. The marker is matched at the start of a line that is not blockquoted. GitHub quote-reply copies the source markdown including HTML comments, so a whole-body search would silently discard the only channel in the design that reaches a person. The sweep is bounded to subjects with a live run, not every issue in every repo the board touches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Board Status decides what a comment is. While an item is Blocked every comment on it is an answer and none of them starts a run — otherwise your answer tries to start a second run, the uniqueness index refuses it, the event retries until it dies, and the blocked run sits waiting for an answer that had already arrived. The event is marked processed and committed before the thread is spawned, never inside the same transaction as it. A thread started inside an open transaction writes to the same SQLite file from another connection while this one holds the write lock; if the commit is what fails, the event rolls back to pending while the thread it already spawned keeps running, and the next dispatch starts a second walker on the same run — two agents in one worktree, reached from inside the check built to prevent it. Marking first would drop the answer if start then failed, which is what the same-transaction rule exists to prevent, so fail_event is the compensation: it returns the event to pending. running? is what stops a retry becoming a second walker, and the cap binds on resumes because a blocked run is not counted as running until its own thread says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
app.rb and config.ru in reps shape: app.rb requires and defines the Roda subclass, config.ru stacks what it needs and ends with App.freeze.app. Plan 4 fills in app/routes and the views against a boot path that already runs. One supervisor, shared. Mill::Workers builds it and hands the same instance to Mill::Poller, because it is the only object holding which process groups mill spawned and which runs have a live thread — a second instance answers "none" to both, and a reaper believing that kills every healthy stage it finds about thirty seconds into every run. The backoff cap is applied after the multiplier, not before. Before it, the real ceiling was three seconds rather than five minutes, so an expired token would have retried twelve hundred times an hour indefinitely. Heartbeats are written under a mutex and read as a snapshot: two worker threads write while a Puma thread reads. The workers are built at class definition and started in config.ru. Starting threads as a side effect of require means any test, console or rake task that loads app.rb silently begins polling a real board. App.freeze is also why they are built rather than memoised on first use. Verified by booting it: both threads alive with fresh heartbeats, bound to 127.0.0.1:9494, nothing raised. 451 runs, 1553 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything Plan 3a added has a way of being quietly wrong, and a red doctor is meant to be what stops it. A secrets file whose mode has drifted off 600 still works — that is the problem. A clone root that does not exist silently becomes "clone it myself" for every repo, and mill then works in a checkout nobody is looking at. A non-loopback bind with an empty admin list puts a kill switch and a worktree deleter on the network with nothing in front of them. And a board whose Status field is missing an option fails at the moment it matters — a run blocking, or finishing — rather than at setup. Checked against the real machine: the three local checks pass, and the board check is red only because MILL_PROJECT is unset. 460 runs, 1573 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both found by running the rehearsal against a real, freshly created board rather than by reading the docs back. Section 3 said to delete the Status field and recreate it with mill options. GitHub refuses both halves: "Only custom fields can be deleted" to the delete, and "Name cannot have a reserved value" to the create. The built-in Status field can be neither removed nor replaced. What does work is updateProjectV2Field, which swaps the whole option list in place; the runbook now carries the working mutation, and the warning that any option omitted from that list is deleted along with its value on every item. Section 4 said to turn off the built-in workflows without saying that this is the one setup step with no API at all. The schema exposes enabled for reading, which is how doctor checks it, but there is no mutation to turn one off — deleteProjectV2Workflow exists and is a different thing. It also now names the six a default project actually ships enabled, rather than a remembered list that did not match. The Status-options check added in Task 13 earned its place immediately: it caught a board still carrying Todo / In Progress / Done, which would have failed at the moment a run first blocked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both from running Plan 3a against a real board rather than a fixture. Doctor reported the schema green against a database three migrations behind. Its check asserted that five tables exist, and they did — what was missing was a column. So the first claim raised "table runs has no column named board_item_id" inside a worker thread, on every tick, while the one command whose job is to say what is wrong before you start said nothing was wrong. It now compares schema_info.version against the highest migration on disk and names the rake task. Mill::Workers built the shared supervisor without a board, so every @board&.want in claim, finish and interrupt was a silent no-op. mill would have run the whole pipeline correctly and never written a Status: the item sits on Ready while a run works, finishes, and opens a pull request, and because a comment only means an answer while the board says Blocked, no blocked run could ever have been resumed. One board is now built once and handed to both the supervisor and the poller. The adversarial review caught the two-supervisor bug and missed this one, which is the same wiring seam one layer down. Worth recording: the failed claim left no orphan row. The insert raised inside the transaction added in Task 6, run_id stayed nil, and nothing was left holding a concurrency slot — which is exactly the CRITICAL that transaction was written for, working on its first real failure. 462 runs, 1575 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found while checking what would happen if I restarted mill against a run that was mid-flight — before doing it, not after. Runner#stage is `@stage ||= route_stages.first`, and @stage was only ever assigned inside restore, which Mill::Run.adopt called only for a blocked run. So the path added in Task 8 — the reaper interrupting a run and starting it again — would have begun the route over. A run interrupted at implement would re-run triage, plan and review:plan: `plan` writes its artifact a second time, and the ledger counts fresh attempts against stages that had already passed clean. restore is now the blocked-run path — guard, reload, and the one sanctioned strike reset — and reload is the part both callers need. A run the supervisor interrupted only reloads: it resumes at the stage it was in with nothing forgiven, because nobody answered anything. Demonstrated rather than argued: the same run reports triage without reload and implement with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first real block posted this to the issue:
Blocked at ``: .
Mill::Supervisor#walk returned the run row from the database, and finish
announces from the runner state — which stage stopped, why, and the
questions it batched. A row carries none of those, so every field came out
nil and the comment said nothing.
Blocking is the mechanism the whole design rests on: the line can always
stop, and asking is free. That comment is the only channel that reaches a
person once you have walked away, so a block that reaches GitHub carrying
nothing is worse than a crash — the board says Blocked, the worktree waits,
and there is no way to learn what for.
Every existing test called finish with a hand-made state hash, which is
exactly why nothing caught it: the walker itself was never driven. The new
test drives the real walker through a scripted launcher and asserts on what
reaches GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught by looking at a live resumed run: status=blocked while pid 58621 was a claude process actively planning. Nothing moved a resumed run back to running. Mill::Runner only writes the status column when it halts or finishes, and the supervisor set it when it claimed — so a run answered from a comment kept the row it had when it stopped, for the whole rest of its route. That lies in three ways at once. The board keeps saying Blocked while mill works, finishes, and opens a pull request. The run does not count against the concurrency cap, so the cap under-counts by however many resumed runs are in flight. And reap queries running rows only, so if that stage died the run was stranded with nothing able to recover it — the exact stranding the identify fix was written to prevent, reached by a different door. The transition belongs to the supervisor, which owns the run lifecycle: it flips the row and tells the board before spawning the walker. 467 runs, 1587 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file read as though all of it worked, which matters more here than elsewhere: CLAUDE.md points people at it first when terminology confuses them, and it had no equivalent of the design doc Where this stands. Five passages described unbuilt behaviour in the present tense. The fast path was the damaging one — it told you to set Ready on an issue with no linked branch, and mill now answers that with a comment telling you to run gh issue develop and commit a spec. Following the doc got you the opposite of what it promised. The mill: PR marker, review-comment and red-check triggers, and Review: Deep are the others; all need routes that have no prompts. They are marked rather than deleted, because the vocabulary is what you will want when Plan 5 lands. Four things were simply stale. Repo said "resolved local clone path" from when mill could only use a clone you already had; it now searches MILL_CLONES and clones into ~/.mill/clones when nothing matches. Project ids are resolved on first write and memoised per process, not at bootstrap. Event has a second terminal state, no_route. And the branch-checked-out warning named ~/code/<repo>, which is the laptop assumption inverted this morning. One gap added rather than corrected: nothing said MILL_PROJECT is the project number and not its node id. That cost real time today — a node id looks like an id, and passing it gets a 404 that suggests nothing about which of the two was wanted. The conversion query is now in the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
triage let a spec through that said "add a report of stock levels, it should be useful, make it fast", and plan spent 29,572 output tokens discovering it was underspecified. triage was right by its own prompt — it was asked about scope and route, and the spec is unambiguously one small feature with a spec on its branch. Two changes. The opener said "when the answer is not obvious, block" and then listed two closed decisions, which tells a stage two different things. It now names the three it judges and says nothing else is its to judge. And it gains a third decision, deliberately narrow: block only when a spec names no exact value anywhere, says nothing about failure, and contains nothing testable — all three together. That is the hopeless case, and it is visible without reading a line of code. The bar is high on purpose. Of the three questions plan asked about that spec, two were answerable from the spec alone but the third — where a threshold lives, given Item carries only sku, name and count — needed the codebase and was the one that changed the public API. A cheap gate catches two, you answer, and plan blocks on the third anyway: two round trips instead of one, which costs a human more than the tokens it saves. So a spec failing one or two of the three goes to plan, and the prompt says why. The prompt tests now compare against whitespace-collapsed text, because every phrase worth asserting on straddles a line break and writing the assertion around the wrapping breaks the moment anyone rewraps a paragraph. 469 runs, 1597 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/reference is for rules you follow; docs/superpowers holds specs and plans. Neither fits an investigation or a contract for something nobody has built yet, so those two were filed as reference or left loose in tmp/. docs/notes/ takes them: nothing in it is binding. - admin-ui-frontend.md moves out of reference. It is the contract Plan 4 will build against, not a rule anyone follows today. - 2026-08-13-agent-convergence-strategies.md moves out of tmp/, where it was one disk failure from gone. One reference updated in the design doc, and the README gains a section so the directory is discoverable rather than folklore. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The poller hit the GraphQL limit during the rehearsal, logged it, backed off and stayed alive — which is the supervising loop working. But it backed off with the same exponential curve it uses for everything else, capped at five minutes, and a GraphQL window can be forty away. That is eight more attempts failing for a reason mill already knew. A rate limit is the one failure that says exactly when to try again, so guessing is strictly worse. Mill::Github#rate_limit_reset asks; the endpoint is itself exempt, so it costs nothing. The wait is bounded at both ends — never shorter than a tick in case the reset just passed, never longer than an hour in case the clocks disagree — and falls back to the old cap when GitHub will not say. The design already had this rule for stages: a rate-limited stage is waiting rather than working, and its deadlines stop counting. This is the same rule for mill own API access, which was the half that had none. The default tick goes from 30 seconds to 60. The board is a queue you touch by hand, so a minute costs nothing in responsiveness and halves what mill spends against a budget measured in points rather than calls. One Github instance is now built by Workers and shared with the board, the supervisor and the poller, which is the same reason the supervisor and board are shared: three of anything is three sets of state to disagree. Caveat on the measurement that prompted this: monitors polling the board every 25 seconds ran alongside the poller for two hours, so the exhausted budget says little about what mill costs unattended. That number needs a night with nobody watching. 474 runs, 1603 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Where this stands flips five rows: the poller, the supervisor, board writes, secrets injection, and the web UI boot path. Each says what is built and what is still missing rather than just "built" — the supervisor has no power assertion, secrets injection has never met a repo that declares any, and the UI is a boot path with one route. The rehearsal record sits beside Plan 2 in the build order. Two pull requests nobody opened by hand, a crash test that recovered twice from an orphaned process group, and nineteen and a half hours unattended overnight surviving eight transient API failures. And the finding worth more than the code: ten defects, six in that day own work, none of which the fixture suite could have caught. Four were the same shape — each component correct alone, each tested alone, the defect in the handoff between two of them. The lesson is recorded where the next plan will read it. An adversarial review of this plan code, before any of it existed, found twelve defects including four that would have stopped the factory silently. It caught that two supervisor instances would make the reaper kill healthy stages. The fix was to share one instance — and the shared instance was built without a board. Reviews catch the layer they are looking at; only running the assembled thing catches the seam below it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The five-hour window closed mid-run and mill charged plan a strike for it. That breaks a stated safety invariant — never charge a strike for something the machine did to a stage — and the design says in as many words that a rate-limited stage is waiting rather than working. The plumbing existed at both ends and nothing joined them, which is the fourth time this branch has found that shape. Mill::Stream already parsed rate_limit_event and exposed rate_limited?. Mill::Ledger::COST already had rate_limited at zero attempts and zero strikes. But Attempt had no delegate and classify never asked, so a refused launch — which exits non-zero — fell through to "return :crashed unless result.success?" and paid. classify now checks it first, ahead of a failed resume, because mill never got far enough to try the session. The runner would then have hot-looped: :rate_limited fell to the generic branch and re-ran immediately, hammering a door that does not open for hours. It now waits for the window the CLI named in resetsAt, which Stream keeps for that purpose — bounded at a minute in case the reset just passed and at an hour in case the clocks disagree, and falling back to the hour when the CLI does not say. Waiting in the run thread is correct rather than lazy: the run is waiting, not working, and the supervisor leaves a run alone while its thread is alive. Free is not unlimited, and a refused launch inserts no row, so there is no counter in the database to bound it. The cap is held in the runner: four waits, then the run blocks and says plainly that nothing was charged. Found because the boundary suite could not run — it hit the same limit, and three of its tests failed rather than passing on an empty transcript, which is the silence-is-never-success rule holding. 483 runs, 1617 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An adversarial review of this branch found four tests that pass for reasons having nothing to do with what they claim to assert. Each sits on a real bug, and each reported green while that bug shipped. test_a_failing_callback_does_not_orphan_the_process_group passed only because this machine can read kern.boottime. Where the host cannot, announce_spawn's rescue calls Spawn.reap, which returns :unknown_boot without signalling — so the group is orphaned and the raise parks behind the child for thirty seconds. The new test forces boot_time to nil and asserts both halves. It stubs on the test thread rather than inside the worker: the worker is the thing that may hang, and a restore that hangs with it would leave every later test in the process reading a nil boot time. test_an_interrupted_run_is_started_again passed only because the default cap of two left headroom for its single run. At MILL_CONCURRENCY=1 the interrupted run's own running row fills the cap and restart's at_cap? check refuses to re-enter it. That guard counts the run being restarted against itself, so it can only ever refuse — it never has capacity to protect. The fail_event compensation test used a fake whose start raised before touching the run row, so it proved nothing about the real supervisor, whose resumed flips the row to running and only then tells the board. It now drives the real Mill::Supervisor with a board that raises the way a misconfigured project does. Every rate-limit test paired rate_limited: true with success: false, so none could see that classify reads the flag before result.success?. A stage throttled at minute two that recovers and exits zero with a valid verdict has its work discarded, and because that path inserts no row the relaunch reuses the log filename and destroys the successful run's log. The review called that flag sticky; it is not — an allowed heartbeat clears it. The reachable case is a refusal that is the last rate-limit event before the result line arrives. The first rewrite of the fail_event test was itself a lie of the same kind: its fake hard-coded the bug's current location, so it would have stayed red under a correct supervisor-side fix and a later session would have concluded the fix had not worked. A fresh reviewer caught it. It now asserts the run is left where a retry can find it, and both candidate repair sites were applied and confirmed to turn it green. The signalling invariant is scoped to stored pgids, because the spawn test otherwise asks for what it forbids. The rule exists for a pgid read back from the database, which may have crossed a reboot; a group this process spawned and still holds the handle for cannot have. Spawn.reap's boot gate is untouched and test_hostile_input still pins it at :unknown_boot. All four are red on purpose, and CI stays red until the bugs beneath them are fixed. Each has a verified fix recorded in the triage note. 485 runs, 1621 assertions, 4 failures — the four, and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four hostile reviewers went over this branch and reported 57 findings, raw and untriaged, in a gitignored scratch file. A fourteen-session work queue should not live somewhere a clean checkout loses. The eight CRITICAL findings are six bugs. Three reviewers independently found the start/reap two-walker race and two found the rate-limit misclassification, which is the strongest signal in the set and was invisible while the findings sat in a flat list. It also records the blocker none of the reviewers could see, because all four ran on this laptop: CI has never been green on this branch. Six TestRepo tests error on a clean runner with "empty ident name", because commit_to_base commits inside a clone that Repo.prepare made with a real git clone and nothing sets an identity there. It passes here only because the author's global gitconfig supplies one — the same shape as the four tests in the previous commit, and the reason it is first in the queue. Twenty HIGH findings are split into eight that block a merge and twelve that do not. Two cases where the obvious fix quietly decides something nobody has decided are recorded as questions rather than bugs, so they are not settled by accident while fixing something adjacent. The suggested order puts the dead reaper third. A reaper that aborts its sweep every tick hides every other symptom, and both the claim orphan and the failed-start bug produce exactly the row that kills it. Merging before the first ten means merging something not safe to run unattended: Workers.enabled? defaults to on, so a stray Ready on the board reaches every critical path in the list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI has never been green on this branch, and the reason was never in the code. Six TestRepo tests error on a clean runner with "empty ident name": git clone copies no config, so a clone has no identity of its own, and commit_to_base commits into one. It passed here because git invents an identity from the macOS account record. The CI runner account has an empty GECOS field, so git has nothing to invent from, and four hostile reviewers all missed it because all four ran on this laptop. The fix that matters is the first one: TestRepo now disables the invention, so the failure reproduces locally instead of waiting for a push to reveal it. Config is only half of that. GIT_AUTHOR_NAME and its three companions sit above config and useConfigOnly does not touch them, so a machine exporting them would satisfy every commit here and still ship a clone the runner cannot commit in — they are cleared alongside it. Verified both ways: with the identity removed and those variables exported, the six still fail. The identity itself goes to the tests that commit, through one helper called from both place_clone and commit_to_base. Repo.resolve makes clones too, and five tests in this file use them; none commits today, and the first one that does would otherwise be green here and red on the runner all over again. Not fixed, and now recorded in the triage note rather than merely observed: mill gives its own clones no identity either. Repo.prepare writes gc.auto and maintenance.auto and nothing else, stages commit inside worktrees of that clone, and prompts/implement.md asks for a commit per task. On a server with no ambient identity that first commit fails and the stage is charged a strike for it. Fixing that means deciding what name mill's commits carry in somebody's real repository, which is a design decision and not this change. This restores signal, not a green check — the earlier note claimed more than it should have. rake test still exits non-zero on the four deliberately-red tests, so the badge stays red until those bugs are fixed. What changed is that the failure list is now four known bugs and nothing else, where before it was four bugs plus six errors that said nothing about the code. 485 runs, 1621 assertions, 4 failures, 0 errors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mill has been authoring commits as whoever runs it. Repo.prepare writes gc.auto and maintenance.auto to the clone and no identity, git clone copies none, and stages commit inside worktrees of that clone — so git walks its usual fallback chain and lands on the operator's ~/.gitconfig. Every commit mill has made so far is indistinguishable in git log and git blame from work typed by hand, and on a server with no gitconfig the implement stage's first commit fails outright and the stage is charged a strike for it. The identity is mill's own setting rather than a .mill.yml key. That file lives in the repo being worked on, so its base branch would otherwise decide the name the operator's credentials push under — and .mill.yml is hardened elsewhere precisely because it is attacker-adjacent. The operator is the author and mill is the committer, which is what git's two identities are for. Blame keeps pointing at the person who wanted the change; log and GitHub both show a machine made it. The committer is named mill and uses the author's address, so the commit still links to the account answerable for it instead of showing as an unrecognised stranger. The hook is GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL in Rules.env_for, beside the secrets. Unset, the author falls back to the machine's git config, so a laptop needs nothing configured. That leaves a server with nothing to fall back to, which is why doctor fails when no identity resolves — a setup that is wrong should say so at setup, not part-way through the first implement stage. Decision only. Nothing is built: this needs the write in prepare, the two committer variables in env_for, the doctor check, and a test that a prepared clone commits with the right author and committer. It sits last in the queue because it is the only item there that is not a bug in shipped behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
classify read the rate-limit flag before anything else, so a stage throttled at minute two that got its launch, worked, and exited with a verdict had that verdict thrown away. Because the free outcome inserts no row, next_attempt never advanced and the relaunch reused the log filename — destroying the log of the run that had just succeeded. Two of four reviewers found this independently, which made it the most-cited finding on the branch. The flag does not say what it looks like it says. rate_limited? reports what the last rate-limit event in the stream was, and an "allowed" heartbeat clears it, so a stage carries it whenever the result line lands before the next heartbeat. What settles whether the limit refused the launch is the verdict: a stage that handed back something mill can read did its work, whatever the limit did around it. Gating on the exit status instead looks equivalent and is not. Nothing here has measured what a refused launch exits with; the incident this file cites is a window closing mid-run, which is a non-zero exit from a launch that did run; and the one refusal mill has measured — a session the CLI would not reopen — is reported in-band. Gating that way would also drop the wait for any refusal exiting cleanly, turning a free outcome into a strike plus an immediate relaunch into a door that will not open for hours. That was the first version of this fix and it was wrong. Seven fixtures paired rate_limited with a readable verdict, which cannot happen — a refused launch hands back no payload, so Verdict.validate fails it. They now say valid: false. Six of the seven had to change for the fix to pass, and together they removed the only combination the change re-priced upward, so that combination is now pinned by a test of its own, along with the new shape exercised through the runner rather than asserted as a classification. Half of the finding remains, and is item 2b. A launch that ran, hit the window partway and handed back nothing still looks from here exactly like one refused outright, so it is still priced as "no launch" and still loses its log and its session. Telling those apart needs the stream — a session id, a model, any turns at all. The test pinning that behaviour says in capitals that it should be deleted by whoever fixes it, because a known bug recorded as intended behaviour is worse than one nobody wrote down. The adjacent case stays free for the same reason the strike ledger exists: charging it needs a premise nothing has measured, and charging on an unproven premise is how a stage gets blocked for a door mill could not open. 488 runs, 1626 assertions, 3 failures — the three remaining deliberately-red tests, unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A commit message is written once and read rarely, so narrative costs nothing there. A code comment and a living note are read every time someone opens the file. I had been putting the first kind of writing into the second two, and it showed: classify carried 33 lines of comment over 16 of code, a three-line assertion carried 20, and the triage note grew by a quarter across two commits in which one bug was fixed. A reviewer had already said so about classify, and the rewrite that followed came out two lines longer than the version it was meant to shorten. Cut: the process narrative. Which wrong turns were taken and in what order, the same session-loss mechanism restated in three places, and a paragraph explaining why the wait cap is no comfort — true, and of no use to anyone who is not reading my reasoning. Kept: everything a future session needs to act on. What the bug was, where, what the fix is, what is still broken, and the one warning worth its lines — do not gate on the exit status instead, because it looks equivalent and silently drops the rate-limit wait. The note still has a structural problem this does not fix. It is an archive of 57 findings, a live work queue, and a decision record in one file, and only the queue is read every session. Splitting it would do more than trimming did. 488 runs, 1626 assertions, 3 failures — unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`interrupt` raised on a running row with no `current_stage`, and the raise escaped `reap`'s `filter_map`. That aborted the whole sweep, so the row was never repaired and it raised again on the next tick, and the next — for every run, not just the broken one. `reap` now handles the nil stage itself: fail the row, announce it through `finish` so the human hears about it rather than only mill's stderr, and carry on with the sweep. Gated on age, because a run claimed but not yet started looks exactly like an orphan. `claim` commits the row as running with no stage and no pid, makes a GraphQL call, and only then does the poller call `start`; until that thread exists `identify` says `:gone`. Failing every stageless row destroyed healthy work in that window. An orphan is minutes old and a run mid-handoff is milliseconds old, so `STAGELESS_GRACE` tells them apart. Replaces the test that asserted the raise reaching the caller, which is the behaviour that had to go. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`start` flipped a blocked run's row to running and then wrote the board — a GraphQL round trip — before registering the thread. Inside that window the run was running, had no thread and had no process, which `identify` read as `:gone`. The reaper charged an interruption nobody earned and started a second walker into the worktree the first was about to enter: two claude processes under acceptEdits, two ledger writers, and whichever finished first tore the worktree down under the other. The supervisor now marks the run before anything slow runs and releases the mark in an `ensure` that fires only once the thread is registered, so the two never both answer no. `running?` reads both. Its other caller in the poller wanted the wider answer already: its comment claimed `running?` was what stopped a retry becoming a second walker, and that only becomes true here. `claim` had the same board write and the longer window — the first write of a process resolves the project fields, so it is three `gh` calls with no timeout. Left unmarked it was worse than the one above rather than milder, because the stageless age guard fails the row and tears off the worktree, and `Run.adopt` does not check status before walking. Both sites are marked. Leaves `restore` and the sanctioned strike reset still unreachable. That reads the row status to decide whether a run is being resumed, and this keeps the flip ahead of the thread on purpose, so the fix belongs with that finding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`reap` asked `Spawn.reap` to kill a foreign process group and threw the answer away. Spawn refuses to signal whenever it cannot prove the group is mill's, and returns `:survived` when TERM and KILL both failed, so the supervisor charged an interruption and re-entered the stage on top of a process that was still running: a second claude under acceptEdits in a worktree the first one still holds. `group_down?` gates on an allowlist, so a tenth answer added to Spawn later is refused rather than read as proof of death. `:rebooted` is deliberately not on it — it returns before Spawn ever checks whether the group is alive, and the only way to reach it here is `identify` having just proved a live process at that pid. On a laptop `lstart` does not move when the clock steps, so an NTP step moves the boot time alone and lands exactly there. The same hole was open on the `:gone` path, which reads the leader pid only. A dead leader over live descendants -- claude exits, the `npm test` it started keeps the worktree and the port -- never reached Spawn at all. On Linux that is also where a clock step lands, because a process start time is computed from /proc/stat btime and moves when btime jitters. So ask Spawn whenever anything is alive under the group, whatever `identify` concluded. A stalled run keeps its concurrency slot. Something is alive under that group and still holding the memory the slot stands for, and starting a second agent beside a process mill could not stop is how a small box runs out of RAM. What was missing was anyone being told, so it comments once naming the pid to stop. Nothing to reply to: kill the group and the next tick finds it gone. Spawn is injected the way git already was, so tests can answer for it without patching a global in a threaded suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The board is a work queue, not a tracker: an item on it at Ready is claimed and launches a run. mill does not build mill yet, so its own issues are for people. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the callback that records a new stage raises - a locked database is the likely way - announce_spawn must kill the group it just started, because nothing else now knows the group exists. It asked Spawn.reap to do it. reap refuses to signal anything until it can read the host boot time, and on a host where that read fails it answered :unknown_boot and signalled nothing. The stage kept running, and popen3 held the raise behind the child until the child chose to exit - about thirty seconds in the test, unbounded in a real run. reap's caution is right for the pgid it was written for: Supervisor#reap reads pgids out of the database, where a number may have crossed a reboot and may now belong to a system daemon. It is wrong for a group this process spawned two lines earlier and is still holding. So the signalling half of reap moves into Spawn.kill_group, reap calls it once its checks pass, and announce_spawn calls it directly. reap's boot gate is untouched and test_hostile_input still pins :unknown_boot. Fixes #9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two repairs to the extraction in ee1d9c9, both from review. The liveness check moved. reap used to ask "is anything in this group alive?" before asking whose group it is; the extraction left that question inside the new method, which runs after identify. Take a pgid whose own group is empty but whose number is now some stranger's non-leader pid: reap used to answer :gone, and answered :recycled or :unverified instead. It still signals nothing either way, so nothing strange gets killed - but the supervisor treats only :gone as proof a run is down, so the run holds its concurrency slot forever and the stall comment names a stranger's process. Test:302 also stopped being a property of the code and started being a property of the host, since a Linux box with pid_max above 99999 may well have pid 99999 running. The name moved too. kill_group was the obvious name to reach for, sat next to a gated method called reap, and Supervisor#stalled already talks about killing the group - the next session would have passed it a pgid straight out of the database. kill_held_group says the precondition at the call site, and the class comment now names which of the two doors takes the checks and which does not. Refs #9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Set Status to
Ready, walk away, come back to a pull request.Mill::Pollerreconciles the board and sweeps comments behind a transactionalcursor.
Mill::Supervisorprepares repos, claims to a cap, owns the worktreelifecycle, walks each run in its own thread, and reaps against a verified
process identity.
Mill::Boardwrites Status and re-drives a write that neverlanded.
Mill::Secretsinjects a repo's environment.app.rbandconfig.rugive both loops a home that Plan 4 fills in.
Plan:
docs/superpowers/plans/2026-08-19-plan-3a-autonomy.md.Design: Where this stands is updated; the rehearsal record sits beside
Plan 2's in the build order.
Demonstrated, not asserted
mill-scratch#4— clean run, six stages, zero strikes, nobody watching.mill-scratch#6— blocked atplanwith three questions, answered in acomment, resumed on its own session, finished. Zero strikes: asking is free.
live process group orphaned. Recovered both times, charging an attempt and no
strike, and re-entering the stage it was in rather than the top of the route.
with both threads alive in the morning.
rake test— 483 runs, 1617 assertions, 0 failures.rake test:boundary— 14 runs, 30 assertions, 0 failures, against the realCLI. Required because this branch changed
Mill::SpawnandMill::Rules.env_for.What running it found
Eleven defects, seven in this branch's own code, and the fixture suite could
not have caught any of the seven. Five were the same shape: each component
correct alone, each tested alone, the defect in the handoff.
Mill::Workersassembled a supervisor without a board, so mill would have runthe whole pipeline and never written a Status — and since a comment only means
an answer while the board says
Blocked, no blocked run could ever have beenresumed.
Supervisor#walkreturned a database row wherefinishexpected the runner'sstate, so the first real block posted
Blocked at: .`` to the issue. millasked three good questions and threw all of them away.
blocked → runningtransition, leaving a resumed runinvisible to the reaper for the rest of its route.
strike, which breaks a stated safety invariant.
Mill::Streamalreadydetected it and
COST[:rate_limited]already priced it at zero;classifynever asked.
The other two: doctor passed a database three migrations behind, because it
checked that tables existed and the missing thing was a column; and the log
scrubber would have corrupted the
stream-jsonit parses back, given a shortvalue like
DEBUG=true.Two runbook steps also turned out not to work as written — the built-in
Statusfield can be neither deleted nor recreated, and disabling the board's workflows
is the one setup step with no API.
The lesson, recorded in the design doc
An adversarial review of this plan's code, before a line of it existed, found
twelve defects including four that would have stopped the factory silently. It
caught that two
Mill::Supervisorinstances would make the reaper kill healthystages. The fix was to share one instance — and the shared instance was built
without a board, which is the first bug listed above.
Reviews catch the layer they are looking at. Only running the assembled thing
catches the seam below it.
Not in this branch
The stall detector, sleep detection, the settle window and
caffeinate, and thelog reaper are Plan 3b. The UI's routes are Plan 4. The
fastanditerateroutes are Plan 5 — only two of the five triggers dispatch, and the rest record
no_route.Untested against reality: branch collisions, the concurrency cap, secrets
injection against a repo that declares any, and the board re-drive. All four
fail loudly in normal use.
mill has still never run on Linux, which is now its primary target.
🤖 Generated with Claude Code