feat(disk): guard root disk space and reclaim build scratch on teardown

[knowttl]

Intent

Deliver two durable, minimal disk-hygiene safeguards for firstmate, after a verified root cause: firstmate's pooled build worktrees live on the small root disk while Docker and the live DB live on a large network share. Every project worktree that builds a frontend gets its own full node_modules (~1.5G each), and these are NOT reclaimed when a crewmate's task tears down or the worktree returns to the pool - they accumulate (one secondmate's pool reached 18G across ~9 worktrees). There was no disk-space guard, so / filled silently until a live deploy failed at its build step; concurrent builds were the trigger.

SAFEGUARD 1 - reclaim build scratch on teardown. When firstmate tears down a completed, LANDED crewmate task (or a worktree is returned to the pool), clean that worktree's regenerable build scratch (the real patterns used across the fleet's projects: node_modules, dist, .next, build caches, etc.), which regenerates on the next install. HARD SAFETY: only ever remove REGENERABLE build scratch, and only for a worktree whose task has LANDED and is being torn down (or a genuinely idle returned pool slot with no live process). NEVER remove source, uncommitted or unlanded work, or anything under a still-live worktree. It must COMPOSE with the existing teardown landed-work test in bin/fm-teardown.sh - reclaim only after landing is confirmed, never a path that bypasses it. Fail closed: if unsure a worktree is safe, skip it.

SAFEGUARD 2 - root-disk-space guard in supervision. Surface a LOUD, rate-limited warning when the root filesystem free space drops below a configurable threshold (a sensible default headroom, overridable), so a fill is caught early instead of failing a build/deploy silently. Wire it into firstmate's existing guard surface (where the watcher-down / worktree-tangle warnings already surface) and/or session-start bootstrap detection. Keep it WARNING/ALARM only - deterministic and idempotent. Do NOT auto-delete caches or auto-pause spawns in this change; early detection is the goal. The threshold and the watched filesystem must be configurable via a gitignored config/ item, documented once in docs/configuration.md.

CONSTRAINTS accepted for this firstmate-repo task: follow the firstmate-coding-guidelines skill (knowledge-placement decision tree, one-owner rule, AGENTS.md size discipline, trigger hygiene, repo style). New config items are gitignored and documented once in their authoritative owner; mechanics live in the script header/--help, not AGENTS.md prose. Colocate tests in tests/ named .test.sh, exercising behavior through the public/executable interface (never asserting implementation bytes): cover the disk-guard threshold logic (warns below, quiet above) and the teardown-reclamation safety (never touches unlanded/live worktrees; only regenerable scratch). shellcheck-clean via bin/fm-lint.sh. One sentence per line in Markdown; plain dash; no agent co-author. Keep it MINIMAL - exactly the two safeguards, nothing more.

IMPLEMENTATION DECISIONS made while doing the work, which a reviewer reading only the diff would not know:

• Reclamation lives in a new standalone bin/fm-reclaim-build-scratch.sh rather than inline in fm-teardown.sh, so its safety contract is testable through a public executable interface.
• Its removal is gated on THREE independent conditions, all required: (1) the worktree is a git worktree with no modified tracked files (a second, independent check on top of teardown's own landed-work test), (2) git itself reports the path IGNORED, and (3) the basename is on a fixed regenerable list. The double gate of "git-ignored AND regenerable name" is deliberate: an ignored .env, credential, or local data directory is precious and must survive, so being ignored alone is intentionally not sufficient.
• The regenerable list is deliberately conservative and fixed, not configurable (minimality): node_modules, .next, .nuxt, .turbo, dist, build, target, pycache, .pytest_cache, .mypy_cache, .ruff_cache. .venv was deliberately EXCLUDED as the more conservative choice.
• Symlinked scratch, or a path resolving outside the worktree, is skipped: scratch is reclaimed in place or not at all.
• The call site in fm-teardown.sh is deliberately placed after every landed-work refusal has passed AND after the existing process reap, and before the treehouse return - verified empirically that treehouse return --force resets tracked content but leaves git-ignored trees in place, which is exactly why returned pool slots keep their install trees. The call is best effort: the script's own refusal, or any failure, never blocks teardown. Process-liveness safety is intentionally left owned by teardown's existing reap step rather than duplicated in the new script (one-owner rule).
• The disk alarm was added to bin/fm-guard.sh only (not also bootstrap), placed alongside the existing tangle alarm and before the in-flight-work early exit so it is independent of fleet state. Adding a second bootstrap diagnostic line would have required a new AGENTS.md trigger and bootstrap-diagnostics entry, which was judged unnecessary surface for this change.
• Rate limiting uses a volatile per-home marker holding the epoch of the last alarm, repeating at most once per FM_DISK_GUARD_REPEAT_SECS (default 3600) and clearing on recovery so the next fill alarms immediately. A read-only guard session reports the alarm without writing home state, matching the existing read-only contract.
• config/disk-guard holds one line, " []", defaulting to 20 GiB on "/". A threshold of 0 disables the alarm; an unusable threshold is reported and falls back to the default rather than silently disabling the check; an unreadable watched path alarms rather than going quietly unmonitored.
• Tests: two new colocated behavior suites plus two composition cases appended to the existing tests/fm-teardown.test.sh (landed work reclaims before the worktree return, proven by a treehouse fake that snapshots real observed state at return time; a teardown refused for unlanded work leaves all scratch intact).

KNOWN, DELIBERATELY OUT OF SCOPE: bin/fm-test-run.sh --check-coverage fails on this host because its comm calls run under the ambient locale while its inputs are sorted with LC_ALL=C. That is pre-existing and unrelated to this change; it was reported rather than fixed, to keep the diff minimal.

What Changed

• Added bin/fm-reclaim-build-scratch.sh, a fail-closed reclaimer that removes a finished worktree's regenerable build scratch (node_modules, dist, .next, tool caches, etc.), gated on three independent conditions: the worktree has no modified tracked files, git reports the path ignored, and its basename is on a fixed regenerable-name list; symlinks and paths resolving outside the worktree are skipped. bin/fm-teardown.sh now invokes it best-effort after every landed-work refusal and process reap have passed, before the worktree returns to the pool.
• Added a low-disk alarm to bin/fm-guard.sh, checked independently of in-flight work alongside the tangle alarm: it prints a loud, rate-limited banner (once per FM_DISK_GUARD_REPEAT_SECS, default 3600, per home; re-armed on recovery) when the watched filesystem's free space falls below the configured headroom. It warns only and never deletes or pauses anything. Threshold and watched path come from the new gitignored config/disk-guard (<min-free-gib> [<path>], default 20 GiB on /; 0 disables; an unusable threshold or unreadable path is reported rather than silently ignored).
• Documented the guard and reclaimer in docs/configuration.md, docs/architecture.md, docs/scripts.md, and the config/disk-guard entry in AGENTS.md; added behavior suites tests/fm-guard-disk-space.test.sh and tests/fm-reclaim-build-scratch.test.sh, plus teardown composition cases in tests/fm-teardown.test.sh.

Risk Assessment

✅ Low: The change is well-bounded and fail-closed: reclamation is triple-gated (clean tracked status, git-reported IGNORED, fixed regenerable-name list) with symlink/escape guards and composes after teardown's landed-work refusals and process reap, while the disk guard is warning-only, rate-limited, and correctly configured, all covered by behavior tests through executable interfaces.

Testing

Ran the three targeted suites covering both safeguards (reclaim behavior, disk-guard threshold/rate-limit, and the two teardown composition cases) - all pass - and produced two CLI transcripts as product-level evidence: the reclaim script clearing a landed frontend worktree's node_modules/dist/.next while preserving source and an ignored .env, then refusing a worktree with modified tracked work; and fm-guard.sh rendering its loud warning-only LOW-DISK banner below the configured headroom, staying silent above it, rate-limiting a repeat, and re-arming on recovery. This is a CLI/operational change with no UI surface, so the appropriate reviewer-visible evidence is the terminal transcripts rather than screenshots. The intent's declared out-of-scope --check-coverage locale failure was not exercised.

Evidence: Safeguard 1 reclaim end-to-end transcript

Source: Safeguard 1 reclaim end-to-end transcript

$ bin/fm-reclaim-build-scratch.sh frontend-worktree reclaim: removed 3 regenerable build-scratch path(s) under frontend-worktree exit=0 SURVIVED : src/app.js SURVIVED : .env SURVIVED : .gitignore RECLAIMED: node_modules RECLAIMED: dist RECLAIMED: .next --- with an UNCOMMITTED tracked edit --- reclaim: REFUSED: frontend-worktree has modified tracked files; leaving all scratch in place exit=1 CONFIRMED: node_modules left intact when tracked work is modified

### SAFEGUARD 1: reclaim build scratch on a LANDED, torn-down frontend worktree

$ du -sh frontend-worktree   # before reclaim (the ~1.5G-class hoard, scaled here)
2.3M	frontend-worktree

$ git -C frontend-worktree status --porcelain --untracked-files=no   # tracked work is clean (landed)
(empty output above = no modified tracked files)

$ bin/fm-reclaim-build-scratch.sh frontend-worktree
reclaim: removed 3 regenerable build-scratch path(s) under frontend-worktree
exit=0

$ du -sh frontend-worktree   # after reclaim
208K	frontend-worktree

--- what SURVIVED (precious) vs what was RECLAIMED (regenerable) ---
SURVIVED : src/app.js
SURVIVED : .env
SURVIVED : .gitignore
RECLAIMED: node_modules
RECLAIMED: dist
RECLAIMED: .next

--- HARD-SAFETY probe: same worktree but with an UNCOMMITTED tracked edit ---
$ bin/fm-reclaim-build-scratch.sh frontend-worktree   # must REFUSE, remove nothing
reclaim: REFUSED: frontend-worktree has modified tracked files; leaving all scratch in place
exit=1
CONFIRMED: node_modules left intact when tracked work is modified

Evidence: Safeguard 2 low-disk alarm banner transcript

Source: Safeguard 2 low-disk alarm banner transcript

●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ● LOW DISK SPACE - BUILDS AND DEPLOYS ARE AT RISK ● / has 23.8G free, below the 999999999G headroom. ● A full disk fails a build or deploy with an error that names anything but the disk. ● Free space before dispatching more build work; finished worktrees are the usual hoard. ● This is a supervision warning only; the guarded operation WILL still run. ●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ guard-exit=0 (repeat inside window: quiet; recovery: CONFIRMED cleared the rate-limit marker)

### SAFEGUARD 2: low-disk alarm surfaced by bin/fm-guard.sh (the existing guard surface)

--- healthy: config asks for 20 GiB headroom, filesystem has more -> guard is silent ---
$ cat config/disk-guard
20 /
$ fm-guard.sh
(no LOW DISK banner above = ample headroom, alarm correctly silent)

--- fill: request an impossible headroom to force the below-threshold branch deterministically ---
$ cat config/disk-guard
999999999 /
$ fm-guard.sh   # LOUD, warning-only banner the operator sees:
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  LOW DISK SPACE - BUILDS AND DEPLOYS ARE AT RISK
●  / has 23.8G free, below the 999999999G headroom.
●  A full disk fails a build or deploy with an error that names anything but the disk.
●  Free space before dispatching more build work; finished worktrees are the usual hoard.
●  This is a supervision warning only; the guarded operation WILL still run.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
guard-exit=0

--- rate limit: an immediate re-run inside the window stays quiet (idempotent, non-spammy) ---
$ fm-guard.sh   # second call, same window
(quiet = alarm rate-limited, not repeated)

--- recovery re-arms: a healthy threshold clears the marker so the next fill alarms immediately ---
CONFIRMED: recovery cleared the rate-limit marker

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-reclaim-build-scratch.test.sh - 6/6 ok (removes only ignored regenerable scratch, preserves .env/local-data/.venv/tracked, refuses dirty/non-worktree/missing-arg, no symlink escape, idempotent)
• bash tests/fm-guard-disk-space.test.sh - 5/5 ok (threshold decides alarm, config selects path/reports invalid threshold, rate-limit + re-arm on recovery, warn-only exit 0, read-only no state write)
• Ran the two new composition cases in tests/fm-teardown.test.sh in isolation (test_landed_teardown_reclaims_build_scratch_before_the_worktree_returns, test_refused_teardown_leaves_build_scratch_untouched) - both ok
• Manual E2E: ran bin/fm-reclaim-build-scratch.sh against a real landed frontend-shaped worktree (node_modules/dist/.next reclaimed, src/.env/.gitignore survived) and against the same worktree with an uncommitted tracked edit (REFUSED, exit 1, node_modules intact) -> /tmp/no-mistakes-evidence/01M0B0ARS8KR2D60DVDQDY1ARJ/reclaim-transcript.txt
• Manual E2E: ran bin/fm-guard.sh against a scratch FM_HOME with config/disk-guard, capturing the silent-above-threshold case, the rendered LOW-DISK banner (exit 0), the rate-limited re-run, and the recovery re-arm -> /tmp/no-mistakes-evidence/01M0B0ARS8KR2D60DVDQDY1ARJ/disk-guard-transcript.txt

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[knowttl]

Closing per captain's decision: not landing the durable disk-hygiene safeguards.