Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ config/backend runtime session-provider backend override for new tasks; LOCAL,
config/calm Pi Calm presentation preference; LOCAL, gitignored, and not inherited; see docs/configuration.md "Pi Calm preference"
config/startup-memory-budget primary-authoritative per-home startup-memory budget; LOCAL, gitignored, materialized as 7,500 estimated tokens by locked primary bootstrap and inherited into secondmate homes; see docs/configuration.md "Startup memory budget"
config/herdr-presentation-spaces optional "off" opt-out from, or "on" opt-in to, Herdr's default-on disposable single-task visual projection, which is unconfigured-default-on only at or above a Herdr version floor; LOCAL, gitignored; inherited by secondmate homes; see docs/herdr-backend.md "Presentation spaces"
config/disk-guard optional free-space headroom and watched filesystem for the low-disk alarm; LOCAL, gitignored; absent means 20 GiB on "/"; see docs/configuration.md "Disk-space guard"
config/trace-context optional presence flag enabling default-off native W3C trace-context propagation to spawned agents; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Trace context propagation" and docs/trace-context.md
config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup")
config/wedge-alarm optional away-mode wedge-alarm active-alert directives; LOCAL, gitignored; absent means auto (macOS Notification Center when available); see docs/wedge-alarm.md
Expand Down
84 changes: 84 additions & 0 deletions bin/fm-guard.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@
# First, always warn if the firstmate primary checkout (FM_ROOT) is on a named
# non-default branch, because that means firstmate-on-itself work landed in the
# primary instead of an isolated worktree.
# Second, always alarm when the watched filesystem's free space has fallen below
# the configured headroom, so a fill is caught before it silently fails a build or
# a deploy. That alarm warns only - it never deletes anything and never pauses
# work - and is rate-limited to once per FM_DISK_GUARD_REPEAT_SECS per FM_HOME,
# re-arming as soon as free space recovers. config/disk-guard owns the threshold
# and the watched path (docs/configuration.md "Disk-space guard").
# Then, if a task is in flight (a state/<id>.meta exists) or X-mode relay
# polling is active (state/x-watch.check.sh exists) and supervision is not
# healthy, prints a loud, clearly delimited banner so the agent cannot skim past
Expand Down Expand Up @@ -46,6 +52,12 @@ CONTINUE_LINE=${FM_GUARD_CONTINUE_LINE:-This is a supervision warning only; the
# Cleared when the home leaves the unhealthy state so a later episode re-arms.
STALE_BANNER_MARKER="$STATE/.guard-watcher-stale-banner"

# Volatile, home-scoped disk alarm marker: one line = the epoch of the last full
# alarm. Removed when free space recovers so the next fill alarms immediately.
DISK_ALARM_MARKER="$STATE/.guard-disk-space-alarm"
DISK_REPEAT_SECS=${FM_DISK_GUARD_REPEAT_SECS:-3600}
case "$DISK_REPEAT_SECS" in ''|*[!0-9]*) DISK_REPEAT_SECS=3600 ;; esac

# shellcheck source=bin/fm-wake-lib.sh
. "$SCRIPT_DIR/fm-wake-lib.sh"
# shellcheck source=bin/fm-tangle-lib.sh
Expand Down Expand Up @@ -121,6 +133,46 @@ fm_guard_clear_stale_banner() {
rm -f "$STALE_BANNER_MARKER" 2>/dev/null || true
}

# Free-space threshold and watched path from config/disk-guard: the first
# non-empty, non-comment line is "<min-free-gib> [<path>]". An absent file means
# the built-in defaults; an unusable threshold is reported and falls back rather
# than silently disabling the alarm. A threshold of 0 disables it.
DISK_MIN_GIB=20
DISK_PATH=/
fm_guard_read_disk_config() {
local line min path
line=$(grep -v -e '^[[:space:]]*#' -e '^[[:space:]]*$' "$CONFIG/disk-guard" 2>/dev/null | head -1 || true)
[ -n "$line" ] || return 0
read -r min path _ <<EOF
$line
EOF
case "$min" in
''|*[!0-9]*)
printf 'WARNING: config/disk-guard: ignoring invalid free-space threshold %s; using %s GiB.\n' \
"$min" "$DISK_MIN_GIB" >&2
;;
*) DISK_MIN_GIB=$min ;;
esac
[ -z "$path" ] || DISK_PATH=$path
}

# Available 1024-byte blocks on the filesystem holding $1, or empty when it
# cannot be read (POSIX df -Pk reports the same units everywhere).
fm_guard_disk_free_kib() {
df -Pk "$1" 2>/dev/null | awk 'NR == 2 { print $4 }'
}

# Rate limit: the full alarm prints when no alarm was recorded, when the record
# is unreadable, or when it is older than the repeat window.
fm_guard_disk_alarm_due() {
local last now
last=$(cat "$DISK_ALARM_MARKER" 2>/dev/null || true)
last=${last%$'\n'}
case "$last" in ''|*[!0-9]*) return 0 ;; esac
now=$(date +%s)
[ "$((now - last))" -ge "$DISK_REPEAT_SECS" ]
}

# Worktree-tangle alarm, checked FIRST and independent of in-flight tasks: the
# firstmate PRIMARY checkout (FM_ROOT) must stay on its default branch. If a
# crewmate's branch/commits landed here instead of in its own isolated worktree,
Expand Down Expand Up @@ -148,6 +200,38 @@ if [ -n "$tangle_branch" ]; then
} >&2
fi

# Low-disk alarm, checked independently of in-flight tasks: pooled build
# worktrees and their install trees share one filesystem, and a fill fails the
# next build or deploy with an error that looks like anything but a full disk.
fm_guard_read_disk_config
if [ "$DISK_MIN_GIB" -gt 0 ]; then
disk_free_kib=$(fm_guard_disk_free_kib "$DISK_PATH")
disk_alarm=
if [ -z "$disk_free_kib" ]; then
disk_alarm=$(printf 'cannot read free space on %s (check config/disk-guard).' "$DISK_PATH")
elif [ "$disk_free_kib" -lt "$((DISK_MIN_GIB * 1024 * 1024))" ]; then
disk_alarm=$(awk -v k="$disk_free_kib" -v p="$DISK_PATH" -v m="$DISK_MIN_GIB" \
'BEGIN { printf "%s has %.1fG free, below the %sG headroom.", p, k / 1048576, m }')
fi
if [ -n "$disk_alarm" ]; then
if fm_guard_disk_alarm_due; then
[ "$READ_ONLY" -eq 1 ] || date +%s >"$DISK_ALARM_MARKER" 2>/dev/null || true
drule='━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
{
printf '●%s\n' "$drule"
printf '● LOW DISK SPACE - BUILDS AND DEPLOYS ARE AT RISK\n'
printf '● %s\n' "$disk_alarm"
printf '● A full disk fails a build or deploy with an error that names anything but the disk.\n'
printf '● Free space before dispatching more build work; finished worktrees are the usual hoard.\n'
printf '● %s\n' "$CONTINUE_LINE"
printf '●%s\n' "$drule"
} >&2
fi
elif [ "$READ_ONLY" -ne 1 ]; then
rm -f "$DISK_ALARM_MARKER" 2>/dev/null || true
fi
fi

# Compute supervision need and watcher-beacon freshness via the shared
# grace-based predicate (bin/fm-supervision-lib.sh). Act when work, an event
# source, or an X-mode relay poll needs supervision.
Expand Down
79 changes: 79 additions & 0 deletions bin/fm-reclaim-build-scratch.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
# Reclaim regenerable build scratch (node_modules, compiled output, tool caches)
# from a worktree that is finished with, so pooled worktrees do not accumulate
# gigabytes of install trees on the disk that holds the pool. `treehouse return`
# resets tracked content but leaves git-IGNORED trees in place, so a returned
# pool slot keeps every install tree its last task built.
#
# Usage: fm-reclaim-build-scratch.sh <worktree-dir>
# Prints one summary line on stdout. Exit 0 = reclaimed (possibly nothing),
# 1 = refused (nothing was removed), 2 = usage error.
#
# Removal is gated on three independent conditions, all of which must hold:
# 1. <worktree-dir> is a git worktree whose TRACKED files are unmodified.
# A tracked modification means live or unfinished work, so this refuses and
# removes nothing. This is a second, independent check: whether the task's
# work has LANDED is owned by bin/fm-teardown.sh, which runs its landed-work
# test (and its process reap) before ever calling this.
# 2. The path is reported IGNORED by git itself, so no tracked file, no
# untracked-but-unignored file, and nothing in the index can ever be a
# candidate.
# 3. Its basename is in the regenerable-scratch list below. Being ignored is
# not sufficient: an ignored `.env`, credential, or local data directory is
# precious, and only names whose contents a plain install or build step
# recreates are removed.
# A symlink, or a directory whose resolved path escapes the worktree, is skipped:
# scratch is reclaimed in place or not at all.
set -eu

# Regenerable when git-ignored: a package install tree, a build output tree, or a
# tool cache that the next install/build/test run recreates.
SCRATCH_NAMES="node_modules .next .nuxt .turbo dist build target __pycache__ .pytest_cache .mypy_cache .ruff_cache"

usage() {
echo "usage: fm-reclaim-build-scratch.sh <worktree-dir>" >&2
exit 2
}

refuse() {
echo "reclaim: REFUSED: $1" >&2
exit 1
}

[ "$#" -eq 1 ] || usage
[ -n "$1" ] || usage
[ -d "$1" ] || refuse "no worktree directory at $1"

ROOT_DIR=$(cd "$1" && pwd -P)
git -C "$ROOT_DIR" rev-parse --is-inside-work-tree >/dev/null 2>&1 \
|| refuse "$ROOT_DIR is not a git worktree"
STATUS=$(git -C "$ROOT_DIR" status --porcelain --untracked-files=no 2>/dev/null) \
|| refuse "cannot inspect $ROOT_DIR for tracked modifications"
[ -z "$STATUS" ] || refuse "$ROOT_DIR has modified tracked files; leaving all scratch in place"

removed=0
while IFS= read -r -d '' rel; do
rel=${rel%/}
[ -n "$rel" ] || continue
case " $SCRATCH_NAMES " in
*" ${rel##*/} "*) ;;
*) continue ;;
esac
path="$ROOT_DIR/$rel"
if [ -L "$path" ] || [ ! -d "$path" ]; then
continue
fi
real=$(cd "$path" && pwd -P) || continue
case "$real" in
"$ROOT_DIR"/*) ;;
*) continue ;;
esac
rm -rf "$real"
removed=$((removed + 1))
done < <(git -C "$ROOT_DIR" ls-files --ignored --exclude-standard --others --directory -z)

if [ "$removed" -eq 0 ]; then
echo "reclaim: no regenerable build scratch under $ROOT_DIR"
else
echo "reclaim: removed $removed regenerable build-scratch path(s) under $ROOT_DIR"
fi
14 changes: 14 additions & 0 deletions bin/fm-teardown.sh
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,14 @@
# root still exists, so the account's healthy LaunchAgent worker and every
# live remote secondmate worker are out of scope. Best effort: a sweep
# failure never blocks this teardown.
# Fix 4 - reclaim regenerable build scratch. `treehouse return` resets tracked
# content but leaves git-ignored trees in place, so each pooled worktree keeps
# the full install tree its last task built (measured 2026-08-18: ~1.5G of
# node_modules per frontend worktree, never reclaimed, filling the pool disk
# until a build failed). bin/fm-reclaim-build-scratch.sh owns the removal
# rules and its own independent safety gate; it runs only here, after every
# landed-work refusal above has passed and the worktree's processes are
# reaped, and its refusal never blocks teardown.
set -eu

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Expand Down Expand Up @@ -2385,6 +2393,12 @@ fi
if [ "$KIND" != secondmate ]; then
conclude_task_no_mistakes_run "$WT"
reap_task_worktree_processes worktree "$WT" "$TASK_TMP"
# Fix 4 (see script header): reclaim regenerable build scratch before the
# worktree goes back to the pool. Best effort - its own refusal, or any
# failure, never blocks this teardown.
if [ -d "$WT" ]; then
"$SCRIPT_DIR/fm-reclaim-build-scratch.sh" "$WT" >&2 || true
fi
fi

# Fix 3 (see script header): sweep remote job workers abandoned by an already
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ It suppresses failed-looking closes when the same identity-matched watcher is he
Cursor's `bin/fm-turnend-guard-cursor.sh` hook is the same between-turns shape in one synchronous step: it parks the awaited `stop` hook on the arm wrapper and translates an actionable close into one `followup_message`, with a generation baton that makes an older park still running after the next `stop` claim stand down instead of leaking a stale duplicate wake.
The existing turn-end guard remains the final backstop for every harness-engine protocol, with pi-signed sharing Pi's protocol, the `--claude` mode cooperating with the auto-arm claim, and Cursor's `--cursor` mode rendering a block as one bounded follow-up because its `stop` step cannot be blocked.
Its `--restart` mode signals only the watcher recorded in the current home's `state/.watch.lock`, so restarting one home cannot kill sibling secondmate watchers.
A pull-based guard (`bin/fm-guard.sh`) warns through supervision tool output if the primary checkout is tangled, if work, process-event sources, or Relay polling has an unhealthy model-aware supervision verdict, or if queued wakes are waiting to be drained.
A pull-based guard (`bin/fm-guard.sh`) warns through supervision tool output if the primary checkout is tangled, if the watched filesystem's free space is below the configured headroom (docs/configuration.md "Disk-space guard"), if work, process-event sources, or Relay polling has an unhealthy model-aware supervision verdict, or if queued wakes are waiting to be drained.
The drain script calls that guard after presenting the queue; records remain durable, and may keep the queued-wakes warning visible, until the exact generation-bound acknowledgement printed by the drain succeeds after handling.
It leads with a prominent bordered tangle banner, while `bin/fm-guard.sh` owns the watcher-down banner and reminder policy so repeated guarded commands stay noisy without reprinting the full banner in the same episode.
On every verified primary harness, tracked hook integration gives the primary session a push-based backstop: when work, a process-event source, or Relay polling needs supervision and no identity-matched watcher lock with a fresh beacon is live, blocking-capable Stop hooks block and nonblocking turn-end integrations force one bounded follow-up.
Expand Down
14 changes: 14 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,19 @@ An absent file means `auto`, i.e. default-on on macOS: the alarm exists precisel
A missing or failing channel logs and falls through to the next, never crashing the daemon.
See [`wedge-alarm.md`](wedge-alarm.md) for the current channel reference, [`verification/supervision.md`](verification/supervision.md#wedge-alarm-channels) for active evidence, and [`examples/wedge-alarm`](examples/wedge-alarm) for a copyable config.

## Disk-space guard (config/disk-guard)

Pooled build worktrees and their install trees share one filesystem, and a fill fails the next build or deploy with an error that names anything but the disk.
`bin/fm-guard.sh` therefore alarms whenever the watched filesystem's free space is below the configured headroom, on every fleet action that already runs the watcher and tangle guards.
The alarm warns only: it never deletes a cache and never pauses a spawn.
It is rate-limited to once per `FM_DISK_GUARD_REPEAT_SECS` (default 3600) per home, and re-arms as soon as free space recovers.

`config/disk-guard` (local, gitignored) holds one non-empty, non-comment line: `<min-free-gib> [<path>]`.
An absent file means 20 GiB of headroom on `/`; a threshold of `0` disables the alarm; an unusable threshold is reported and falls back to the default rather than disabling the check silently.
A watched path whose free space cannot be read alarms as well, so a mistyped mount point is visible rather than quietly unmonitored.

Reclaiming that space when a task finishes is a separate, independent safeguard: `bin/fm-teardown.sh` calls `bin/fm-reclaim-build-scratch.sh` once a torn-down task's landed-work checks have passed, and that script's header owns which paths it will and will not remove.

## Trace context propagation (config/trace-context / FM_TRACE_CONTEXT)

The optional local, gitignored `config/trace-context` presence flag enables default-off native W3C trace-context propagation.
Expand Down Expand Up @@ -545,6 +558,7 @@ FM_STARTUP_NETWORK_TIMEOUT=120 # seconds bounding the whole deferred network s
FM_TASKS_AXI_COMPATIBLE= # internal one-hop handoff of an already-computed tasks-axi compatibility verdict (0 or 1); consumed when bin/fm-tasks-axi-lib.sh is sourced
FM_GUARD_READ_ONLY=0 # internal/read-only guard mode: keep alarms but suppress drain, supervision repair, and checkout repair commands
FM_GUARD_CONTINUE_LINE='This is a supervision warning only; the guarded operation WILL still run.' # banner continuation line; fm-send.sh overrides it to name the requested message specifically
FM_DISK_GUARD_REPEAT_SECS=3600 # seconds between repeats of the low-disk alarm in one home; see "Disk-space guard"
FM_POLL=15 # seconds between watcher poll cycles
FM_HEARTBEAT=600 # base seconds between heartbeat scans; no-change heartbeats are absorbed while idle
FM_HEARTBEAT_MAX=7200 # heartbeat backoff cap
Expand Down
3 changes: 2 additions & 1 deletion docs/scripts.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize
| `fm-test-run.sh` | Behavior-test runner: selection, portable lanes, proven-isolated `--jobs`, coverage guard, timing/JSON |
| `fm-test-isolation-proof.sh` | Concurrent isolation proof and proven-isolated candidate set owner |
| `fm-ensure-agents-md.sh` | Ensure a project's real `AGENTS.md`, its `CLAUDE.md` `@AGENTS.md` pointer, and the canonical self-governance section |
| `fm-guard.sh` | Warn on primary-checkout tangles, pending queued wakes, and unhealthy supervision |
| `fm-guard.sh` | Warn on primary-checkout tangles, low root-disk free space, pending queued wakes, and unhealthy supervision |
| `fm-primary-scope-lib.sh` | Shared marker-or-plain-checkout primary-home predicate for tracked hooks |
| `fm-session-lock-lib.sh` | Shared session-lock harness identity (ancestry walk and holder liveness) for fm-lock.sh and the Claude Stop auto-arm |
| `fm-claude-stop-autoarm.sh` | Claude Stop `asyncRewake` hook owning tokenless watcher continuity with single-flight exit-2 rewake (docs/watcher-continuity.md) |
Expand Down Expand Up @@ -108,6 +108,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize
| `fm-pr-merge.sh` | Record PR metadata, then merge a task's canonical full GitHub URL |
| `fm-promote.sh` | Promote a scout task in place to a protected ship task with an explicit delivery mode |
| `fm-teardown.sh` | Fail-closed teardown: return landed ship worktrees, require completed scout deliverables, retire secondmate homes |
| `fm-reclaim-build-scratch.sh` | Fail-closed reclaim of a finished worktree's regenerable build scratch (git-ignored install trees, build output, tool caches) |
| `fm-harness.sh` | Detect the running harness and resolve crew or secondmate harness, model, and effort |
| `fm-lock.sh` | Per-home firstmate session lock |
| `fm-x-lib.sh` | Shared Relay config, relay, and reply-threading helpers |
Expand Down
Loading
Loading