Skip to content

test: the macOS-local red baseline has grown far beyond #370's four tracked cases, including a guarded-update rehearsal family #423

Description

@evanharmon1

Invariant

Every locally-failing-but-CI-green test:skills assertion on a healthy macOS
machine is tracked by an issue that names it, so a branch author can tell a
pre-existing environmental failure from a regression their change introduced by
citation rather than by re-deriving the baseline each session.

Current violation (observed 2026-08-11)

370 tracks four diff-template sweep cases (observed 2026-08-10). During the

358/#381/#359 work, three independent sessions on the same macOS host

(Darwin 24.6.0) each measured the clean-main task verify failure set at
53 test:skills assertions — byte-identical with and without their
changes, while CI verify on the same commits stayed green. The excess over

370's four includes at least one coherent untracked family: guarded-update

rehearsal assertions (flagged during PR #409's rounds as distinct from #370's
symlink/ignore shape). Working sessions currently absorb this by diffing
their failure set against a clean-main baseline run — machinery every session
rebuilds because no issue names the full set.

Verify

git stash --include-untracked 2>/dev/null; git checkout main
task test 2>&1 | grep -c '^✗'   # or the test:skills target directly

On macOS: a count materially above the four cases #370 enumerates means the
untracked remainder still exists. On CI (or Linux): the same commit reports
zero, confirming the failures are environmental.

Acceptance criteria

  • The full macOS-local failure set on clean main is enumerated once —
    either by widening test: four diff-template sweep cases fail locally on macOS while CI verify is green #370 or in this issue — grouped into families with
    the suspected environmental cause per family (the guarded-update
    rehearsal family explicitly among them).
  • Each family gets a disposition: fix the assertion to be
    platform-honest, guard it to CI with a documented reason (the AGENTS.md
    CI-only carve-out), or fix the underlying macOS incompatibility.
  • After the dispositions land, task verify on a clean macOS checkout of
    main is green, or every remaining red line points at a tracking
    issue in its output.

Refs #370.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions