[finding] `ci-failure.mjs`'s self-test prose is parsed as a workflow command — every PR's green lint check carries a real `failure` annotation

[os-steve]

Found while measuring annotation visibility for #10803 (PR #11883). Not fixed there — out of scope for that card.

What happens

scripts/pm/ci-failure.mjs:2343 prints a line of self-test prose that names a workflow-command token literally:

'    tail with no `##[error]` in it is labelled a window rather than an anchor. And a `fix`\n' +

The Actions runner does not know the token is being quoted. It parses it, and emits a genuine annotation. Measured on a real PR head sha (f9ed462900fbeb07383386bc9ca2bbfc3de5d294, run 32783785081, job Lint & Repo Gates), via GET /repos/{owner}/{repo}/check-runs/97611285494/annotations:

level=failure  path=.github  line=32  title=
  msg=` in it is labelled a window rather than an anchor. And a `fix`

That is an annotation_level: failure on a check run whose conclusion is success.

Why it is worth fixing

1. It is wrong on its face. A failure-level annotation on a green required check is a claim that something failed. Nothing did. The step (PM ci-failure self-test, Fix CI test failures: syntax error and undefined schema references #33) passes.
2. It fires on every pull request. The step has no paths: filter, so this annotation is minted on every PR run in the repo, permanently.
3. It degrades the annotation surface that other gates depend on. This was found precisely because [finding] check-release-section-coverage --strict has no standing caller — the advisory half is the only half that runs #10803 is about whether an advisory ::warning:: is visible enough on a PR. On the head measured, there were 5 annotations across 31 check runs — and 1 of the 5 is this artifact. A repo that teaches its readers that failure-level annotations on green checks are meaningless is training them to skim exactly the surface check-release-section-coverage reports into.
4. It is self-referential: the script whose job is reading CI failure output is manufacturing a fake one.

Likely shape of the fix

Do not print the literal token. The usual options, cheapest first: break the token so the runner's parser does not match it while a human still reads it (e.g. an escaped or zero-joined spelling), or route the self-test's prose through a helper that neutralises ##[...] and ::...:: before printing. A regression pin belongs with it — this class is invisible to every local run, because the transformation only happens on a runner.

Worth checking whether any sibling script prints the same token in prose; this was found by reading one job's annotations, not by a scan.

---

Generated by Claude Code

[os-zhuang]

Triage: lands in scripts/ (ci-failure.mjs self-test prose emitting a live failure workflow annotation on green lint runs); routed finding + domain:devx, awaiting first-touch grading.

---

Generated by Claude Code

[claude]

Concentrated triage batch: finding → pm:queue, Bug, S (domain:devx stands) — a failure-level annotation on every green lint run is wrong on its face and trains readers to ignore annotations. Fix: neutralize the workflow-command token in the printed self-test prose (split or escape the literal at ci-failure.mjs:2343) so the runner cannot parse it; pin that the self-test output contains no ##[-parseable sequence. Read the one comment at claim.

---

Generated by Claude Code

[os-steve]

Claim: devx PM seat, session session_015ahemw8RcTgqtxrj15PEZx, branch claude/issue-11886-ci-failure-selftest-annotation.

Dispatching now. Label swapped pm:queue → pm:dispatched.

---

Generated by Claude Code

[claude]

{
  "issue": 11886,
  "status": "done",
  "branch": "claude/issue-11886-ci-failure-selftest-annotation",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/12018",
  "premise_still_valid": true,
  "summary": "Premise held in full and got WORSE on re-measure: the card measured the artifact as 1 of 5 annotations on one PR head; on today's main head (d63b01436, both runs) it is 1 of 1 — the two sibling `Release section coverage` warnings have since been fixed, so the only annotation `Lint & Repo Gates` carries is this one. Verified rather than inherited: lint.yml triggers on pull_request with no `paths:` filter, and neither the `lint` job nor the step carries an `if:`. Did the sweep the card said had not been done — ran all 128 self-tests CI runs and scanned their real stdout+stderr: exactly ONE live site (`ci-failure.mjs`), zero line-start `::` sites, and two mid-line `::error::`/`::warning::` lines in check-prerelease-pin-watch.mjs that are measured INERT. Two runner facts measured on real runs decided the design: the legacy `##[...]` form is parsed ANYWHERE in a line (the token sat at column 18 and was still consumed, everything before it dropped, the remainder taken as the message — which is why the annotation starts mid-sentence on a stray backtick), while `::...::` is parsed ONLY at line start (the two mid-line ones print on every PR and mint zero annotations). `failure` is the only level minted, mechanically: `error` is the only registered command name appearing in a printed token anywhere in the population. Fix: reworded the one live line so the PRINTED BYTES carry no token — recorded in code that splitting the JS string would change nothing, since the runner parses the printed line, not the source. Class closed by a new gate, `scripts/check-self-test-workflow-commands.mjs`, which RUNS the self-tests CI runs and scans real output, so 'is this token printed?' is answered by printing it rather than by a rule about source; a static prefilter only selects which to run (~14s vs ~4min for the whole population), and membership is imported from check-self-test-wired.mjs rather than re-derived. No registered-command list on purpose (GitHub's to change; a stale copy under-matches in silence) — the detectors match command SHAPE, over-matching in the safe direction, with zero measured false positives. The two inert sites are deliberately NOT touched: they are one re-indentation away from being real and the gate reddens the moment they move, so flagging them where they sit would be a false positive the next author would rightly delete.",
  "tests": "All readings at final commit e018b0024 (tree clean), exit codes captured before any pipe throughout; gate verdicts quoted from the gates' own printed lines, never a bare $?.\n\nABLATION (three legs, scripted under `trap restore EXIT INT TERM`). NO REBUILD LEG APPLIES and this is not a skipped step: both mutated files are read directly off disk — `node <path>` for the spawned self-test and a relative source import for the shared population helper — so nothing resolves through a dist/ and the mutation reaches the measurement with no build in between. Each leg confirmed on disk BEFORE any reading, by grepping BOTH the injected and the removed text (never a bare `git diff --stat`, never an editor exit code): leg A `fixed-text=0 defect-text=1 detector=1`; leg C `fixed-text=0 defect-text=1 detector=0` plus ablation-marker present=1; leg B (restore) `fixed-text=1 defect-text=0 detector=1` plus marker absent=0. Readings — A (fix reverted, detector intact): production exit=1 printing `check-self-test-workflow-commands: 1 finding(s)`, --self-test exit=1 printing `FAILED: ... scripts/pm/ci-failure.mjs --self-test still prints a line the runner would parse`. C (defect STILL on disk, detector ablated to /ZZ_ABLATED_NEVER_MATCHES_ZZ/): production exit=0 GREEN — this is what makes A mean something, proving the red came from the detector matching and not from a refusal or an unrelated path. B (both restored): exit=0 / exit=0. Direction was predicted before running and observed as predicted (turn red). Post-ablation the tree was re-verified: `git diff --stat` shows only the 2 intended tracked files and `grep -c ZZ_ABLATED` = 0.\n\nNEW GATE: `node scripts/check-self-test-workflow-commands.mjs --self-test` exit=0; production exit=0 printing `✓ check-self-test-workflow-commands: no self-test CI runs prints a line the Actions runner would parse as a workflow command.` / `scope: 128 script(s) CI runs ship a --self-test; 14 mention a workflow-command token in code (comments masked) and were RUN, and their real stdout+stderr was scanned.` Measured wall time 13.97s.\n\nFIXED SITE: `node scripts/pm/ci-failure.mjs --self-test` exit=0, and its printed line now reads `tail carrying no runner error line at all is labelled a window rather than an anchor.`\n\nDERIVED FAMILIES: `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` (no paths passed — the script took its own change set from the merge base; --repo assertion held against this checkout's origin), 3 paths, 163 families discovered, 23 local. All re-run at e018b0024, all exit=0: check-self-test-wired (+ --self-test), step-collectors, required-contexts (alias + direct), workflow-status-functions, shard-attestation, aggregator-roster, ci-filter-parity, entry-guard, parse-guard, nul-bytes, cross-package-test-inputs (alias + direct), pnpm-filter-targets, agent-test-spelling, node-version, pnpm-acquisition, whole-set-label-write, drift-comment, type-check-coverage, type-check-debt, ci-failure --self-test, check-self-test-workflow-commands.\n\nRATCHET: `check:type-check-debt` first REFUSED (`--re-measure cannot run: 56 workspace dependencies ... have no built type entry point on disk`) — a prerequisite refusal by design on an unbuilt tree, not a finding, and NOT narrowed around: built the closure under the shared lock (`os-verify-lock.sh -c \"turbo run build --filter='./packages/*' --filter='./packages/*/*' --concurrency=4\"`, VERDICT command-exit 0, 70/70 successful, held 281s) and re-ran for real at HEAD — `check-type-check-coverage --re-measure: OK — 32 ledger entr(ies) re-measured in 204.8s, 1898 raw tsc error(s) total, none above its recorded number.` Concurrency 4 rather than 2 stated as a deviation: under the shared lock this is the only heavy job by construction, and 2 risked exceeding the foreground cap and yielding no reading at all.\n\nLINT: FULL repo `pnpm lint` (= `eslint . --no-inline-config`), run under the lock at HEAD, VERDICT command-exit 0 in 76s. NOT narrowed — no narrowing is declared anywhere in this card.\n\nCONTROL BYTES: `check:nul-bytes` exit=0, plus the out-of-gate self-scan `grep -naP '[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f]'` over all three touched files — no matches.\n\nLIVE MEASUREMENTS (GitHub API, read not assumed): annotations on check-run 97611285494 (the card's head) = 3, one of them level=failure title=\"\" message=\"` in it is labelled a window rather than an anchor. And a `fix`\"; on today's main head d63b01436, check-runs 97678882948 and 97676097238 each carry exactly 1 annotation and it is that same artifact.",
  "open_questions": [],
  "out_of_scope_findings": []
}

---

Generated by Claude Code