fix(ci): distinguish infrastructure cancellation from lane failure in verify

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• The pr.yml workflow's verify job is the single required check that gates every merge into master
• Right now, verify cannot tell "an upstream lane genuinely failed" apart from "an upstream lane was cancelled by CI infrastructure before it produced a result" — both collapse into the same test "$X" = "success" exit-1
• That matters right now specifically: an ongoing incident (tracked as internal ticket BLO-20867 in Paperclip's own tracker, not a GitHub issue in this repo — repo-wide pr.yml cancellation, ARC runs-on: default listener pod churn correlated with mass mid-job cancellations across unrelated PRs) means every affected author sees an identical red verify failure and reasonably assumes their own diff broke something
• This pull request splits the verify step's outcome into a distinct "lane failure" vs "infrastructure cancellation" annotation, so the check output tells the author which one actually happened
• The benefit is fewer authors misattributing an infra outage to their own code, and a machine-checkable signal for future measurement of the underlying incident (does verify still say "cancellation" after infra is fixed?)

Linked Issues or Issue Description

No GitHub issue exists in this repo for the underlying incident — it's tracked in Paperclip's internal issue tracker as BLO-20867 (PR CI runs are being cancelled fleet-wide), which isn't a GitHub-repo issue so it can't be tagged with Fixes #/Closes #. Describing inline per the bug report template instead:

What happened?

pr.yml's verify job (the legacy required-check name that gates merge) runs a bash test "$X_RESULT" = "success" per upstream lane. When a lane's result is cancelled — which is currently happening fleet-wide due to ARC runs-on: default listener-pod churn interrupting jobs mid-run, unrelated to any PR's diff — verify fails exactly the same way as when a lane genuinely fails a test. The PR author sees one plain red "failure" either way and cannot tell an infra outage from a real regression without digging into gh run view job timings themselves.

Expected behavior

verify should surface which outcome actually happened — a distinct annotation (or, ideally, a distinct check conclusion) for "an upstream lane was cancelled by infrastructure" versus "an upstream lane failed because of this PR's code" — so authors don't misattribute infra noise to their own diff.

Steps to reproduce

1. Open any PR against Blockcast/paperclip while the ARC arc-default/arc-dind runner-scale-set listener is churning (see BLO-20867 for the live incident evidence: k8s events showing the listener pod killed/rescheduled to a different node twice within ~2 minutes, exactly correlated with a fleet-wide mass-cancellation window).
2. Watch any runs-on: default job (typecheck_release_registry, general_tests, worktree_install, build) get cancelled mid-run with no test having actually failed.
3. Observe the verify required check reports a plain failure, indistinguishable in the GitHub UI from a real test failure.

Paperclip version or commit

This repo at master as of 2026-08-02 (base commit 3250910c4).

Deployment mode

Self-hosted server (GitHub Actions on this repo's own self-hosted ARC runners, not a local/dev concern).

What Changed

• .github/workflows/pr.yml: the verify job's "Fail if any split verify lane failed" step now buckets each upstream lane result into cancelled vs everything-else-non-success, and emits a distinct ::error title=verify: infrastructure cancellation:: vs ::error title=verify: lane failure:: annotation before exiting non-zero. A run with both a real failure and a cancellation reports as a failure (the real problem takes priority).
• scripts/__tests__/pr-verify-lane-outcome.test.mjs: extracts the actual run: shell script out of pr.yml and executes it under bash for four scenarios (all-success, one cancelled, one failed, mixed) — asserts exit code and which annotation is emitted, so the logic is verified by more than eyeballing YAML.
• Wired the new test into policy's existing node --test list in pr.yml, next to the sibling release-verify-workflow.test.mjs check.
• scripts/__tests__/npmrc-devdeps-guard.test.mjs: updated an existing assertion that matched the old literal test "$WORKTREE_INSTALL_RESULT" = "success" string (this PR's own CI run caught this — the guard broke on first push). Now asserts the new lane_results map still wires in worktree_install, preserving the actual guarantee (a non-success worktree_install result still gates verify) without depending on the removed literal.

Verification

• node --test ./scripts/__tests__/pr-verify-lane-outcome.test.mjs ./scripts/__tests__/npmrc-devdeps-guard.test.mjs — all pass locally (verified both before and after the guard-test fix above).
• Manually reviewed the edited run: block for indentation/whitespace (no tabs, consistent block-scalar indent) since I could not run a full YAML parser in this environment (no pyyaml/js-yaml installed and no network pip/npm install available in my worktree) — the extraction-and-execute test exercises the real script text, which also guards against a syntax error in the block silently no-oping.
• This PR's own first CI push caught a real regression (the npmrc-devdeps-guard test) and the policy job is green on the follow-up push — see run history on this PR.

Risks

• Low risk, CI-only change; no product code touched. Worst case if the bucketing logic has a bug: verify still fails on any non-success lane (same net effect as before), just possibly with a less accurate annotation.
• pr.yml itself is unreliable right now — this repository is mid-incident (BLO-20867: 0/60 successful pr.yml runs in the trailing ~5.5h at time of writing, cause: ARC arc-default/arc-dind listener pod churn). I can't guarantee a fully green run on this PR before merge for reasons unrelated to this diff. verify itself runs on arc-light (the pool BLO-20867's findings say is not the one churning), so it should be able to complete even mid-incident, but I can't promise that from here.

Model Used

Claude Sonnet 5 (claude-sonnet-5[1m]), Claude Code / Agent SDK, agent role PlatformSREEngineer. Standard (non-extended) reasoning mode, tool use (bash, k8s read-only MCP, GitHub MCP) for the underlying BLO-20867 investigation that motivated this change.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above (none found touching this specific verify step)
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — N/A, no UI
• I have updated relevant documentation to reflect my changes — inline comments explain the "why" in the workflow file and the guard test
• I have considered and documented any risks above
• All Paperclip CI gates are green — cannot confirm, see Risks: repo is mid fleet-wide pr.yml cancellation incident (BLO-20867)
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending automated review
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-20867

[allyblockcast]

🔗 Paperclip issue: BLO-20867

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 9110c56

Important Issues (1)

• [pr-review-toolkit / gstack / native-codex] .github/workflows/pr.yml:386 — skipped lanes are misreported as real PR failures. GitHub Actions exposes needs.<job>.result as success, failure, cancelled, or skipped, but the wildcard branch puts skipped into failed_lanes. Because all four split lanes depend on policy, an infrastructure cancellation of policy makes those lanes skipped; verify then emits "This reflects a real problem with the PR diff," recreating the misdiagnosis this change is meant to prevent. The new tests cover success, failure, and cancellation, but not skipped dependency propagation.
  • Handle skipped explicitly without attributing it to the PR. For accurate root-cause classification, include policy in verify.needs and use its result to distinguish policy failure from policy cancellation, then add fixtures for skipped lanes after each policy outcome.

Strengths

• The test executes the shell extracted from the workflow rather than duplicating its classification logic.
• Failure precedence for a directly failed lane plus a directly cancelled lane is explicit and covered.

Recommended Action

1. Address the Important issue this cycle and add the missing skipped propagation tests.

The PR author is app/allyblockcast, so the Ally GitHub App cannot review its own PR. Reopen this exact head under an independent author before an App approval is possible.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 1418bdc

Prior Findings Dispositioned (1)

• prior:9110c56 important 1 — fixed — .github/workflows/pr.yml:394 — skipped now has a dedicated classification and annotation instead of falling through to failed_lanes; the extracted-script tests also cover skipped-only and mixed-result precedence.

Looks good. No Critical or Important issues remain.

Strengths

• The tests execute the shell extracted from the workflow itself, avoiding drift between test logic and production CI logic.
• Failure, skipped, and cancelled outcomes have explicit precedence and dedicated regression coverage, including unknown-result fail-closed behavior.

The PR author is app/allyblockcast, so the Ally GitHub App cannot review its own PR. This exact head must be reopened under an independent author before an App approval is possible.

[allyblockcast]

Closing as a duplicate of #964 — same fix (verify job cancelled-vs-failed lane classification, BLO-20867), opened independently by an earlier heartbeat of mine before I'd noticed #964 already existed with an active Ally review thread. #964 now has the more complete version (handles skipped lanes distinctly, and uses neutral wording for cancelled per Ally's 12:07Z review rather than asserting it's definitely infrastructure) — that's the one to land. Consolidating so review effort isn't split across two copies of the same change.

[allyblockcast]

Duplicate of #964.