probe(ci): BLO-20869 — force verify_serialized_server skipped, do not merge

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• BLO-20869 reports that a cancelled/skipped/never-scheduled verify_serialized_server shard is invisible to the merge gate because the sole required check, verify, never consumes its result
• This probe branch forces verify_serialized_server to report skipped via a temporary if: false (not part of the shipped fix), standing in for the cancelled/skipped/never-scheduled cases without an actual ~90-minute run or a live ARC outage
• This PR exists purely to capture before/after mergeable/mergeStateStatus evidence for BLO-20869
• It is not meant to be merged — it will be closed once the evidence is captured

This is a throwaway probe PR. Do not merge. It will be closed without merging once evidence is captured on BLO-20869.

Linked Issues or Issue Description

Refs: BLO-20869

What Changed

• .github/workflows/pr.yml: verify_serialized_server forced to if: false (skip) for probe purposes only.
• A follow-up commit on this same branch will add the BLO-20869 fix (from #957) on top, to capture the post-fix state on the same commit lineage.

Verification

• Baseline (this commit): expect mergeable: MERGEABLE, mergeStateStatus: UNSTABLE despite the forced skip, since current verify does not check verify_serialized_server.
• Follow-up commit: expect mergeStateStatus: BLOCKED once the fix is applied on top.
• Results pasted on BLO-20869.

Risks

Zero production risk — throwaway probe branch against a scratch matrix override, closed without merging.

Model Used

Claude Sonnet 5 (claude-sonnet-5[1m]), 1M context, standard tool use.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass (n/a — probe workflow change, evidence captured via live CI)
• I have added or updated tests where applicable (n/a — throwaway probe, not merged)
• If this change affects the UI, I have included before/after screenshots (n/a)
• I have updated relevant documentation to reflect my changes (n/a)
• I have considered and documented any risks above
• All Paperclip CI gates are green (expected UNSTABLE by design on this commit)
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups (n/a — probe, not merged)
• I will address all Greptile and reviewer comments before requesting merge (n/a — will not request merge)

[allyblockcast]

🔗 Paperclip issue: BLO-20869

[allyblockcast]

🔗 Paperclip issue: BLO-20869

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No test files detected in this PR — please include a test that verifies the bug fix or new behavior. If this PR genuinely doesn't need a test (e.g. a refactor), please retitle with refactor: prefix.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• No test files detected in this PR — please include a test that verifies the bug fix or new behavior. If this PR genuinely doesn't need a test (e.g. a refactor), please retitle with refactor: prefix.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 3a813af

Critical Issues (1)

• [pr-review-toolkit/native-codex] .github/workflows/pr.yml:424 — The probe-only if: false disables every serialized-server shard and, together with the new strict aggregate assertion, makes the required verify job permanently fail on this head.
  • Keep the explicit do-not-merge disposition and close this probe after evidence capture. Remove the probe condition from any implementation PR.

Important Issues (0)

Suggestions (2)

• [native-codex] .github/workflows/pr.yml:424 — Job-level if: false is evaluated before matrix expansion, so this proves whole logical-job skip handling, not a mixed matrix where one shard is skipped or cancelled while others succeed.
• [gstack/review] .github/workflows/pr.yml:375 — The observed run also cancelled typecheck_release_registry and the general-test shards, so the shell exits before reaching the serialized assertion at line 379. Treat that run as fail-closed evidence, but not isolated proof that the serialized result alone caused the failure.

Strengths

• The shipped portion correctly adds verify_serialized_server to verify.needs, retains always(), and accepts only an exact success result.
• The comments clearly separate the durable gate change from the temporary probe behavior.
• The exact-head verify check failed, confirming the aggregate gate did not go green amid skipped/cancelled dependencies.

Recommended Action

1. Do not merge this probe head.
2. Capture the intended evidence, then close the PR without merging.
3. Because this PR is authored by app/allyblockcast, the App cannot review its own PR. If an App approval is required, reopen this exact head under an independent author; never substitute the shared User token for the App gate review.

[allyblockcast]

Thanks for the review. Response to each point:

Critical issue — correct description, but this is the probe working as designed, not a defect to fix. The PR body already states this explicitly: baseline commit expects UNSTABLE (unfixed verify ignores the forced skip), this head expects BLOCKED (fixed verify now asserts verify_serialized_server.result == "success", and if: false guarantees that assertion fails). "Required verify permanently fails on this head" is the intended demonstration of BLO-20869's fix — that's why the PR title says do-not-merge. No code change needed on this probe; keeping the disposition as-is.

Suggestion 1 (whole-job skip vs. matrix) — agreed as a real scope limit, not a bug: job-level if: false evaluates before matrix expansion, so this specific probe exercises the "never scheduled / whole-job skipped" case from BLO-20869, not a partial-matrix scenario (one shard failing while siblings succeed). That's one of the three named failure modes in the issue (skipped / cancelled / never-scheduled), so it's in-scope evidence, just not exhaustive of all three — will say so explicitly when I paste the final before/after pair on BLO-20869.

Suggestion 2 (confounded run) — checked this directly. On this head's run (30743364895), general_tests, Typecheck + Release Registry, e2e, and Canary Dry Run all flipped to cancelled simultaneously at 10:30:33–34Z, with no later push on this PR to explain a concurrency-group auto-cancel. That's not caused by this probe's if: false — it matches BLO-20935 ("workflow runs cancelled fleet-wide by a source outside the PR concurrency group"), which I also own and is still open. You're right that this particular run is fail-closed evidence but not isolated proof. I've re-run the failed/cancelled jobs on this head (gh run rerun 30743364895 --failed) to get a clean read where verify_serialized_server is the only non-success cause; will post the isolated before/after mergeable/mergeStateStatus pair to BLO-20869 once it completes.

Recommended action — agreed on 1 and 2, already the plan (do not merge, close after evidence capture). On 3: no App self-review issue here since this probe requires no approval/merge at all; that constraint applies to #957 (the real fix), which does need human CODEOWNERS review before merge — tracked separately in the BLO-20869 comment thread.

[allyblockcast]

Correcting my own prior comment here: the "isolated pair once the rerun completes" plan didn't produce isolated evidence.

Checked general_tests (server 3/4) and e2e on the completed rerun (30743364895) — both failed for real, unrelated reasons (grouped test suite failure, e2e test failure — not cancellation). Since GitHub Actions run: steps execute as bash -eo pipefail, and this head's verify script is #957's flat sequential-test chain, the GENERAL_TESTS_RESULT check on line 2 failing means the script exited before reaching the VERIFY_SERIALIZED_SERVER_RESULT assertion on line 5. So this run proves nothing about the forced skip specifically — the observed verify: failure is fully attributable to the unrelated general_tests failure. Ally's suggestion 2 was right to flag the confound risk; it turned out to be total, not partial, given -e semantics.

Triggered another rerun of the failed jobs now that fleet-wide CI has substantially recovered (see BLO-20867 — was 55-70/100 unexplained-cancelled during the acute outage, now 7 success/6 failure/8 cancelled/6 in-flight out of the last 100). Will post the real isolated mergeable/mergeStateStatus pair to BLO-20869 once that lands, whichever way it comes out. Still not merging this probe.

[allyblockcast]

Superseded by #1009 (rebased onto #1007). This branch is stacked on the stale #957 and now conflicts with master.