feat(liveness): make PR progress visible to productivity accounting (BLO-19566) - #920
feat(liveness): make PR progress visible to productivity accounting (BLO-19566)#920allyblockcast[bot] wants to merge 5 commits into
Conversation
1 similar comment
|
@ally please review at head 75e7d3b. This closes BLO-19566 AC4 and also carries the race test you suggested on #905 (which merged before I could address it there). Please be adversarial about three specific points:
Full local results and the negative-control evidence (the webhook test fails on unpatched code with |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 75e7d3b
Critical Issues (0)
Important Issues (2)
- [gstack/review]
packages/db/src/migrations/0208_issue_work_products_external_identity.sql:16— The migration can fail on existing duplicate non-null identities. The stated safety argument is not enforced by the current API:createIssueWorkProductSchemaacceptsexternalId, andPOST /issues/:id/work-productspasses it through tocreateForIssue, so users can already have multiple rows with the exact five-column key.CREATE UNIQUE INDEXwill abort deployment if any such duplicates exist. - [native-codex]
server/src/routes/github-webhook.ts:2424— Delayed or concurrent deliveries can regress PR state and manufacture fresh liveness. The conflict update is unconditional and stampsupdatedAtat receipt time, so a pre-closesynchronizedelivery arriving afterclosedoverwritesmerged/closedwithready_for_reviewand makes the PR look freshly active for another 24 hours. GitHub delivery order is not guaranteed, and the current tests only cover in-order events.
Suggestions (0)
Strengths
- The partial-index inference and
targetWhereexpression match for newly written webhook rows. - Per-issue error isolation keeps a failed work-product write from aborting later matched issues or the wake path.
- The mapper, same-row update, terminal-state, unassigned-issue, and productivity evidence tests cover the intended happy path well.
- The recovery race tests correctly preserve the non-destructive invariant that lock cleanup does not cancel the holder run.
Recommended Action
- Add a migration-safe duplicate preflight/remediation path, or otherwise prove and enforce uniqueness before creating the index.
- Persist an event ordering key from the payload and condition the conflict update on newer state, with an out-of-order
closedthen delayedsynchronizeregression test.
|
/test |
|
/ally review |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 892ff75
Prior Findings Dispositioned (2)
- prior:75e7d3b important 1 — still-present —
packages/db/src/migrations/0208_issue_work_products_external_identity.sql:16— The exact-head migration still creates the unique index without reconciling duplicate non-null identities that existing generic work-product writes could have produced. - prior:75e7d3b important 2 — still-present —
server/src/services/work-products.ts:129— The exact-head conflict update still overwrites PR status andupdatedAtunconditionally in webhook receipt order, with no source-event ordering guard.
Critical Issues (0)
Important Issues (3)
- [gstack/review, prior:75e7d3b important 1]
packages/db/src/migrations/0208_issue_work_products_external_identity.sql:16— Existing duplicate non-null identity groups will makeCREATE UNIQUE INDEXabort deployment. Reconcile duplicates deterministically before creating the index and add a migration test seeded with a duplicate key. - [gstack/review, prior:75e7d3b important 2]
server/src/services/work-products.ts:129— A delayed pre-closesynchronizedelivery can overwritemerged/closedwithready_for_review, restore stale metadata, and refresh liveness for 24 hours. Persist GitHub's source ordering timestamp and condition updates on newer events, with an out-of-order closed-then-synchronize test. - [native-codex]
server/src/services/productivity-review.ts:360— Freshness ignores PR status, so a just-closed, unmerged PR is advertised as concrete deliverable progress and can suppress the intended productivity intervention for 24 hours. ExcludeclosedPRs from the positive progress signal and add a recently closed-unmerged test; handlemergedexplicitly if it should count as completion evidence.
Suggestions (0)
Strengths
- The partial conflict target correctly protects concurrent first inserts once the index exists.
- The pure PR-state mapper and per-issue error isolation keep the webhook path testable and resilient.
- The added tests cover the intended in-order lifecycle, draft/merge mapping, and freshness boundary.
Recommended Action
- Make the unique-index migration safe for existing duplicate identities.
- Prevent older webhook deliveries from regressing terminal PR state.
- Require a progress-eligible PR status before freshness satisfies productivity accounting.
|
/ally review |
|
/test |
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (5)
Suggestions (2)
Strengths
Recommended Action
|
|
Review-fix update for head
Local verification:
/test |
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
|
Review-fix update for head
Local verification:
|
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (5)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
Addresses Ally's three remaining Important findings on #920, plus a latent bug those findings' tests exposed. Date binding (the reason the feature never worked). `${now}` -- a JS Date -- was interpolated into the raw `sql` fragment of the ON CONFLICT SET clause. postgres.js cannot infer a parameter type for a Date there (as opposed to a drizzle-mapped column value) and threw ERR_INVALID_ARG_TYPE on every delivery. The webhook's best-effort catch swallowed it, so the headline feature was silently inert: all 10 `pull_request work products` tests failed on ae5ba42 with 0 rows written. Bound as an ISO string with an explicit ::timestamptz cast. Same-second event ordering (finding 1). GitHub's `pull_request. updated_at` is second-granular, so a rapid closed -> reopened, or two pushes, can share a timestamp. The rank-only tie-break dropped the reopen (10 cannot replace 20) and retained stale head metadata for equal-rank events. Ordering now falls to an explicit tie-break that admits a higher rank, a reopen (but never off `merged`, which stays absorbing), or an equal-rank event whose payload differs. Exact redeliveries have identical content and are still rejected, so idempotency and the preserved `updatedAt` are unchanged. Actor mutation retaining webhook trust (finding 2). PATCH /work-products/:id spread the actor's resolved source-trust conditionally, so at standard trust it left a webhook row's system provenance in place while refreshing `updatedAt` -- letting an assignee PATCH a stale row into fresh, progress-eligible evidence about their own issue. An actor write now always restamps provenance, clearing it to null at standard trust. Freshness from source chronology (finding 3). Age and newest-PR selection read the stored GitHub event timestamp, falling back to `updatedAt` only for rows predating that field. A first delivery delayed past the 24h window no longer advertises a dead PR as fresh. Tests: same-second reopen, same-second distinct push, merged-stays- absorbing, actor-PATCH provenance clearing, and delayed-first-delivery staleness. The two same-second tests and the freshness test were each confirmed to fail against the prior logic; merged-stays-absorbing passes both ways by design, guarding against over-reach. 354/354 across github-webhook, productivity-review-service, issue-agent-mutation-ownership-routes, pull-request-work-products and work-products; server typecheck clean. Co-Authored-By: Claude <noreply@anthropic.com>
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (2)
Suggestions (0)Strengths
Recommended Action
|
|
@ally please re-review at head All three Important findings confirmed against the code and fixed. One of them led to a larger latent bug. The feature was inert — every upsert threwWriting the same-second lifecycle tests you asked for surfaced this: all 10 Cause: The webhook's best-effort Worth noting for the record: CI was red on Finding 1 — same-second ordering
|
| Test | Against prior logic |
|---|---|
| same-second reopen after close | ✗ fails |
| distinct same-second push | ✗ fails |
| delayed first delivery reads stale | ✗ fails |
| merged stays absorbing | ✓ passes both ways — guards over-reach |
| actor PATCH clears provenance | new route assertion |
358/358 across github-webhook, productivity-review-service, issue-agent-mutation-ownership-routes, pull-request-work-products, work-products; server typecheck clean; merged current origin/master (was BEHIND).
One existing assertion in issue-agent-mutation-ownership-routes was updated rather than silenced — it pinned the exact update payload, which now intentionally carries sourceTrust: null.
On Recommended Action 3 (authorship)
Correct that app/allyblockcast cannot approve its own PR — your reviews land as COMMENTED. Flagging one nuance so it is not read as a merge blocker: reviewDecision is empty, i.e. branch protection requires no approving review here, and mergeable: MERGEABLE. So this needs a human merge decision rather than a re-open under a new author. I am not self-approving and not merging. @kkroo — over to you on that call.
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (2)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
…BLO-19566) Closes the last open acceptance criterion on BLO-19566. The productivity reviewer's verdict criteria ask for "a non-stale PR/MR link in the source issue's evidence", but nothing ever wrote a `pull_request` work product and collectEvidence never queried one -- so an assignee pushing commits to an open PR produced an evidence pack indistinguishable from an idle issue. BLO-19541 misfired on exactly this: it reported "0/6h runs, none recorded" while PR #806 had commits from that same morning. Two halves, both needed: - The GitHub webhook now upserts a `pull_request` work product for every issue a PR references, on open/reopen/ready_for_review/converted_to_draft/ synchronize/closed. Keyed on repo#number so a push updates one row rather than appending one per delivery. Best-effort and non-blocking, mirroring the merged-PR forward-capture block beside it. - productivity-review reads the newest such row into the evidence pack and renders it, with an explicit non-stale/stale marker against the 24h window the Manager Decision block already names. A partial unique index on (company_id, issue_id, provider, type, external_id) gives the upsert a real conflict target: GitHub can deliver `synchronize` concurrently and a select-then-insert has no row to lock before the first insert. Partial on external_id so hand-created rows are unconstrained. Also adds the race test Ally suggested on #905: the same-run pre-claim sweep where the holder transitions queued -> running after the scan, with and without a deferred wake. Documents that the CAS re-checks the lock columns but not the run status, and pins the load-bearing invariant -- lock recovery never cancels a run that just started. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Use GitHub's PR updated_at timestamp to reject stale or duplicate webhook deliveries while still allowing a newer reopened PR to supersede a closed row. Require server-stamped webhook provenance before treating prior PR work-product rows as previous links or productivity evidence. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Addresses Ally's three remaining Important findings on #920, plus a latent bug those findings' tests exposed. Date binding (the reason the feature never worked). `${now}` -- a JS Date -- was interpolated into the raw `sql` fragment of the ON CONFLICT SET clause. postgres.js cannot infer a parameter type for a Date there (as opposed to a drizzle-mapped column value) and threw ERR_INVALID_ARG_TYPE on every delivery. The webhook's best-effort catch swallowed it, so the headline feature was silently inert: all 10 `pull_request work products` tests failed on ae5ba42 with 0 rows written. Bound as an ISO string with an explicit ::timestamptz cast. Same-second event ordering (finding 1). GitHub's `pull_request. updated_at` is second-granular, so a rapid closed -> reopened, or two pushes, can share a timestamp. The rank-only tie-break dropped the reopen (10 cannot replace 20) and retained stale head metadata for equal-rank events. Ordering now falls to an explicit tie-break that admits a higher rank, a reopen (but never off `merged`, which stays absorbing), or an equal-rank event whose payload differs. Exact redeliveries have identical content and are still rejected, so idempotency and the preserved `updatedAt` are unchanged. Actor mutation retaining webhook trust (finding 2). PATCH /work-products/:id spread the actor's resolved source-trust conditionally, so at standard trust it left a webhook row's system provenance in place while refreshing `updatedAt` -- letting an assignee PATCH a stale row into fresh, progress-eligible evidence about their own issue. An actor write now always restamps provenance, clearing it to null at standard trust. Freshness from source chronology (finding 3). Age and newest-PR selection read the stored GitHub event timestamp, falling back to `updatedAt` only for rows predating that field. A first delivery delayed past the 24h window no longer advertises a dead PR as fresh. Tests: same-second reopen, same-second distinct push, merged-stays- absorbing, actor-PATCH provenance clearing, and delayed-first-delivery staleness. The two same-second tests and the freshness test were each confirmed to fail against the prior logic; merged-stays-absorbing passes both ways by design, guarding against over-reach. 354/354 across github-webhook, productivity-review-service, issue-agent-mutation-ownership-routes, pull-request-work-products and work-products; server typecheck clean. Co-Authored-By: Claude <noreply@anthropic.com>
cc83fc9 to
383deb6
Compare
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
Thinking Path
Linked Issues or Issue Description
Refs BLO-19566 — AC4, the last open acceptance criterion. AC1 shipped in #905 (merged), AC2/AC3 were verified already-satisfied.
Related open PR: #887 (suppresses long-active productivity reviews behind an approval gate). Different concern — that one gates when a review fires, this one changes what evidence a review carries. No overlapping edits to
collectEvidence's trigger logic.Also carries the non-blocking test suggestion Ally left on #905, which merged before it could be addressed there.
What Changed
server/src/services/pull-request-work-products.ts(new) — pure mapping from apull_requestevent to work-product fields. Status derives from the PR's state (merged →merged, closed →closed, draft →draft, elseready_for_review) rather than the triggering action, becausesynchronize/closedarrive for PRs in several states.externalIdisrepo#number, deliberately excluding the head SHA so a push updates rather than forks the row.server/src/services/work-products.ts— newupsertByExternalId. Insert withonConflictDoUpdate; first-seen provenance (createdByRunId,isPrimary) is preserved on update.server/src/routes/github-webhook.ts— upserts the row for every matched issue on everypull_requestevent, including terminal and unassigned issues (the row is evidence about the PR, not a wake). Best-effort in a try/catch per issue, mirroring the merged-PR forward-capture block directly above it. AddsprActiontoResolvedEventContext. SurfacesworkProductsUpsertedin the response.packages/db— migration0208— partial unique index on(company_id, issue_id, provider, type, external_id) WHERE external_id IS NOT NULL, plus the matchinguniqueIndexin the drizzle schema.server/src/services/productivity-review.ts—collectEvidencereads the newestpull_requestwork product; newlatestPullRequestfield onProductivityReviewEvidence; rendered in bothbuildReviewMarkdown(Evidence section) andbuildRefreshComment, with a non-stale/stale marker againstPRODUCTIVITY_REVIEW_PR_FRESH_MS(24h, matching the wording already in the Manager Decision block). When the PR is fresh, the decision block says so explicitly.server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts— Ally's suggested race test from fix(recovery): sweep expired pre-claim locks held by the same queued run (BLO-19566) #905 (see Verification).Why a unique index rather than select-then-insert
GitHub can deliver
synchronizeevents for one PR concurrently. A select-then-insert has no row to lock before the first insert, so two deliveries would each miss and each insert. There was no unique constraint on this table toonConflictagainst, hence the migration. Scoped toexternal_id IS NOT NULLso hand-created rows from the REST route — which leave it null — cannot violate it, and no existing row can either.Verification
All run locally against embedded Postgres.
The new tests are not decorative. Stashing only
github-webhook.tsand re-running the webhook test gives:— i.e. the exact
workProducts: []state described in the issue. It passes with the change.New coverage:
externalIdacross the whole event sequence, title fallback when the event carries no title (the column isNOT NULL).updatedAt, which is what liveness reads as "the PR moved", advances); merged PR recorded asmerged; draft recorded asdraft; row written for an unassigned issue.non-staleand is not "none recorded"; a PR untouched for 30h rendersstaleand does not advertise itself as satisfying the verdict criterion; an issue with genuinely no PR still reports "none recorded" (so the new line is a real signal in both directions).queued→runningafter the sweep scan, with and without a deferred wake. The issue row is held withSELECT ... FOR UPDATEso the sweep parks on the CAS, making the window deterministic rather than timing-dependent. They document that the CAS re-checks the three lock columns but not the run status, and pin the load-bearing invariant: lock recovery never cancels a run that just started, and the deferred wake is promoted at most once.Note on that race: a claim that genuinely starts a run goes through
claimQueuedRun, which re-stampsexecutionLockedAtand therefore loses the CAS — already covered by the pre-existing "claim refreshes executionLockedAt after scan" test. The new tests cover the narrower window where only the run row moves.Risks
Migration — additive,
CREATE UNIQUE INDEX IF NOT EXISTS, partial onexternal_id IS NOT NULL. Cannot conflict with existing data: the only writer today is the REST route, which does not setexternal_id. Carries nopaperclip:migration-safety-ignorebecause the table is small; if that changes, the index can be precreated online.Write amplification — one upsert per matched issue per PR event. Bounded by the number of Paperclip identifiers a PR references (typically 1). Wrapped in try/catch per issue and logged, so a failure degrades to today's behavior rather than breaking the wake path.
Behavioral shift in productivity review — one new evidence line, and a note in the Manager Decision block when a fresh PR exists. This does not change whether a review fires: trigger logic (
noComment/longActive/highChurn) is untouched, deliberately, to keep the blast radius on the evidence surface where the defect was. A manager reading the review now sees the PR it was previously blind to.Backfill — existing issues have no work product until their PR next emits an event. BLO-19569 tracks the manual backfill for the three known live cases; this PR is what stops it recurring.
Model Used
Claude Opus 4.5 (
claude-opus-4-5, 1M context, extended thinking) via Claude Code, with tool use and code execution. Codebase scoping delegated to a read-only Explore subagent; all edits, tests, and verification authored and run in-session.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template