Skip to content

feat: add the reviewer registry that makes the triage queue answerable - #407

Merged
mlorentedev merged 1 commit into
masterfrom
feat/reviewer-registry
Aug 25, 2026
Merged

feat: add the reviewer registry that makes the triage queue answerable#407
mlorentedev merged 1 commit into
masterfrom
feat/reviewer-registry

Conversation

@mlorentedev

Copy link
Copy Markdown
Owner

Closes #403.

The problem

$ dotf pr triage-queue
pr triage-queue: read reviewer registry: open harness/review-attestation.json: no such file or directory
$ echo $?
1

That command exits non-zero for two different reasons — work is pending, or the question could not be answered — deliberately, so an unanswerable queue is never mistaken for a clear one. Here it was always the second. The consequence: hive could produce no queue-clear evidence for any PR, ever, so every PR closed with the Definition of Done's evidence item skipped rather than met.

The non-obvious part: which marker identifies a review

This is not a copy of the dotfiles registry, and the reason is worth reading before reviewing the file.

CodeRabbit files proper reviews through the reviews API on this repo. Measured on #401: a 5-finding review reading **Actionable comments posted: 5**, sitting in reviews[]. But the queue reads gh pr list --json comments, which returns issue comments only and never reviews[]. Matching the review's own text would have left the queue permanently empty on a repository that is actively being reviewed — the exact false clear the mechanism exists to prevent.

So the registered marker is the walkthrough comment CodeRabbit posts alongside every review, which is comment-shaped. Verified present on both PRs open at the time of writing:

PR Walkthrough marker Rate-limit marker What it was
#401 a real 5-finding review
#406 a decline

The limitation this leaves is stated in the file, not left to be rediscovered: the walkthrough is edited rather than reposted on re-review, and createdAt does not move on an edit — so a second review after a push may not re-open the queue for a PR already triaged once. The queue is a floor, not a ceiling. Closing that needs a consumer that reads reviews[], which is a change to the dotfiles CLI, not to this file.

advisory: true, measured here rather than inherited

On #405 and #406, CodeRabbit was rate limited, posted "Review limit reached", and its commit status read success. A green check that means no review ran is precisely the failure this whole mechanism exists to catch. Advisory means its decline does not refuse the change — a quota is a billing state of a third party, not a property of the diff — while its review still attests.

Note what the two settings do together, and it is intended: the decline notice also carries the walkthrough marker, so a rate-limited PR reads as output awaiting disposition rather than as a clear queue. That is the honest answer.

Deliberately absent

The escape and exempt blocks dotfiles carries. Those are read by its review-attestation CI gate, which hive does not run — prtriage reads reviewers and triage.marker and nothing else. A registry declaring policy no consumer here enforces is drift with extra steps. The practical consequence: release-please PRs are not exempted and will queue for a one-line disposition.

Verified end to end

$ dotf pr triage-queue
1 pull request(s) awaiting a disposition:

  #405   chore(master): release 4.1.1
         coderabbitai reviewed, never triaged, 2026-08-24 20:47

Both directions confirmed: #406 correctly omitted (it carries a ## Review triage comment) and #405 correctly surfaced (it does not). The queue found a real open PR I did not know existed.

AGENTS.md gains two bullets in the PR workflow section: how to run the queue and where to record the disposition, and the warning that CodeRabbit's green status can mean it never reviewed.

Closes #403.

`dotf pr triage-queue` reads harness/review-attestation.json to decide
which open PRs carry reviewer output nobody has dispositioned. The file
did not exist here, so the command could not answer and exited 1 on
every run — and since it exits non-zero both when work is pending and
when the question is unanswerable, hive could produce no queue-clear
evidence for any PR at all. Every PR in this repo closed with the
Definition of Done's evidence item skipped rather than met.

Only coderabbitai is registered. dotfiles additionally declares
github-actions for PR-Agent; no such workflow runs here, and declaring
a reviewer that cannot run would assert a review that never happens.

The non-obvious part is which marker identifies its output. CodeRabbit
files proper reviews through the reviews API — #401 carries a 5-finding
review reading "Actionable comments posted: 5" in reviews[] — but the
queue reads `gh pr list --json comments`, which never returns reviews[].
Matching the review text would leave the queue permanently empty on a
repository that is actively being reviewed. The registered marker is
therefore the walkthrough comment posted alongside every review, which
is comment-shaped; verified present on both open PRs, #401 (a review)
and #406 (a decline). The limitation this leaves — the walkthrough is
edited rather than reposted on re-review, and createdAt does not move on
an edit — is stated in the file rather than left to be rediscovered.

advisory: true on the decline, measured here and not inherited: on #406
and #405 CodeRabbit was rate limited and its commit status read
`success`. A green check meaning no review ran is the failure this
mechanism exists to catch, so AGENTS.md now says so in the PR workflow
alongside how to run the queue.

Verified end to end: the queue now answers, correctly omits #406 (which
carries a triage comment) and correctly surfaces #405 (which does not).
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 26 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 63202e68-df10-4440-bc39-984e82b50153

📥 Commits

Reviewing files that changed from the base of the PR and between 66dbf17 and a8f2b6c.

📒 Files selected for processing (2)
  • AGENTS.md
  • harness/review-attestation.json

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mlorentedev

Copy link
Copy Markdown
Owner Author

Review triage

CI: all 7 check runs green (check 3.12/3.13, cross_worker_lock ubuntu + windows, docker, GitGuardian, add-to-project).

Review: none ran. CodeRabbit's commit status reads success; its comment is "Review limit reached". Third PR in a row today with that pattern (#405, #406, this one), which is the measurement behind this PR's own advisory: true setting. This PR is unreviewed.

There is a circularity worth naming rather than glossing: the change that teaches this repo not to trust CodeRabbit's green status is itself merging on an unreviewed green status. That is not a reason to hold it — it is the argument for it — but it does mean the registry lands without an independent read of its marker choice, which is the one judgement call in the file.

Findings: none to dispose of.

Verified in this session, not just asserted:

One wrinkle found while testing, worth knowing before merge: the registry path is CWD-relative (harness/review-attestation.json), so dotf pr triage-queue only answers from a branch that has the file. Running it from fix/hive-384-spec-artifacts mid-session gave the original "no such file" error. This is invisible on dotfiles because the file merged long ago, and it self-resolves here once this hits master — but until then, the queue's answerability depends on which branch is checked out. Not worth a code change; worth not being surprised by.

Recommendation: merge on the human's call, with the unreviewed status on the record.

@mlorentedev
mlorentedev merged commit 2492151 into master Aug 25, 2026
8 checks passed
@mlorentedev
mlorentedev deleted the feat/reviewer-registry branch August 25, 2026 03:23
mlorentedev added a commit that referenced this pull request Aug 25, 2026
Lesson 096, from the #403 / #407 work. It merged before this was
written; this carries the part worth keeping.

## The finding

`dotf pr triage-queue` calls `gh pr list --json comments`. GitHub
exposes a PR's discussion through **three separate surfaces** — issue
comments, the reviews API (`reviews[]`), and inline review comments —
and that call returns **only the first**.

On this repo CodeRabbit files *proper* reviews. #401's five findings
live in `reviews[]` as `**Actionable comments posted: 5**`; `comments`
held only the walkthrough. So registering the review's own text as the
marker would have produced a queue that was **permanently, silently
empty on a repository that was being reviewed the whole time**.

The dotfiles registry does not hit this because its second reviewer,
PR-Agent, publishes *through* the comments API — which is why its marker
does not transfer, and why "port the config" was the wrong mental model
for #403.

## Why it is worth a lesson rather than a commit message

The direction of failure. A wrong marker does not error — it returns
*"no reviewer output yet"*, which is also what a genuinely quiet PR
looks like. Same family as
[087](docs/lessons/lesson-087-a-green-review-check-can-mean-did-not-review.md)
and
[094](docs/lessons/lesson-094-a-verification-command-that-selects-zero-tests.md):
**a check that cannot see its subject reports the subject is fine.**

Two rules fall out:

1. When porting a config between repos, verify the **observable** it
keys on against live data in the destination. One `gh api
repos/…/pulls/N/reviews` versus `…/issues/N/comments` settles it in ten
seconds; reading the source registry for an hour does not.
2. Prove a matcher in **both** directions. This one was confirmed by
watching the queue correctly omit a triaged PR *and* correctly surface
an untriaged one it found on its own — #405, a real open release PR
nobody had looked at, invisible until the registry existed.

Docs only. All wikilinks verified against the working tree.
mlorentedev added a commit that referenced this pull request Aug 26, 2026
🤖 I have created a release *beep* *boop*
---


## [4.2.0](v4.1.0...v4.2.0)
(2026-08-25)


### Features

* add the reviewer registry that makes the triage queue answerable
([#407](#407))
([2492151](2492151))


### Bug Fixes

* **spec:** record measured evidence on HIVE-384, and stop AC5 reading
green ([#408](#408))
([5fd5aa6](5fd5aa6))


### Documentation

* correct lesson-094's premise and three reviewer findings from
[#401](#401)
([#406](#406))
([66dbf17](66dbf17))
* record five lessons from the delegate-verb work
([#401](#401))
([8bc66eb](8bc66eb))
* record why the reviewer registry could not be copied
([#409](#409))
([2b920ba](2b920ba))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hive has no reviewer registry, so dotf pr triage-queue cannot answer and exits 1 on every run

1 participant