docs: competitive gap analysis for CodeRabbit-class replacement

.github/workflows/ci.yml

@@ -42,8 +42,23 @@ jobs:
       - name: Fixture catalog present
         run: |
           test "$(wc -l < evals/fixtures/catalog.jsonl | tr -d ' ')" = "50"
+          cp evals/fixtures/catalog.jsonl /tmp/catalog-before.jsonl
           python3 scripts/generate_fixture_catalog.py
           test "$(wc -l < evals/fixtures/catalog.jsonl | tr -d ' ')" = "50"
+          # Regenerating must not shrink protected rows if any exist.
+          python3 - <<'PY'
+          import json
+          from pathlib import Path
+          before = [json.loads(line) for line in Path('/tmp/catalog-before.jsonl').read_text().splitlines() if line.strip()]
+          after = [json.loads(line) for line in Path('evals/fixtures/catalog.jsonl').read_text().splitlines() if line.strip()]
+          protected = {row['case_id'] for row in before if row.get('status') in {'ready','adjudicated','retired'}}
+          after_ids = {row['case_id'] for row in after}
+          missing = protected - after_ids
+          assert not missing, missing
+          PY
+
+      - name: Fixture batch runner help
+        run: python3 scripts/run_fixture_batch.py --help
 
       - name: Build release
         run: cargo build --release

AGENTS.md

@@ -25,15 +25,15 @@ Flow:
 1. A consumer workflow calls this Action.
 2. Docker starts `entrypoint.sh`.
 3. `entrypoint.sh` runs `prbot review`.
-4. The CLI talks to GitHub and OpenRouter, runs multi-agent review, then posts PR feedback.
+4. The CLI talks to GitHub and OpenRouter, runs a primary review plus independent verification, then posts PR feedback.
 
 Important paths:
 
 - `src/main.rs` - CLI entrypoint and subcommands
 - `src/config.rs` - review config, defaults, and `.prbot.toml` loading
 - `src/types.rs` - shared review, finding, and outcome types
 - `src/review/` - review orchestration (args, events, commands, contextual and legacy engines)
-- `src/agents/` - router, specialist tasks, verifier, and prompts
+- `src/agents/` - primary reviewer, verifier, and prompts
 - `src/llm.rs` / `src/llm/` - OpenRouter client and token/cost budgets
 - `src/repository/` - ephemeral Git store, diffs, syntax context, and read-only tools
 - `src/reporting/` - anchors, dedupe, and summary comment rendering

CHANGELOG.md

@@ -3,6 +3,19 @@
 
 ## [Unreleased]
 
+### Added
+- OpenRouter HTTP 429/5xx retries with reserve-once budgeting
+- Configurable collaborator `min_permission` (`admin` default; `maintain`/`write` supported)
+- Risk-scaled review depth and optional multipass primary review with majority merge
+- Human-oriented walkthrough section in the summary comment
+- Resolution-rate tracking across incremental fingerprint lifecycle
+- Fixture batch runner and catalog preservation for adjudicated eval cases
+
+### Changed
+- Docs and package description align with the live primary reviewer + independent verifier flow
+- Specialist routing remains an eval harness and future option, not the shipped review engine
+- Summary state version is now 4 for resolution-rate fields
+
 ## [1.0.2] - 2026-07-28
 
 ### Changed

Cargo.toml

@@ -2,7 +2,7 @@
 name = "prbot"
 version = "1.0.2"
 edition = "2021"
-description = "Open-source multi-agent PR reviewer for GitHub Actions"
+description = "Precision-first PR reviewer for GitHub Actions (primary review + independent verification)"
 license = "MIT"
 repository = "https://github.com/jaibhasin/prbot"
 readme = "README.md"

README.md

@@ -48,8 +48,8 @@ After the first publish, set that package visibility to **Public** in the repo P
 
 ## How it works
 
-- PRBot automatically reviews pull requests authored by users with GitHub `admin` permission.
-- Users with GitHub `admin` permission can request a review on any pull request by commenting `/prbot review`.
+- PRBot automatically reviews pull requests authored by collaborators who meet `min_permission` (default: GitHub `admin`).
+- Collaborators who meet `min_permission` can request a review on any pull request by commenting `/prbot review`.
 - It fetches the exact pull request revisions and analyzes them as read-only Git data.
 - It maps related code and tests, reviews the relevant changes, and independently verifies each potential finding.
 - It posts one GitHub review with verified comments and a `PRBot review` check.
@@ -66,9 +66,14 @@ Add a trusted `.prbot.toml` file to narrow review paths or provide repository-sp
 exclude = ["**/generated/**", "**/*.lock"]
 instructions = ["Prioritize user-visible correctness regressions."]
 max_comments = 8
+min_permission = "write"
+primary_passes = 1
 ```
 
-Supported owner commands:
+Set `min_permission` to `write` or `maintain` when ordinary collaborators should be able to trigger reviews.
+The Action default remains `admin` for backward-compatible installs.
+
+Supported commands:
 
 ```text
 /prbot review

action.yml

@@ -51,6 +51,14 @@ inputs:
     description: "Maximum number of verified inline findings published per run."
     required: false
     default: "12"
+  primary_passes:
+    description: "Number of diversified primary review passes (1-3). Default 1."
+    required: false
+    default: "1"
+  min_permission:
+    description: "Minimum collaborator permission for auto-review and /prbot commands (admin, maintain, or write). Default admin."
+    required: false
+    default: "admin"
   engine:
     description: "Review engine: contextual (default) or legacy rollback."
     required: false

docs/research/competitive_gap_analysis.md

@@ -0,0 +1,255 @@
+# Competitive Gap Analysis: Replacing CodeRabbit-Class Reviewers
+
+Date: 2026-07-28
+
+Scope: PRBot codebase vs CodeRabbit, Cursor Bugbot, Greptile, and GitHub Copilot Code Review.
+
+Related:
+
+- [`future_checklist.md`](./future_checklist.md)
+- [`implementation_plan_do_now.md`](./implementation_plan_do_now.md)
+
+## Verdict
+
+PRBot already has a strong precision-first core: tool-using primary review, independent verification, anchor resolution, fingerprint dedupe, incremental re-review, and hard budgets.
+
+That architecture is closer to Cursor Bugbot than to CodeRabbit.
+
+CodeRabbit wins today on product surface (walkthroughs, chat, learnings, lint hybrid, polish), not on a fundamentally different review idea.
+
+You should not chase CodeRabbit feature parity next.
+
+You should chase measured review confidence: adjudicated evals, resolution rate, multi-pass recall, and a walkthrough that makes humans trust the bot.
+
+Until those land, PRBot should remain a precision complement, not a full replacement.
+
+## What PRBot already does well
+
+### Precision-first review loop
+
+Current live flow is `PR -> one primary reviewer -> independent verifier -> anchored comments`.
+
+This matches the product claim in `README.md` and `docs/research/future_checklist.md`.
+
+It does not match older marketing/history that still describes routed specialists.
+
+Key files:
+
+- `src/agents/mod.rs` - single primary pass over selected bundles, then verifier
+- `src/agents/verifier.rs` - rejects low-confidence and P3 findings
+- `src/reporting/anchors.rs` - exact line anchors, file-level fallback, fingerprints
+- `src/review/incremental.rs` - re-review only affected bundles
+
+### Agentic context without executing PR code
+
+Repository tools (`list_tree`, `read_file`, `read_diff`, `search_code`, `find_symbol`, `find_references`, `get_pr_context`) give Cursor-like exploration inside a read-only sandbox.
+
+This is a real differentiator versus pure diff-paste reviewers.
+
+Safety posture is strong: no PR code execution, no model-chosen shell, base-branch config trust, AGENTS.md protected from being a finding target.
+
+### Production wiring that many Action prototypes lack
+
+Already shipped:
+
+- formal GitHub review + summary comment + check run
+- stale-head detection and one retry
+- command gate (`/prbot review|ask|explain`)
+- `.prbot.toml` include/exclude/path instructions
+- AGENTS.md ingestion as trusted instructions
+- cost/time/token budgets
+- dry-run and eval-json paths
+
+This is more than a prompt wrapper.
+
+It is already an Action product skeleton.
+
+## Competitive landscape (2026)
+
+| Product | Strength | Weakness | Closest PRBot analog |
+| --- | --- | --- | --- |
+| CodeRabbit | Walkthrough UX, path instructions, learnings, 40+ linters, chat/autofix, multi-SCM | Can be noisy; less transparent internals | Summary + path rules, but much thinner UX |
+| Cursor Bugbot | High precision, resolution-rate hill-climbing, multi-pass then agentic tools | Cursor ecosystem / GitHub-focused | Primary + verifier + tools is the closest peer |
+| Greptile | Whole-repo semantic index for cross-file bugs | Heavier infra; recall/noise tradeoffs vary by bench | Related-file graph + tools, without durable index |
+| Copilot Code Review | Zero-friction GitHub-native | Weaker customization and depth | Not the target; different distribution model |
+
+Sources informing this table:
+
+- [Building a better Bugbot](https://cursor.com/blog/building-bugbot)
+- [CodeRabbit walkthroughs](https://docs.coderabbit.ai/pr-reviews/walkthroughs)
+- [CodeRabbit path instructions](https://docs.coderabbit.ai/configuration/path-instructions)
+- Industry comparisons from Monterail, Context Rankings, Macroscope, and MorphLLM (2026)
+
+## Gap map
+
+### A. Confidence and quality (highest leverage)
+
+#### 1. No real quality hill-climb loop
+
+Bugbot's published lesson is blunt: qualitative taste plateaus; resolution rate unlocked improvement.
+
+PRBot has gate thresholds in `scripts/evaluate.py`, but:
+
+- `evals/fixtures/catalog.jsonl` is still `pending_adjudication`
+- Qodo scoreboard is empty
+- router evals target a specialist architecture that is not live
+
+Without an adjudicated corpus and an online resolution metric, every prompt/model change is guesswork.
+
+**Improve now:** adjudicate a first real fixture set (start with 50), instrument resolution rate from prior fingerprints vs later commits, and treat that as the primary quality KPI.
+
+#### 2. Single-pass recall ceiling
+
+Bugbot's early breakthrough was eight parallel passes with majority voting before the validator.
+
+PRBot does one primary pass (max 6 tool steps) then one verifier.
+
+The verifier improves precision; it cannot recover bugs the primary never proposed.
+
+**Improve now:** add 2-3 diversified primary passes (different diff order or bundle subsets), cluster findings, keep majority or high-confidence unions, then verify.
+
+Do this before rebuilding a specialist router.
+
+#### 3. Prompt and tool harness are still thin
+
+`src/agents/prompts/primary.rs` and `verifier.rs` are short generic contracts.
+
+They are good for safety and speed, weak for deep defect hunting.
+
+Bugbot found agentic loops need aggressive investigation prompts, not only restraint.
+
+Also: `max_steps: 6` is low for large cross-file PRs, and `max_concurrency` is mostly unused for review parallelism because primary and verifier are sequential.
+
+**Improve now:** expand investigation playbooks by category, raise tool-step budgets for high-risk bundles, and parallelize work where cost allows.
+
+#### 4. OpenRouter resilience gap
+
+GitHub calls retry on 429/5xx.
+
+OpenRouter fails fast.
+
+Transient provider rate limits can fail an entire review stage and produce incomplete coverage failures.
+
+This is already on `future_checklist.md` and remains high priority.
+
+### B. Human reviewer experience (why CodeRabbit feels irreplaceable)
+
+CodeRabbit's moat is not only bug finding.
+
+It is reducing human cognitive load.
+
+Missing relative to CodeRabbit:
+
+1. PR walkthrough with file/cohort narrative and optional sequence diagrams
+2. High-level summary that orients reviewers before they read diffs
+3. Conversational thread replies on findings (`chat.auto_reply`)
+4. Learnings from dismissed or corrected comments
+5. Committable suggestions / autofix hooks
+6. Hybrid deterministic tools (linters/SAST) alongside LLM findings
+
+PRBot posts verified findings and a status summary.
+
+It does not yet help a human understand the change.
+
+`/prbot ask` and `/prbot explain` exist, but they are command-gated, not ambient conversation on review threads.
+
+**Improve now:** ship a strong walkthrough summary (change narrative + risk highlights + file groups).
+
+Defer autofix, Slack, and Change Stack-like UI until trust in findings is high.
+
+### C. Product/policy friction
+
+#### Admin-only gate is too strict for replacement use
+
+README and code require GitHub `admin` for auto-review and commands.
+
+CodeRabbit and Copilot typically serve ordinary collaborators.
+
+For many teams, admin-only means PRBot will not run on the PRs that need it most.
+
+**Improve now:** support `write`/`maintain` with a config knob, keep a separate stricter mode for high-security repos.
+
+#### Review event is always COMMENT
+
+Blocking is only via check conclusion.
+
+Teams used to CodeRabbit/pre-merge checks may want optional `REQUEST_CHANGES` or richer pre-merge policy.
+
+Lower priority than quality and walkthrough, but needed for "replacement" feel.
+
+#### Docs and runtime drift
+
+`AGENTS.md`, `CHANGELOG`, and Cargo description still imply routed specialists.
+
+Runtime has only `ReviewAgent::Primary`.
+
+This confuses contributors and oversells current capability.
+
+**Improve now:** align docs to the live architecture; keep specialists as an eval-gated future item.
+
+### D. Context depth
+
+Strengths already exist: related-file graph, syntax helpers, bundle risk, path instructions, linked issue context.
+
+Gaps:
+
+- no durable repo index (Greptile-style) for very large monorepos
+- markdown/docs files are mostly outside the supported review extension set, which weakens "stale docs" goals
+- linked-issue parsing is naive (first `#N`)
+- risk levels inform prompts but do not change depth, specialists, or budgets
+- caching of git objects/context by base SHA is still future work
+
+**Improve now:** review documentation files when path rules ask for it, and use risk to scale depth (steps/passes/budget), not just prompt text.
+
+## What not to prioritize yet
+
+These are attractive CodeRabbit features, but they will not create replacement confidence by themselves:
+
+- multi-platform SCM support
+- poem / tone / dashboard polish
+- autofix coding agents
+- unit-test or docstring generation
+- full specialist router with many agents
+- Change Stack-like external review UI
+
+Add specialists only when evals show a quality gain worth the cost, as `future_checklist.md` already states.
+
+## Recommended roadmap
+
+### Now (confidence blockers)
+
+1. Adjudicate eval fixtures and wire CI to a real (even small) quality gate beyond smoke.
+2. Define and track resolution rate from fingerprint state across PR lifetime.
+3. Add multi-pass primary review with clustering + majority/confidence merge, then existing verifier.
+4. Retry OpenRouter 429/5xx with backoff.
+5. Align docs/architecture claims with the live primary+verifier design.
+6. Add a human-oriented walkthrough section to the summary comment.
+
+### Next (replacement readiness)
+
+1. Relax auth policy with configurable collaborator permissions.
+2. Ambient replies on PRBot review threads (not only `/prbot ask`).
+3. Persist finding lifecycle: open, fixed, outdated, dismissed.
+4. Risk-scaled depth and budgets; optional one high-risk specialist.
+5. Suggested patches for high-confidence findings.
+6. Optional deterministic lint/SAST ingestion as extra evidence, not as noisy comments.
+
+### Later (scale and moat)
+
+1. Cache Action image/Git/context by base SHA.
+2. Durable codebase index for large repos.
+3. Online experiment harness for prompt/model/tool changes.
+4. Learnings store from human dismissals and corrections.
+5. Autofix only after resolution rate is stable and high.
+
+## Design principle going forward
+
+Optimize for bugs humans fix, not comments humans ignore.
+
+Bugbot more than doubled resolved bugs per PR by measuring resolution rate and experimenting ruthlessly.
+
+CodeRabbit remains sticky because humans feel oriented and can converse with the review.
+
+PRBot can win a third position: Action-native, model-flexible, precision-first, and measurable.
+
+That is a credible replacement path for teams that want CodeRabbit-like GitHub presence without SaaS lock-in, but only after quality is proven rather than asserted.