Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 11 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,22 @@
# PRBot

PRBot is a precision-first, multi-agent pull request reviewer that runs entirely as a GitHub Action.
PRBot is a precision-first pull request reviewer that runs entirely as a GitHub Action.
It uses OpenRouter models, an ephemeral local Git object store, syntax-aware related-file discovery, bounded read-only repository tools, and independent finding verification.

Status: experimental.

## How reviews work

PRBot does more than send GitHub patch fragments to one model.
PRBot gives one primary reviewer the complete selected change set and lets it investigate with bounded repository tools.

1. It authorizes the triggering GitHub user before making any LLM call.
2. It fetches the exact pull request base and head into an ephemeral bare Git repository.
3. It computes the authoritative local diff, including deletions, renames, and multiline changes.
4. It builds a relationship map from imports, symbols, references, matching tests, manifests, and directory structure.
5. It assigns every eligible changed hunk to a semantic review bundle.
6. It asks a routing agent to select architecture, security, performance, and documentation specialists for relevant bundles.
7. It always runs correctness reviewers, then runs the selected specialists concurrently with bounded read-only tools.
8. It independently verifies every candidate finding.
9. It resolves exact diff anchors, removes duplicates, creates one sectioned GitHub review, updates one persistent summary, and publishes a check against the pull request head.
6. It sends every selected bundle to one primary reviewer with bounded read-only tools.
7. It independently verifies every candidate finding.
8. It resolves exact diff anchors, removes duplicates, creates one GitHub review, updates one persistent summary, and publishes a check against the pull request head.

Syntax-aware symbol extraction supports Rust, TypeScript, JavaScript, Python, and Go.
Other supported source and configuration files use import heuristics and bounded code search.
Expand Down Expand Up @@ -93,14 +92,14 @@ Action inputs are hard ceilings:

| Input | Default | Purpose |
| --- | ---: | --- |
| `review_model` | `deepseek/deepseek-v4-flash` | Routing and specialist model |
| `review_model` | `deepseek/deepseek-v4-flash` | Primary review model |
| `verification_model` | `deepseek/deepseek-v4-flash` | Independent verification model |
| `max_review_minutes` | `15` | Wall-clock deadline |
| `max_input_tokens` | `500000` | Total estimated input-token ceiling |
| `max_cost_usd` | `3.00` | Estimated model-cost ceiling |
| `max_concurrency` | `8` | Concurrent model calls |
| `max_comments` | `12` | Maximum published inline findings |
| `engine` | `contextual` | Default multi-agent engine; set `legacy` to roll back |
| `engine` | `contextual` | Default primary-review engine; set `legacy` to roll back |
| `dry_run` | `false` | Build and print the manifest without LLM or GitHub writes |

PRBot currently uses `deepseek/deepseek-v4-flash` for both review and independent verification.
Expand Down Expand Up @@ -130,13 +129,12 @@ Hierarchical `AGENTS.md` files from the base revision are also applied to matchi
## Review output

PRBot publishes at most one formal review per run.
The review contains separate correctness, architecture, security, performance, and documentation sections.
Each section reports whether its agent was completed, skipped by the router, or failed.
The review contains one Precision review section that reports whether the primary reviewer completed, skipped, or failed.
It supports right-side additions, left-side deletions, context lines, multiline anchors, and file-level fallback when an anchor is ambiguous.
The model supplies exact anchor text, while deterministic code resolves and validates the GitHub line range.

The Documentation Steward reports concrete drift in README files, `docs/**/*.md`, and user-facing examples.
It names the required correction but never writes repository files and never requests changes to `AGENTS.md`.
The primary reviewer can report concrete documentation drift in README files, `docs/**/*.md`, and user-facing examples.
It never receives `AGENTS.md` patch content or direct access to those files.

PRBot publishes a `PRBot review` check against the exact pull request head.
The check succeeds only when coverage is complete and no verified findings remain.
Expand Down Expand Up @@ -185,7 +183,7 @@ Important source boundaries:
```text
src/review/ Event authorization and orchestration
src/repository/ Git snapshots, diffs, context graph, and read-only tools
src/agents/ Routing, parallel specialist reviewers, and verification
src/agents/ Primary review, verification, and prompts
src/reporting/ Anchor resolution, fingerprints, and summary state
src/github/ Paginated GitHub API client and batched publishing
src/llm.rs OpenRouter tool loop, concurrency, and budget ledger
Expand Down
4 changes: 2 additions & 2 deletions action.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: "PRBot"
description: "Multi-agent PR reviewer powered by OpenRouter, runs as a GitHub Action."
description: "Precision-first PR reviewer powered by OpenRouter, runs as a GitHub Action."
author: "PRBot contributors"

branding:
Expand All @@ -24,7 +24,7 @@ inputs:
required: false
default: "false"
review_model:
description: "OpenRouter model used by routing and specialist review agents."
description: "OpenRouter model used by the primary review agent."
required: false
default: "deepseek/deepseek-v4-flash"
verification_model:
Expand Down
23 changes: 23 additions & 0 deletions docs/research/future_checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Future Review Checklist

Current flow: `PR -> one reviewer -> verifier -> comments`.

## Later improvements

- Cache the Action image, Git objects, diff, file reads, and context by base SHA.
- Reuse a completed review only when head SHA, base SHA, models, config, and PRBot version match.
- Keep one reviewer and verifier by default.
- Add one specialist only for high-risk changes: auth, payments, migrations, APIs, or concurrency.
- Set per-task limits for cost, time, tokens, and tool calls.
- Settle reserved budget against actual provider usage, and stop optional tasks before the verifier budget is at risk.
- Retry only HTTP 429 and 5xx responses with `Retry-After` or jittered backoff.
- Record stage latency, tokens, cost, retries, completion rate, precision, P0/P1 recall, and resolution rate.
- Add multi-pass or multi-agent review only if evals prove a quality gain worth the additional cost.

## Industry ideas

- Cursor Bugbot: dynamic context, validation, deduplication, and resolution-rate optimization.
- Codex: adapt depth to PR complexity, follow repository instructions, and optionally validate risky changes in a sandbox.
- GitHub Copilot: repository-wide and path-specific review instructions.

Sources: [Cursor Bugbot](https://cursor.com/blog/building-bugbot), [Codex](https://openai.com/index/introducing-upgrades-to-codex/), and [Copilot](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/request-a-code-review/use-code-review?tool=vscode).
142 changes: 116 additions & 26 deletions src/agents/integration_tests.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use super::review_manifest;
use super::{review_bundles, review_manifest};
use crate::config::ReviewConfig;
use crate::llm::{Budget, LlmClient};
use crate::repository::{GitRepository, RepositoryTools};
Expand All @@ -16,11 +16,9 @@ use std::thread;
use std::thread::JoinHandle;

#[tokio::test]
async fn routes_reviews_and_verifies_findings_end_to_end() {
async fn primary_reviewer_verifies_findings_end_to_end() {
let (address, server) = mock_server(vec![
r#"{"assignments":[{"agent":"architecture","bundle_ids":["bundle"],"rationale":"public contract changed"}]}"#,
r#"{"findings":[{"path":"src/lib.rs","side":"RIGHT","anchor":"pub fn value() -> i32 { 2 }","priority":"P1","category":"correctness","title":"Changed result","body":"Existing callers require one.","evidence":[],"confidence":0.95}]}"#,
r#"{"findings":[]}"#,
r#"{"accepted_indices":[0]}"#,
]);

Expand All @@ -41,33 +39,22 @@ async fn routes_reviews_and_verifies_findings_end_to_end() {
let result = review_manifest(&client, tools, &manifest, &config).await;

assert!(result.failed_bundles.is_empty());
assert!(!result.router_fallback);
assert_eq!(result.findings.len(), 1);
assert_eq!(result.findings[0].agent, ReviewAgent::Correctness);
let architecture = result
.agent_runs
.iter()
.find(|run| run.agent == ReviewAgent::Architecture)
.expect("architecture run");
assert_eq!(architecture.bundle_ids, ["bundle"]);
let security = result
.agent_runs
.iter()
.find(|run| run.agent == ReviewAgent::Security)
.expect("security run");
assert!(security.bundle_ids.is_empty());
assert_eq!(result.findings[0].agent, ReviewAgent::Primary);
assert_eq!(result.agent_runs.len(), 1);
assert_eq!(result.agent_runs[0].agent, ReviewAgent::Primary);
assert_eq!(result.agent_runs[0].bundle_ids, ["bundle"]);
let requests = server.join().expect("server");
assert_eq!(requests.len(), 4);
assert!(requests[0].contains("Route these review bundles"));
assert!(requests[3].contains("accepted_indices"));
assert_eq!(requests.len(), 2);
assert!(requests[0].contains("Review these selected pull-request bundles"));
assert!(!requests[0].contains("Route these review bundles"));
assert!(requests[1].contains("accepted_indices"));
}

#[tokio::test]
async fn verifier_failure_marks_review_coverage_incomplete() {
let (address, server) = mock_server(vec![
r#"{"assignments":[{"agent":"architecture","bundle_ids":["bundle"],"rationale":"public contract changed"}]}"#,
r#"{"findings":[{"path":"src/lib.rs","side":"RIGHT","anchor":"pub fn value() -> i32 { 2 }","priority":"P1","category":"correctness","title":"Changed result","body":"Existing callers require one.","evidence":[],"confidence":0.95}]}"#,
r#"{"findings":[]}"#,
"not-json",
]);
let fixture = RepositoryFixture::new();
Expand Down Expand Up @@ -95,10 +82,84 @@ async fn verifier_failure_marks_review_coverage_incomplete() {
let correctness = result
.agent_runs
.iter()
.find(|run| run.agent == ReviewAgent::Correctness)
.expect("correctness");
.find(|run| run.agent == ReviewAgent::Primary)
.expect("primary");
assert_eq!(correctness.candidate_findings, 1);
assert_eq!(server.join().expect("server").len(), 4);
assert_eq!(server.join().expect("server").len(), 2);
}

#[tokio::test]
async fn primary_reviewer_receives_every_selected_bundle_in_one_request() {
let (address, server) = mock_server(vec![r#"{"findings":[]}"#]);
let fixture = RepositoryFixture::new();
let repository = Arc::new(
GitRepository::from_worktree(&fixture.root, &fixture.base, &fixture.head)
.expect("repository"),
);
let tools = Arc::new(RepositoryTools::new(repository, "PR context".to_owned()));
let budget = Arc::new(Budget::new(1, 100_000, 10.0));
let client =
LlmClient::new("key", Some(format!("http://{address}/chat")), budget, 1).expect("client");
let result = review_manifest(
&client,
tools,
&manifest_with_two_bundles(),
&ReviewConfig::default(),
)
.await;

assert!(result.failed_bundles.is_empty());
assert!(result.findings.is_empty());
assert_eq!(result.agent_runs[0].bundle_ids, ["bundle", "second"]);
let requests = server.join().expect("server");
assert_eq!(requests.len(), 1);
assert!(requests[0].contains("src/lib.rs"));
assert!(requests[0].contains("src/second.rs"));
}

#[tokio::test]
async fn primary_reviewer_failure_skips_verification() {
let (address, server) = mock_server(vec!["not-json"]);
let fixture = RepositoryFixture::new();
let repository = Arc::new(
GitRepository::from_worktree(&fixture.root, &fixture.base, &fixture.head)
.expect("repository"),
);
let tools = Arc::new(RepositoryTools::new(repository, "PR context".to_owned()));
let budget = Arc::new(Budget::new(1, 100_000, 10.0));
let client =
LlmClient::new("key", Some(format!("http://{address}/chat")), budget, 1).expect("client");
let result = review_manifest(&client, tools, &manifest(), &ReviewConfig::default()).await;

assert_eq!(result.failed_bundles, ["primary-reviewer"]);
assert_eq!(
result.agent_runs[0].status,
crate::types::AgentStatus::Failed
);
assert_eq!(server.join().expect("server").len(), 1);
}

#[tokio::test]
async fn empty_bundle_selection_skips_model_calls() {
let fixture = RepositoryFixture::new();
let repository = Arc::new(
GitRepository::from_worktree(&fixture.root, &fixture.base, &fixture.head)
.expect("repository"),
);
let tools = Arc::new(RepositoryTools::new(repository, "PR context".to_owned()));
let budget = Arc::new(Budget::new(1, 100_000, 10.0));
let client = LlmClient::new("key", Some("http://127.0.0.1:1/chat".to_owned()), budget, 1)
.expect("client");
let manifest = manifest();
let result = review_bundles(&client, tools, &manifest, &[], &ReviewConfig::default()).await;

assert!(result.findings.is_empty());
assert!(result.failed_bundles.is_empty());
assert_eq!(result.agent_runs.len(), 1);
assert_eq!(
result.agent_runs[0].status,
crate::types::AgentStatus::Skipped
);
}

fn manifest() -> ReviewManifest {
Expand Down Expand Up @@ -141,6 +202,35 @@ fn manifest() -> ReviewManifest {
}
}

fn manifest_with_two_bundles() -> ReviewManifest {
let mut manifest = manifest();
manifest.files.push(ChangedFile {
path: "src/second.rs".to_owned(),
old_path: None,
status: FileStatus::Added,
patch: "@@ -0,0 +1 @@\n+pub fn second() {}\n".to_owned(),
hunks: vec![DiffHunk {
header: "@@ -0,0 +1 @@".to_owned(),
old_start: 0,
new_start: 1,
lines: vec![DiffLine {
side: DiffSide::Right,
old_line: None,
new_line: Some(1),
content: "pub fn second() {}".to_owned(),
}],
}],
});
manifest.bundles.push(ReviewBundle {
id: "second".to_owned(),
paths: vec!["src/second.rs".to_owned()],
hunk_count: 1,
risk: RiskLevel::Low,
related_files: Vec::new(),
});
manifest
}

struct RepositoryFixture {
_temp: tempfile::TempDir,
root: std::path::PathBuf,
Expand Down
Loading
Loading