Skip to content

[stack 11/40] Consolidated benchmark corpus III #214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sarthakagrawal927 merged 2 commits into from
Sep 3, 2026
Merged

[stack 11/40] Consolidated benchmark corpus III #214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
feacf0b
test: stage consolidated benchmark corpus 3/3
sarthakagrawal927 Sep 2, 2026
369bd7f
Merge branch 'stack/09-benchmark-corpus-02' into stack/09-benchmark-c…
sarthakagrawal927 Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions benchmarks/public-catch-rate/reviews-raw/ts-hardcoded-credentials.codevetter.raw.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
{
"agent": "claude",
"changed_lines": 14,
"coordinator_used": false,
"diff_range": "HEAD~1..HEAD",
"duration_ms": 55313,
"evidence_candidates": [
{
"affected_files": ["source.ts"],
"caveats": ["The user may have run verification outside indexed agent sessions."],
"confidence": 0.66,
"evidence_refs": [
{
"detail": null,
"kind": "history",
"label": "No fresh command/test/browser proof found in compact history"
}
],
"id": "no-fresh-verification-evidence",
"kind": "not_verified",
"open_questions": ["What is the smallest relevant command or browser task for this diff?"],
"scale": "1 changed file(s)",
"severity_hint": "medium",
"suggested_checks": [
"Attach a fresh command, test, log, screenshot, or trace before marking the finding fixed."
],
"why_it_matters": "A review without fresh verification evidence can only prove static plausibility, not that the change works."
}
],
"evidence_procedure_steps": [
{
"action": "Choose the smallest relevant repo command or browser task, rerun it, and attach the fresh output.",
"artifact": "command log, test report, screenshot, trace, or QA run artifact",
"blocked_on": ["fresh verification artifact"],
"candidate_ids": ["no-fresh-verification-evidence"],
"gate": "No candidate is marked confirmed or fixed using stale or missing verification.",
"id": "rerun_relevant_verification",
"input": "Failed, stale, or missing command/test/browser evidence.",
"output": "Fresh pass/fail evidence linked to the review.",
"procedure": "rerun_relevant_verification",
"status": "blocked"
}
],
"findings": [
{
"confidence": 0.99,
"filePath": "source.ts",
"line": 7,
"severity": "critical",
"suggestion": "Rotate the production password immediately (it is already in git history). Load credentials from environment/secret manager: `password: process.env.DB_PASSWORD` with no default, and fail fast if unset. Add secret-scanning to CI to prevent recurrence.",
"summary": "Line 7 commits `password: 'P@ssw0rd-prod-2024!'` for the production host `db.prod.internal` (line 3) directly into source. The line-6 comment self-labels this as a BUG, so the stated intent contradicts the committed code. The secret is already in git history (commit d354ed0 'agent change under review'), meaning it is exposed to anyone with repo access and must be rotated, not merely deleted from source. There is no .env or .gitignore to mitigate it.",
"title": "Hardcoded production database password committed in plaintext"
},
{
"confidence": 0.93,
"filePath": "source.ts",
"line": 13,
"severity": "high",
"suggestion": "Use a real Postgres client instead of fetch, e.g. `const client = new (await import('pg')).Client(dbConfig); await client.connect(); return client;`. Then actually run connect() against a test/mocked DB and attach the output.",
"summary": "Lines 12-13 build a `postgres://...` URL and pass it to `fetch(url)`. `fetch` only supports http/https/data/blob/file schemes, so this throws `TypeError: ... URL scheme \"postgres\" is not supported` at runtime — the function cannot establish a DB connection. 0 callers (newly added, no regression), but the symbol's entire purpose is broken, and the throw shows it was never executed/verified.",
"title": "connect() uses fetch() on a postgres:// URL and can never connect"
}
],
"findings_count": 2,
"qa_evidence": [],
"review_id": "07fce861-da5f-4ba3-8f7f-e4babb582f89",
"review_memory_graph": {
"edges": [
{
"confidence": 0.74,
"from": "file-source-ts",
"kind": "has_history_context",
"to": "history-context"
},
{
"confidence": 0.68,
"from": "file-source-ts",
"kind": "has_blast_radius",
"to": "blast-radius"
},
{
"confidence": 0.66,
"from": "file-source-ts",
"kind": "raises_candidate",
"to": "candidate-no-fresh-verification-evidence"
},
{
"confidence": 0.86,
"from": "candidate-no-fresh-verification-evidence",
"kind": "requires_gate",
"to": "gate-rerun-relevant-verification"
}
],
"nodes": [
{
"detail": "changed file",
"file_path": "source.ts",
"id": "file-source-ts",
"kind": "file",
"label": "source.ts"
},
{
"detail": "306 chars in prompt section",
"file_path": null,
"id": "history-context",
"kind": "history_context",
"label": "Prior commits, decisions, agents, and command evidence"
},
{
"detail": "computed from repo relationships",
"file_path": null,
"id": "blast-radius",
"kind": "blast_radius",
"label": "Blast-radius summary"
},
{
"detail": "not_verified · confidence 0.66",
"file_path": "source.ts",
"id": "candidate-no-fresh-verification-evidence",
"kind": "evidence_candidate",
"label": "no-fresh-verification-evidence"
},
{
"detail": "blocked · No candidate is marked confirmed or fixed using stale or missing verification.",
"file_path": null,
"id": "gate-rerun-relevant-verification",
"kind": "procedure_gate",
"label": "rerun_relevant_verification"
}
],
"schema_version": 1,
"scope": "review_changed_files",
"truncated": false
},
"review_mode": "specialist-lite",
"risk_tier": "lite",
"score": 70.0,
"sensitive_paths": [],
"specialists": ["assumption-integrity", "product-safety", "agent-handoff"],
"summary": "Risk-tiered review (lite) completed.\n\nNew file source.ts contains a critical, self-acknowledged security defect: a production database password hardcoded in plaintext (line 7) that is already committed to git history and requires rotation. Separately, connect() (lines 11–13) is non-functional because it passes a postgres:// URL to fetch(), which only supports http/https and will throw at runtime. Both findings are confirmed by static inspection; no fresh runtime verification was run but none is needed to prove either defect.\n\nNew 14-line file with two real defects. Critical: plaintext production DB credentials committed at source.ts:7 (author-flagged) — full prod access leak, now permanent in git history. Product-safety-scope correctness bug: connect() (source.ts:13) calls fetch() on a postgres:// URL, which throws at runtime because Fetch only supports http/https — the connect flow is non-functional. connect() currently has 0 callers, limiting live blast radius. The no-fresh-verification-evidence gate is still open; I did not execute the code — the credential leak is proven statically, the fetch/postgres failure is derived from the Fetch spec.\n\nNew file source.ts contains a critical committed secret (plaintext prod DB password, line 7) that must be rotated, not just deleted, and a broken connect() that calls fetch() on a postgres:// URL (lines 12-13) so it can never work. No test or run evidence exists — consistent with the blocked verification gate — and the code as written would throw on first execution.",
"talk_id": "a3efe414-b70e-46ed-ace1-f92ddcde85f0"
}
133 changes: 133 additions & 0 deletions benchmarks/public-catch-rate/reviews-raw/ts-insecure-cookie.codevetter.raw.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
{
"agent": "claude",
"changed_lines": 9,
"coordinator_used": false,
"diff_range": "HEAD~1..HEAD",
"duration_ms": 46578,
"evidence_candidates": [
{
"affected_files": ["source.ts"],
"caveats": ["The user may have run verification outside indexed agent sessions."],
"confidence": 0.66,
"evidence_refs": [
{
"detail": null,
"kind": "history",
"label": "No fresh command/test/browser proof found in compact history"
}
],
"id": "no-fresh-verification-evidence",
"kind": "not_verified",
"open_questions": ["What is the smallest relevant command or browser task for this diff?"],
"scale": "1 changed file(s)",
"severity_hint": "medium",
"suggested_checks": [
"Attach a fresh command, test, log, screenshot, or trace before marking the finding fixed."
],
"why_it_matters": "A review without fresh verification evidence can only prove static plausibility, not that the change works."
}
],
"evidence_procedure_steps": [
{
"action": "Choose the smallest relevant repo command or browser task, rerun it, and attach the fresh output.",
"artifact": "command log, test report, screenshot, trace, or QA run artifact",
"blocked_on": ["fresh verification artifact"],
"candidate_ids": ["no-fresh-verification-evidence"],
"gate": "No candidate is marked confirmed or fixed using stale or missing verification.",
"id": "rerun_relevant_verification",
"input": "Failed, stale, or missing command/test/browser evidence.",
"output": "Fresh pass/fail evidence linked to the review.",
"procedure": "rerun_relevant_verification",
"status": "blocked"
}
],
"findings": [
{
"confidence": 0.98,
"filePath": "source.ts",
"line": 8,
"severity": "high",
"suggestion": "Set the security attributes: `res.cookie('session', token, { maxAge: 86400000, httpOnly: true, secure: true, sameSite: 'lax' })` (use `sameSite: 'strict'` if no cross-site flows require the cookie). Consider deriving `secure` from environment so local HTTP dev still works while production stays HTTPS-only.",
"summary": "Line 8 `res.cookie('session', token, { maxAge: 86400000 })` passes only `maxAge`; Express defaults `httpOnly`/`secure`/`sameSite` to off, so the session token is JS-readable (XSS theft), sent over plaintext HTTP (MITM), and attached cross-site (CSRF). The comment on lines 5–7 confirms this is the intended-to-be-fixed vulnerability, not desired behavior.",
"title": "Session cookie set without HttpOnly, Secure, or SameSite"
}
],
"findings_count": 1,
"qa_evidence": [],
"review_id": "b80521d7-a9de-4606-a3a0-9570128242bb",
"review_memory_graph": {
"edges": [
{
"confidence": 0.74,
"from": "file-source-ts",
"kind": "has_history_context",
"to": "history-context"
},
{
"confidence": 0.68,
"from": "file-source-ts",
"kind": "has_blast_radius",
"to": "blast-radius"
},
{
"confidence": 0.66,
"from": "file-source-ts",
"kind": "raises_candidate",
"to": "candidate-no-fresh-verification-evidence"
},
{
"confidence": 0.86,
"from": "candidate-no-fresh-verification-evidence",
"kind": "requires_gate",
"to": "gate-rerun-relevant-verification"
}
],
"nodes": [
{
"detail": "changed file",
"file_path": "source.ts",
"id": "file-source-ts",
"kind": "file",
"label": "source.ts"
},
{
"detail": "306 chars in prompt section",
"file_path": null,
"id": "history-context",
"kind": "history_context",
"label": "Prior commits, decisions, agents, and command evidence"
},
{
"detail": "computed from repo relationships",
"file_path": null,
"id": "blast-radius",
"kind": "blast_radius",
"label": "Blast-radius summary"
},
{
"detail": "not_verified · confidence 0.66",
"file_path": "source.ts",
"id": "candidate-no-fresh-verification-evidence",
"kind": "evidence_candidate",
"label": "no-fresh-verification-evidence"
},
{
"detail": "blocked · No candidate is marked confirmed or fixed using stale or missing verification.",
"file_path": null,
"id": "gate-rerun-relevant-verification",
"kind": "procedure_gate",
"label": "rerun_relevant_verification"
}
],
"schema_version": 1,
"scope": "review_changed_files",
"truncated": false
},
"review_mode": "assumption-first",
"risk_tier": "trivial",
"score": 90.0,
"sensitive_paths": [],
"specialists": ["assumption-integrity", "general"],
"summary": "Risk-tiered review (trivial) completed.\n\nSingle newly-added function `setSessionCookie`. It sets a session-token cookie with only `maxAge`, omitting `httpOnly`, `secure`, and `sameSite` — a real session-security hole (XSS exfiltration, plaintext transmission, CSRF). The in-file comment explicitly flags it as a BUG, so stated intent and code agree; the fix is to add the missing attributes on line 8. Currently 0 callers, so no live path yet, but it is the intended session-cookie helper.\n\nSingle new file adding a session-cookie helper. It sets the 'session' cookie with only maxAge, omitting httpOnly, secure, and sameSite — a real, statically-verifiable session-security defect (XSS theft, plaintext transmission, CSRF). Currently 0 callers, so runtime blast radius is nil today, but this is an auth-path helper whose intended use makes the defect high severity. No other issues in the 9 changed lines.",
"talk_id": "6d5a3ccc-46fa-436f-91f7-cb681b5d6c8a"
}
Loading
Loading