[stack 10/40] Consolidated benchmark corpus II

benchmarks/public-catch-rate/README.md

@@ -0,0 +1,154 @@
+# CodeVetter Public Benchmark
+
+A public, hand-labeled benchmark for measuring whether code review / security
+analysis tools actually catch known issues. Each case is a small code snippet
+with one or more **hand-labeled** expected findings. The cases are intentionally
+synthetic and self-contained so anyone can reproduce a score: drop a tool's
+output into `reviews/<case-id>.json` and run the scorer.
+
+This exists so enterprise claims about CodeVetter (or any reviewer) are backed
+by **external, repeatable proof** instead of internal fixtures that cannot be
+audited.
+
+## Layout
+
+```
+benchmarks/public-catch-rate/
+  cases/
+    <case-id>/
+      source.<ext>   # the code snippet with known issues
+      label.json     # hand-labeled ground truth: type, severity, location, description
+  reviews/           # gitignored; drop a reviewer's output here per case
+    <case-id>.json
+  README.md          # this file
+```
+
+Each `label.json` has the shape:
+
+```json
+{
+  "id": "ts-sql-injection",
+  "title": "SQL injection via string concatenation in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "security",
+  "ground_truth": [
+    {
+      "id": "sql-injection-email-concat",
+      "type": "sql_injection",
+      "severity": "high",
+      "location": { "file": "source.ts", "lines": [14, 14] },
+      "description": "User-controlled emailInput is concatenated directly into the SQL query string ..."
+    }
+  ]
+}
+```
+
+A reviewer output file (`reviews/<case-id>.json`) has the shape:
+
+```json
+{
+  "case_id": "ts-sql-injection",
+  "reviewer": "codevetter",
+  "findings": [
+    {
+      "id": "f-1",
+      "type": "sql_injection",
+      "severity": "high",
+      "file": "source.ts",
+      "lines": [14, 14],
+      "title": "SQL injection via string concatenation",
+      "matched_ground_truth": ["sql-injection-email-concat"],
+      "rationale": "Identifies the same concatenated user input into the SQL string."
+    }
+  ]
+}
+```
+
+`matched_ground_truth` lists the ground-truth ids the finding catches. Findings
+with an empty `matched_ground_truth` count as false positives.
+
+## Cases (27)
+
+| Case | Language | Category | Issue type |
+| --- | --- | --- | --- |
+| ts-sql-injection | TypeScript | security | sql_injection |
+| py-hardcoded-secret | Python | security | hardcoded_secret |
+| go-race-condition | Go | concurrency | race_condition |
+| ts-xss | TypeScript | security | xss |
+| py-path-traversal | Python | security | path_traversal |
+| js-eval-injection | JavaScript | security | code_injection |
+| rust-integer-overflow | Rust | bug | integer_overflow |
+| ts-dead-code | TypeScript | maintainability | dead_code |
+| py-command-injection | Python | security | command_injection |
+| go-errcheck | Go | bug | unchecked_error |
+| ts-hardcoded-credentials | TypeScript | security | hardcoded_secret |
+| py-weak-hash | Python | security | weak_crypto |
+| java-insecure-random | Java | security | insecure_random |
+| ts-prototype-pollution | TypeScript | security | prototype_pollution |
+| py-sql-injection | Python | security | sql_injection |
+| go-sql-injection | Go | security | sql_injection |
+| ts-missing-await | TypeScript | bug | missing_await |
+| py-bare-except | Python | bug | swallowed_error |
+| js-open-redirect | JavaScript | security | open_redirect |
+| ts-insecure-cookie | TypeScript | security | insecure_cookie |
+| py-ssrf | Python | security | ssrf |
+| go-hardcoded-credentials | Go | security | hardcoded_secret |
+| ts-regex-dos | TypeScript | security | regex_dos |
+| py-zip-bomb | Python | security | resource_exhaustion |
+| ts-type-confusion | TypeScript | bug | type_confusion |
+| py-insecure-deserialization | Python | security | insecure_deserialization |
+| go-nil-pointer | Go | bug | nil_dereference |
+
+Coverage spans TypeScript, JavaScript, Python, Go, Rust, and Java across
+security, concurrency, bug, and maintainability categories.
+
+## Running the scorer
+
+From the repo root:
+
+```bash
+# Validate every case and print a scorecard of the labeled ground truth.
+# This requires no reviewer output and always works.
+npm run bench:public
+
+# Score a reviewer's output after dropping files into benchmarks/public-catch-rate/reviews/.
+npm run bench:public -- --reviewer=codevetter
+
+# Emit a JSON scorecard.
+npm run bench:public -- --reviewer=codevetter --json
+
+# Write a Markdown scorecard to disk.
+npm run bench:public -- --reviewer=codevetter --format=markdown --out=artifacts/public-benchmark.md
+
+# Gate on minimum catch rate (exits non-zero when below threshold).
+npm run bench:public -- --reviewer=codevetter --min-rate=0.8
+```
+
+## How to evaluate a tool against this benchmark
+
+1. For each case in `benchmarks/public-catch-rate/cases/<case-id>/`, feed `source.<ext>` to your
+   reviewer (CodeVetter or any comparator).
+2. Normalize the reviewer's findings into the `reviews/<case-id>.json` shape
+   above, filling `matched_ground_truth` with the ground-truth ids each finding
+   catches (leave empty for findings that do not match any labeled issue).
+3. Run `npm run bench:public -- --reviewer=<name>` to get catch-rate, precision,
+   F1, false-positive, and per-severity metrics, plus a per-case breakdown.
+
+## Metrics
+
+- **Catch rate**: matched ground-truth issues / total expected issues.
+- **Precision**: matched issues / (matched + false positives + redundant matches).
+- **F1**: harmonic mean of catch rate and precision.
+- **False positives**: reviewer findings with empty `matched_ground_truth`.
+- **Redundant matches**: repeated matches to an issue already caught in the same case.
+- **By-severity catch rate**: catch rate grouped by `severity`.
+
+## Notes
+
+- Cases are synthetic and self-contained; they are not tied to a specific PR or
+  repo. They exist to make the benchmark reproducible by anyone, anywhere.
+- The sibling `benchmarks/agent-prs/` harness measures catch rate on real public
+  agent-generated PRs with preserved review artifacts. This `benchmarks/public-catch-rate/` set
+  complements it with broad, language- and issue-type coverage that is cheap to
+  re-run.

benchmarks/public-catch-rate/cases/ts-dead-code/label.json

@@ -0,0 +1,23 @@
+{
+  "id": "ts-dead-code",
+  "title": "Unreachable code after unconditional return",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "maintainability",
+  "ground_truth": [
+    {
+      "id": "unreachable-branch-after-return",
+      "type": "dead_code",
+      "severity": "medium",
+      "location": { "file": "source.ts", "lines": [12, 15] },
+      "description": "The if (score >= 70) branch and final return are unreachable because an unconditional return at line 8 always exits the function first."
+    },
+    {
+      "id": "unused-helper-function",
+      "type": "dead_code",
+      "severity": "low",
+      "location": { "file": "source.ts", "lines": [18, 20] },
+      "description": "neverCalled is defined but never referenced anywhere, so it is dead code that should be removed."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-dead-code/source.ts

@@ -0,0 +1,21 @@
+// Case: Dead/unreachable code after an unconditional return.
+export function classify(score: number): string {
+  if (score >= 90) {
+    return 'A';
+  }
+  if (score >= 80) {
+    return 'B';
+  }
+  return 'C';
+
+  // BUG: everything below this point is unreachable. The unconditional return
+  // above means this branch can never execute, and the helper is never used.
+  if (score >= 70) {
+    return 'D';
+  }
+  return 'F';
+}
+
+function neverCalled(): void {
+  console.log('this function has no callers');
+}

benchmarks/public-catch-rate/cases/ts-hardcoded-credentials/label.json

@@ -0,0 +1,16 @@
+{
+  "id": "ts-hardcoded-credentials",
+  "title": "Hardcoded database password in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "security",
+  "ground_truth": [
+    {
+      "id": "hardcoded-db-password",
+      "type": "hardcoded_secret",
+      "severity": "high",
+      "location": { "file": "source.ts", "lines": [8, 8] },
+      "description": "The production database password is committed in plaintext inside the source file, exposing credentials to anyone with repository access."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-hardcoded-credentials/source.ts

@@ -0,0 +1,14 @@
+// Case: Hardcoded database credentials in a TypeScript service config.
+export const dbConfig = {
+  host: 'db.prod.internal',
+  port: 5432,
+  user: 'admin',
+  // BUG: the production database password is committed in plaintext.
+  password: 'P@ssw0rd-prod-2024!',
+  database: 'orders',
+};
+
+export async function connect() {
+  const url = `postgres://${dbConfig.user}:${dbConfig.password}@${dbConfig.host}:${dbConfig.port}/${dbConfig.database}`;
+  return fetch(url);
+}

benchmarks/public-catch-rate/cases/ts-insecure-cookie/label.json

@@ -0,0 +1,16 @@
+{
+  "id": "ts-insecure-cookie",
+  "title": "Insecure session cookie attributes in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "security",
+  "ground_truth": [
+    {
+      "id": "cookie-missing-secure-httponly-samesite",
+      "type": "insecure_cookie",
+      "severity": "high",
+      "location": { "file": "source.ts", "lines": [8, 8] },
+      "description": "The session cookie is set without Secure, HttpOnly, or SameSite attributes, so it is transmitted over HTTP, readable by JavaScript/XSS, and vulnerable to CSRF."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-insecure-cookie/source.ts

@@ -0,0 +1,9 @@
+// Case: Session cookie set without Secure, HttpOnly, or SameSite attributes.
+import type { Response } from 'express';
+
+export function setSessionCookie(res: Response, token: string): void {
+  // BUG: the cookie is set without Secure (sent over HTTP), HttpOnly (readable
+  // by JS/XSS), and SameSite (vulnerable to CSRF). A stolen cookie value is a
+  // stolen session.
+  res.cookie('session', token, { maxAge: 86400000 });
+}

benchmarks/public-catch-rate/cases/ts-missing-await/label.json

@@ -0,0 +1,16 @@
+{
+  "id": "ts-missing-await",
+  "title": "Missing await on async operation in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "bug",
+  "ground_truth": [
+    {
+      "id": "fire-and-forget-delete-session",
+      "type": "missing_await",
+      "severity": "medium",
+      "location": { "file": "source.ts", "lines": [8, 8] },
+      "description": "deleteSession returns a promise but is not awaited, so a rejection becomes an unhandled promise rejection and logout resolves before the session is actually deleted, leaving stale sessions."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-missing-await/source.ts

@@ -0,0 +1,10 @@
+// Case: Missing await on a rejected promise swallows an error.
+import { deleteSession } from './session';
+
+export async function logout(userId: string): Promise<void> {
+  // BUG: deleteSession returns a promise but is not awaited. If it rejects,
+  // the rejection becomes an unhandled promise rejection and logout resolves
+  // as if the session were deleted, leaving stale sessions behind.
+  deleteSession(userId);
+  console.log('user logged out');
+}

benchmarks/public-catch-rate/cases/ts-prototype-pollution/label.json

@@ -0,0 +1,16 @@
+{
+  "id": "ts-prototype-pollution",
+  "title": "Prototype pollution via recursive merge in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "security",
+  "ground_truth": [
+    {
+      "id": "proto-pollution-merge",
+      "type": "prototype_pollution",
+      "severity": "high",
+      "location": { "file": "source.ts", "lines": [11, 19] },
+      "description": "The recursive merge walks user-supplied keys without blocking __proto__/constructor/prototype, so a payload like {\"__proto__\": {\"admin\": true}} pollutes Object.prototype and escalates privileges across the application."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-prototype-pollution/source.ts

@@ -0,0 +1,21 @@
+// Case: Prototype pollution via recursive object merge.
+function isObject(v: unknown): v is Record<string, unknown> {
+  return typeof v === 'object' && v !== null;
+}
+
+// BUG: the merge walks user-supplied keys without blocking __proto__,
+// constructor, or prototype. A payload like {"__proto__": {"admin": true}}
+// pollutes Object.prototype and escalates privileges app-wide.
+export function merge(target: Record<string, unknown>, source: unknown): Record<string, unknown> {
+  if (!isObject(source)) return target;
+  for (const key of Object.keys(source)) {
+    const tv = target[key];
+    const sv = source[key];
+    if (isObject(tv) && isObject(sv)) {
+      merge(tv, sv);
+    } else {
+      target[key] = sv;
+    }
+  }
+  return target;
+}

benchmarks/public-catch-rate/cases/ts-regex-dos/label.json

@@ -0,0 +1,16 @@
+{
+  "id": "ts-regex-dos",
+  "title": "Catastrophic backtracking regex (ReDoS) in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "security",
+  "ground_truth": [
+    {
+      "id": "redos-nested-quantifier",
+      "type": "regex_dos",
+      "severity": "high",
+      "location": { "file": "source.ts", "lines": [4, 4] },
+      "description": "The regex uses a nested + quantifier ((...+)+) causing exponential backtracking on non-matching inputs; a long crafted string hangs the event loop and denies service to all other requests."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-regex-dos/source.ts

@@ -0,0 +1,10 @@
+// Case: Catastrophic backtracking regex (ReDoS).
+// This regex is used to validate user-supplied email-like strings.
+export const emailLikePattern = /^([a-zA-Z0-9._%+-]+)+$/;
+
+// BUG: the nested + quantifier ((...+)+) creates exponential backtracking on
+// non-matching inputs. A long string like "a".repeat(30) + "!" hangs the event
+// loop and denies service to all other requests.
+export function isEmailLike(input: string): boolean {
+  return emailLikePattern.test(input);
+}

benchmarks/public-catch-rate/cases/ts-sql-injection/label.json

@@ -0,0 +1,16 @@
+{
+  "id": "ts-sql-injection",
+  "title": "SQL injection via string concatenation in TypeScript",
+  "language": "typescript",
+  "source_file": "source.ts",
+  "category": "security",
+  "ground_truth": [
+    {
+      "id": "sql-injection-email-concat",
+      "type": "sql_injection",
+      "severity": "high",
+      "location": { "file": "source.ts", "lines": [14, 14] },
+      "description": "User-controlled emailInput is concatenated directly into the SQL query string, allowing injection of arbitrary SQL by breaking out of the single-quoted value."
+    }
+  ]
+}

benchmarks/public-catch-rate/cases/ts-sql-injection/source.ts

@@ -0,0 +1,16 @@
+// Case: SQL injection via string concatenation in a TypeScript query builder.
+import { db } from './db';
+
+interface User {
+  id: number;
+  email: string;
+}
+
+export async function findUserByEmail(emailInput: string): Promise<User | null> {
+  // BUG: user-controlled emailInput is concatenated directly into the SQL
+  // string, allowing an attacker to break out of the quoted value and append
+  // arbitrary SQL (e.g. "' OR '1'='1").
+  const sql = `SELECT id, email FROM users WHERE email = '${emailInput}' LIMIT 1`;
+  const rows = await db.query<User>(sql);
+  return rows[0] ?? null;
+}