One small API with six deliberately seeded bugs, three code-review tools pointed at it, every raw output kept.
We publish this because CodeTruss's entire argument is "check our claims," and a
competitive comparison you cannot re-run is exactly the kind of claim we tell
people not to accept. The fixture is here, the seeded bugs are marked in the
source, the exact commands are here, and the raw stdout/stderr of every run is
in raw/. If you think a verdict below is wrong, you have everything
you need to show it.
Our CodeTruss row is re-measured against the currently published binary
(0.2.43), not against a build from our tree. The tarball is downloaded from
the public download URL and its sha256 checked against the published digest
before install (raw/2026-08-07-codetruss-0.2.43/tarball-sha256.txt).
1. This is not a same-day comparison, and you should not cite the competitor
rows as current. Our row was measured on 2026-08-07 against CodeTruss CLI
0.2.43. The Semgrep and CodeRabbit rows were measured on 2026-08-06
against Semgrep 1.164.0 and CodeRabbit CLI 0.6.5, and we have not
re-run them since — deliberately, because we are not neutral and a competitor
number we refresh at our convenience is worth nothing. Both tools ship
constantly; CodeRabbit's own output on the day already advertised an update from
0.6.5 to 0.7.2. Their rows are a dated snapshot. Re-run all three yourself —
the scripts for our row are in scripts/, and the competitor
invocations are written out in PROTOCOL.md.
2. We wrote the fixture, and we are not neutral. 127 lines, five files, six bugs, chosen to span classes we detect and classes we do not. n = 1. This is a worked example, not a benchmark. Anyone's fixture would flatter its author; ours presumably flatters us in ways we cannot see, which is the reason to hand you the fixture rather than a chart.
3. CodeRabbit is an LLM ensemble and is not deterministic. We ran it once. A second run could find more or fewer things. Semgrep and CodeTruss are deterministic rule engines and should reproduce.
4. Free tiers only, one operator, one machine. Paid tiers of Semgrep and CodeRabbit were not run and nothing here says anything about them. CodeTruss's hosted product was not run either — this is the free local CLI.
5. Two of the six bugs are misses for us, and the section explaining why is longer than the section on what we catch. That is on purpose. See What CodeTruss misses.
taskflow-api — a small multi-tenant Express + pg + raw-SQL TypeScript
service. Baseline is committed; a "task search" feature is staged on top as the
change under review. Every seeded bug carries an in-source comment naming its
number and class.
| # | Bug | Where | Class | In-source marker |
|---|---|---|---|---|
| 1 | SQL injection — search string concatenated into the query | fixture/src/routes/tasks.ts:16 |
injection / taint | // BUG 1 (SQLi) at line 11 |
| 2 | Missing org scope on GET /tasks/:id — cross-tenant read (IDOR) |
fixture/src/routes/tasks.ts:26 |
authorization / dataflow | // BUG 2 (missing org scope / IDOR) at line 24 |
| 3 | Audit insert never awaited — floating DB write | fixture/src/routes/tasks.ts:52 |
async correctness | // BUG 3 (floating write) at line 50 |
| 4 | Empty catch swallows a completion failure, client still gets 200 | fixture/src/routes/tasks.ts:69-71 |
error handling | // BUG 4 (empty catch) at line 70 |
| 5 | Scope drift — billing.ts edited by a change that claims to add task search |
fixture/src/lib/billing.ts |
changeset scope | // BUG 5 (scope drift) at line 1 |
| 6 | Hardcoded Stripe live secret | fixture/src/lib/billing.ts:5 |
secrets | // BUG 6 (secrets) at line 5 |
Bug 2 is introduced by the change: the baseline GET /tasks/:id has
AND org_id = $2, and the feature diff drops it.
About bug 6's key. It is fabricated and has never been a real credential, but GitHub's push protection will not host a string in that shape — reasonably — so
fixture/src/lib/billing.tscarriesSTRIPE_LIVE_KEY_PLACEHOLDERandscripts/make-fixture.shsubstitutes the literal when it builds the fixture. The value it writes is byte-for-byte the one every run in this teardown analysed, and you do not have to take that on trust: the staged tree hash below only comes out right if the substitution produced exactly those bytes. For the same reason the key is masked in one Semgrep output file and the receipts'.patchevidence files are not published — both noted where they occur inraw/.
One difference from the bytes the competitors saw. On 2026-08-06 bug 6 had no marker of its own — it was described in the bug-5 comment block above it. For this repo we appended
// BUG 6 (secrets): hardcoded live Stripe secret key.to the end of line 5 so every bug is marked in-source. It is a trailing comment, so no line numbers moved. We re-ran CodeTruss 0.2.43 on both versions of the fixture: identical findings, identical rule ids, identical lines, identical delta. Git tree hashes, so you can tell which is which: baselined6dd96912634cf2fa2771ff4b2621c2dd871ce15, this repo's staged treee3c6ddaad2086d9624f0d4183f8a5cd13f0940dc, the 2026-08-06 staged tree4ff245521a34ae6135491a6c0ebdfedf43b8ea18.
Read again: the CodeTruss column is one day newer than the other two, and the other two were not re-run. Semgrep and CodeRabbit were measured 2026-08-06 at the versions in the header; CodeTruss was measured 2026-08-07 at 0.2.43. This is not a same-day head-to-head, and a reader who cares about the answer should re-run all three.
| # | Bug | Semgrep 1.164.0, free OSS rules 2026-08-06 |
CodeRabbit CLI 0.6.5, free 2026-08-06 |
CodeTruss CLI 0.2.43 2026-08-07 |
|---|---|---|---|---|
| 1 | SQL injection | MISS | CATCH — critical, with a fix diff | CATCH — sql-injection, CRITICAL, CWE-89, with the taint path |
| 2 | Cross-tenant read (IDOR) | MISS | MISS | MISS |
| 3 | Floating DB write | MISS | CATCH — major | CATCH — unawaited-persistence, MEDIUM, CWE-252 |
| 4 | Empty catch | MISS | CATCH — major | MISS — by policy; the rule ships and fires, see below |
| 5 | Scope drift | MISS | MISS — reviewed the file, no concept of changeset scope | CATCH — but imprecise without --allow, see below |
| 6 | Hardcoded Stripe secret | CATCH | CATCH — critical, with a fix diff | CATCH — HIGH, with a fix diff |
| Total | 1 / 6 | 4 / 6 | 4 / 6 | |
| False positives | 0 | 0 | 0 |
Scoring rules are in PROTOCOL.md. Briefly: a CATCH means the tool named the defect at the right place in its default output; a finding is a false positive only if it is untrue of this code.
4–4 is a tie, and a tie is not a win. CodeRabbit got its four from one command with no configuration, and it caught the empty catch that we deliberately stay quiet about. What our column shows is a local, sub-second, signed run that now finds the injection — not a better reviewer.
Same date labels apply: the first two columns are 2026-08-06, the last is 2026-08-07.
| Semgrep 1.164.0 | CodeRabbit CLI 0.6.5 | CodeTruss CLI 0.2.43 | |
|---|---|---|---|
| Getting a first run | --config auto refused ("Cannot create auto config when metrics are off"); named rule packs work and are fetched over the network; p/express returned 404 |
two runs failed until a base branch was configured; the review is performed on CodeRabbit's servers | install script verifies the published sha256, --ignore-scripts; no account needed for this run |
| Wall clock, this fixture | 3.05 s (215 rules) | 78.01 s | 0.49 s on the recorded run; 0.49–0.61 s across five repeats (raw) |
| Evidence artifact | terminal output (we did not retain a SARIF/JSON artifact, so we claim nothing about its format) | prose findings with fix diffs, in the terminal | ed25519-signed receipt; codetruss verify latest → verified, with no account (raw). A stranger can re-check ours with codetruss verify-receipt … --public-key (CLI 0.2.44+) against the published key, which exits 0 — but see what that key does and does not prove. We did not test verification with the network pulled |
| Does the code leave the machine? | not established by our outputs; rule configs were fetched over the network | yes — the progress log shows the CLI connecting to CodeRabbit, and the review comes back from there | the receipt asserts "No source code or diff left the machine". That is our own binary's claim about itself; we did not put a network monitor on it |
| Deterministic? | rule engine; we did not re-run it to confirm | no — LLM ensemble, single run | yes on this fixture: three runs in three working directories produced the same findings, rule ids and lines |
Do not turn the wall-clock row into a ratio. Those numbers are from different days on a machine under different load. The same 0.2.38 binary measured 0.392 s earlier on 2026-08-07 and 0.49–0.57 s a few hours later (control run). The defensible statement is the shape, not the multiple: a local sub-second run versus a 78-second cloud round trip, measured a day apart.
This is the part that matters. Two of six.
GET /tasks/:id looks a task up by id with no org_id filter, so any
authenticated user of any tenant can read any task by guessing an id. It is
arguably the most exploitable bug in the file. CodeTruss does not find it on
this fixture, and neither did the two competitor configurations we ran.
Why we miss it: CodeTruss's hosted analysis does have a detector for this class, and it works two ways that this fixture defeats.
- It enumerates externally reachable entry points from framework convention —
Next.js App Router
route.tsmethod exports,'use server'actions, andpages/apihandlers. This fixture is a bareexpress.Router()withtasks.get("/tasks/:id", ...). There is no convention here to enumerate, so the entry list is empty. - It decides what "belongs to a tenant" from a machine-readable schema (Prisma/Drizzle model definitions). This fixture talks to Postgres with raw SQL strings and ships no schema, so there is nothing to derive tenancy from, and the detector's designed behaviour with no schema is to produce no finding rather than guess.
So: the class is built, and on an Express + raw-SQL codebase like this one it does not fire. That is a miss on this fixture, and the hosted detector was not run against this fixture as part of this teardown — nothing here is evidence that it would have caught it.
What we will not say about this bug. We will not say "no tool catches this."
What we can say from the raw outputs is narrower: the two competitor
configurations we ran did not report it. Those were (a) Semgrep 1.164.0 on free
community rule packs, 215 rules over the 5 files, and (b) CodeRabbit CLI 0.6.5
on the free tier, one run. Semgrep's paid tiers, CodeRabbit's paid tiers, and
every tool we did not run are outside what this fixture measured. CodeRabbit got
notably close without landing it: it flagged an org-scope concern on the
adjacent complete route (tasks.ts:64-68) while saying nothing about the
unscoped read on line 26.
CodeRabbit reported this and we did not, so it scores as a miss. But it is not that the rule is missing.
The swallowed-error rule ships in the local pass and fires on this exact code.
It treats a comment inside the catch block as a record of deliberate intent and
stays silent — the same convention its own remediation text tells users to
follow when a swallow really is intended. The fixture's catch contains the
seeded-bug comment, which is precisely that shape.
You can watch it flip. scripts/probe-swallowed-error.sh
deletes that one comment line and changes nothing else:
local-sast findings with the comment removed:
sql-injection | CRITICAL | src/routes/tasks.ts:12
unawaited-persistence| MEDIUM | src/routes/tasks.ts:52
swallowed-error | LOW | src/routes/tasks.ts:69
Whether honouring an in-catch comment is the right policy is a fair argument to have — a comment saying "BUG" is not a claim of intent, and a smarter rule would read it. On this fixture, against a tool that reported it, we score it a miss.
The bare default run does flag the drift, with no configuration. But it names both changed files:
- 2 file(s) changed outside approved scope: src/lib/billing.ts, src/routes/tasks.ts
src/routes/tasks.ts is the task. Only the --allow 'src/routes/**' run
isolates billing.ts and nothing else. So a first-time user on an unconfigured
repository is told "everything you touched is unexpected," which is a weaker
signal than the table's CATCH suggests.
The rule behind it, from scripts/probe-scope-inference.sh:
the CLI infers an allowed scope from the turn's own working set only when that
would produce a single root. A one-file change infers src and stays quiet; a
two-file change inside src/routes infers src/routes and stays quiet; this
fixture touches src/routes and src/lib, so inference is withheld and
everything classifies unexpected. We count it CATCH because the drifting file
is named and the earlier protocol scored the identical behaviour that way, and
we count the over-naming as a known UX gap rather than a false positive — but if
you score it as "flagged the wrong thing too," our total is 3.5, not 4, and we
will not argue with you.
The previous internal measurement was 0.2.38, also on 2026-08-07. We re-ran the same protocol on the currently published 0.2.43 and the headline did not move.
- Still 4 of 6, still 0 false positives, same rule ids, same severities,
same lines (
sql-injectionCRITICAL attasks.ts:12,unawaited-persistenceMEDIUM attasks.ts:52, secrets HIGH atbilling.ts:5, scope drift naming both files by default andbilling.tsalone with--allow). - Two analyzers appeared that 0.2.38 did not run (
comment-slop,overengineering), taking the profile string fromlocal-registry-v2tolocal-registry-v4and the receipt's own wording from "13 deterministic registry analyzers … plus a local security pass" to "15 … plus a local security pass". (0.2.30, for reference, said 13 with no security pass at all.) They produce one new INFO finding on this fixture: "3 export(s) with no consumer in src/lib/billing.ts". It is true — nothing in the fixture importsbilling.ts— but it restates thedead-codeLOW already reported on the same file, so read it as redundancy, not as new information. - The finding delta changed from
3 introduced / 0 worsened / 6 recurring / 0 resolvedto3 introduced / 0 worsened / 7 recurring / 1 resolved. The extra recurring is the newoverengineeringfinding. The one resolved is real and correct: at baseline,src/db.tsexportedpoolwith no consumer; the feature change importspool, so that finding legitimately goes away. - No measurable speed difference. 0.2.43 across five repeats: 0.494–0.609 s. The 0.2.38 binary re-run on the same machine minutes later as a control: 0.494–0.574 s. The 0.392 s recorded for 0.2.38 earlier the same day was a less-loaded machine, not a faster build.
Needs node >= 20.9, npm, git, curl. Nothing is installed outside the
checkout — the CLI goes into work/ctprefix, and HOME, the XDG dirs and the
npm cache are redirected into work/fakehome.
git clone https://github.com/CodeTruss/verification-teardown.git
cd verification-teardown
sh scripts/reproduce.shThat downloads the published 0.2.43 tarball, verifies its sha256 against the published digest (and refuses to install on a mismatch), builds the fixture as a Git repo with the change staged, runs both recorded invocations, verifies the signed receipt, and runs the two probes. It takes about a minute, most of it download.
Individually:
sh scripts/install-codetruss.sh # download + checksum + isolated install
sh scripts/make-fixture.sh # -> work/fixture (prints the tree hashes)
sh scripts/run-codetruss.sh # the two recorded invocations
sh scripts/probe-swallowed-error.sh # bug 4: why we stay quiet
sh scripts/probe-scope-inference.sh # bug 5: when scope is inferred
sh scripts/repeat-timings.sh # five wall-clock repeatsTo reproduce against a different CLI version: sh scripts/install-codetruss.sh 0.2.44. Anything other than the pinned 0.2.43 is checked against whatever
digest the site publishes at that moment.
The two receipts under raw/2026-08-07-codetruss-0.2.43/
are signed and unmodified. The supported way to check a receipt you were handed
is codetruss verify-receipt, which needs CLI 0.2.44 or later and no
account, no checkout and no config:
codetruss verify-receipt raw/2026-08-07-codetruss-0.2.43/receipts-default \
--public-key codetruss-teardown-2026-08-07.pubThat exits 0. It reports two claims and never merges them:
- integrity — these bytes have not changed since they were signed. Checkable from the receipt alone.
- provenance — the key you named is the key that signed them. Checkable only
against a
--public-keyyou supply.
Drop --public-key and it exits 1: integrity established, provenance not.
That is the right answer when you have no key, because the only key left is the
one the receipt carries, and anyone can generate a keypair and sign a receipt of
their own with it.
codetruss verify is a different command and still refuses these receipts
("receipt signer … does not match trusted key"). That is correct: verify
measures a receipt against the key your repository pins, which is the right
behaviour for your own history and useless for ours.
codetruss-teardown-2026-08-07.pub is the
ed25519 public half that signed both 0.2.43 receipts. Its fingerprint:
698b57e85de6458e
That is the evidence.keyFingerprint recorded inside both receipt JSONs, and
the signing key verify-receipt prints.
Read what this key is, because it is less than the phrase "signing key" suggests.
- It is a run key, not a company key. The CLI generates a signing key per
install, under
~/.config/codetruss. These receipts came fromscripts/reproduce.sh, which redirectsHOMEinto the checkout, so this key was minted for that sandbox and signs nothing else. The 0.2.30 and 0.2.38 receipts inraw/carry two different fingerprints —48970f3dc155ea50and3925e0f558fc61cb— for exactly this reason, and when you runreproduce.shyou will mint a fourth. There is no long-lived CodeTruss signing identity behind this file. Do not build one on top of it. - We published it in the same repository as the receipts it verifies, so the provenance check here is close to circular. You are trusting GitHub to deliver both, from us, unmodified. Anyone able to alter the receipts in this repo could alter this file in the same push. A key is worth something when it reached you by a route the receipts did not; this one did not.
- What it does buy you is that the claim is now explicit and falsifiable rather than implicit. Before this file the only key available was the one inside the receipt, and the verifier could not evaluate provenance at all.
So: integrity is fully checkable by a stranger. Provenance, here, is not — and
no arrangement inside a single repository can make it so. For provenance that
means something, run sh scripts/reproduce.sh and verify a receipt your own
machine signed.
scripts/verify-published-receipt.mjs
still works and needs no CodeTruss install, just Node:
node scripts/verify-published-receipt.mjs raw/2026-08-07-codetruss-0.2.43/receipts-defaultUse it if you would rather not install our binary in order to check our claims,
which is a fair thing to want. But it checks strictly less, and one gap
matters. It confirms the .md matches the digest recorded in the signed JSON
and stops there; it never re-derives the .md from that JSON. verify-receipt
re-renders the Markdown from the signed structured data and compares it byte for
byte, so a signer cannot hand-write a flattering summary, record that digest,
and sign it. We tested that difference: a bundle whose .md reads "PASS —
everything looked great" over a signed JSON that says FAILED passes the script
above at exit 0 and fails verify-receipt at exit 2. verify-receipt also
checks that evidence.keyFingerprint matches the embedded key, validates the
receipt's structure, and checks the patch's byte count and truncation metadata
rather than only its digest.
Neither tool proves the run happened or that its conclusions are right. Reproduce it for that.
The competitor commands are in PROTOCOL.md. Running them needs your own Semgrep install and your own CodeRabbit account.
Every number above traces to a file here.
| Directory | What is in it |
|---|---|
raw/2026-08-06-semgrep/ |
the --config auto refusal, the 215-rule scan (1 finding: the Stripe key), the p/express 404, and the SQLi-focused re-run |
raw/2026-08-06-coderabbit/ |
two failed runs, then the 78-second review with all six findings |
raw/2026-08-06-codetruss-0.2.30/ |
the original CodeTruss run, when the CLI had no SAST and said so |
raw/2026-08-07-codetruss-0.2.38/ |
the previous re-measurement, kept as history |
raw/2026-08-07-codetruss-0.2.43/ |
our row: checksum record, both runs, both signed receipts, both probes, timings, the 0.2.38 control, machine details |
Absolute paths from the operator's machine were replaced with <WORKDIR> in the
2026-08-06 and 0.2.38 files. That edits their bytes, so the historical receipts
reproduced there no longer match their signatures and their .sig files are
not included — treat them as transcripts. The 0.2.43 receipts were generated in
a neutral directory, are byte-for-byte unmodified, ship with their .sig, and
verify.
The internal write-up this repo is built from made several claims we could not substantiate from the retained raw outputs. They are not here:
- "Semgrep's SQLi-focused packs also found only the secret." The second Semgrep run's stderr survived (153 ts rules, 5 files, 1 finding) but its stdout did not, so we cannot show which finding. We publish the count and stop there. Bug 1 is scored MISS for Semgrep on the strength of the 215-rule run, whose full output is retained and whose only finding is the Stripe key.
- "A minimal direct
pool.queryconcatenation sink also produced 0 findings under free rules." The probe file exists; no output for it was retained. Dropped entirely. - "The good taint-mode Postgres SQLi rules are Pro-tier." A claim about Semgrep's commercial packaging that this teardown never measured. Dropped.
- "Semgrep emits SARIF/JSON, unsigned." We retained terminal output only. Dropped; the artifact row now says what we actually have.
- Which rule packs each Semgrep run used. The invocations were not captured in the retained outputs — only the rule counts were. PROTOCOL.md records the packs as the operator's notes, explicitly labelled as unverified from artifacts.
- Exact wall-clock repeats for 0.2.38 (
0.409 / 0.415 / 0.430 s). The retained timing files from that session show0.457 / 0.462 / 0.505 s. We use the retained files and the fresh same-machine control instead. - "CodeRabbit requires an account to run at all." Our retained output shows a base-branch failure and a signed-in review; it does not establish what happens signed out. Softened to what the logs show.
- The operator's account identity and machine paths. Removed as irrelevant to the result.
Fixture and scripts: MIT (see LICENSE). Raw outputs are transcripts of tool runs and are reproduced for verification.
Published by CodeTruss. We are the vendor of one of the three tools. Everything above was designed to make that bias checkable rather than to hide it.