Audit the auditor: evidence rules, substance checks, waiver injection fix (3.4.0)

.claude-plugin/marketplace.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
   "name": "clean-code-toolkit",
-  "version": "3.3.0",
+  "version": "3.4.0",
   "description": "Clean-code and product-handoff tools for AI-assisted builders.",
   "owner": {
     "name": "Tarik Moody"
@@ -10,7 +10,7 @@
     {
       "name": "clean-code-toolkit",
       "description": "Review code, assess product readiness, refactor safely, and prepare a developer handoff.",
-      "version": "3.3.0",
+      "version": "3.4.0",
       "author": {
         "name": "Tarik Moody"
       },

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "clean-code-toolkit",
-  "version": "3.3.0",
+  "version": "3.4.0",
   "description": "Practical clean-code, product-readiness, and developer-handoff workflows for AI-assisted projects.",
   "author": {
     "name": "Tarik Moody"

CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## 3.4.0
+
+An audit of the auditor. A deliberately hollow repo, every control a costume, scored 78 out of 100 with nine passing checks, five of them false. Fixing that exposed more.
+
+**Confidence now follows evidence, structurally.**
+- A pass with an empty evidence list can no longer claim to be `verified`. The downgrade is automatic, so the hand-maintained `WEAK_PASS_IDS` set no longer has to be remembered when a check is added.
+- Checks that always had real evidence now carry it: `log-1` and `log-2` cite the matched dependency, `dep-1` the lockfile, `ci-6` the script, `res-2` the matching line. `sec-3`, `sec-4` and `ms-1` cite the scope they searched, which is the honest evidence for a proof of absence.
+
+**Existence is not function.** Six checks passed on controls that existed in name only.
+- `ci-2` (critical) was satisfied by a CI step of `echo "test skipped"`. It now requires a real test-runner invocation, with string literals stripped first.
+- `ci-6` was satisfied by a `test` script of `echo "no tests yet" && exit 0`.
+- `res-1` was satisfied by an empty `runbook.md`. It now requires 50 words of prose.
+- `res-2` was satisfied by an unchecked `- [ ] figure out rollback` TODO box.
+- `ms-2` was a bag-of-words test marked verified; it is now labelled a text match.
+- `log-1`/`log-2` matched substrings of a serialized dependency blob, so `@types/pino` scored a verified pass. They now match package names, excluding type stubs.
+
+**Security: the waiver file is untrusted input.** `.prod-audit-waivers.json` lives in the audited repo, and its text was written into the Markdown report unescaped. A crafted `reason` field forged a second `## Repository Controls Score: 100/100` heading in the report. Waiver text is now flattened to a single length-capped line with pipes escaped and leading markup stripped, at the point it is loaded rather than at each render site.
+
+**The generated documents are now linted.** `scripts/check_report.py` checks a finished audit against the JSON it was written from: a quoted file path must appear in the scan, a `weak` pass must not be written up as a win, every `critical` must be addressed in the fix brief, a waived finding must stay visible, no placeholders, no em dashes. The skill runs it before sharing. Run against real generated documents that had already been reviewed by hand, it found six problems.
+
+**Refactor.** `prod_audit.py` was 1836 lines, more than twice this project's own 800-line ceiling, with three hand-maintained tables far from the checks they described. It is now an `audit/` package of eight modules, largest 503 lines, plus a thin entry point that re-exports the public names so every documented command and import keeps working. 77 unused imports removed.
+
+**House style.** Em dashes are gone from every generated string, so the report linter no longer rejects the documents this toolkit produces.
+
+Tests: 59, up from 37.
+
+Known and not yet fixed: `claude plugin eval` is in early access and could not be run, so the five prompt-only skills still have no automated coverage. Each was tested once by hand against a real repository; the findings are tracked for 3.5.0.
+
 ## 3.3.0
 
 Two verified defects in how the coach treats evidence, and the trap they created in the fix brief.

skills/prod-readiness-coach/README.md

@@ -9,6 +9,8 @@ python3 scripts/prod_audit.py --repo /path/to/repo --output report.md --json rep
 python3 scripts/prod_audit.py --help   # --profile, --context, --fail-on
 ```
 
+`scripts/check_report.py` lints a finished audit against the JSON it was written from: invented file paths, a text match sold as a win, a critical finding missing from the fix brief, a waived finding quietly dropped. The skill runs it before sharing.
+
 Tests: `python3 -m unittest discover skills/prod-readiness-coach/tests` from the toolkit root.
 
 Everything else (what it checks, profiles, confidence labels, limits, when to run it) lives in the [toolkit README](../../README.md) and [how it works](../../docs/how-it-works.md), so there is one place to keep current. Decisions that shaped this skill are in [`docs/decisions/`](docs/decisions/).

skills/prod-readiness-coach/SKILL.md

@@ -259,6 +259,25 @@ severity before starting the next phase"). Substance rules:
 
 ### 8. Review both documents before sharing
 
+**Run the linter first.** It checks the rules below mechanically, so you
+only have to think about the ones it cannot:
+
+```bash
+python3 <skill-folder>/scripts/check_report.py \
+  AUDIT_PLAIN_ENGLISH.md FIX_BRIEF_FOR_CLAUDE_CODE.md --json <the audit json>
+```
+
+It fails if a quoted file path never appeared in the scan, if a `weak`
+pass is written up as a win, if a `critical` finding is missing from the
+brief, if a waived finding was quietly dropped, or if a placeholder or an
+em dash survived. Fix everything it reports before sharing. If you
+verified a path by hand rather than from the scan, say so in the sentence
+that uses it, which is the rule in step 7 and what the linter is asking
+you to make explicit.
+
+Then check by eye what it cannot:
+
+
 - Scan for any leftover jargon that isn't immediately explained — if you
   used a technical term, either the glossary already covers it or you
   need to add a one-clause explanation inline.

skills/prod-readiness-coach/references/report-templates.md

@@ -2,7 +2,7 @@
 
 Two documents get generated per audit run. Follow these structures. Fill
 every `{{...}}` placeholder using the JSON output from `prod_audit.py` and
-the plain-English glossary — never paste raw technical `detail` strings from
+the plain-English glossary, never paste raw technical `detail` strings from
 the JSON without translating them first.
 
 ---
@@ -12,7 +12,7 @@ the JSON without translating them first.
 ```markdown
 # How Production-Ready Is {{repo_name}}?
 
-*A plain-English walkthrough — no jargon left untranslated.*
+*A plain-English walkthrough, no jargon left untranslated.*
 
 ## The short version
 
@@ -33,7 +33,7 @@ what it scored; it goes in the "Looks fine, but only from a text match"
 list below instead. A category whose checks
 are all `n/a` has `score: null`; it is neither a win nor a gap, so leave
 it out. E.g. "**Your secrets are safe.** You're not
-accidentally leaking passwords or API keys into your code — a mistake even
+accidentally leaking passwords or API keys into your code, a mistake even
 experienced teams make. Nice work."}}
 
 ## Looks fine, but only from a text match
@@ -60,7 +60,7 @@ medium → low), write ONE section using this exact structure:}}
 
 **What we checked:** {{one sentence, plain English, what this check looks for}}
 
-**What we found:** {{plain English restatement of the technical finding —
+**What we found:** {{plain English restatement of the technical finding,
 name specific files/evidence when available, so it feels concrete not
 abstract}}
 
@@ -70,23 +70,23 @@ hint of this, not proof." Then say what you did (or did not do) to
 confirm it. The warm voice stays; the uncertainty is stated first, not
 tucked at the end.}}
 
-**Why it matters:** {{a real, concrete consequence — what actually happens
+**Why it matters:** {{a real, concrete consequence, what actually happens
 to you or your users if this stays unfixed. Avoid abstractions like "poses
 a risk"; say what the risk *looks like* in practice.}}
 
 **The concept, in one paragraph:** {{a short teaching moment explaining the
-underlying idea using the glossary — written so someone who's never heard
+underlying idea using the glossary, written so someone who's never heard
 the term walks away actually understanding it, not just able to repeat it}}
 
-**How urgent is this really?** {{one honest sentence calibrating — is this
+**How urgent is this really?** {{one honest sentence calibrating, is this
 "fix before you have real users" or "nice to have before you scale past a
 few hundred users"? Don't inflate urgency for effect.}}
 
 ---
 
 ## Your next-lesson roadmap
 
-{{A short ordered list — 3-6 items — framing the fixes as a learning
+{{A short ordered list, 3-6 items, framing the fixes as a learning
 sequence, e.g.:
 1. Learn what a CI pipeline does by adding your first one (30-60 min)
 2. Learn why "silent failures" are dangerous by adding error tracking
@@ -95,7 +95,7 @@ Each item should feel achievable, not overwhelming.}}
 
 ---
 
-*This audit was generated by a static-analysis tool — it reads your code
+*This audit was generated by a static-analysis tool, it reads your code
 and config files, it doesn't run your app. Some findings may need a quick
 human double-check. Full technical findings with file-level evidence and
 best-practice citations are in the companion report.*
@@ -115,20 +115,20 @@ code appear.
 # Fix Brief: {{repo_name}}
 
 Paste this into Claude Code inside this repo. **This brief is split into
-phases by severity — do not start Phase 2 until every Phase 1 acceptance
+phases by severity, do not start Phase 2 until every Phase 1 acceptance
 criterion is checked off and you've re-run the audit to confirm it. Same
-gate between every later phase.** This isn't just pacing — a phase can
+gate between every later phase.** This isn't just pacing, a phase can
 change what a later fix should even look like (e.g. adding CI in Phase 1
 changes how you verify a Phase 2 fix), so skipping ahead risks doing
 Phase 2 work against a moving target.
 
 ## Before you start
 
 {{2-3 sentences setting expectations: what stack this repo uses (from
-stack_fingerprint in the JSON — name the actual frameworks/deploy
+stack_fingerprint in the JSON, name the actual frameworks/deploy
 surfaces detected, e.g. "Next.js on Vercel with a Convex backend"), and a
 reminder that it's fine to ask Claude Code to explain any step before
-running it — that's how you learn what it's doing instead of just
+running it, that's how you learn what it's doing instead of just
 trusting it blindly.}}
 
 {{If stack_fingerprint.adapters_matched is non-empty, add one sentence
@@ -140,55 +140,55 @@ for the fact that rolling back one doesn't roll back the other."}}
 
 {{Group every failing/warning check into phases by severity: Phase 1 =
 all `critical` findings, Phase 2 = all `high` findings, Phase 3 = all
-`medium`/`low` findings. Skip a phase entirely if it has zero findings —
+`medium`/`low` findings. Skip a phase entirely if it has zero findings,
 don't manufacture filler. Open each phase with this block, then the
 numbered sections for just that phase's findings:}}
 
 ## Phase {{n}}: {{phase title, e.g. "Release Blockers"}} {{severity badge}}
 
-**Acceptance criteria for this phase — all must be true before moving on:**
+**Acceptance criteria for this phase, all must be true before moving on:**
 {{One checkbox line per finding in this phase, phrased as an observable
 outcome, not a task. E.g. "- [ ] `python3 prod_audit.py --repo . --json
 /tmp/audit.json` shows zero remaining `critical` findings" rather than
 "- [ ] Added CI pipeline."}}
 
-**Gate — do not proceed to Phase {{n+1}} until:**
+**Gate, do not proceed to Phase {{n+1}} until:**
 1. Every checkbox above is true.
 2. You've re-run: `python3 prod_audit.py --repo . --fail-on critical` (or
-   `--fail-on high` for a Phase 2 gate) and it exits 0 — **or** every
+   `--fail-on high` for a Phase 2 gate) and it exits 0, **or** every
    finding still failing at that severity is one whose control genuinely
    lives outside this repo, and the human has added a waiver for it in
    `.prod-audit-waivers.json`. If you cannot reach 0 and cannot honestly
    waive, stop and say so. Do not add a file whose only purpose is to
    satisfy the scan.
 3. Anything on the separate "Manual steps for a human" list tied to this
    phase (see below) has been done or explicitly deferred with the
-   human's sign-off — an AI agent should not silently skip these and
+   human's sign-off, an AI agent should not silently skip these and
    report the phase complete.
 
 {{One numbered section per finding in this phase:}}
 
 ## {{n}}. {{plain-English title}} {{severity badge, e.g. "🔴 Fix this first"}}
 
-**In plain English:** {{1-2 sentences — what's missing and why, using the
+**In plain English:** {{1-2 sentences, what's missing and why, using the
 glossary voice, no unexplained jargon}}
 
 **What to ask Claude Code to do:**
 
-{{A precise, technical, copy-pasteable instruction block — specific file
+{{A precise, technical, copy-pasteable instruction block, specific file
 paths from the audit evidence, specific commands, specific acceptance
 criteria. This part CAN and SHOULD use correct technical terminology,
 because this is the part an AI agent executes. Model it on this shape:}}
 
 > Add {{specific thing}} to {{specific file/location}}. It should
 > {{specific behavior}}. Verify by {{specific verification step}}.
 
-**What you'll learn from this fix:** {{1-2 sentences — the transferable
+**What you'll learn from this fix:** {{1-2 sentences, the transferable
 skill/concept this teaches, so the user internalizes *why*, not just
-*what*. E.g. "This is your first taste of CI/CD — once you've set this up
+*what*. E.g. "This is your first taste of CI/CD, once you've set this up
 once, you'll reuse this exact pattern on every future project."}}
 
-**How to know it worked:** {{a concrete, observable check — a green
+**How to know it worked:** {{a concrete, observable check, a green
 checkmark, a log line, a webpage that now loads, etc.}}
 
 ---
@@ -197,31 +197,31 @@ checkmark, a log line, a webpage that now loads, etc.}}
 
 {{Closing note: remind them to re-run the full audit (not just
 --fail-on critical) to see the score improve, and to feel good about the
-delta — cite the before/after score if known.}}
+delta, cite the before/after score if known.}}
 
 ## Manual steps for a human (not for the AI agent to execute)
 
 {{A separate list, pulled out here so it's never buried inside a phase an
 agent might run through unattended. Populate this from anything flagged
 during drafting as: interactive-only (a setup wizard that requires a
 human at a keyboard), plan/tier-dependent (e.g. GitHub branch protection
-on a private repo needs a paid plan — confirm the plan before promising
+on a private repo needs a paid plan, confirm the plan before promising
 this fix), or a judgment call with real consequences (e.g. "should we
 actually run this destructive migration's rollback, or restore from
-backup instead?"). If this list is empty, state that plainly — don't
+backup instead?"). If this list is empty, state that plainly, don't
 invent an entry to fill the section.}}
 ```
 
 ## Formatting rules for both documents
 
-- Use `##`/`###` headers exactly as templated — don't invent deeper nesting.
+- Use `##`/`###` headers exactly as templated, don't invent deeper nesting.
 - Never leave a `{{placeholder}}` unfilled in the final output.
-- Keep every "Why it matters" and "In plain English" block under ~60 words —
+- Keep every "Why it matters" and "In plain English" block under ~60 words,
   concise beats thorough here.
 - If a category has zero failing/warning checks, skip the "needs attention"
-  section entirely for it — don't manufacture filler content.
+  section entirely for it, don't manufacture filler content.
 - Preserve file-path evidence from the JSON output verbatim (e.g.
-  `app/api/ats-check/route.ts`) inside the technical instruction blocks —
+  `app/api/ats-check/route.ts`) inside the technical instruction blocks,
   precision matters there even though the surrounding prose is plain English.
 - Any fact that came from the user rather than the tool gets attributed
   inline: "you told me you're on Vercel Pro, so...". Any hand-verification

skills/prod-readiness-coach/scripts/audit/__init__.py

@@ -0,0 +1 @@
+"""Production-readiness audit engine."""