diff --git a/docs/canon/architecture.md b/docs/canon/architecture.md new file mode 100644 index 0000000..5269ddf --- /dev/null +++ b/docs/canon/architecture.md @@ -0,0 +1,75 @@ +# Architecture — reqdrive (canon) + +Stamped at commit `7420d69`, 2026-07-24. Requirements are canon; where the code +contradicts them the requirement is recorded as intended and the divergence is +logged in [requirements/drift-report.md](requirements/drift-report.md). + +reqdrive is a **Bash orchestration monolith**, not a service or a library: one +entry point (`bin/reqdrive`, 727 lines) sourcing eleven `lib/*.sh` modules +(2,683 lines), with a 3,242-line assertion suite and a freeze gate around it. It +takes one markdown requirement file and produces one GitHub pull request, using +the `claude` CLI as a stateless subprocess per story. + +## Routing table + +| Path pattern | Module doc | +|--------------|------------| +| `bin/**` | [modules/cli.md](modules/cli.md) | +| `lib/run.sh` | [modules/pipeline.md](modules/pipeline.md) | +| `lib/verification.sh`, `lib/policy.sh` | [modules/evidence.md](modules/evidence.md) | +| `lib/pr-create.sh` | [modules/pr.md](modules/pr.md) | +| `lib/config.sh`, `lib/schema.sh`, `lib/validate.sh`, `lib/init.sh`, `templates/**` | [modules/config.md](modules/config.md) | +| `lib/sanitize.sh`, `lib/preflight.sh`, `lib/errors.sh` | [modules/safety.md](modules/safety.md) | +| `tests/**` | [modules/test-harness.md](modules/test-harness.md) | +| `.github/**`, `install.sh`, `scripts/**` | [modules/ci-and-install.md](modules/ci-and-install.md) | +| `skills/**`, `archive/**` | [modules/skills-and-archive.md](modules/skills-and-archive.md) | +| *(additive)* any shell file | [conventions/shell-conventions.md](conventions/shell-conventions.md) | +| *(additive)* `lib/**`, `bin/**` | [conventions/sanitization.md](conventions/sanitization.md), [conventions/artifact-contracts.md](conventions/artifact-contracts.md) | +| *(additive)* `tests/**`, `.github/**` | [conventions/testing-and-freeze.md](conventions/testing-and-freeze.md) | + +## System overview + +`bin/reqdrive` parses arguments and dispatches to one of ten commands +(`bin/reqdrive:679-727`); everything that touches an agent flows into +`run_pipeline` (`lib/run.sh:855`). That function runs pre-flight gates +(`lib/preflight.sh:165`), cuts `reqdrive/` from the configured base branch +(`lib/run.sh:911`), then **Phase 1** invokes `claude` once to write `prd.json` +(`lib/run.sh:962-997`, hard-aborting with exit 5 if no PRD appears). **Phase 2** +loops: `select_next_story` picks the highest-priority incomplete story by jq +(`lib/run.sh:413`), a quoted-heredoc prompt is built with `@@TOKEN@@` +substitution (`lib/run.sh:281`), one fresh `claude` process implements it +(`lib/run.sh:465`), then `testCommand` and a commit-format check run as +observation-only counters (`lib/run.sh:1059-1082`) and the scope check runs as +the loop's only hard gate (`lib/run.sh:1088-1093`). **Phase 3** re-runs the test +suite for real (`lib/verification.sh:85`) and writes `verification-summary.json` +(`lib/verification.sh:98`). The draft decision is taken at `lib/run.sh:1177-1178` +and passed to `create_pr` (`lib/pr-create.sh:78`), which assembles the body and +calls `gh`. **Phase 4** review is off by default and runs *after* the PR exists, +so it cannot influence the draft flag (`lib/run.sh:1195`). + +State is per requirement under `.reqdrive/runs//` +(`lib/run.sh:685,691`); each Claude call is stateless and carries context only +through files (`lib/run.sh:447-478`). + +## Requirement governance + +Canonical docs: `tests/BEHAVIOR-SPEC.md` (202 stories), +`docs/superpowers/specs/2026-07-23-…-design.md` (140 reqs), `README.md` + +`docs/INTEGRATION.md` (150), `CLAUDE.md` (110) — **602 requirements** total. See +[requirements/register.md](requirements/register.md) for coverage and the nine +canon-vs-canon conflicts, and [requirements/drift-report.md](requirements/drift-report.md) +for **48 confirmed drift findings plus 8 confirmed defects that no requirement +covers**. [claims-audit.md](claims-audit.md) grades the project's stated +capabilities (summary and README claims) against this canon. + +## Coverage & freshness + +Read, not inferred: all of `bin/`, `lib/`, `.github/`, `install.sh`, +`templates/`, `skills/`, `archive/`, and the harness files under `tests/`; +`tests/simple-test.sh` was sampled structurally rather than line-by-line. +`bash tests/simple-test.sh` was executed on this machine: **202 passed, 0 failed, +exit 0**. Every drift entry carries an adversarial verifier verdict; 3 candidate +findings were refuted and are listed in the drift report so they are not +re-raised. Files matched by no module glob: the repo-root docs (`README.md`, +`CLAUDE.md`, `ROADMAP.md`, `LICENSE`, `docs/**`, dotfiles) — these are canon +*sources*, not code, and are governed rather than governing. diff --git a/docs/canon/claims-audit.md b/docs/canon/claims-audit.md new file mode 100644 index 0000000..098e9ad --- /dev/null +++ b/docs/canon/claims-audit.md @@ -0,0 +1,179 @@ +# Claims Audit — stated capabilities vs. canon + +What the project says about itself, graded against +[the canon](architecture.md) at commit `7420d69` (2026-07-24). Every grade is +anchored; the evidence lives in [requirements/drift-report.md](requirements/drift-report.md) +and the module docs. + +Grades: **Supported** (true as written) · **Supported, with caveat** (true, but a +reader would want the caveat) · **Overstated** (the load-bearing word does more +work than the code earns) · **Unsupported** (not true today). + +## Claims under audit + +The one-line descriptions the project is summarized by, in the two forms it is +usually stated: + +> *reqdrive — a requirements-to-PR pipeline that generates a PRD, then implements +> story-by-story deterministically and opens a PR with a validation checklist.* + +> *reqdrive — a requirements-to-PR pipeline: takes a written spec, generates a +> PRD, implements story-by-story deterministically, and opens a PR with a +> validation checklist — the ambiguity-to-working-code loop, automated.* + +And the strongest form of the summary claim: + +> *…a pipeline that turns written requirements into **reviewed, working** +> pull requests.* + +## Verdict table + +| # | Claim | Grade | Evidence | +|---|-------|-------|----------| +| 1 | "requirements-to-PR pipeline" / "takes a written spec" | **Supported** | markdown req in → GitHub PR out, end to end (lib/run.sh:855 → lib/pr-create.sh:261) | +| 2 | "generates a PRD" | **Supported** | Phase 1 writes `prd.json` and hard-aborts with exit 5 if the agent produces none (lib/run.sh:962-997) | +| 3 | "implements story-by-story" | **Supported** | one stateless `claude` process per story, fresh each time (lib/run.sh:1019-1046, :447-478) | +| 4 | "deterministically" | **Supported, with caveat** | see below | +| 5 | "opens a PR with a validation checklist" | **Supported, with caveat** | see below | +| 6 | "**reviewed** … pull requests" | **Overstated** | see below | +| 7 | "**working** pull requests" | **Overstated** | see below | +| 8 | "the ambiguity-to-working-code loop, automated" | **Supported, with caveat** | the loop is genuinely automated end to end; "working" carries the same weight as #7 | + +--- + +### #4 "deterministically" — Supported, with caveat + +**What is true, and it is the non-obvious engineering:** story selection is a +pure jq function of `prd.json` — `[.userStories[] | select(.passes != true and +((.attempts // 0) < $max))] | sort_by(.priority) | first | .id` +(lib/run.sh:423-428). The agent is never asked what to work on; the shell selects +at `lib/run.sh:1028` *before* building the prompt at `:1043`, and the prompt says +"implement ONLY story `@@STORY_ID@@`". Three consecutive runs on a fixed PRD were +verified identical. That is a real property most agent harnesses do not have. + +**The caveat:** `prd.json` — the sole input to that function — is written by the +agent, and the implementation prompt explicitly invites it to *"update priorities +in prd.json"* (lib/run.sh:366). The agent therefore influences future selection +one iteration ahead, can remove a story from the queue by setting `passes: true` +(nothing re-verifies that per story), and can end Phase 2 early with +`COMPLETE` (lib/run.sh:1112). A story omitting `priority` sorts +first under jq's null ordering. + +**Fair phrasing:** "deterministic story selection" or "the shell, not the agent, +picks the next story" — both fully earned. "Deterministic pipeline" is not. +→ [CLD-001](requirements/drift-report.md#CLD-001) + +### #5 "opens a PR with a validation checklist" — Supported, with caveat + +**Verified true, end to end.** A dedicated verifier tried to refute this and +could not: `lib/pr-create.sh:112-121` reads each story's `id`, `title` and +`acceptanceCriteria[]` and emits + +``` +### Functional Verification + +### US-001: Login form +- [ ] User can submit email and password +- [ ] Invalid creds show an error +``` + +spliced at `:246` into the body that reaches `gh pr create --body` at `:267`. +Rendered end-to-end under a fake `gh`. The PR body *also* carries a Pipeline +Verification table from `verification-summary.json` (`:160-178`) and a scope-findings +section (`:183-191`) — both more than the claim promises. + +**The caveats a careful reader would want:** +- It degrades **silently**. A story whose `acceptanceCriteria` is missing ships as a heading with zero checkboxes (jq's error goes to stderr inside a process substitution, so `set -e` never fires); `prd.json` absent, `userStories: []` and `acceptanceCriteria: []` all render an identical empty section. A reviewer cannot tell "nothing to verify" from "the content was lost". Schema validation *would* catch it but is warn-only at every call site (lib/run.sh:916, 1000, 1106). +- Nothing downstream reads the checklist back or gates a merge on it — it is a prompt to a human, which is the honest reading of "validation checklist" but worth knowing. +- Zero test coverage: no assertion anywhere checks that a `- [ ]` line ever appears. +- One line in the same body is factually wrong: `**Stories completed:** ` has no `.passes` filter, so it contradicts the verification table directly above it. → [PR-OVERSTATES-COMPLETION](requirements/drift-report.md#PR-OVERSTATES-COMPLETION) + +### #6 "reviewed … pull requests" — Overstated + +This is the weakest claim in the set, and the project's own decision log +already says so (CLAUDE.md:190): + +- **Off by default.** `reviewCommand` defaults to `""` (lib/config.sh); with the default config no review happens at all. +- **Not an independent grader.** The review runs on the *same* `$model` as the implementer (lib/run.sh:1195 → :610) — writer and grader are the same system. +- **Cannot influence the outcome.** It runs *after* `gh pr create` (lib/run.sh:1193 then :1195) and returns 0 on every path (:521, 543, 626, 666), so it can never change the draft flag or abort anything. +- **And the output is currently broken.** The findings table's heading, header row, delimiter and first finding collapse onto one line (lib/pr-create.sh:40-42), so GFM renders no table; the same section is passed through `sanitize_for_prompt` before `gh pr edit`, publishing literal `\$` and turning code spans into single quotes. → [US-REV-05](requirements/drift-report.md#US-REV-05) + +**Fair phrasing:** "opens a PR with a verification table and a validation +checklist for human review", or "with an optional post-PR review pass". Drop +"reviewed" as an unqualified property of the output. + +### #7 "working pull requests" — Overstated + +"Working" implies the pipeline will not present unverified work as ready. The +design decision that would earn it — D2, *"the draft-PR gate becomes fail-closed: +draft by default, cleared only on positive evidence"* — is the project's stated +L2 fix and its headline readiness claim (`docs/STATUS.md:16-19`). + +**The gate's shape is right; three of its inputs can report success without +success occurring.** Two independent verifiers reproduced **non-draft PRs with +real incomplete work** using the repo's own test harness: + +| Path | What happens | Observed | +|------|--------------|----------| +| Unparseable / fenced / `.userStories`-less `prd.json` | every jq in `verify_collect` ends `\|\| echo "0"` (lib/verification.sh:68-75), so "cannot count" reads as "zero remaining"; `prd_present` is a bare `[ -f ]` | `gh pr create` with **no `--draft`**, body reading "Verification passed / Stories 0 / 0 completed" | +| Agent deletes a story it cannot implement | nothing binds `prd.json`'s story set to the plan — no freeze, hash or count invariant | non-draft PR asserting "Stories 1 / 1 completed" while US-002 was never implemented | +| `userStories: []` | passes `validate_prd_schema` (no minimum-count check) | non-draft PR, zero commits, "Verification passed" | +| `testCommand: " "` or `"# npm test (disabled)"` | `[ -n … ]` says configured, `eval` exits 0 | `verification_passed: true` instead of `null` → non-draft | + +Two further facts a reader should weigh: a Claude **timeout or crash exits 0** +(lib/run.sh:465-478 — measured), so an integrator branching on exit status cannot +see agent failure; and nothing ever independently checks a story's self-reported +`passes: true` (lib/verification.sh:69). + +**What genuinely holds:** the three states the gate was written for — missing +PRD, correctly-counted incomplete stories, and `verification_passed` null/false — +*do* force a draft; the final test suite really is executed (`eval +"$REQDRIVE_TEST_COMMAND"`, lib/verification.sh:85), not merely re-read from the +agent's self-report; and no other code path creates a PR. + +**Fair phrasing:** "opens a draft PR unless the story set is complete and the +test suite passes" — accurate, still impressive, and does not rest on the +fail-open paths. → [DES-002](requirements/drift-report.md#DES-002) + +--- + +## README and in-repo publicity + +| Claim | Grade | Note | +|-------|-------|------| +| "Version 0.3.0" | **Supported** | `bin/reqdrive:11`, measured | +| Command table (10 commands) | **Supported** | every documented command exists and every existing command is documented — machine-gated (tests/simple-test.sh:2963) | +| Config field table (12 fields, defaults) | **Supported** | all 12 read at lib/config.sh:53-91 with matching defaults; the README table is *more* accurate than CLAUDE.md's, which omits `maxStoryRetries` and `policy` | +| `orchestrate` "**Not implemented** — prints a coming soon notice and exits 0" | **Supported** | measured. Honest self-description | +| `verify` exit codes 0/9/3/4/10 | **Supported** | all five measured | +| Risk-tier prefix semantics (`src/auth` matches `src/auth/login.ts`, not `src/authorization/x.ts`) | **Supported** | measured exactly as documented — one of the better-specified parts of the repo | +| Scope check "warn by default, block opt-in", with the rationale | **Supported** | code matches; it is CLAUDE.md:212 and STATUS.md that wrongly call it a hard gate → [CLD-086](requirements/drift-report.md#CLD-086) | +| "Requirement content is scanned for … **path traversal**" (README.md:159, CLAUDE.md:56/262, PIPELINE-ANALYSIS.md:187) | **Unsupported** | `validate_file_path` has zero runtime callers and no pattern matches `../`. Mitigating: content is never used as a path, so the check would protect nothing — the real gap is the unvalidated REQ-ID argument → [CLD-006](requirements/drift-report.md#CLD-006) | +| "Windows Users: … Git Bash or WSL2" (README.md:25) | **Unsupported by evidence** | all six CI jobs are `ubuntu-latest`; nothing verifies MSYS2/WSL2. The suite does pass locally under Git-Bash (202/202, ~10 min) | +| "all 152 tests must pass" (CLAUDE.md:232) / "157 tests" (CLAUDE.md:26) | **Unsupported** | measured **202** in the suite, the lock and BEHAVIOR-SPEC; `docs/STATUS.md:97`'s "202/202" is the correct figure | +| "Simplification from 17 scripts to 5" (CLAUDE.md:131) | **Unsupported** | `lib/` is 11 files / 2,683 lines — 52% *larger* than the 1,770-line v0.1.x lib it replaced → [CLD-059](requirements/drift-report.md#CLD-059) | +| "L3 on the Readiness Ladder" (docs/STATUS.md:5) | **Overstated** | L3 rests on the L2 fix (the fail-closed gate), which does not hold → #7 above. L1 *is* now genuinely held: the harness can report a failure (202/202 correct assertion form, 0 deviations) and is frozen behind whole-file hashes | + +## What to do with this + +**Defensible as written, no change needed:** #1, #2, #3, and every README row +graded Supported. The determinism property and the risk-tier semantics are the +two things here that are more rigorous than the average project of this kind, and +both survived adversarial verification intact. + +**Worth rewording wherever the project is described** — one sentence covers both: + +> *reqdrive — a requirements-to-PR pipeline: takes a written spec, generates a +> PRD, then implements story-by-story with deterministic shell-side story +> selection, opening a draft PR unless the story set is complete and the test +> suite passes — with a validation checklist and verification table in the body.* + +That drops "reviewed" and unqualified "working", keeps everything the code +earns, and is arguably a stronger claim because "draft unless proven" is a more +specific engineering statement than "working". + +**Worth fixing in the repo first**, if the claim matters more than the wording: +[DES-002](requirements/drift-report.md#DES-002) is four bounded changes (fail +closed on a jq error instead of `|| echo "0"`; require `total > 0`; pin the story +set in the checkpoint; trim `testCommand`). Fixing them would make "working" +defensible as written. diff --git a/docs/canon/conventions/artifact-contracts.md b/docs/canon/conventions/artifact-contracts.md new file mode 100644 index 0000000..9f92a81 --- /dev/null +++ b/docs/canon/conventions/artifact-contracts.md @@ -0,0 +1,40 @@ +# Convention: artifact contracts (canon) + +## Rule + +Machine state is **JSON**; human/agent context is **Markdown**. Every JSON +artifact the pipeline writes carries `"version": "0.3.0"` (CLAUDE.md:234). All +run state lives under `.reqdrive/runs//` (lib/run.sh:685,691). +Schema predicates in `lib/schema.sh` police the boundary; enforcement strength is +the call site's choice. + +## Applies to + +`lib/**`, `bin/**`, `tests/fixtures/**` + +## Examples + +| Artifact | Written by | Version field? | +|----------|-----------|----------------| +| `run.json` | lib/run.sh:24 | **no** — the sole exception | +| `prd.json` | the agent (schema at lib/schema.sh:125) | yes, agent-supplied | +| `checkpoint.json` | lib/run.sh:88 | yes (:107) | +| `verification-summary.json` | lib/verification.sh:98 | yes (:137) | +| `iteration-N.summary.json` | lib/run.sh:170 | agent-supplied | +| `reqdrive.json` | lib/init.sh:57 | yes (:59) | +| `progress.txt`, `prompt.md`, `iteration-N.log`, `scope-findings.txt`, `output.log` | various | markdown/plain — no version | + +- Atomic write, done right: `verify_write_summary` writes `.tmp` then `mv` (lib/verification.sh:135-164) +- Tri-state kept honest: `verification_passed` is the JSON literal `true`/`false`/`null`, never a quoted string (lib/verification.sh:160) +- `remaining` is `null` exactly when `prd_present` is false (lib/verification.sh:107-112) + +## Exceptions + +- `run.json` carries no `version` field, unlike every other pipeline-written JSON. (lib/run.sh:71-83) +- `cmd_migrate` rewrites files with `echo "$tmp" > "$file"` — truncate-then-write, not the atomic pattern used elsewhere. (bin/reqdrive:363-364,383-384) +- `reqdrive init` interpolates raw answers into its heredoc with no JSON escaping, so it can emit an unparseable `reqdrive.json`. → [drift RDM-012](../requirements/drift-report.md#RDM-012) +- `run.json.pr_url` is JSON-*escaped* correctly (`jq -Rn`, lib/run.sh:48) but holds the wrong *value* — a two-line string including `create_pr`'s progress output. → [drift INT-030](../requirements/drift-report.md#INT-030) +- The PRD schema does not validate `attempts` at all, and makes `passes` and `priority` optional. → [drift ATTEMPTS-UNVALIDATED](../requirements/drift-report.md#ATTEMPTS-UNVALIDATED) +- Every `verify_collect` jq read ends `|| echo "0"`, so an unreadable `prd.json` is indistinguishable from a complete one. This is the mechanism behind the headline drift finding. → [drift DES-002](../requirements/drift-report.md#DES-002) +- `README.md:113-121` lists a strictly smaller run-directory contents than `docs/INTEGRATION.md:397-414`, and neither lists `scope-findings.txt`. Recorded as a canon-vs-canon conflict in the register. +- Two independent `0.3.0` literals exist — `bin/reqdrive:11` and `lib/schema.sh:4` — with nothing asserting they agree. diff --git a/docs/canon/conventions/sanitization.md b/docs/canon/conventions/sanitization.md new file mode 100644 index 0000000..83528b3 --- /dev/null +++ b/docs/canon/conventions/sanitization.md @@ -0,0 +1,33 @@ +# Convention: sanitization (canon) + +## Rule + +All user- and agent-supplied content passes through `sanitize_for_prompt` +before reaching a prompt; PR labels pass through `sanitize_label`; requirement +content is scanned against `DANGEROUS_PATTERNS`. CLAUDE.md:56 declares this a +**security boundary** covering "shell injection, path traversal, and dangerous +commands". + +## Applies to + +`lib/**`, `bin/**` + +## Examples + +- PRD fields sanitized before token substitution: lib/run.sh:299-302 +- Requirement content sanitized: lib/run.sh:738 +- Labels sanitized before `gh`, then expanded as a quoted array: lib/pr-create.sh:200,210,265 +- Both prompt heredocs quoted, so nothing is re-expanded: lib/run.sh:213, :324 +- The prompt file reaches the agent by stdin redirect, never `eval`: lib/run.sh:465 + +## Exceptions + +**The boundary is narrower than the docs claim. Treat the following as true:** + +- **No path-traversal protection exists at runtime.** `validate_file_path` (lib/sanitize.sh:112) has zero callers in `lib/` or `bin/`, and none of the 15 `DANGEROUS_PATTERNS` (lib/sanitize.sh:8-30) matches `../`, `..\`, a leading `/`, `~/`, or a symlink. Verified by execution: traversal probes pass the scan clean. → [drift CLD-006](../requirements/drift-report.md#CLD-006) + - *Correctly scoped:* requirement **content** is never used as a filesystem path — it is only `cat`'d into the prompt — so scanning it for traversal would protect nothing. The genuine gap is the unvalidated **REQ-ID argument**, which reaches both the requirement-file glob (lib/run.sh:704) and `mkdir -p` (lib/run.sh:691,729). Verified: a REQ-ID of `../../../../pwndir` creates a directory outside the project root. That is operator-triggered, not attacker-reachable — the operator already has shell access. +- **Content scanning is unconditionally warn-only.** The single caller omits the `strict` argument (lib/run.sh:721), making the strict branch and the `--force` bypass that guards it both dead code. → [drift CLD-110](../requirements/drift-report.md#CLD-110) +- **`sanitize_for_prompt` now corrupts two of its three consumers.** Its `$`→`\$` escaping was needed for unquoted heredocs; the implementation prompt reverses it (lib/run.sh:317-321), but `build_planning_prompt` (lib/run.sh:276) and the PR review body (lib/pr-create.sh:52) do not. Measured: `Budget is $500 for ${TEAM}` reaches `prompt.md` as `Budget is \$500 for \${TEAM}`. The backtick→`'` rule also destroys code spans in published PR text. → [drift SANITIZE-PLANNING](../requirements/drift-report.md#SANITIZE-PLANNING) +- **One agent-controlled value reaches a glob unquoted.** `lib/run.sh:1077` interpolates the PRD-authored story id into a `[[ == ]]` pattern, and `lib/schema.sh:159` puts no charset constraint on `.id`. Verified: an id of `US-*` credits a stale prior-iteration commit. Blast radius is a reporting counter only — nothing gates on it. → [drift COMMIT-CHECK-FORGEABLE](../requirements/drift-report.md#COMMIT-CHECK-FORGEABLE) +- **Config strings are executed by design**: `eval "$REQDRIVE_TEST_COMMAND"` (lib/run.sh:1062, lib/verification.sh:85) and `bash -c` on `completionHook` (lib/run.sh:502) and `reviewCommand` (:646). A repo-local `reqdrive.json` is therefore arbitrary code execution — and the agent can write that file. +- `realpath` is used with inconsistent guards (lib/sanitize.sh:124 fails closed, :130 unguarded); unreachable today because the function is dead. diff --git a/docs/canon/conventions/shell-conventions.md b/docs/canon/conventions/shell-conventions.md new file mode 100644 index 0000000..109cfb6 --- /dev/null +++ b/docs/canon/conventions/shell-conventions.md @@ -0,0 +1,31 @@ +# Convention: shell conventions (canon) + +## Rule + +The entry point sets `set -euo pipefail` (bin/reqdrive:6); libraries are +**sourced into that process**, never executed, so errexit governs all of them +regardless of their own headers. Libraries are nominally required to set `set -e` +(CLAUDE.md:230). Every modified `.sh` must pass `bash -n` before commit +(CLAUDE.md:231). + +## Applies to + +`bin/**`, `lib/**`, `tests/**`, `scripts/**`, `install.sh` + +## Examples + +- Entry point: `set -euo pipefail` (bin/reqdrive:6) +- Compliant libraries: `lib/run.sh:6`, `lib/validate.sh:5`, `lib/init.sh:4`, `lib/policy.sh:10`, `lib/verification.sh:54` +- Sourcing model — every reference is `source`, never `bash lib/x.sh` (bin/reqdrive:14,83,169-193,343-344,433-436,476-541) +- Logging: `log_info`/`log_warn`/`log_error` are defined in `lib/run.sh:15-17`; `bin/reqdrive:482-483` defines its own shims for `cmd_verify`, which never sources run.sh +- Errors: named `EXIT_*` constants from `lib/errors.sh:7-17`, not bare integers + +## Exceptions + +- **Six of eleven libraries set no shell mode** — `lib/config.sh`, `lib/errors.sh`, `lib/pr-create.sh`, `lib/preflight.sh`, `lib/sanitize.sh`, `lib/schema.sh` (measured). No concrete bug follows, because sourcing means the entry point's `set -euo pipefail` already applies. This is a documentation-conformance gap, not a defect. → [drift CLD-098](../requirements/drift-report.md#CLD-098) +- The inverse hazard is more real: the five libraries that *do* `set -e` at file top level mutate their sourcer's options. `tests/simple-test.sh` runs with `set +e` and works around this with a `(set -e; source …)` subshell at :374. +- `errexit` does **not** propagate into command substitutions without `shopt -s inherit_errexit`, which this repo never sets — so `x=$(failing_function)` continues silently. This is load-bearing: it is why a failing `select_next_story` does not abort the pipeline. (verified by execution) +- `set -e` is suppressed inside `if` conditions, which is how an agent timeout at `lib/run.sh:465` becomes a warning rather than an abort. +- `bash -n` coverage in CI omits `tests/**` and `scripts/**` (.github/workflows/ci.yml:31-44), which is how `scripts/setup-validation-env.sh` rotted undetected — though only semantically; it still parses. → [drift CLD-099](../requirements/drift-report.md#CLD-099) +- `lib/config.sh:36-39` calls `exit`, not `return`, so a sourcing caller cannot recover from a config failure. +- bash 4+ is assumed (`declare -A` at lib/errors.sh:21) but never version-checked. → [drift RDM-005](../requirements/drift-report.md#RDM-005) diff --git a/docs/canon/conventions/testing-and-freeze.md b/docs/canon/conventions/testing-and-freeze.md new file mode 100644 index 0000000..b9f61db --- /dev/null +++ b/docs/canon/conventions/testing-and-freeze.md @@ -0,0 +1,35 @@ +# Convention: testing and freeze (canon) + +## Rule + +Every assertion in `tests/simple-test.sh` has exactly one `### US-` story in +`tests/BEHAVIOR-SPEC.md`, and the pairing is frozen in `tests/oracle.lock.json` +together with whole-file sha256 hashes. The suite must be able to **report** a +failure — the assertion form is `set +e` at file scope with `set -e` as the +first statement inside each `( … )` subshell, invoked as a simple command. + +## Applies to + +`tests/**`, `.github/**` + +## Examples + +- The one correct assertion form (measured 202/202 conforming, 0 deviations): file-scope `set +e` at tests/simple-test.sh:11, subshell body opening with `set -e`, closing `)` followed by `test_result "name" $?` +- Totality: 202 test names == 202 lock entries == 202 spec stories, enforced by `tests/spec-map.sh` +- Freeze: four whole-file hashes — `simple-test.sh`, `oracle-gate.sh`, `pipeline-harness.sh`, `spec-map.sh` (tests/oracle-gate.sh:118-137) +- Gate rules R0–R3, R6, R7 all evaluate (no short-circuit) at tests/oracle-gate.sh:114-184; `tests/gate-selftest.sh` proves R7, R2, R6 and R0 fire +- Doc-coverage gates, frozen as US-DOC-01/02/03: every command (tests/simple-test.sh:2963), every config field (:2979) and every flag (:3005) must appear in README.md +- Warn-before-enforce ladder: log → checkpoint annotation → hard gate (CLAUDE.md:54); `testCommand` and the commit check sit at rung one, the scope check at rung one-and-a-half (warn default, `block` opt-in) + +## Exceptions + +- The freeze covers **four files**. All 52 bats tests, `launch-lifecycle.sh`, `gate-selftest.sh` and `mutate.sh` are unfrozen. +- `--accept` regenerates the lock without checking the suite's exit status, so it will lock a red suite (tests/oracle-gate.sh:51-88). *Refuted as a gate hole:* R2 catches any failing locked name on the very next enforce run, so the exposure is a stale-lock window, not a green-on-red gate. +- `spec-map.sh` and `mutate.sh` are linted by CI but never executed by it — the mutation evidence behind the freeze is a one-off human result. +- `bash -n` in CI omits `tests/**`. +- 24 assertions end in a terminal negative and cannot detect a broken setup; `tests/FINDINGS.md` and `docs/STATUS.md:43` still say 18. → [drift DES-059](../requirements/drift-report.md#DES-059) +- `spec-map.sh` checks name totality only — a story whose *body* has drifted from its test's behavior is invisible to every gate. +- The lock records `environment.claude: true` but CI installs no `claude`, and the gate never reads that field. → [drift DES-008](../requirements/drift-report.md#DES-008) +- The suite exercises the real `run_pipeline` in only 12 of 202 assertions; `tests/lib/pipeline-harness.sh` fakes just `claude` and `gh` (git, config load and all of `lib/` are real). The bats "e2e" suite does not use that harness and never reaches PR creation. +- No test asserts on the rendered PR body's checklist or scope-findings sections — which is why two rendering defects survived. → [drift RDM-077](../requirements/drift-report.md#RDM-077) +- No Windows/macOS CI runner exists, so the MSYS2 support README.md:25 promises is unverified. diff --git a/docs/canon/index.json b/docs/canon/index.json new file mode 100644 index 0000000..36eaa8e --- /dev/null +++ b/docs/canon/index.json @@ -0,0 +1,110 @@ +{ + "version": 1, + "generated": "2026-07-24", + "commit": "7420d69", + "architecture_doc": "docs/canon/architecture.md", + "modules": [ + { + "name": "cli", + "doc": "docs/canon/modules/cli.md", + "globs": ["bin/**"], + "responsibility": "argument parsing, command dispatch, and the offline commands (status, logs, migrate, verify, launch)", + "governing_reqs": 101 + }, + { + "name": "pipeline", + "doc": "docs/canon/modules/pipeline.md", + "globs": ["lib/run.sh"], + "responsibility": "the requirement-to-PR lifecycle: planning, the story loop, verification orchestration, PR creation, review", + "governing_reqs": 212 + }, + { + "name": "evidence", + "doc": "docs/canon/modules/evidence.md", + "globs": ["lib/verification.sh", "lib/policy.sh"], + "responsibility": "verification math, the verification-summary artifact, risk-tier path classification and the scope check", + "governing_reqs": 98 + }, + { + "name": "pr", + "doc": "docs/canon/modules/pr.md", + "globs": ["lib/pr-create.sh"], + "responsibility": "PR body construction (validation checklist, verification table, scope findings) and submission via gh", + "governing_reqs": 44 + }, + { + "name": "config", + "doc": "docs/canon/modules/config.md", + "globs": ["lib/config.sh", "lib/schema.sh", "lib/validate.sh", "lib/init.sh", "templates/**"], + "responsibility": "manifest discovery and loading, JSON schema predicates, the validate command, the init wizard", + "governing_reqs": 128 + }, + { + "name": "safety", + "doc": "docs/canon/modules/safety.md", + "globs": ["lib/sanitize.sh", "lib/preflight.sh", "lib/errors.sh"], + "responsibility": "input sanitization, pre-run gates, and the exit-code vocabulary", + "governing_reqs": 113 + }, + { + "name": "test-harness", + "doc": "docs/canon/modules/test-harness.md", + "globs": ["tests/**"], + "responsibility": "the 202-assertion suite, the oracle freeze gate, the pipeline harness, and the bats suites", + "governing_reqs": 92 + }, + { + "name": "ci-and-install", + "doc": "docs/canon/modules/ci-and-install.md", + "globs": [".github/**", "install.sh", "scripts/**"], + "responsibility": "the six CI gate jobs and the installer", + "governing_reqs": 30 + }, + { + "name": "skills-and-archive", + "doc": "docs/canon/modules/skills-and-archive.md", + "globs": ["skills/**", "archive/**"], + "responsibility": "inert peers: four Claude Code skills and the archived v0.1.x implementation", + "governing_reqs": 4 + } + ], + "conventions": [ + { + "topic": "shell-conventions", + "doc": "docs/canon/conventions/shell-conventions.md", + "globs": ["bin/**", "lib/**", "tests/**", "scripts/**", "install.sh"], + "responsibility": "shell modes, sourcing model, logging and error propagation" + }, + { + "topic": "artifact-contracts", + "doc": "docs/canon/conventions/artifact-contracts.md", + "globs": ["lib/**", "bin/**", "tests/fixtures/**"], + "responsibility": "JSON for machine state, Markdown for agent context, schemaVersion 0.3.0" + }, + { + "topic": "sanitization", + "doc": "docs/canon/conventions/sanitization.md", + "globs": ["lib/**", "bin/**"], + "responsibility": "what is sanitized, where the security boundary actually is, and where it is not" + }, + { + "topic": "testing-and-freeze", + "doc": "docs/canon/conventions/testing-and-freeze.md", + "globs": ["tests/**", ".github/**"], + "responsibility": "one behavior-spec story per assertion, whole-file freeze hashes, doc-coverage gates" + } + ], + "requirements": { + "register": "docs/canon/requirements/register.md", + "drift_report": "docs/canon/requirements/drift-report.md", + "claims_audit": "docs/canon/claims-audit.md" + }, + "unbuilt_features": [ + "Vision-based QA agent (Playwright + Claude Vision) — CLD-092 — roadmap, 0% built, deliberately deferred", + "Multi-requirement parallelism via git worktrees (`reqdrive orchestrate`) — CLD-093 — roadmap, stub only at bin/reqdrive:579-586", + "PR rejection feedback loop — CLD-094 — roadmap, 0% built", + "CI integration via `gh pr checks` polling — CLD-095 — roadmap, 0% built", + "Cost tracking / token budgets — CLD-096 — roadmap, 0% built", + "Adaptive retry policies — CLD-097 — roadmap, 0% built" + ] +} diff --git a/docs/canon/modules/ci-and-install.md b/docs/canon/modules/ci-and-install.md new file mode 100644 index 0000000..67605d8 --- /dev/null +++ b/docs/canon/modules/ci-and-install.md @@ -0,0 +1,84 @@ +# ci-and-install (canon) + +`.github/**`, `install.sh`, `scripts/**` — 30 governing requirements. + +## Purpose + +Runs the enforcement gates on every push/PR to `main`, and installs the CLI onto +a developer's PATH. + +## Public interface + +| Symbol | Kind | Anchor | Notes | +|--------|------|--------|-------| +| `lint` job | ci | .github/workflows/ci.yml:10 | shellcheck over an **explicit hand-maintained file list** (:29) | +| `syntax-check` job | ci | .github/workflows/ci.yml:31 | `bash -n` on `bin/`, `lib/`, `install.sh` only | +| `test-simple` job | ci | .github/workflows/ci.yml:46 | `bash tests/simple-test.sh` | +| `test-bats` job | ci | .github/workflows/ci.yml:55 | unit → e2e → assert zero e2e skips (:69-75) | +| `oracle-gate` job | ci | .github/workflows/ci.yml:77 | `oracle-gate.sh` + `gate-selftest.sh` | +| `launch-lifecycle` job | ci | .github/workflows/ci.yml:89 | LAUNCH-TEST-PLAN cases 1/4/6 | +| `install.sh` | cli | install.sh:1 | prereq loop, clone-or-pull, chmod, rc-file PATH append | +| `scripts/shellcheck` | cli | scripts/shellcheck:1 | docker wrapper pinning `koalaman/shellcheck:stable` | +| `scripts/setup-validation-env.sh` | cli | scripts/setup-validation-env.sh:1 | **dead code** — see gotchas | + +## Invariants + +- All six jobs run on `ubuntu-latest`, none declares `needs:`, so they run in parallel and any failure fails the build. (.github/workflows/ci.yml:12,33,48,57,79,91) +- CI never swallows a test exit code: `continue-on-error` appears zero times, and the only `|| true` absorbs `grep`'s zero-match status on a step already covered by an unmasked one. (.github/workflows/ci.yml:73) +- CI triggers only on push to `main` and PRs targeting `main` — no tag, schedule, or manual dispatch. (.github/workflows/ci.yml:3-7) +- The three README doc-coverage rules are frozen in the oracle lock as US-DOC-01/02/03, so weakening them trips the gate. (tests/simple-test.sh:2962-3021) +- `install.sh`'s PATH append is idempotent. (install.sh:47-48) +- `install.sh` checks tool **presence** only — never version, never authentication — for all five tools. (install.sh:12-17) + +## Data flows + +- **CI**: push/PR → 6 parallel jobs → `test-simple` runs the suite; `oracle-gate` runs it again inside `oracle-gate.sh:46` and compares against `tests/oracle.lock.json` +- **install**: install.sh:12 prereq loop → :25 clone/ff-pull → :29 chmod +x → :33-57 rc-file PATH append (zsh/bash/fish only) + +## Dependencies + +Internal: the whole `test-harness` module (its scripts are the CI steps), +`bin/`+`lib/` (lint and chmod targets), `README.md` (asserted against). +External: GitHub Actions `ubuntu-latest`, `actions/checkout@v4`, apt +`shellcheck` and `bats`, `jq`/`git`/`sha256sum`/`timeout`, `gh` (presence only), +`claude` (required by install.sh, **absent from CI**), docker (scripts/shellcheck). + +## Gotchas + +- `tests/oracle.lock.json` records `environment.claude: true`, but CI installs no `claude` and the gate never reads the `environment` field — the lock was generated where it is not enforced, contradicting design decision D8. (tests/oracle-gate.sh:64,69) → [drift DES-008](../requirements/drift-report.md#DES-008) +- `scripts/setup-validation-env.sh` would fail at runtime: it copies `templates/prompt.md.tpl` (:212, gone), sources `lib/prd-gen.sh`/`agent-run.sh`/`verify.sh` (:262,270,275, all archived), calls `reqdrive_load_config_path` (:299, undefined), and emits a v0.1.x manifest. It passes `bash -n` — the rot is semantic, so extending CI's syntax check would **not** catch it. → [drift CLD-099](../requirements/drift-report.md#CLD-099) +- `scripts/**` is invisible to both the shellcheck list (:29) and the `bash -n` job (:37-44). +- `install.sh` clones `https://github.com/user/reqdrive.git` — a literal placeholder matching README.md:30. (install.sh:25) +- `install.sh` demands `claude` before installing anything, inverting the CLI's own deliberate deferral. (install.sh:12 vs bin/reqdrive:26-39) +- `reqdrive launch` never calls `check_claude`; the detached child does, so the dependency error lands in `output.log` while `launch` exits 0 reporting success. (bin/reqdrive:588,641) +- The e2e skip assertion runs bats a third time and pipes into `grep`, so an empty bats output makes the check pass vacuously. (.github/workflows/ci.yml:69-75) +- No job sets `timeout-minutes`. +- `install.sh` writes no rc file for shells outside {zsh, bash, fish}. (install.sh:34-39) + +## Requirement coverage + +30 governing reqs: **15 satisfied · 11 drifted · 2 intent-met · +1 unimplemented (roadmap) · 1 not-code-verifiable**. This module has the highest +drift density in the repo, and it is almost entirely *unenforced prerequisites*. + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| RDM-008 / INT-006 | drifted | `gh` is authenticated with push and PR-create permission | [drift-report](../requirements/drift-report.md#RDM-008) | +| RDM-009 | drifted | `claude` is needed only for `run`/`launch` | [drift-report](../requirements/drift-report.md#RDM-009) | +| RDM-010 | drifted | `timeout` and `sha256sum` are required prerequisites | [drift-report](../requirements/drift-report.md#RDM-010) | +| RDM-005 | drifted | bash 4.0+ is required | [drift-report](../requirements/drift-report.md#RDM-005) | +| INT-007 | drifted | `claude` is installed **and authenticated** | [drift-report](../requirements/drift-report.md#RDM-008) | +| CLD-058 / CLD-106 | drifted | the documented dependency set is the real one | [drift-report](../requirements/drift-report.md#CLD-058) | +| CLD-099 | drifted | every modified `.sh` passes `bash -n` | [drift-report](../requirements/drift-report.md#CLD-099) | +| DES-008 | drifted | the lock is generated in the CI environment | [drift-report](../requirements/drift-report.md#DES-008) | +| DES-120 | drifted | the WORKFLOW.md L2→L0 correction is applied | [drift-report](../requirements/drift-report.md#DES-120) | +| CLD-095 | unimplemented | `gh pr checks` polling | roadmap — index.json#unbuilt_features | + +**Coverage hole:** there is no Windows/MSYS2 or macOS runner. All six jobs are +`ubuntu-latest`, yet MSYS2 is the design doc's primary platform and README.md:25 +promises Git-Bash/WSL2 support. Nothing verifies it. + +## Pointers + +What the gates enforce: [test-harness.md](test-harness.md). Prerequisite +contract: [safety.md](safety.md). diff --git a/docs/canon/modules/cli.md b/docs/canon/modules/cli.md new file mode 100644 index 0000000..9b3b845 --- /dev/null +++ b/docs/canon/modules/cli.md @@ -0,0 +1,87 @@ +# cli (canon) + +`bin/reqdrive` — 101 governing requirements. + +## Purpose + +The single entry point: parses arguments, enforces the tool preconditions, +dispatches ten commands, and implements every command that does **not** invoke +the agent (`status`, `logs`, `migrate`, `verify`, `launch`, `orchestrate`). + +## Public interface + +| Symbol | Kind | Anchor | Notes | +|--------|------|--------|-------| +| `reqdrive init` | cli | bin/reqdrive:680 | interactive wizard → [config.md](config.md) | +| `reqdrive run ` | cli | bin/reqdrive:86 | `-i/--interactive` (default), `--unsafe`, `--dangerously-skip-permissions`, `--force`, `--resume` | +| `reqdrive launch ` | cli | bin/reqdrive:588 | `nohup … run --unsafe` (:641); prints PID, exits 0 | +| `reqdrive status [REQ-ID]` | cli | bin/reqdrive:196 | PID liveness via `kill -0` (:256-259) | +| `reqdrive logs ` | cli | bin/reqdrive:650 | tails `output.log` | +| `reqdrive validate` | cli | bin/reqdrive:176 | exit 3 on failure | +| `reqdrive migrate` | cli | bin/reqdrive:342 | adds `version` to pre-0.3.0 config/PRD | +| `reqdrive plan ` | cli | bin/reqdrive:394 | Phase 1 only; **does** require `claude` (:395) | +| `reqdrive verify [--ref B]` | cli | bin/reqdrive:440 | exits 0 pass / 9 fail / 3 missing / 4 branch mismatch / 10 run live | +| `reqdrive orchestrate` | cli | bin/reqdrive:579 | pure stub: prints "Coming soon", exits 0 | +| `--version` / `--help` | cli | bin/reqdrive:716 / :719 | prints `reqdrive 0.3.0` | +| `check_tool` / `check_claude` | function | bin/reqdrive:18 / :31 | presence probes only | + +## Invariants + +- Every dispatch label is documented in README — machine-enforced by a doc-coverage rule. (bin/reqdrive:679-727 / tests/simple-test.sh:2963-2976) +- Every flag label accepted anywhere in the file is documented in README — likewise gated. (tests/simple-test.sh:3005-3021) +- Interactive is the default agent mode; only an explicit unsafe flag clears it, and the consumer fails safe when the variable is unset. (bin/reqdrive:92,102-105 / lib/run.sh:458) +- `launch` can never run in a permission-prompting mode: `--unsafe` is a hardcoded literal on the nohup line. (bin/reqdrive:641) +- All run-state paths are `.reqdrive/runs//`, computed identically here and in `pipeline`. (bin/reqdrive:241,486,603,663 / lib/run.sh:685,691) +- `cmd_verify` performs every refusal check *before* sourcing `lib/verification.sh`, so no refusal path can write a summary. (bin/reqdrive:490-539 precede :541) +- `jq`, `git` and `gh` are required for every invocation, including offline ones. (bin/reqdrive:26-28) + +## Data flows + +- **dispatch**: bin/reqdrive:679 → `cmd_*` → `reqdrive_load_config` (lib/config.sh:34) → `run_pipeline` (lib/run.sh:855) for run/plan only +- **launch**: bin/reqdrive:632 (duplicate-PID check) → :641 `nohup "$reqdrive_bin" run "$ID" --unsafe > output.log` → child re-enters `cmd_run` +- **verify**: bin/reqdrive:491 (run dir) → :496 (summary) → :509 (live PID → 10) → :519-537 (branch/`--ref` → 4) → :541 source verification.sh → :566/570 exit 0/9 + +## Dependencies + +Internal: `safety` (errors.sh sourced unconditionally at :14), `config`, +`pipeline`, `evidence`. External: bash 4+ (associative array in errors.sh), +`jq`/`git`/`gh` (hard-required at startup), `claude` (deferred to run/plan), +`nohup`, `kill`, `tail`, `tr`. + +## Gotchas + +- `VERSION` is duplicated: `bin/reqdrive:11` (what `--version` prints) and `lib/schema.sh:4` (what stamps artifacts). Nothing asserts they agree. +- `cmd_run`/`cmd_plan` call `check_claude` **before** parsing arguments, so `--help`-style mistakes on a claude-less machine fail with a dependency error. (bin/reqdrive:88,395) +- `launch` and `logs` are dispatched with only `"${1:-}"`, silently discarding extra arguments where `run`/`plan`/`verify` reject them. (bin/reqdrive:710,714) +- `reqdrive verify --ref ` runs a real `git checkout` and never restores the caller's original branch. (bin/reqdrive:519) +- The same "no reqdrive.json" condition yields **two different exit codes**: 3 from `cmd_validate`, 1 from `reqdrive_load_config`. (bin/reqdrive:183 vs lib/config.sh:38) → [drift INT-013](../requirements/drift-report.md#INT-013) +- `kill -0` liveness cannot distinguish a recycled PID, and is unreliable under MSYS2. (bin/reqdrive:256-259) +- `cmd_migrate` rewrites files with `echo "$tmp" > "$file"` — truncate-then-write, not atomic. (bin/reqdrive:363-364,383-384) +- `gh` is a startup hard dependency even for wholly offline commands (`validate`, `migrate`, `status`, `logs`). (bin/reqdrive:26-28) +- The unsafe confirmation prompt only fires on a TTY, so piped/CI `run --unsafe` proceeds unprompted. (bin/reqdrive:159-166) + +## Requirement coverage + +101 governing reqs: **92 satisfied · 5 drifted · 1 unimplemented (roadmap) · +1 intent-met · 2 not-code-verifiable**. Canon (intended) behavior for the +drifted ones — all of them exit-code or prerequisite contracts owned by +[safety](safety.md) and [ci-and-install](ci-and-install.md): + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| INT-013 | drifted | exit 3 means bad `reqdrive.json` or missing requirement file | [drift-report](../requirements/drift-report.md#INT-013) | +| INT-014 | drifted | exit 4 means a git operation failed | [drift-report](../requirements/drift-report.md#INT-014) | +| INT-017 | drifted | exit 7 means the user sent SIGINT | [drift-report](../requirements/drift-report.md#INT-017) | +| RDM-009 | drifted | `claude` is needed only for `run` and `launch` | [drift-report](../requirements/drift-report.md#RDM-009) | +| CLD-106 | drifted | require bash 4.0+, jq, git, and an **authenticated** gh | [drift-report](../requirements/drift-report.md#RDM-008) | +| CLD-093 | unimplemented | `orchestrate` builds multi-requirement parallelism | roadmap — index.json#unbuilt_features | + +Verified-good, and worth stating because they are commonly assumed broken: +`--version` prints `reqdrive 0.3.0` (measured); `orchestrate` exits 0 (measured); +`verify`'s five exit codes match the documented 0/9/3/4/10 exactly; every +documented command exists and every existing command is documented. + +## Pointers + +Pipeline internals: [pipeline.md](pipeline.md). Exit-code vocabulary: +[safety.md](safety.md). Config loading: [config.md](config.md). diff --git a/docs/canon/modules/config.md b/docs/canon/modules/config.md new file mode 100644 index 0000000..0b497e9 --- /dev/null +++ b/docs/canon/modules/config.md @@ -0,0 +1,88 @@ +# config (canon) + +`lib/config.sh`, `lib/schema.sh`, `lib/validate.sh`, `lib/init.sh`, +`templates/**` — 128 governing requirements. + +## Purpose + +Finds and loads `reqdrive.json` into the `REQDRIVE_*` environment contract, +provides the JSON schema predicates for config/PRD/checkpoint, backs the +`validate` command, and writes the initial config in `init`. + +## Public interface + +| Symbol | Kind | Anchor | Notes | +|--------|------|--------|-------| +| `reqdrive_find_manifest` | function | lib/config.sh:11 | walks upward from `pwd` to `/` | +| `reqdrive_load_config` | function | lib/config.sh:34 | exports 12 `REQDRIVE_*` vars + 2 derived; **exits**, never returns non-zero | +| `reqdrive_get_req_file` | function | lib/config.sh:97 | first `*.md` glob match | +| `REQDRIVE_SCHEMA_VERSION` | config | lib/schema.sh:4 | `"0.3.0"` | +| `check_schema_version` | function | lib/schema.sh:10 | major-only compatibility | +| `validate_config_schema` | function | lib/schema.sh:50 | pure predicate; **not called at load time** | +| `validate_prd_schema` | function | lib/schema.sh:125 | top-level then per-story checks | +| `validate_checkpoint_schema` | function | lib/schema.sh:191 | soft-ignored by the resume path | + +**Config fields actually read** (lib/config.sh:53-91), with real defaults: +`version` (no default), `requirementsDir` `docs/requirements`, `testCommand` `""`, +`model` `claude-sonnet-4-20250514`, `maxIterations` `10`, `baseBranch` `main`, +`prLabels` `["agent-generated"]`, `projectName` `""`, `completionHook` `""`, +`maxStoryRetries` `3`, `reviewCommand` `""`, `policy` `{}` (`scopeCheck` `warn`). +All twelve match the README table exactly; `maxStoryRetries` and `policy` are +absent from the CLAUDE.md table. + +## Invariants + +- The manifest is discovered by walking upward and terminating at the filesystem root. (lib/config.sh:15-29) +- `REQDRIVE_PROJECT_ROOT` is always `dirname(REQDRIVE_MANIFEST)`. (lib/config.sh:42-43) +- Every config field is optional — every schema check is `has()`-guarded, so a minimal `{}` config validates. (lib/schema.sh:62-81) +- Schema functions are pure predicates: they print to stderr and return 0/1, never `exit`. Enforcement is the call site's choice. (lib/schema.sh:117,183,219) +- Version compatibility is major-only: any `0.x.x` loads. (lib/schema.sh:33-42) +- `REQDRIVE_POLICY_SCOPE_CHECK` is CR-stripped after extraction because native Windows `jq` emits CRLF. (lib/config.sh:89-91) +- Both `validate.sh` failure paths exit `EXIT_CONFIG_ERROR` (3), never a bare 1. (lib/validate.sh:22,76 — measured) +- `passes` and `priority` are optional in the PRD schema; the pipeline compensates with `select(.passes != true)`. (lib/schema.sh:165-170, lib/run.sh:424) + +## Data flows + +- **load**: bin/reqdrive:170 → `reqdrive_find_manifest` lib/config.sh:11 → `check_schema_version` lib/config.sh:46 → 12 `jq -r … // default` reads :53-91 → exported env +- **validate**: bin/reqdrive:176 → lib/validate.sh:22 (`jq empty`) → :27 `validate_config_schema` → :76 exit 3 + +## Dependencies + +Internal: `safety` (errors.sh for `EXIT_CONFIG_ERROR`), consumed by every other +module through the `REQDRIVE_*` contract. External: `jq` (every read and every +validation), coreutils `dirname`/`basename`/`cut`/`find`/`wc`. + +## Gotchas + +- `reqdrive validate` reports "Validation PASSED" and exits 0 for a manifest whose version `reqdrive run` refuses — `validate.sh` never calls `check_schema_version`. Measured. (lib/validate.sh:19-36) → [drift VALIDATE-FALSE-PASS](../requirements/drift-report.md#VALIDATE-FALSE-PASS) +- `validate` always prints "(1 errors)" regardless of violation count: `ERRORS` is incremented once per failed *check group*, and only one group exists. Cosmetic — each violation is still printed and the exit code is right. (lib/validate.sh:27-36) +- `reqdrive init` performs **no JSON escaping**; an answer containing `"` yields a `reqdrive.json` that fails `jq empty` while init prints "Done!". A crafted answer can also inject sibling keys. Measured. (lib/init.sh:57-68) → [drift RDM-012](../requirements/drift-report.md#RDM-012) +- `attempts` is never schema-validated, yet `select_next_story` compares it numerically — `"attempts": "lots"` makes a story permanently unselectable and the loop then logs "All stories complete!". Measured. (lib/schema.sh:125-184) → [drift ATTEMPTS-UNVALIDATED](../requirements/drift-report.md#ATTEMPTS-UNVALIDATED) +- `validate_config_schema` type-checks 9 fields but not `version` and not `maxStoryRetries`. (lib/schema.sh:70-81) +- PRD story-level validation is gated on the top-level checks passing, so one missing top-level key silently skips all per-story checks. (lib/schema.sh:154) +- `policy.riskTiers` keys are unrestricted, so a tier name the matcher never probes validates clean. (lib/schema.sh:105-113) +- Because `validate_config_schema` is not wired into load (deliberate — CLD-075), an out-of-enum `policy.scopeCheck: "blocking"` loads and silently degrades the gate to warn. Measured. (lib/policy.sh:82) +- `reqdrive_get_req_file` returns the **first** glob match, so `REQ-1` can match `REQ-10-*.md`. (lib/config.sh:101-106) +- `reqdrive_load_config` calls `exit`, not `return` — anything sourcing it cannot recover. (lib/config.sh:36-39) + +## Requirement coverage + +128 governing reqs: **122 satisfied · 5 drifted · 1 intent-met**. + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| RDM-012 / CLD-030 | drifted | `init` creates a valid `reqdrive.json` | [drift-report](../requirements/drift-report.md#RDM-012) | +| INT-013 | drifted | exit 3 means a bad `reqdrive.json` | [drift-report](../requirements/drift-report.md#INT-013) | +| CLD-091 | drifted | `pr-create.sh` consumes the policy exports | [drift-report](../requirements/drift-report.md#CLD-091) | +| DES-081 | drifted | `DOC_EXEMPT` holds the specified 2–3 names | [drift-report](../requirements/drift-report.md#DES-081) | +| CLD-098 | drifted | library scripts set `set -e` | [drift-report](../requirements/drift-report.md#CLD-098) | +| RDM-038 | intent-met | `version` defaults to `"0.3.0"` — code warns on absence instead of silently defaulting, which is stronger; amend the req. See the register's spec-stale table | + +`CLD-075` (config-load-time validation stays deferred) is **satisfied**: the +omission is a recorded decision at CLAUDE.md:187, not drift. + +## Pointers + +Consumers of the env contract: [pipeline.md](pipeline.md), +[evidence.md](evidence.md). Artifact versioning: +[../conventions/artifact-contracts.md](../conventions/artifact-contracts.md). diff --git a/docs/canon/modules/evidence.md b/docs/canon/modules/evidence.md new file mode 100644 index 0000000..09ba03c --- /dev/null +++ b/docs/canon/modules/evidence.md @@ -0,0 +1,79 @@ +# evidence (canon) + +`lib/verification.sh`, `lib/policy.sh` — 98 governing requirements. + +## Purpose + +The pipeline's evidence layer: it counts stories, actually re-runs the test +suite, writes `verification-summary.json`, classifies changed paths into risk +tiers, and raises scope findings. Everything the draft decision and the PR's +verification table rest on is computed here. + +## Public interface + +| Symbol | Kind | Signature | Anchor | Notes | +|--------|------|-----------|--------|-------| +| `verify_collect` | function | ` ` | lib/verification.sh:56 | sets `VERIFY_STORIES_{TOTAL,COMPLETED,FAILED,REMAINING}`, `VERIFY_PRD_PRESENT` | +| `verify_run_tests` | function | `` | lib/verification.sh:79 | **tri-state**: 0 passed, 1 failed, 2 no testCommand. Never collapse to a boolean | +| `verify_write_summary` | function | ` ` | lib/verification.sh:98 | atomic write via `.tmp` + `mv` | +| `policy_tier_for_path` | function | ` → high\|medium\|low\|none` | lib/policy.sh:13 | literal path-prefix match | +| `policy_classify_paths` | function | `…` | lib/policy.sh:35 | **dead code — no callers** | +| `policy_scope_check` | function | ` ` | lib/policy.sh:54 | returns 1 only in block mode | + +## Invariants + +- `verification_passed` is always one of the JSON literals `true`, `false`, `null` — never a quoted string. (lib/verification.sh:160) +- `remaining` is JSON `null` exactly when `prd_present` is false; otherwise an integer. (lib/verification.sh:107-112) +- A tier pattern matches only at a directory boundary: `path == pattern`, or `path` begins with `pattern/`. `src/auth` matches `src/auth/login.ts` but never `src/authorization/x.ts`. (lib/policy.sh:24, verified by execution) +- Tier precedence is the literal probe order `high, medium, low`, returning on first match. (lib/policy.sh:19-27, verified by execution) +- With no policy configured, `REQDRIVE_POLICY_JSON` is `{}`, every path classifies `none`, and no finding is possible. (lib/policy.sh:15) +- Warn mode never changes the pipeline's exit code; block mode's only escalation is returning 1, which the caller turns into exit 8. (lib/policy.sh:82-87 → lib/run.sh:1089-1093) +- `merge` mode never rewrites `iterations`/`tests`/`commits` from the accumulators — it reads them back from the existing file. (lib/verification.sh:116-125) +- The summary file is replaced atomically. (lib/verification.sh:135-164) + +## Data flows + +- **verification**: lib/run.sh:1131 → `verify_collect` lib/verification.sh:56 (reads prd.json) → `verify_run_tests` lib/verification.sh:85 (`eval "$REQDRIVE_TEST_COMMAND"`) → `verify_write_summary` lib/verification.sh:98 → draft gate lib/run.sh:1177 → PR table lib/pr-create.sh:168 +- **scope check**: lib/run.sh:1088 (sourced per iteration) → `git diff --name-only HEAD~1 HEAD` lib/policy.sh:64 → `policy_tier_for_path` lib/policy.sh:71 → `scope-findings.txt` lib/policy.sh:79 → PR body lib/pr-create.sh:183-191 + +## Dependencies + +Internal: `pipeline` (sole `full`-mode caller and owner of the `RUN_SUMMARY_*` +accumulators), `cli`'s `cmd_verify` (sole `merge`-mode caller, which defines its +own `log_info`/`log_warn` shims at bin/reqdrive:482-483), `config` (the +`REQDRIVE_TEST_COMMAND`/`REQDRIVE_POLICY_*` exports), `pr` (re-reads the +artifacts). External: `jq`, `git`, `date -Iseconds`, and the user's +`testCommand` executed via `eval`. + +## Gotchas + +- The final test suite genuinely executes — `eval "$REQDRIVE_TEST_COMMAND"` at lib/verification.sh:85 — but it runs in the **caller's CWD**, not the project root. +- Nothing here independently checks the agent's self-reported `passes: true`: `stories_completed` is a raw count of the agent's own claims. (lib/verification.sh:69) +- `git diff --name-only HEAD~1 HEAD` inspects the *last commit only*, not the iteration's full work — an agent making two commits hides the first from the scope check. (lib/policy.sh:64) +- A glob in config is silently inert: `riskTiers.high = ["src/auth/**"]` matches nothing, disabling the tier with no warning. Measured. (lib/policy.sh:24) → [drift](../requirements/drift-report.md#GLOB-FOOTGUN) +- Only the tier names `high`, `medium`, `low` are probed; a `critical` tier passes schema validation and is inert. (lib/policy.sh:19) → [drift](../requirements/drift-report.md#TIER-NAME-FOOTGUN) +- Only `high` ever produces a finding — `medium` and `low` are classification-only. (lib/policy.sh:72) +- When `testCommand` is unset, `verify_run_tests` returns 2 **without truncating** `verification.test.log`, so a stale log survives. (lib/verification.sh:92-95) +- `reqdrive verify req-01` (lowercase) rewrites the summary's `req_id` from `REQ-01` to `req-01`. (bin/reqdrive:561) +- `merge` mode does not validate the file it reads; a corrupt summary aborts `verify` with jq's status. (lib/verification.sh:120-125) + +## Requirement coverage + +98 governing reqs: **89 satisfied · 5 drifted · 2 intent-met · 2 +not-code-verifiable**. Canon (intended) behavior for the drifted ones: + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| DES-114 | drifted | post-iteration path classification is recorded | [drift-report](../requirements/drift-report.md#DES-114) | +| DES-116 | drifted | a scope finding is recorded in `checkpoint.json` | [drift-report](../requirements/drift-report.md#DES-116) | +| DES-131 | drifted | warn-mode findings reach the checkpoint as well as the PR | [drift-report](../requirements/drift-report.md#DES-116) | +| CLD-086 | drifted | *(register conflict)* CLAUDE.md:212 calls the scope check a hard gate; README/D6/code make it warn-by-default. The **code is right**, the doc is wrong | [drift-report](../requirements/drift-report.md#CLD-086) | +| RDM-077 | drifted | warn findings render under a `### Scope findings` heading | [drift-report](../requirements/drift-report.md#RDM-077) | + +`INT-069` (PID liveness may be unreliable on MSYS2) is `not-code-verifiable` — +a runtime-platform property, excluded from drift and gap counts. + +## Pointers + +Draft gate consumer: [pipeline.md](pipeline.md). Rendering: +[pr.md](pr.md). Config keys: [config.md](config.md). diff --git a/docs/canon/modules/pipeline.md b/docs/canon/modules/pipeline.md new file mode 100644 index 0000000..ec893cf --- /dev/null +++ b/docs/canon/modules/pipeline.md @@ -0,0 +1,85 @@ +# pipeline (canon) + +`lib/run.sh` — 212 governing requirements. + +## Purpose + +Owns the whole requirement→PR lifecycle (setup, Phase 1 planning, Phase 2 story +loop, Phase 3 verification, PR creation, Phase 4 review) and all durable run +state. The shell decides *which* story is worked and whether the PR opens as a +draft; the Claude subprocess only writes code. + +## Public interface + +| Symbol | Kind | Anchor | Notes | +|--------|------|--------|-------| +| `run_pipeline` | function | lib/run.sh:855 | the full pipeline; sole caller of every phase | +| `run_plan` | function | lib/run.sh:743 | Phase 1 only, for `reqdrive plan` | +| `pipeline_setup` | function | lib/run.sh:676 | config, slug, agent_dir, req-file resolution | +| `write_run_status` | function | lib/run.sh:24 | writes `run.json`; preserves `started_at`; `pr_url` JSON-escaped via `jq -Rn` | +| `save_checkpoint` / `load_checkpoint` | function | lib/run.sh:88 / :123 | 8-field checkpoint; resume warns (never fails) on HEAD/SHA divergence | +| `build_planning_prompt` | function | lib/run.sh:209 | quoted heredoc (`<<'PROMPT_PLAN'`, :213) | +| `build_implementation_prompt` | function | lib/run.sh:281 | quoted heredoc (:324) + ordered `@@TOKEN@@` substitution (:399-403); output pinned by `tests/fixtures/golden-impl-prompt.md` | +| `select_next_story` | function | lib/run.sh:413 | jq: highest-priority story with `passes != true` and `attempts < max` (:423-428) | +| `run_claude_iteration` | function | lib/run.sh:447 | one stateless `claude` process under `timeout 1800` (:465) | +| `run_review_phase` | function | lib/run.sh:510 | Phase 4; returns 0 on every path (:521,543,626,666) | +| `run_completion_hook` | function | lib/run.sh:484 | `bash -c` on the configured hook; exports `REQ_ID/STATUS/PR_URL/BRANCH/EXIT_CODE` | +| `extract_iteration_summary` | function | lib/run.sh:170 | lifts the fenced iteration-summary JSON out of agent output | + +## Invariants + +- Story selection is a pure jq function of `prd.json`; the agent is never asked which story is next. (lib/run.sh:423-428) +- Every Claude invocation is a fresh process — no `--continue`/`--resume`; cross-iteration context flows only through files. (lib/run.sh:447-478) +- Both prompt heredocs are quoted; PRD-derived values reach a prompt only via `sanitize_for_prompt` + token substitution. (lib/run.sh:213, 324, 299-321, 399-403) +- Planning failure (no `prd.json` after 2 attempts) hard-aborts with `EXIT_AGENT_ERROR` before any PR exists. (lib/run.sh:992-997) +- Phase 3 always writes `verification-summary.json` before PR creation. (lib/run.sh:1164 → create_pr at :1193) +- Phase 4 review always returns 0, so it can never abort the pipeline — and runs after the PR, so it can never change the draft flag. (lib/run.sh:1195, 666) +- Per-iteration `testCommand` and commit-format checks are observation-only: counters plus warnings, never abort, never retry. (lib/run.sh:1059-1082) +- The scope check is the loop's only hard gate, and only under `policy.scopeCheck: "block"`. (lib/run.sh:1088-1093, lib/policy.sh:82-87) +- `run.json.status` vocabulary is exactly `running|completed|failed|interrupted`. (lib/run.sh:874, 877-879, 994, 1196, 1200) +- All run state is confined to `.reqdrive/runs//`. (lib/run.sh:685, 691) + +## Data flows + +- **run**: bin/reqdrive:86 → lib/run.sh:855 → preflight lib/preflight.sh:165 → branch lib/run.sh:911 → plan lib/run.sh:962 → loop lib/run.sh:1019 → verify lib/verification.sh:85 → gate lib/run.sh:1177 → PR lib/pr-create.sh:78 → review lib/run.sh:1195 +- **one iteration**: select lib/run.sh:1028 → prompt lib/run.sh:1043 → claude lib/run.sh:465 → summary lib/run.sh:170 → test lib/run.sh:1062 → commit check lib/run.sh:1077 → scope check lib/run.sh:1089 → attempts++ lib/run.sh:1096 → checkpoint lib/run.sh:1101 + +## Dependencies + +Internal: `safety` (errors, sanitize, preflight — sourced :9-11), `evidence` +(policy sourced per-iteration at :1088; verification at :1131), `pr` (:1174), +`config` (the `REQDRIVE_*` env contract). External: `claude`, `jq`, `git`, `gh` +(via `pr`), coreutils `timeout`/`tee`/`seq`/`date -Iseconds`. + +## Gotchas + +- `create_pr`'s progress line is echoed to **stdout**, and `lib/run.sh:1193` captures that stdout as the PR URL — so `run.json.pr_url` is a two-line string. (lib/pr-create.sh:95) → [drift INT-030](../requirements/drift-report.md#INT-030) +- The agent can end Phase 2 early by printing `COMPLETE`; the shell greps for it and `break`s with no re-check of remaining stories. The live prompt never asks for it, so the signal is vestigial. (lib/run.sh:1112) +- `select_next_story` sorts by `.priority` with no default, and jq orders `null` before numbers — a story omitting `priority` sorts first. (lib/run.sh:426) +- `build_planning_prompt` appends sanitized content verbatim, so the **planning** prompt still contains `\$` escapes that the implementation path reverses. (lib/run.sh:276) → [drift](../requirements/drift-report.md#SANITIZE-PLANNING) +- An agent timeout or crash is warn-only; `run_claude_iteration` never propagates the failure, so the run can exit 0. (lib/run.sh:465-478) +- The INT/TERM/HUP traps write status `interrupted` but do **not** fire the completion hook. (lib/run.sh:877-879) +- `RUN_SUMMARY_*` accumulators are initialised to 0 *before* Phase 1, so a planning failure writes a zeroed summary where `null` is specified. (lib/run.sh:935 vs :53) +- `prompt.md` is overwritten every iteration and shared with `run_plan`, so post-run it reflects only the last one. (lib/run.sh:732, 1043) +- `run.json` is the only pipeline-written JSON artifact with no `version` field. (lib/run.sh:71-83) + +## Requirement coverage + +212 governing reqs: **200 satisfied · 6 drifted · 5 unimplemented (all Tier-3 +roadmap) · 1 intent-met**. Canon (intended) behavior for the drifted ones: + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| DES-002 | drifted | draft by default, cleared **only** on positive evidence | [drift-report](../requirements/drift-report.md#DES-002) | +| CLD-001 | drifted | the shell controls what to work on **and when to stop** | [drift-report](../requirements/drift-report.md#CLD-001) | +| INT-030 | drifted | `run.json.pr_url` is the GitHub PR URL | [drift-report](../requirements/drift-report.md#INT-030) | +| INT-048 | drifted | the hook's `PR_URL` is the PR URL, empty on failure | [drift-report](../requirements/drift-report.md#INT-030) | +| INT-015 | drifted | exit 5 means the Claude invocation failed (timeout, crash, no PRD) | [drift-report](../requirements/drift-report.md#INT-015) | +| INT-031 | drifted | `run.json.summary` is `null` when the pipeline never reached implementation | [drift-report](../requirements/drift-report.md#INT-031) | +| CLD-059 | drifted | keep the library at the simplified ~5-script shape | [drift-report](../requirements/drift-report.md#CLD-059) | +| CLD-092..097 | unimplemented | Tier-3 roadmap — expected, not a gap | index.json#unbuilt_features | + +## Pointers + +Draft gate and evidence math: [evidence.md](evidence.md). PR body: +[pr.md](pr.md). Prompt-safety rules: [../conventions/sanitization.md](../conventions/sanitization.md). diff --git a/docs/canon/modules/pr.md b/docs/canon/modules/pr.md new file mode 100644 index 0000000..f6c633e --- /dev/null +++ b/docs/canon/modules/pr.md @@ -0,0 +1,81 @@ +# pr (canon) + +`lib/pr-create.sh` — 44 governing requirements. + +## Purpose + +Assembles the pull-request body — summary, commit list, Pipeline Verification +table, validation checklist, scope findings — pushes the branch, and submits via +`gh`. Also appends review findings to an existing PR. This module produces the +project's only user-visible output. + +## Public interface + +| Symbol | Kind | Signature | Anchor | Notes | +|--------|------|-----------|--------|-------| +| `create_pr` | function | ` ` | lib/pr-create.sh:78 | never decides draft itself — forwards `$5` | +| `update_pr_with_review` | function | ` ` | lib/pr-create.sh:13 | `gh pr edit --body` (:69) | + +## Invariants + +- The draft flag is fail-closed *at this layer*: `create_pr` only forwards what the caller passed, expanded as `${draft_flag:+"$draft_flag"}` at :262 and :276 — verified to survive the label-dropping retry and `set -u`. The **decision** is [pipeline](pipeline.md)'s. (lib/run.sh:1177-1190) +- Every label reaching `gh` has passed `sanitize_label`, and labels expand as a quoted array. (lib/pr-create.sh:200,210,265) +- The Pipeline Verification section exists iff `verification-summary.json` exists. (lib/pr-create.sh:131) +- The `### Scope findings` section exists iff `scope-findings.txt` exists and is non-empty. (lib/pr-create.sh:185) +- Review findings are **appended to**, never substituted for, the existing body — the current body is fetched first. (lib/pr-create.sh:65-69) +- A `create_pr` failure is always terminal: the caller writes status `failed`, fires the completion hook, and exits `EXIT_PR_ERROR`. (lib/run.sh:1198-1203) + +## Data flows + +- **body assembly**: prd.json → checklist lib/pr-create.sh:112-121 · verification-summary.json → table :138-178 · scope-findings.txt → :183-191 · git log → commits · all spliced into the heredoc at :223-255 → `gh pr create --body` :261 +- **review append**: lib/run.sh:661 → `gh pr view` :65 → concat :67 → `gh pr edit --body` :69 + +## Dependencies + +Internal: `safety` (`sanitize_label`, `sanitize_for_prompt`, auto-sourced +:6-8), `pipeline` (sole caller, :1174), `evidence` (produces both artifacts it +reads), `config` (`REQDRIVE_PR_LABELS`). External: `gh` (`pr create`, `pr view`, +`pr edit`), `git` (push, log), `jq`. + +## Gotchas + +- `echo " Pushing branch …"` goes to **stdout** while every other diagnostic in the file uses `>&2`; the caller captures that stdout as the PR URL. One-line fix: add `>&2` to :95. (lib/pr-create.sh:95) → [drift INT-030](../requirements/drift-report.md#INT-030) +- Three sections are built as `section=$(printf '…\n\n')` — command substitution strips the trailing newlines, fusing the next line onto the heading. Affects `### Scope findings` (:186) and the review findings table (:40-42). (lib/pr-create.sh:40) → [drift RDM-077](../requirements/drift-report.md#RDM-077), [US-REV-05](../requirements/drift-report.md#US-REV-05) +- `sanitize_for_prompt` is applied to PR markdown at :52, publishing literal `\$` into the stored body and rewriting backticks to single quotes — destroying code spans in review findings. (lib/pr-create.sh:52) → [drift](../requirements/drift-report.md#SANITIZE-PR-BODY) +- PR-number extraction is `grep -o '[0-9]*$'` on the URL, so a trailing slash or query string breaks it. (lib/pr-create.sh:56) +- The label-dropping retry drops **all** labels, including the REQ-specific one. (lib/pr-create.sh:275-280) +- `jq` output on MSYS carries CRLF; `lib/policy.sh:21` strips `\r` explicitly for exactly this reason — this file does not. (lib/pr-create.sh:45) +- This file sets neither `set -e` nor `set -u`, unlike run.sh/policy.sh/verification.sh. Inert in practice: it is only ever sourced into `bin/reqdrive`, which sets `set -euo pipefail`. (lib/pr-create.sh:1-8) + +## Requirement coverage + +44 governing reqs: **28 satisfied · 11 drifted · 3 intent-met · +2 not-code-verifiable**. Canon (intended) behavior for the drifted ones: + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| US-PR-01 | drifted | `create_pr` emits the PR URL on stdout | [drift-report](../requirements/drift-report.md#INT-030) | +| INT-030 / INT-048 | drifted | `run.json.pr_url` and the hook's `PR_URL` are the PR URL | [drift-report](../requirements/drift-report.md#INT-030) | +| US-REV-05 | drifted | review findings render as a markdown table | [drift-report](../requirements/drift-report.md#US-REV-05) | +| RDM-077 | drifted | scope findings render under a `### Scope findings` heading | [drift-report](../requirements/drift-report.md#RDM-077) | +| RDM-045 | drifted | `projectName` supplies the PR title | [drift-report](../requirements/drift-report.md#RDM-045) | +| INT-016 | drifted | exit 6 means PR creation failed **after retry** | [drift-report](../requirements/drift-report.md#INT-016) | +| DES-114 / DES-131 | drifted | classification and findings reach the checkpoint | [drift-report](../requirements/drift-report.md#DES-116) | +| CLD-084 | drifted | the PR body carries an iteration-log summary | [drift-report](../requirements/drift-report.md#CLD-084) | +| CLD-091 | drifted | `pr-create.sh` consumes the `REQDRIVE_POLICY_*` exports | [drift-report](../requirements/drift-report.md#CLD-091) | +| CLD-098 | drifted | library scripts set `set -e` | [drift-report](../requirements/drift-report.md#CLD-098) | + +**Verified TRUE, and load-bearing for the project's public claims:** the PR body +really does contain a validation checklist rendered from the PRD's +`acceptanceCriteria` (built :112-121, spliced :246, reaching `gh pr create +--body` :267 — rendered end-to-end under a fake `gh`), a Pipeline Verification +table from `verification-summary.json` (:160-178), a scope-findings section +(:183-191), and appended review findings (:69). See +[claims-audit.md](../claims-audit.md) for how far that carries the stated claim, +and drift entry [`PR-OVERSTATES-COMPLETION`](../requirements/drift-report.md#PR-OVERSTATES-COMPLETION) +for the body's one factually false line. + +## Pointers + +Draft decision: [pipeline.md](pipeline.md). Artifacts it reads: +[evidence.md](evidence.md). diff --git a/docs/canon/modules/safety.md b/docs/canon/modules/safety.md new file mode 100644 index 0000000..dd471e9 --- /dev/null +++ b/docs/canon/modules/safety.md @@ -0,0 +1,80 @@ +# safety (canon) + +`lib/sanitize.sh`, `lib/preflight.sh`, `lib/errors.sh` — 113 governing +requirements. **Declared security boundary** (CLAUDE.md:56) — read +[../conventions/sanitization.md](../conventions/sanitization.md) alongside this. + +## Purpose + +Sanitizes user/agent content before it reaches a prompt or `gh`, gates the run +on pre-flight conditions, and owns the exit-code vocabulary. + +## Public interface + +| Symbol | Kind | Anchor | Notes | +|--------|------|--------|-------| +| `DANGEROUS_PATTERNS` | config | lib/sanitize.sh:8 | 15 shell-injection/dangerous-command regexes | +| `sanitize_for_prompt` | function | lib/sanitize.sh:36 | backtick→`'`, `$`→`\$` — **exactly two** substitutions | +| `sanitize_label` | function | lib/sanitize.sh:50 | ≤50 chars, strips `$ \ ; \| & > < "` and backtick | +| `validate_requirement_content` | function | lib/sanitize.sh:79 | ` [strict]`; strict is unreachable in production | +| `validate_file_path` | function | lib/sanitize.sh:112 | **dead code — zero runtime callers** | +| `run_preflight_checks` | function | lib/preflight.sh:165 | 5 blocking + 2 advisory checks | +| `EXIT_*` / `EXIT_MESSAGES` / `get_exit_message` | config/function | lib/errors.sh:7 / :21 / :38 | codes 0–10, incl. 9 `VERIFICATION_FAILED`, 10 `CONCURRENT_RUN` | +| `die` / `die_on_error` | function | lib/errors.sh:44 / :58 | **dead code — zero callers in lib/ or bin/** | + +## Invariants + +- `sanitize_for_prompt` transforms exactly two character classes and leaves everything else untouched. (lib/sanitize.sh:40,43 — verified by execution) +- `sanitize_label` output is ≤50 chars and contains no shell metacharacters. (lib/sanitize.sh:58-69) +- `run_preflight_checks` short-circuits: once a blocking check fails no later check runs, so only the first failure is reported. (lib/preflight.sh:176-192) +- `--force` skips preflight **wholesale** rather than relaxing individual checks. (lib/run.sh:694-700) +- Every exit code 0–10 has both a constant and an `EXIT_MESSAGES` entry, with `"Unknown error"` as the fallback. (lib/errors.sh:7-33,40) +- No sanitizer output is re-parsed by a shell: the prompt heredocs are quoted and the prompt file reaches `claude` via stdin redirect. (lib/run.sh:324, :465) +- Blocking preflight checks: git repo (:176), clean tree (:179), base branch exists (:183), requirements dir (:187), requirement file (:191). Advisory (cannot fail): branch conflicts (:196), testCommand configured (:200). + +## Data flows + +- **requirement content**: lib/run.sh:719 `cat` → :721 `validate_requirement_content` (warn-only) → :738 `sanitize_for_prompt` → prompt.md +- **PRD fields**: lib/run.sh:299-302 `sanitize_for_prompt` → :306-310 strip `@@` → :317-321 reverse `\$` → :399-403 token substitution +- **labels**: lib/config.sh `REQDRIVE_PR_LABELS` → lib/pr-create.sh:200,210 `sanitize_label` → :265 quoted array → `gh` + +## Gotchas + +- `validate_file_path` is defined and tested but **never called** from `lib/` or `bin/` — the documented path-traversal protection does not exist at runtime. (lib/sanitize.sh:112) → [drift CLD-006](../requirements/drift-report.md#CLD-006) +- `die`/`die_on_error` are likewise dead code. (lib/errors.sh:44,58) +- `validate_requirement_content`'s strict mode is unreachable: the only caller omits the `strict` argument, so the failure branch and the `--force` bypass guarding it are both dead. (lib/sanitize.sh:100-105 ← lib/run.sh:721) → [drift CLD-110](../requirements/drift-report.md#CLD-110) +- `sanitize_for_prompt`'s `$`→`\$` escaping is now **output corruption** everywhere it is not manually reversed — the implementation prompt reverses it, the planning prompt and the PR review body do not. (lib/run.sh:276, lib/pr-create.sh:52) → [drift](../requirements/drift-report.md#SANITIZE-PLANNING) +- The commit-verification signal is agent-forgeable: `lib/run.sh:1077` interpolates the PRD-authored story id unquoted into a `[[ == ]]` glob, and `schema.sh:159` puts no charset constraint on `.id`. (lib/run.sh:1077) → [drift COMMIT-CHECK-FORGEABLE](../requirements/drift-report.md#COMMIT-CHECK-FORGEABLE) +- `--force` also suppresses the "all PRs will be drafts" warning, because it skips `run_preflight_checks` entirely. (lib/run.sh:694-700) +- `realpath` is called twice with inconsistent guards — :124 fails closed with a misleading message on non-GNU hosts, :130 is unguarded and could fail *open*. Unreachable today only because the function has no callers. (lib/sanitize.sh:124,130) +- None of the three files sets a shell mode; inert because they are only sourced into `bin/reqdrive`, which sets `set -euo pipefail`. (lib/sanitize.sh:1) + +## Requirement coverage + +113 governing reqs: **98 satisfied · 14 drifted · 1 intent-met**. The drift +concentrates in two clusters — the security-boundary claims and the exit-code +contract. Canon (intended) behavior: + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| CLD-006 / CLD-108 / RDM-070 | drifted | content is scanned for shell injection **and path traversal** | [drift-report](../requirements/drift-report.md#CLD-006) | +| CLD-110 | drifted | content validation is warn-only, with `--force` as the sole full bypass | [drift-report](../requirements/drift-report.md#CLD-110) | +| INT-013 | drifted | exit 3 = bad config / missing requirement file | [drift-report](../requirements/drift-report.md#INT-013) | +| INT-014 | drifted | exit 4 = git operation failed | [drift-report](../requirements/drift-report.md#INT-014) | +| INT-015 | drifted | exit 5 = Claude invocation failed | [drift-report](../requirements/drift-report.md#INT-015) | +| INT-016 | drifted | exit 6 = PR creation failed after retry | [drift-report](../requirements/drift-report.md#INT-016) | +| INT-017 | drifted | exit 7 = user sent SIGINT | [drift-report](../requirements/drift-report.md#INT-017) | +| INT-005 | drifted | clean working tree **on baseBranch** | [drift-report](../requirements/drift-report.md#INT-005) | +| DES-116 | drifted | block-mode findings reach the checkpoint | [drift-report](../requirements/drift-report.md#DES-116) | +| CLD-106 / RDM-009 / RDM-010 | drifted | the documented prerequisite set is enforced | [drift-report](../requirements/drift-report.md#RDM-008) | +| CLD-098 | drifted | library scripts set `set -e` | [drift-report](../requirements/drift-report.md#CLD-098) | + +Verified good: `sanitize_label` is applied to every label reaching `gh`; +`sanitize_for_prompt` is applied to all four PRD fields and to requirement +content; the prompt heredocs are quoted; codes 9 and 10 exist with messages. + +## Pointers + +Where the sanitizers are called from: [pipeline.md](pipeline.md), +[pr.md](pr.md). Boundary analysis: +[../conventions/sanitization.md](../conventions/sanitization.md). diff --git a/docs/canon/modules/skills-and-archive.md b/docs/canon/modules/skills-and-archive.md new file mode 100644 index 0000000..68a15e3 --- /dev/null +++ b/docs/canon/modules/skills-and-archive.md @@ -0,0 +1,71 @@ +# skills-and-archive (canon) + +`skills/**`, `archive/**` — 4 governing requirements. **Index module:** many +small independent peers, one row each. + +## Purpose + +Two quarantined bodies of content: four Claude Code skills a human invokes +interactively, and the archived v0.1.x implementation retained as design +reference. Neither is reachable from the running pipeline. + +## Peer inventory + +| Peer | Path | Kind | One line | +|------|------|------|----------| +| design-to-prd | skills/design-to-prd/SKILL.md | skill | design docs → requirements; writes markdown `tasks/prd-.md`, **not** the pipeline's `prd.json` | +| prd | skills/prd/SKILL.md | skill | feature idea → PRD; same markdown output shape | +| project-journal | skills/project-journal/SKILL.md | skill | maintains `FOR[username].md`; competes with the `docs/STATUS.md` convention | +| verification-workflow | skills/verification-workflow/SKILL.md | skill | overlaps Phase 3 / `reqdrive verify`, but is human-invoked | +| errors.sh | archive/v1-complex/lib/errors.sh | archived-lib | 385 lines; the only archived file that still resolves its own symbols (`ERR_*`, `log_*`) | +| worktree.sh | archive/v1-complex/lib/worktree.sh | archived-lib | 176 lines; the stated reason archive/ is kept — see gotchas | +| orchestrate.sh | archive/v1-complex/lib/orchestrate.sh | archived-lib | 181 lines; parallel run driver | +| run-single-req.sh | archive/v1-complex/lib/run-single-req.sh | archived-lib | 279 lines; sources a `pr-create.sh` that is not in the archive (:252) | +| verify.sh | archive/v1-complex/lib/verify.sh | archived-lib | 168 lines; the name `lib/verification.sh` exists to avoid colliding with it | +| prd-gen · agent-run · agent-tasks · deps · check-deps · find-next-reqs · status · clean | archive/v1-complex/lib/ | archived-lib | 123/122/79/79/63/56/37/22 lines | +| prompt.md.tpl · prompt-tasks.md.tpl · reqdrive.json.tpl | archive/v1-complex/templates/ | template | v0.1.x prompt and 30-field manifest templates | +| PHASE1-VALIDATION-GUIDE · VALIDATION-PLAN · SESSION-STATE-2025-01-30 | archive/docs/ | archived-doc | frozen history | + +## Invariants + +- No file under `archive/` is sourced, executed, or otherwise reachable from `bin/` or `lib/` at runtime — the only `archive` mention in live code is a naming comment. (lib/verification.sh:4-5; repo-wide grep: 2 hits, both comments) +- Git worktree support is absent from the active pipeline — zero occurrences of `worktree` in `bin/`, `lib/`, `tests/`. (grep) +- `skills/` is never loaded by the pipeline; zero occurrences of `skills` in `bin/`, `lib/`, `tests/`. They load only through interactive Claude Code invocation. (skills/README.md:3) +- The v0.1.x command surface has not returned: dispatch exposes exactly `init|run|validate|status|migrate|plan|verify|orchestrate|launch|logs`. (bin/reqdrive:680-722) +- `reqdrive orchestrate` remains a pure stub touching no worktree code. (bin/reqdrive:579-586) + +## Dependencies + +Internal: none at runtime — that is the point. `bin/reqdrive:579-586` is the +placeholder for what `orchestrate.sh` + `worktree.sh` would become. External: +Claude Code `/skill` invocation (skills only); `git worktree` (archived only). + +## Gotchas + +- `archive/v1-complex/` is **not a faithful v0.1.x snapshot**: the v0.1.x `config.sh` was deleted rather than archived, and seven archived scripts still `source "${REQDRIVE_ROOT}/lib/config.sh"` — which now silently resolves to the *live* 0.3.0 loader. (archive/v1-complex/lib/check-deps.sh:8 and six siblings) +- Consequently `worktree.sh` is not drop-in revivable: `reqdrive_resolve_path` (:81), `reqdrive_timestamp` (:97), `$REQDRIVE_PATHS_AGENT_DIR` and `$REQDRIVE_AGENT_WORKTREE_PREFIX` are defined **nowhere** in the repo. Its other dependencies (`ERR_WORKTREE`, `ERR_GIT`, `log_info`, `log_warn`) *do* resolve, via the co-archived `errors.sh:32-50`. Reviving it means porting a path resolver and two env vars — not a rewrite. → [drift ARCHIVE-ROT](../requirements/drift-report.md#ARCHIVE-ROT) +- `skills/README.md:29-30` is stale: it lists `reqdrive plan` as "(coming soon)" and `reqdrive verify` as "(future)". Both shipped. (bin/reqdrive:394, :440) +- The `cmd_orchestrate` stub promises **sequential** processing while CLAUDE.md:222 defines `orchestrate` as **parallelism via git worktrees** — the two descriptions of the same unbuilt command disagree. (bin/reqdrive:580-585) +- The `prd` and `design-to-prd` skills emit markdown PRDs, not the `prd.json` the pipeline consumes — they are adjacent tools, not pipeline components. (skills/prd/SKILL.md:134-138 vs lib/run.sh:224-240) +- Nothing enforces the quarantine: no test asserts that `archive/` and `skills/` stay unreachable, so a future `source .../archive/...` would pass CI. + +## Requirement coverage + +4 governing reqs: **2 satisfied · 1 drifted · 1 unimplemented (roadmap)**. + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| CLD-061 | satisfied | worktree support stays archived, not reintroduced | — | +| CLD-070 | satisfied | multi-requirement parallelism stays deferred | — | +| CLD-059 | drifted | keep the library at the simplified ~5-script shape | [drift-report](../requirements/drift-report.md#CLD-059) | +| CLD-093 | unimplemented | `orchestrate` builds worktree parallelism | roadmap — index.json#unbuilt_features | + +Scale, measured: `archive/` is 3,446 lines across 19 files and `skills/` 1,639 +lines — 5,085 inert lines against 3,410 live (`bin/` + `lib/`). Live `lib/` +(2,683 lines / 11 files) is now **larger** than the 1,770-line / 13-file v0.1.x +`lib/` the simplification replaced. + +## Pointers + +The stub that would consume this: [cli.md](cli.md). The simplification claim: +[../requirements/drift-report.md#CLD-059](../requirements/drift-report.md#CLD-059). diff --git a/docs/canon/modules/test-harness.md b/docs/canon/modules/test-harness.md new file mode 100644 index 0000000..19b0f66 --- /dev/null +++ b/docs/canon/modules/test-harness.md @@ -0,0 +1,86 @@ +# test-harness (canon) + +`tests/**` — 92 governing requirements. + +## Purpose + +The project's evidence about itself: a 202-assertion dependency-free suite, a +tamper-evidence freeze gate over it, a pipeline harness that fakes only `claude` +and `gh`, and optional bats suites. + +## Public interface + +| Symbol | Kind | Anchor | Notes | +|--------|------|--------|-------| +| `bash tests/simple-test.sh` | cli | tests/simple-test.sh:1 | the primary suite; exits `[ "$FAIL" -eq 0 ]` (:3242) | +| `test_result` / `test_skip` | function | tests/simple-test.sh:34 / :47 | 202 / 2 call sites | +| `bash tests/oracle-gate.sh [--accept]` | cli | tests/oracle-gate.sh:1 | enforces rules R0–R3, R6, R7 | +| `bash tests/spec-map.sh [--list]` | cli | tests/spec-map.sh:1 | test-name ↔ story totality | +| `bash tests/gate-selftest.sh` | cli | tests/gate-selftest.sh:1 | proves the gate fires | +| `bash tests/mutate.sh ` | cli | tests/mutate.sh:1 | **mutates the tree** — never run casually | +| `ph_setup` / `ph_fake_claude` / `ph_fake_gh` / `ph_run` / `ph_gh_args` | function | tests/lib/pipeline-harness.sh:8 | fakes only `claude` (:54-106) and `gh` (:108-119) | +| `bash tests/launch-lifecycle.sh` | cli | tests/launch-lifecycle.sh:1 | Linux only; prints `SKIP:` elsewhere (:9-12) | +| `tests/oracle.lock.json` | config | — | 202 `{name, story}` entries + 4 whole-file sha256 | + +## Invariants + +- **The suite can report a failure.** `set +e` at file scope (:11) with `set -e` as the first statement inside every assertion subshell, each invoked as a simple command whose status feeds `test_result`. Measured: 202/202 correct form, 0 incorrect (no `if ( … )`, no `( … ) && ||`, no subshell missing `set -e`). (tests/simple-test.sh:11) +- Failure propagates to the exit code. (tests/simple-test.sh:3242) +- `TEST_TEMP` is guarded twice before any assertion runs, so the historical `rm -rf .git`-in-the-repo hazard cannot recur. (tests/simple-test.sh:56-57) +- Suite names == lock names == BEHAVIOR-SPEC stories == **202**, enforced by spec-map totality. (measured on all four sources) +- The four frozen files hash identically in worktree, in git, and in the lock. (tests/oracle.lock.json) +- `conditional` is a closed enum whose only member is `claude`; any other value is a hard R3 failure. (tests/oracle-gate.sh:152-164) +- A developer **cannot** silently delete or rename one of the 202 — R7 plus R1/R6 catch it, and `--accept` additionally requires spec-map totality. + +## Data flows + +- **gate**: tests/oracle-gate.sh:46 run suite → :36 `parse_results` (strip ANSI, split on `": "`) → R7 file hashes :118-137 → R2 locked-test-failed :158 → R3 conditional :152 → R6 unlocked-name :181 → R1/R0 truncation :176,:182 +- **pipeline test**: tests/lib/pipeline-harness.sh:125 `set -euo pipefail` → :136 source lib/run.sh → fake claude/gh on PATH → real git repo, real bare origin, real config load + +## Dependencies + +Internal: every `lib/*.sh`, `bin/reqdrive`, `README.md` (read by the three +doc-coverage rules), `tests/BEHAVIOR-SPEC.md` (read by spec-map). External: +bash 4+, `jq`, `git`, **`sha256sum`** (hard requirement of the gate), +`mktemp`/`comm`/`sort`/`awk`, `tar` and GNU `sed -i` (selftest/mutate), +bats-core (optional), `claude` (optional — gates 2 assertions). + +## Gotchas + +- The freeze surface is exactly four files: `simple-test.sh`, `oracle-gate.sh`, `pipeline-harness.sh`, `spec-map.sh`. All 52 bats tests, `launch-lifecycle.sh`, `gate-selftest.sh` and `mutate.sh` are **unfrozen**. (tests/oracle-gate.sh:118-137) +- `--accept` locks whatever ran, including a red suite — it stores `{name, story}` with no verdict and never checks `SUITE_RC`. The gate itself still catches a red suite via R2 on the next enforce run, so the exposure is narrow. (tests/oracle-gate.sh:51-88) +- `gate-selftest.sh` demonstrates R7, R2, R6 and R0 — never R1 or R3. (tests/gate-selftest.sh:71-75) +- `tests/mutate.sh` is linted by CI but never executed by it, so the mutation evidence behind the freeze is a one-off human result. (.github/workflows/ci.yml) +- `tests/spec-map.sh` is likewise never a CI step — only invoked transitively under a human `--accept`. (tests/oracle-gate.sh:52-55) +- `bash -n` in CI covers `bin/`, `lib/` and `install.sh` but **not** `tests/*.sh`. +- The suite is very slow under MSYS2/Git-Bash (≈10+ min here vs seconds in CI). +- Many assertions use `source … 2>/dev/null || true`, which swallows sourcing failures. (tests/simple-test.sh:1881, 3094) +- `tests/run-tests.sh` counts failing **suites**, not failing tests. (tests/run-tests.sh:118-123) +- 24 assertions end in a terminal negative and so cannot detect a broken setup — `tests/FINDINGS.md` F4 and `docs/STATUS.md:43` both still say 18. (measured twice, independently) → [drift DES-059](../requirements/drift-report.md#DES-059) + +## Requirement coverage + +92 governing reqs: **60 satisfied · 8 drifted · 17 intent-met · +7 not-code-verifiable**. Canon (intended) behavior for the drifted ones: + +| REQ | Status | Canon (intended) | Link | +|-----|--------|------------------|------| +| DES-059 / DES-121 | drifted | the pure-negative count is accurate and triaged | [drift-report](../requirements/drift-report.md#DES-059) | +| DES-008 / DES-060 | drifted | the lock is generated in the CI environment configuration | [drift-report](../requirements/drift-report.md#DES-008) | +| DES-101 | drifted | the characterization test pins the summary's identity | [drift-report](../requirements/drift-report.md#DES-101) | +| DES-072 | drifted | the `prd_present == 0` draft branch has a dedicated test | [drift-report](../requirements/drift-report.md#DES-072) | +| DES-081 / US-DOC-02 | drifted | `DOC_EXEMPT` holds the specified 2–3 names, not 5 | [drift-report](../requirements/drift-report.md#DES-081) | + +The 17 `intent-met` reqs are mostly P0/P2 mechanism specs the implementation met +by a better route; they are listed in the register's spec-stale table, not here. +The 7 `not-code-verifiable` reqs are process obligations (e.g. "run the suite +before committing") that source cannot confirm. + +**Refuted during verification** (do not re-raise): "the oracle gate passes +against a red suite" — R2/R6 catch every failing locked name, and +`gate-selftest.sh:73` demonstrates it. + +## Pointers + +CI wiring: [ci-and-install.md](ci-and-install.md). Freeze rules: +[../conventions/testing-and-freeze.md](../conventions/testing-and-freeze.md). diff --git a/docs/canon/requirements/drift-report.md b/docs/canon/requirements/drift-report.md new file mode 100644 index 0000000..53b4b64 --- /dev/null +++ b/docs/canon/requirements/drift-report.md @@ -0,0 +1,197 @@ +# Drift Report (canon vs. as-built) + +Requirements are canon; each entry is a place the code contradicts them — a +suspected bug or debt, **not** intended behavior. Canon documented these; it +changed no code. + +**48 confirmed drifted requirements + 8 confirmed defects that no requirement +covers.** 2 candidate findings were refuted in verification and 1 was materially +corrected; all three are recorded at the bottom so they are not re-raised. Every +entry below carries a Phase-3 adversarial verifier verdict. + +> Only `drifted` reqs appear here. `intent-met` (21) and `not-code-verifiable` +> (10) live in [register.md](register.md). All 6 unimplemented reqs are Tier-3 +> roadmap and are **not** gaps. + +## Summary — fix list, most severe first + +| # | REQ / Finding | Sev | Module | One-line | +|---|---------------|-----|--------|----------| +| 1 | **DES-002** | **critical** | pipeline | the fail-closed draft gate is fail-**open** four ways; non-draft PRs reproduced with real incomplete work | +| 2 | INT-030 · INT-048 · US-PR-01 | high | pr | `run.json.pr_url` and the hook's `PR_URL` hold a two-line string, not a URL | +| 3 | PR-OVERSTATES-COMPLETION | high | pr | the PR body states "Stories completed: " with no `.passes` filter, contradicting its own table | +| 4 | GLOB-FOOTGUN | high | evidence | a glob in `riskTiers` silently matches nothing — the protection is inert, and `validate` says PASSED | +| 5 | RDM-008 · INT-006 · INT-007 | high | ci-and-install | `gh`/`claude` auth is never checked; failure surfaces after a full agent cycle | +| 6 | INT-013 | high | safety | a bad `reqdrive.json` exits 1 (or 5), never the documented 3 | +| 7 | INT-014 | high | safety | exit 4 is unreachable from `run`; a failed checkout leaks raw 128 | +| 8 | INT-015 | high | safety | a Claude timeout or crash exits **0** | +| 9 | CLD-006 · CLD-108 · RDM-070 | medium | safety | three docs promise path-traversal scanning that does not exist | +| 10 | SANITIZE-PLANNING · SANITIZE-PR-BODY | medium | safety | `$`→`\$` escaping corrupts the planning prompt and the published PR body | +| 11 | RDM-012 · CLD-030 | medium | config | `reqdrive init` writes unescaped JSON; a quote breaks the file and init reports success | +| 12 | VALIDATE-FALSE-PASS | medium | config | `validate` passes a config `run` refuses | +| 13 | ATTEMPTS-UNVALIDATED | medium | config | a non-numeric `attempts` makes a story permanently unselectable | +| 14 | CLD-086 | medium | evidence | CLAUDE.md and STATUS.md call the scope check a hard gate; it is warn-by-default | +| 15 | CLD-110 | medium | safety | content-scan strict mode and its `--force` bypass are both dead code | +| 16 | US-REV-05 · RDM-077 | medium | pr | two PR sections lose their newlines; the review table never renders as a table | +| 17 | DES-116 · DES-131 · DES-114 | medium | evidence | scope findings never reach `checkpoint.json`; `policy_classify_paths` is dead | +| 18 | TIER-NAME-FOOTGUN | medium | evidence | an unrecognised tier name validates clean and is inert | +| 19 | INT-005 | medium | safety | preflight never checks HEAD is `baseBranch`; a mis-based existing branch is adopted silently | +| 20 | RDM-005 · RDM-009 · RDM-010 · CLD-058 · CLD-106 | medium | ci-and-install | the documented prerequisite set is neither accurate nor enforced | +| 21 | CLD-059 | medium | pipeline | "17 scripts to 5" — `lib/` is 11 files / 2,683 lines, larger than the 1,770 it replaced | +| 22 | DES-008 · DES-060 | medium | test-harness | the oracle lock was generated where it is not enforced | +| 23 | DES-059 · DES-121 | medium | test-harness | the pure-negative count is 24, not the 18 in FINDINGS.md and STATUS.md | +| 24 | DES-101 · DES-072 | medium | test-harness | the characterization test pins presence, not values; the `prd_present==0` branch is untested | +| 25 | CLD-001 | medium | pipeline | "the agent never decides what to work on next" overstates a real but narrower guarantee | +| 26 | CLD-099 · ARCHIVE-ROT | medium | ci-and-install | `scripts/**` is unchecked and has rotted; `worktree.sh` is not revivable as-is | +| 27 | INT-031 · INT-056 · RDM-045 · CLD-084 · CLD-091 · INT-016 · DES-081 · US-DOC-02 | low | various | artifact/field mismatches — see compact entries | +| 28 | COMMIT-CHECK-FORGEABLE | low | safety | agent-authored story id reaches a glob unquoted; reporting-only blast radius | +| 29 | CLD-098 · DES-120 · INT-017 | low | various | convention and documentation conformance | + +--- + +## DES-002 — the draft-PR gate is fail-open {#DES-002} + +- **Canon (intended):** *"Make the draft-PR gate fail-closed: create a draft PR by default and clear the draft flag only on positive evidence, rather than enumerating individual fail-open conditions."* (design doc, decision D2, :89). Restated as the project's readiness claim in `docs/STATUS.md:16-19`. +- **As-built (drift):** the gate's *shape* is right — `local draft_flag="--draft"` cleared only when `prd_present==1 && final_remaining==0 && verification_passed=="true"` (lib/run.sh:1177-1178) — but three of its inputs can report success without success occurring: + - **(A) An unreadable PRD scores as complete.** Every jq in `verify_collect` ends `|| echo "0"` (lib/verification.sh:68-75), so invalid, truncated, markdown-fenced, or `.userStories`-less JSON yields `remaining=0`, while `prd_present=1` is a bare `[ -f ]`. "I cannot read the plan" is scored identically to "the plan is complete." + - **(B) An empty story list scores as complete.** `{"userStories":[]}` passes `validate_prd_schema` (lib/schema.sh:125-183, no minimum-count check) → total=0, remaining=0. + - **(C) Agent story-deletion scores as complete.** Nothing binds `prd.json`'s story set to the plan — no freeze, hash, or count invariant exists — so an agent that deletes a story it cannot implement yields remaining=0. The attempts-increment jq at lib/run.sh:1096 is a silent no-op on a vanished id. + - **(D) A blank or commented-out `testCommand` reads as a pass.** `[ -n "${REQDRIVE_TEST_COMMAND:-}" ]` (lib/verification.sh:83) treats `" "` or `"# npm test (disabled)"` as configured, and `eval` of either exits 0 → `verification_passed: true` rather than `null`. +- **Severity:** critical — this is the guarantee the project's L2/L3 readiness rests on. +- **Verifier verdict:** confirmed by **two independent skeptics**, both reproducing it end-to-end with the repo's own `tests/lib/pipeline-harness.sh` in scratch directories. Observed `gh pr create` argv with **no `--draft`** for: a story-shrinking agent (`"Stories 1 / 1 completed"` while US-002 was never implemented), a resumed run with a fenced `prd.json` (`"stories": {"total": 0 … }`, `verification_passed: true`), an empty story list, and both disabled-`testCommand` forms. The second skeptic classified all four **real-run reachable**, not unit-only. +- **What is genuinely sound and should not be "fixed":** the three states the gate was written for — missing PRD, correctly-counted incomplete stories, and `verification_passed` null/false — all correctly force a draft; `${draft_flag:+"$draft_flag"}` survives `set -u` and the label-dropping retry; and no other code path creates a PR. +- **Suggested reconciliation:** distinguish "counted zero" from "could not count" — drop the `|| echo "0"` fallbacks and fail closed on a jq error; require `total > 0`; record the planned story-id set in `checkpoint.json` at first write and require the final set to be a superset; trim `testCommand` and reject a whitespace/comment-only value. +- **Note on the suite:** the shipped positive control (`tests/simple-test.sh`, *"draft gate: full evidence produces non-draft PR"*) uses `"testCommand": "true"` — a vacuous-evidence case — so the suite would not notice the gate degrading to trusting agent self-report. + +## INT-030 · INT-048 · US-PR-01 — `pr_url` is not a URL {#INT-030} + +- **Canon (intended):** `run.json.pr_url` is the GitHub PR URL, `null` when no PR was created (docs/INTEGRATION.md:128); the completion hook receives `PR_URL` (:183); `create_pr` emits the URL on stdout (`tests/BEHAVIOR-SPEC.md`, US-PR-01). +- **As-built (drift):** `lib/pr-create.sh:95` echoes `" Pushing branch ..."` to **stdout** without `>&2` — the only diagnostic in the file that does not redirect. `lib/run.sh:1193` captures that stdout wholesale with no extraction, so the value is two lines. +- **Severity:** high — an integrating pipeline reading `.pr_url` gets an unusable value; `bin/reqdrive:293` prints it raw in `status`; the same string is exported to `completionHook` and to any external `reviewCommand` (lib/run.sh:499, :641). +- **Verifier verdict:** confirmed, measured under fake `git`/`gh`: `" Pushing branch reqdrive/req-01...\nhttps://github.com/test/repo/pull/42"` (72 bytes, literal `\n` at offset 0o43). `run.json` stays *valid* JSON — `jq -Rn` escapes it faithfully — so this is a wrong-value defect, not a parse failure. The review phase survives by accident: its `grep -o '[0-9]*$'` is line-oriented and the push line ends in `...`. +- **Suggested reconciliation:** add `>&2` to lib/pr-create.sh:95 (and :97). One line. Note that `tests/BEHAVIOR-SPEC.md:953-958` currently *freezes* the polluted value as expected, so the spec story and the oracle lock must be updated with the fix. + +## PR-OVERSTATES-COMPLETION — the PR body contradicts itself {#PR-OVERSTATES-COMPLETION} + +- **Canon (intended):** no requirement governs this line — which is itself the finding. The PR body is the product's only user-visible output. +- **As-built (drift):** `lib/pr-create.sh:109-110` builds `story_count` and `story_ids` from `.userStories` with **no `.passes` filter**, rendered at :227 under the literal label `**Stories completed:**`. The Pipeline Verification table in the *same body* (:168) uses the `.passes`-filtered count from `lib/verification.sh:69`. +- **Severity:** high — a draft PR for a run where 1 of 3 stories passed states `**Stories completed:** US-01, US-02, US-03 (3 stories)` directly above `| Stories | 1 / 3 completed |`. +- **Verifier verdict:** confirmed; reproduced by running the real `create_pr` with stubbed `git`/`gh` and capturing `--body`. `grep passes lib/pr-create.sh` returns zero hits. +- **Assessed and cleared in the same pass:** the validation checklist rendering every criterion as an unchecked `- [ ]` regardless of pass state is **correct** — unchecked is the conservative direction and the surrounding prose establishes the human as the ticker. Its weakness is omission, not overstatement: nothing marks which stories were never attempted. +- **Suggested reconciliation:** filter by `.passes == true`, or relabel to `**Stories in scope:**` and add a completed count. + +## GLOB-FOOTGUN — a glob silently disables a risk tier {#GLOB-FOOTGUN} + +- **Canon (intended):** `policy.riskTiers` values are path **prefixes**, not globs (README.md:167-172), and `reqdrive validate` validates the policy block (lib/schema.sh:105-113). +- **As-built (drift):** patterns are matched literally (lib/policy.sh:24), so the natural-looking `"high": ["src/auth/**"]` matches only a file *named* `src/auth/**`. Nothing warns. Measured: `src/auth`, `src/auth/login.ts` and `src/auth/sub/deep.ts` all classify `none`, while `reqdrive validate` reports `✓ Schema valid` / `Validation PASSED` (exit 0). +- **Severity:** high — a security-adjacent control that silently does nothing, with a config that the project's own validator blesses. The code comments (lib/policy.sh:4-9) and README explain the semantics, so the gap is validation, not intent. +- **Verifier verdict:** confirmed, measured. +- **Suggested reconciliation:** reject `*`, `?` and `[` in tier patterns in `validate_config_schema`, or warn at load time. + +## RDM-008 · INT-006 · INT-007 — authentication is never checked {#RDM-008} + +- **Canon (intended):** `gh` must be authenticated with push and PR-create permission (README.md:21, docs/INTEGRATION.md:33); `claude` must be installed **and authenticated** (docs/INTEGRATION.md:34). +- **As-built (drift):** only presence is checked (`command -v`, install.sh:13, bin/reqdrive:19,26). `gh auth` appears nowhere in executable code — only in human checklists in `docs/VERIFICATION-PLAN.md`. The first credential-requiring call is `git push` at lib/pr-create.sh:96, after branch creation, planning, every implementation iteration and verification. +- **Severity:** high — a read-only token or expired SSO burns a full unattended agent cycle before failing. +- **Verifier verdict:** confirmed. +- **Suggested reconciliation:** add `gh auth status` to preflight; it is one command and fails in under a second. + +## INT-013 · INT-014 · INT-015 · INT-016 · INT-017 — the exit-code contract {#INT-013} + +`docs/INTEGRATION.md:440-460` explicitly instructs integrators to branch on +these codes, so a code that never fires as documented is a contract violation. +All five confirmed, all **measured**. + +| REQ | Canon (intended) | As-built (drift) | Sev | +|-----|------------------|------------------|-----| +| **INT-013** {#INT-013b} | 3 = bad `reqdrive.json` **or** missing requirement file | a missing/incompatible manifest exits **1** (lib/config.sh:37-38,47-48); a *malformed-JSON* manifest exits **5**, colliding with `EXIT_AGENT_ERROR` — which INTEGRATION.md:445 marks retry-worthy, so a broken config yields an infinite retry loop. Only the missing-req-file half emits 3 | high | +| **INT-014** {#INT-014} | 4 = a git operation (checkout, commit, push) failed | 4 is emitted only by `cmd_verify`; a failed checkout aborts under `set -e` with git's raw **128** — a value absent from the contract entirely (lib/run.sh:906-912); push failure becomes 6 | high | +| **INT-015** {#INT-015} | 5 = the Claude invocation failed (timeout, crash, no PRD) | only the no-PRD case exits 5. Measured: an agent exiting 42 on every turn → pipeline **exit 0**; an agent timing out on every implementation turn → **exit 0** with a PR created. Partial mitigation: incomplete stories force `--draft`, so the PR is flagged even though the exit code says success | high | +| **INT-016** {#INT-016} | 6 = PR creation failed **after retry** | accurate for the `gh` path (one label-dropping retry, lib/pr-create.sh:273-284), but a `git push` failure also returns 6 with **no** retry — and INTEGRATION.md:54 assigns push failures to code 4 | med | +| **INT-017** {#INT-017} | 7 = the user sent SIGINT | SIGINT exits **130** (lib/run.sh:877). 7 fires only when a user declines the `--unsafe` prompt, which is TTY-gated and therefore unreachable in the unattended mode the contract targets | low | + +- **Suggested reconciliation:** make `reqdrive_load_config` exit `EXIT_CONFIG_ERROR`; wrap the checkout/push sites to exit `EXIT_GIT_ERROR`; propagate `run_claude_iteration`'s failure to `EXIT_AGENT_ERROR`; document 128/130 or trap them. + +## CLD-006 · CLD-108 · RDM-070 — promised path-traversal scanning does not exist {#CLD-006} + +- **Canon (intended):** "requirement content is scanned for dangerous patterns (shell injection, **path traversal**)" — README.md:159, CLAUDE.md:56 (declared a *security boundary*), CLAUDE.md:262. `docs/PIPELINE-ANALYSIS.md:187` goes further, tabulating `validate_file_path()` as a "Hard error / unbypassable" check. +- **As-built (drift):** `validate_file_path` (lib/sanitize.sh:112) has **zero callers** in `lib/` or `bin/` — its only references are tests, the spec, the lock and that analysis doc. None of the 15 `DANGEROUS_PATTERNS` matches `../`, `..\`, a leading `/`, `~/`, or `ln -s`. Measured: traversal probes pass the scan clean with no warning. +- **Severity:** medium — **downgraded from high on verification.** The verifier established that requirement *content* is never used as a filesystem path (it is only `cat`'d into the prompt), so scanning content for traversal would protect nothing; and PRD-derived fields reach only jq selectors and template tokens. **Not attacker-reachable.** +- **The real, separate gap the docs do not mention:** the **REQ-ID CLI argument** is unvalidated (`req_id="$1"`, bin/reqdrive:121) and reaches both the requirement-file glob (lib/run.sh:704) and `mkdir -p` (lib/run.sh:691,729). Measured: a REQ-ID of `../../../../pwndir` created a directory outside the project root, and `../../../secrets/pwned` passed preflight and read an out-of-tree `.md`. Operator-triggered today; genuinely exploitable if a future `reqdrive orchestrate` ever sources REQ-IDs from a file or remote input. +- **Verifier verdict:** confirmed with that reframing. +- **Suggested reconciliation:** either wire `validate_file_path` into the REQ-ID path and add a traversal pattern, or amend all four docs to say "shell injection and dangerous commands" and state the REQ-ID trust assumption explicitly. + +## SANITIZE-PLANNING · SANITIZE-PR-BODY — escaping that now corrupts {#SANITIZE-PLANNING} + +- **Canon (intended):** design decision D9 — the `\$` corruption is fixed in its own enumerated step (design doc:96). `sanitize_for_prompt` exists to make content safe for a prompt, not to alter it. +- **As-built (drift):** the fix landed for the implementation prompt only. `build_planning_prompt` (lib/run.sh:276) appends content sanitized at :738 into a **quoted** heredoc (:213), so the escaping buys nothing and the backslashes survive; `lib/pr-create.sh:52` sanitizes review findings immediately before `gh pr edit --body`, an execve argv that is never shell-re-evaluated. +- **Severity:** medium — it degrades the text the planning agent reads and the text published to GitHub. +- **Verifier verdict:** both confirmed, measured end-to-end. `Budget is $500 for ${TEAM} and run \`make test\`` reaches `prompt.md` as `Budget is \$500 for \${TEAM} and run 'make test'` (`od -c` shows literal `0x5C` before each `0x24`). For the PR body, GFM renders `\$` as a plain `$` so the *rendered* impact is small — but the stored/API body carries the backslashes, and the backtick→`'` rule destroys code spans unconditionally. +- **Why it went unnoticed:** the guarding test (tests/simple-test.sh:1831-1845) feeds **raw** content directly to `build_planning_prompt`, bypassing `sanitize_for_prompt` entirely, and asserts with a substring match that passes on both the correct and corrupted forms. +- **Suggested reconciliation:** drop the `$`→`\$` substitution from `sanitize_for_prompt` now that no consumer is a shell context (and delete the reversal at lib/run.sh:317-321), or move the escaping to the one call site that needs it. + +## CLD-086 — the scope check is warn-by-default, not a hard gate {#CLD-086} + +- **Canon (intended):** `policy.scopeCheck` defaults to `"warn"`; `"block"` is opt-in (README.md:181-197, design decision D6, `templates/reqdrive.json.example:18`). +- **As-built (drift):** the **code is correct** (`local mode="${REQDRIVE_POLICY_SCOPE_CHECK:-warn}"`, lib/policy.sh:56). The drift is documentary: `CLAUDE.md:212` calls `policy_scope_check` a "hard gate, not advisory", and `docs/STATUS.md:21-22` and `:94` repeat it. +- **Severity:** medium — it misrepresents the project's actual enforcement posture in the two docs a maintainer reads first, and contradicts the "warn before enforce" principle at CLAUDE.md:54. +- **Verifier verdict:** confirmed; the drift is wider than first reported (STATUS.md too). No doc-coverage gate checks CLAUDE.md, only README. +- **Suggested reconciliation:** replace CLAUDE.md:212 with "warn-only by default (`policy.scopeCheck: "warn"`), promotable to a hard gate with `"block"` (aborts with `EXIT_PREFLIGHT_FAILED`/8)", and correct STATUS.md. + +## Compact entries + +Each confirmed, each with a verifier verdict; grouped to keep the fix list usable. + +**config / init** +- **RDM-012 · CLD-030** {#RDM-012} *(medium)* — canon: `init` creates a valid `reqdrive.json`. Drift: `lib/init.sh:57-68` interpolates raw answers with no JSON escaping. Measured: `My "Quoted" Project` → `jq empty` parse error while init prints "Done!" and exits 0; `C:\reqs` parses but silently becomes `C:eqs`; a crafted answer can inject sibling keys. Fix: `jq -n --arg`. +- **VALIDATE-FALSE-PASS** {#VALIDATE-FALSE-PASS} *(medium, no governing req)* — `lib/validate.sh` never calls `check_schema_version`, so a `version: "1.0.0"` config reports "Validation PASSED" (exit 0) while `reqdrive run` refuses it (exit 1). Measured both. +- **ATTEMPTS-UNVALIDATED** {#ATTEMPTS-UNVALIDATED} *(medium, no governing req)* — `attempts` is absent from `lib/schema.sh` entirely. Measured: `"attempts": "lots"` passes validation, and jq's cross-type ordering makes `(("lots" // 0) < 3)` false, so the story is permanently skipped and the loop logs "All stories complete!". Contained: Phase 3 still counts it remaining, so the PR drafts. It also flips `verification.sh:74`'s `>= max` true, mislabelling it retry-exhausted. +- `reqdrive validate` always prints "(1 errors)" — `ERRORS` increments once per failed *check group* and only one group exists (lib/validate.sh:27-36). Cosmetic; each violation is still printed and exit 3 is right. + +**evidence / policy** +- **DES-116 · DES-131** {#DES-116} *(medium)* — canon: a scope finding is recorded in `checkpoint.json`. Drift: `save_checkpoint` (lib/run.sh:88-121) has no scope field in any mode, and block mode exits at :1092 *before* the save at :1101. Findings survive only in `scope-findings.txt` (both modes) and the PR body (warn only). +- **DES-114** {#DES-114} *(low)* — canon: post-iteration path classification is recorded. Drift: `policy_classify_paths` (lib/policy.sh:35) has zero callers anywhere, including tests — a knowing bypass (see the comment at :48-51), but `tests/BEHAVIOR-SPEC.md:1454` still describes it as the matcher the scope check consumes. Only the high-tier violating subset is persisted. +- **TIER-NAME-FOOTGUN** {#TIER-NAME-FOOTGUN} *(medium, no governing req)* — only `high`/`medium`/`low` are probed (lib/policy.sh:19); a `critical` tier validates clean and is inert. Measured. +- **RDM-077** {#RDM-077} *(medium)* — canon: warn findings render under a `### Scope findings` heading. Drift: `$(printf '\n### Scope findings\n\n')` at lib/pr-create.sh:186 loses its trailing newlines, fusing the first finding onto the heading. Measured with `cat -A`. Findings 2..n render fine. +- **US-REV-05** {#US-REV-05} *(medium)* — same command-substitution bug at lib/pr-create.sh:40-42 collapses the review findings heading, header row, delimiter and first finding onto one line, so GFM renders no table at all. + +**pipeline / cli artifacts** +- **CLD-001** {#CLD-001} *(medium)* — canon: "the shell controls what to work on, when to stop… the agent never decides what to work on next." Verified true in the narrow sense: given a fixed `prd.json`, `select_next_story` is a pure deterministic function and the agent cannot influence its own assignment within an iteration (selection at lib/run.sh:1028 precedes prompt-building at :1043). Verified **false** in the causal sense: the agent authors `prd.json`, the prompt at lib/run.sh:366 explicitly invites it to reorder priorities, and `COMPLETE` (:1112) ends Phase 2 with no re-check. A missing `priority` sorts first (jq null ordering, measured). Contained by the fail-closed intent of Phase 3 — see DES-002 for how far that containment actually holds. Fix: amend the principle to "reproducible given a fixed PRD, plus a fail-closed outcome gate", and correct the stale `lib/run.sh:347` citation (it is :413). +- **INT-031** *(low)* — `RUN_SUMMARY_*` initialise to 0 before Phase 1 (lib/run.sh:935), so a planning failure writes a zeroed `.summary` where `null` is specified. +- **INT-056** *(low)* — `verification.test.log` and `iteration-N.test.log` are absent, or stale from a previous run, when no `testCommand` is configured (lib/verification.sh:92-95). +- **RDM-045** {#RDM-045} *(medium)* — canon: `projectName` supplies the PR title. Drift: `lib/pr-create.sh` never reads it; the title comes from the PRD's `.project` or `" Implementation"`. +- **CLD-084** {#CLD-084} *(low)* — canon: the PR body carries an iteration-log summary. Drift: only an `iterations run/max` table row exists. +- **CLD-091** {#CLD-091} *(low)* — canon: `pr-create.sh` consumes `REQDRIVE_POLICY_*`. Drift: it reads `scope-findings.txt` directly; zero references to either variable. Fix the doc, not the code. +- **INT-005** {#INT-005} *(medium)* — canon: a clean tree **on baseBranch**. Drift: nothing compares HEAD to baseBranch (lib/preflight.sh:165-201). Harmless for a fresh run (the branch is cut from the configured base at lib/run.sh:911), but on the existing-branch path lib/preflight.sh:40-45 downgrades a pre-existing `reqdrive/` to a warning — measured: an unrelated commit from a mis-based branch landed in the PR. + +**prerequisites** +- **RDM-005** {#RDM-005} *(medium)* — bash 4.0+ is documented and never checked; `declare -A` at lib/errors.sh:21 is sourced unconditionally, so stock macOS bash 3.2 fails at source time for *every* command, including `--help`. +- **RDM-009** {#RDM-009} *(medium)* — canon: `claude` only for `run`/`launch`. Drift in both directions: `cmd_plan` requires it (bin/reqdrive:395); `install.sh:12` makes it an install-time hard requirement, blocking `validate`/`init`/`status`/`logs`/`migrate`; `cmd_launch` never checks it, deferring the error into the detached child's `output.log` while `launch` exits 0. +- **RDM-010** {#RDM-010} *(medium)* — `timeout`/`sha256sum` documented, checked by neither installer nor CLI. Worse than reported: `lib/run.sh:6` sets `set -e` without `pipefail`, so a missing `timeout` in `timeout … | tee` yields `tee`'s status 0 — the failure is **silent** and misreported as "Agent failed to create PRD after 2 attempts". (`sha256sum` *is* self-checked at tests/oracle-gate.sh:23.) +- **CLD-058** {#CLD-058} *(low)* — CLAUDE.md:128 and :250-256 claim "bash/jq/git/gh"; the real runtime set adds `claude` and `timeout`. README.md:23 already concedes this; CLAUDE.md was never updated. +- **CLD-099** {#CLD-099} *(medium)* — canon: every modified `.sh` passes `bash -n`. Drift: CI covers `bin/`, `lib/`, `install.sh` only. Note the two halves are causally independent — `scripts/setup-validation-env.sh` **passes** `bash -n`; its rot is semantic (missing `templates/prompt.md.tpl` :212; sources `lib/prd-gen.sh`/`agent-run.sh`/`verify.sh` :262,270,275, all archived; calls undefined `reqdrive_load_config_path` :299; emits a v0.1.x manifest), so closing the CI gap would not have caught it. + +**self-description** +- **CLD-059** {#CLD-059} *(medium)* — canon: keep the library at the simplified ~5-script shape ("17 scripts to 5", CLAUDE.md:131; `docs/SIMPLIFICATION-SUMMARY.md:231`). Measured: `lib/` is **11 files / 2,683 lines** vs the archived v0.1.x `lib/`'s **13 files / 1,770 lines** — 913 lines (52%) *larger* than what it replaced. The archived figure quoted in both docs (1,679) is itself 91 lines under the on-disk total. The anti-goal holds (no v0.1.x module returned); the numeric claim does not. +- **ARCHIVE-ROT** {#ARCHIVE-ROT} *(low, corrected on verification)* — CLAUDE.md:139 keeps `archive/` so `worktree.sh` can be revived for `orchestrate`. It is not drop-in revivable, but **less broken than first reported**: `ERR_WORKTREE`, `ERR_GIT`, `log_info` and `log_warn` *do* resolve via the co-archived `errors.sh:32-50`. Genuinely unresolvable: `reqdrive_resolve_path`, `reqdrive_timestamp`, `$REQDRIVE_PATHS_AGENT_DIR`, `$REQDRIVE_AGENT_WORKTREE_PREFIX` — all from the v0.1.x config layer, which was **deleted rather than archived**. Reviving means porting a path resolver and two env vars. Related: seven archived scripts still `source "${REQDRIVE_ROOT}/lib/config.sh"`, now silently resolving to the live 0.3.0 loader. Also `skills/README.md:29-30` still calls `reqdrive plan` "coming soon" and `reqdrive verify` "(future)"; both shipped. +- **DES-120** {#DES-120} *(low)* — the WORKFLOW.md L2→L0 correction was never applied; it lives parked at `docs/STATUS.md:59-83`. That file is outside this checkout, so this is not code-fixable here. + +**test harness** +- **DES-008 · DES-060** {#DES-008} *(medium)* — canon: decision D8, "lock generated in the CI environment configuration". Drift: `tests/oracle.lock.json` records `environment.claude: true` and `generated: 2026-07-24` — it was generated on a workstation with `claude` on PATH. CI installs no `claude`, and `tests/oracle-gate.sh` reads `.environment` **only** in the `--accept` writer (:69), never at enforce time. The field is decorative; the two `conditional: "claude"` entries are what actually keep CI green. +- **DES-059 · DES-121** {#DES-059} *(medium)* — canon: the pure-negative count is measured and triaged. Drift: **24** today (13 terminal `!` + 11 `[ -z `), derived twice by independent methods; `tests/FINDINGS.md:18` and `docs/STATUS.md:43` both say 18, and the design doc says ~21. The suite grew 157→202 and the figure was never remeasured. (A stricter reading that excludes 3 guard-form hits gives 21 — still not 18.) +- **DES-101** {#DES-101} *(medium)* — canon: the characterization test proves the extraction did not change the summary. Drift: of its 12 `jq -e` assertions (tests/simple-test.sh:2669-2685), 10 are bare `has()` presence checks; the other two pin a constant string and a non-null. No computed value is pinned — flipping `.stories.completed` from 3 to 0, or `prd_present` from true to false, leaves it green. `prd_present` is exactly the field the draft gate keys on. +- **DES-072** {#DES-072} *(low)* — the "missing prd.json forces draft" test was retargeted (commit `551d454`) to assert a planning-abort (rc==5, no `pr create`), so the `prd_present != 1` branch at lib/run.sh:1178/:1181 has no exercising test. `tests/FINDINGS.md:19` (F6) records this honestly. +- **DES-081 · US-DOC-02** {#DES-081} *(low)* — `DOC_EXEMPT` has **5** members (tests/simple-test.sh:2987); the design spec pins 2 (:264) and `tests/BEHAVIOR-SPEC.md:1362` pins 3. The two extra names are `REQDRIVE_POLICY_JSON`/`REQDRIVE_POLICY_SCOPE_CHECK`; `policy` *is* documented at README.md:101, so nothing undocumented is being concealed — but neither spec was updated to authorise them. + +**conventions** +- **COMMIT-CHECK-FORGEABLE** {#COMMIT-CHECK-FORGEABLE} *(low — downgraded from high)* — `lib/run.sh:1077` interpolates the agent-authored story id unquoted into a `[[ == ]]` RHS pattern, and `lib/schema.sh:159` constrains `.id` to existence only. Measured: id `US-*` credits a stale prior-iteration commit; id `US-[0-9]` likewise. **Bounded**: the `feat:\ \[` prefix is backslash-escaped and literal-anchored, so no id matches a non-conforming message. **Downgraded** because nothing gates on the counter (the draft gate at :1178 ignores it), and because the signal was already trivially forgeable — the agent writes the commit messages, so `git commit --allow-empty -m 'feat: [US-2] done'` passes with a perfectly literal id. Worth a one-line quote as hygiene. +- **CLD-098** {#CLD-098} *(low)* — canon: library scripts set `set -e`. Drift: 6 of 11 do not (`config`, `errors`, `pr-create`, `preflight`, `sanitize`, `schema`). **No concrete bug follows** — every lib is sourced into `bin/reqdrive`, which sets `set -euo pipefail`. Documentation-conformance only. The inverse is the live hazard: the 5 that *do* set it at file top level mutate their sourcer's options, which `tests/simple-test.sh:374` already works around. + +--- + +## Refuted or corrected in verification — do not re-raise + +- **"The oracle gate passes against a red suite."** *Refuted.* `oracle-gate.sh` does capture `SUITE_RC` (:47) and never gates on it, and `--accept` will lock a red suite — but R2 (:158-164) fails the gate for any locked test that FAILs, R6 for any unlocked name that ran, and `tests/gate-selftest.sh:73` demonstrates R2 firing. A lock taken while red reddens on the very next enforce run. Residual hole is narrow: a suite emitting all 202 expected PASS lines yet exiting non-zero for an unrelated reason. +- **"`reqdrive_load_config` not calling `validate_config_schema` is drift."** *Refuted* — it is a recorded decision (CLAUDE.md:187) and the code matches it exactly. The *consequence* is real and documented in [modules/config.md](../modules/config.md): jq's `//` fires only on null/false, so wrong-typed truthy values pass through — measured, `policy.scopeCheck: "blocking"` loads and silently degrades the gate to warn; `maxIterations: "ten"` breaks `seq` so the loop body never runs. +- **"`archive/v1-complex/lib/worktree.sh` calls 8 symbols that exist nowhere."** *Corrected to 4 of 8* — see [ARCHIVE-ROT](#ARCHIVE-ROT). +- **"The validation checklist does not exist / is fake."** *Refuted* — it is real and reaches `gh pr create --body`; see [claims-audit.md](../claims-audit.md) for its verified caveats. diff --git a/docs/canon/requirements/register.md b/docs/canon/requirements/register.md new file mode 100644 index 0000000..b5e2530 --- /dev/null +++ b/docs/canon/requirements/register.md @@ -0,0 +1,101 @@ +# Requirements Register (canon) + +Source of truth for intended behavior. **602 requirements from 5 documents**, +extracted 2026-07-24 at commit `7420d69`. Per-REQ anchors live in each module +doc's *Requirement coverage* table — this file is the index, the conflict +record, and the spec-stale record. + +## Canonical sources and precedence + +| Doc | Prefix | Count | Governs | Horizon | Precedence note | +|-----|--------|-------|---------|---------|-----------------| +| `tests/BEHAVIOR-SPEC.md` | `US-*` | 202 | lib/**, bin/** | current | behavioral contract; 1:1 with test names, frozen in `tests/oracle.lock.json` | +| `docs/superpowers/specs/2026-07-23-…-design.md` | `DES-` | 140 | whole pipeline | current (3 roadmap) | states intent + decisions D1–D12; most specific on the P0–P7 work | +| `README.md` | `RDM-` | 81 | whole repo | current | user-facing contract; normative sections only | +| `CLAUDE.md` | `CLD-` | 110 | whole repo | current (12 roadmap) | architecture + roadmap + decision log | +| `docs/INTEGRATION.md` | `INT-` | 69 | CLI contract, run artifacts | current | integrator contract: exit codes, artifacts, hook env | + +Descriptive, **not** canon (context only): `ROADMAP.md` (explicitly superseded), +`docs/superpowers/plans/…` (sequencing, not obligations), `docs/PIPELINE-ANALYSIS.md`, +`docs/SIMPLIFICATION-SUMMARY.md`, `docs/audits/…`, `docs/STATUS.md`, +`docs/QUICKSTART.md`, `docs/VERIFICATION-PLAN.md`, `docs/LAUNCH-TEST-PLAN.md`, +`tests/README.md`, `tests/FINDINGS.md`, `skills/README.md`. + +## Coverage by module + +Attribution is the requirement's **first governing module**; the offending code +may live elsewhere (the drift report records the real anchor). + +| Module | n | satisfied | drifted | intent-met | unimplemented | not-code-verifiable | +|--------|---|-----------|---------|------------|---------------|---------------------| +| pipeline | 149 | 136 | DES-002, CLD-001, CLD-059, INT-030, INT-031, INT-048, INT-056 | INT-068, CLD-063 | CLD-092, CLD-096, CLD-097 *(roadmap)* | DES-133 | +| config | 108 | 105 | RDM-045, CLD-091 | RDM-038 | — | — | +| test-harness | 88 | 56 | US-DOC-02, DES-008, DES-059, DES-060, DES-072, DES-081, DES-101, DES-121 | DES-011, -019, -029, -049, -050, -051, -052, -061, -082, -087, -099, -102, -124, -125, -126, -127 | — | DES-001, -033, -055, -058, -096, -122, -123, -128 | +| safety | 85 | 75 | CLD-006, CLD-108, CLD-110, RDM-070, INT-005, INT-013, INT-014, INT-015, INT-016, INT-017 | — | — | — | +| cli | 77 | 70 | RDM-012, CLD-030, CLD-058, CLD-098, CLD-106 | — | CLD-093 *(roadmap)* | INT-069 | +| evidence | 55 | 50 | DES-114, DES-116, DES-131, RDM-077, CLD-086 | — | — | — | +| ci-and-install | 20 | 11 | DES-120, RDM-005, RDM-008, RDM-009, RDM-010, INT-007, CLD-099 | RDM-011, CLD-101 | — | — | +| pr | 19 | 13 | US-PR-01, US-REV-05, INT-006, CLD-084 | — | CLD-094, CLD-095 *(roadmap)* | — | +| skills-and-archive | 1 | 1 | — | — | — | — | + +**Totals:** 517 satisfied · **48 drifted** (→ [drift-report.md](drift-report.md)) +· 21 intent-met · 6 unimplemented (**all 6 are `horizon: roadmap`** → see +`index.json#unbuilt_features`; the real gap count is **0**) · 10 +not-code-verifiable. + +> **DES-002 override.** The module explorer scored DES-002 (*"make the draft-PR +> gate fail-closed"*) `satisfied` from reading the gate condition. Two +> independent Phase-3 verifiers refuted that reading with end-to-end +> reproductions, so it is recorded here as **drifted**. This is the single most +> consequential entry in the canon — see +> [drift-report.md#DES-002](drift-report.md#DES-002). + +## Mechanism drift / spec is stale (`intent-met`) + +Code achieves the requirement's *goal* by a different — usually superior — +mechanism than the spec names. **Not bugs; not in the drift report.** The fix is +to amend the requirement. + +| REQ | Canon (spec mechanism) | As-built (better mechanism) | Suggested amendment | +|-----|------------------------|-----------------------------|---------------------| +| RDM-038 | `version` defaults to `"0.3.0"` | no loader default; absence *warns* rather than silently assuming a version (lib/config.sh:46) | amend README's default column to "(none — absence warns)" | +| CLD-063 | rely on `sanitize_for_prompt` to mitigate expansion risk in a variable-bearing prompt | the prompt is a **quoted** heredoc with `@@TOKEN@@` substitution (lib/run.sh:324), a strictly stronger guarantee | delete CLD-063; it is superseded by CLD-088 | +| INT-068 | per-iteration `testCommand` failures never influence any gate | they feed the scope check, which under `"block"` *can* abort (lib/run.sh:1089) — a stronger control the doc predates | amend INT-068 to name the scope-check exception | +| RDM-011, CLD-101 | (process/build obligations) | met by CI configuration rather than the named mechanism | no change needed | +| DES-011 … DES-127 (16 reqs, test-harness) | specific P0/P2 harness mechanisms | implementation chose equivalent-or-stronger forms (e.g. whole-file hashing subsuming per-body hashing) | fold into a single "freeze mechanism" requirement | + +## Not code-verifiable (infra / process) + +Excluded from both drift and gap counts. + +- `INT-069` — PID liveness may be unreliable on MSYS2 (runtime platform property). +- `DES-133` — a deferral decision about future work, not present behavior. +- `DES-001, -033, -055, -058, -096, -122, -123, -128` — process obligations + (sequencing of the P0–P7 effort, "run the suite before committing", "record the + reason for each deferral"): source code cannot confirm they were honored. + +## Requirement conflicts + +**Nine canon-vs-canon contradictions.** An unresolved spec conflict is itself a +finding; none of these is a code defect. + +| # | Contested point | Doc A | Doc B | Which is right | +|---|-----------------|-------|-------|----------------| +| 1 | Is there a Phase 3 verification stage before PR creation? | README.md:140-148 lists five stages, no verification | docs/INTEGRATION.md:145 places `verification-summary.json` at "the end of Phase 3, before PR creation" | **INTEGRATION** — Phase 3 exists (lib/run.sh:1131-1164). README's stage list is incomplete | +| 2 | The full exit-code set | README.md:56 documents 9 and 10 for `verify` | docs/INTEGRATION.md:46-58 stops at 8 | **README** — `lib/errors.sh:7-17` defines 0–10 | +| 3 | Meaning of exit 8 | README.md:187-190 reuses 8 for a `scopeCheck: "block"` abort | docs/INTEGRATION.md:58 defines 8 as git state only, and :451 advises "do not retry on 8 — git state issue" | **README** — the reuse is real (lib/run.sh:1092); INTEGRATION's retry advice misclassifies it | +| 4 | Can a per-iteration `testCommand` failure abort the run? | README.md:174-190 — yes, under `block` | docs/INTEGRATION.md:459 — "logged but don't abort or force retries" flatly | **README** — see conflict 3. Tracked as `intent-met` on INT-068 | +| 5 | Is the scope check a hard gate? | CLAUDE.md:212 "hard gate, not advisory"; docs/STATUS.md:21-22, :94 repeat it | README.md:181-197, design D6, and `lib/policy.sh:56` all say warn-by-default | **README/D6/code** — CLAUDE.md and STATUS.md are wrong. → [CLD-086](drift-report.md#CLD-086) | +| 6 | Required dependencies | README.md:23 adds `timeout` and `sha256sum` | docs/INTEGRATION.md:34,52 and CLAUDE.md:128 omit them | **README** — `timeout` is invoked at lib/run.sh:465, `sha256sum` at tests/oracle-gate.sh:23 | +| 7 | Run-directory contents | README.md:113-121 lists 6 artifacts | docs/INTEGRATION.md:397-414 lists 11 | **INTEGRATION**, and even it is incomplete — neither lists `scope-findings.txt` | +| 8 | Test count | CLAUDE.md:26 "157 tests"; CLAUDE.md:232 "all 152 tests must pass"; design doc:109 "157 test names" | docs/STATUS.md:97 "202/202" | **STATUS** — measured 202 in the suite, the lock and BEHAVIOR-SPEC, and a local run reports 202/0 | +| 9 | What `orchestrate` will do | `bin/reqdrive:580-585` promises **sequential** multi-requirement processing | CLAUDE.md:222 defines it as **parallelism via git worktrees** | unresolved — both describe an unbuilt command; pick one before building it | + +Two further internal inconsistencies in `CLAUDE.md` are recorded as drift rather +than conflicts because code settles them: the heredoc entry (Decision Log :147 +and Known Pitfalls :268 describe an *unquoted* heredoc at `lib/run.sh:274`, +while Tier 2 :214 correctly reports the quoted form at :324 — the code is quoted, +so :147/:268 are stale), and `select_next_story` cited at `lib/run.sh:347` +(:46, :50) when it is at **:413**. `reqdrive verify` is missing from the CLAUDE.md +Commands table, and `maxStoryRetries` and `policy` are missing from its +Configuration table.