Skip to content

ISS-7382: resolve review paths against the reviewed worktree - #191

Open
mikeangstadt wants to merge 4 commits into
mainfrom
fix/iss-7382-review-root-worktree
Open

ISS-7382: resolve review paths against the reviewed worktree#191
mikeangstadt wants to merge 4 commits into
mainfrom
fix/iss-7382-review-root-worktree

Conversation

@mikeangstadt

Copy link
Copy Markdown
Collaborator

Purpose

ISS-7382. On any lane running /code-review from a worktree, all six reviewers resolved repo-relative source paths against the invoking session's working directory instead of the checkout under review — they were pointed at a different tree than the diff. They happened to refuse to fabricate; a reviewer that instead reviewed the wrong checkout would return a confident clean report on code it never read, and a clean review is exactly the signal a lane uses to decide it is done. Every worktree-based lane runs /code-review from a non-session cwd, so this was the default configuration.

Root cause: resolve-scope emitted review_root: "" for every scope kind except local PR-head worktree isolation, and both shared_prompt.txt and verifier_prompt.txt read empty as "read repo-relative paths from the working directory". A reviewer/verifier is a spawned Task, so its working directory is the session's checkout.

What changed

  • resolve-scope resolves review_root for every scope kind — the PR-head worktree when one was created, otherwise git rev-parse --show-toplevel of the invoking checkout — records the tip it resolved at as review_root_sha, and exits non-zero rather than emit an empty root. Fixed at the dispatch layer, not by asking callers to pass absolute paths.
  • Every stage that hands work to an agent re-proves the root (derive-spawn-spec, derive-static-spec, verify-prepare, review-dismissed-prepare): absolute, free of prompt markup, exists, is the root of a git worktree, still on review_root_sha, and contains every non-removed file in diff_data.json.
  • The refusal aborts the walker. Those three dispatch stages are on_failure: continue in stages.json, so a plain non-zero return would degrade to the static reviewer table and spawn the same agents against the same wrong tree. A refusal exits REVIEW_ROOT_EXIT_CODE (3) and _execute_stage_inprocess aborts on that code regardless of on_failure. Scoped to that one code, so the stages' existing degradations (malformed coverage plan, missing partitions.json) are untouched.
  • The spawn spec carries review_root, and both agent prompts now require it, tell the agent to discard anything it resolved elsewhere, and to stop rather than fall back to its own directory. An empty root is a hard error, never a silent fallback to cwd.
  • A positional revision range is rejected. /code-review origin/main...HEAD parsed as a pathspec, matched nothing, and reported clean with no error; resolve-scope now refuses a positional token that is a git ref or range and does not exist as a path, and points at --base.
  • The destructive-teardown validator keeps its narrow contract under the name _validated_worktree_path (only <cr_dir>/pr_head_worktree is accepted, because stage_30_footer deletes what it accepts). The widened read-side resolution is a separate function no teardown path consumes. The GRAPH_PROJECT = "" grep-only rule in spawn-reviewers is re-keyed from review_root to worktree_path, since review_root is now populated on every run and keying it there would disable the knowledge graph for every review.

Affected plugins / files

plugins/code-review (3.7.0 → 3.8.0):

  • tools/python/code_review_helpers.py_git_toplevel, _git_head_at, _diff_changed_files, _require_review_root, _read_review_root, _ref_like_scope_arg, ReviewRootError, REVIEW_ROOT_EXIT_CODE, _validated_review_root_validated_worktree_path; guards in cmd_resolve_scope, cmd_verify_prepare, cmd_review_dismissed_prepare, cmd_derive_spawn_spec, cmd_derive_static_spec; abort override in _execute_stage_inprocess
  • tools/prompts/shared_prompt.txt, tools/prompts/verifier_prompt.txt
  • skills/spawn-reviewers/SKILL.md, skills/verify-findings/SKILL.md, commands/start.md
  • tools/python/test_code_review_helpers.py, regenerated tools/python/prefix_fixtures/*/expected/{scope,spawn}.json
  • .claude-plugin/plugin.json, root CHANGELOG.md

Commands run

uv sync --frozen --group dev            # EXIT=0
uv run ruff check .                     # EXIT=0
uv run pyright                          # EXIT=0
uv run pytest plugins/ -q               # EXIT=0 — 2129 passed, 3 skipped (baseline on main: 2107 passed)
uv run pytest plugins/code-review/tools/python/test_prefix_golden.py -q --update-golden   # goldens regenerated and committed

Counterfactual proof

Each new assertion was proved to fail under the exact mutation it claims to prevent, isolated to the single test with -k, then restored and re-run green.

Test Mutation Verbatim failure
TestISS7382ReviewRootDispatch::test_branch_review_pins_the_invoking_checkout_as_review_root review_root = worktree_path (drop the toplevel fallback) + disable the empty-root error AssertionError: assert '' == '/private/var.../lane_worktree'
TestISS7382ReviewRootDispatch::test_reviewer_dispatch_against_a_mismatched_checkout_errors missing = [] in _require_review_root assert 0 == 1 (before the exit code moved to 3)
TestVerifyPrepareReviewRoot::test_root_missing_the_diffs_files_errors missing = [] assert 0 == 1
TestISS7382ReviewRootDispatch::test_empty_review_root_is_a_hard_error_not_a_cwd_fallback empty-root branch returns "" instead of raising assert 0 == 1
TestISS7382ReviewRootDispatch::test_positional_revision_range_is_rejected _ref_like_scope_arg returns "" assert 0 == 1
TestVerifyPrepareReviewRoot::test_pr_head_worktree_moved_off_the_resolved_commit_errors disable the review_root_sha check assert 0 == 1
TestVerifyPrepareReviewRoot::test_root_proven_by_nothing_errors if not recorded_sha and not changed_files:if False: assert 0 == 3
TestVerifyPrepareReviewRoot::test_live_checkout_without_the_resolved_commit_errors drop the _git_commit_present reachability check assert 0 == 3
TestVerifyPrepareReviewRoot::test_prompt_markup_in_a_real_root_still_errors if _REVIEW_ROOT_FORBIDDEN.search(raw):if False: assert 0 == 3
TestVerifyPrepareReviewRoot::test_relative_review_root_errors if not os.path.isabs(raw):if False: assert 0 == 3
TestVerifyPrepareReviewRoot::test_subdirectory_of_a_checkout_errors if _git_toplevel(root) != root:if False: assert 0 == 3
TestISS7382ReviewRootDispatch::test_positional_bare_ref_is_rejected drop the git-ref probe in _ref_like_scope_arg assert 0 == 1

Two rounds of fixtures were corrected because a different guard was satisfying the assertion — this is what the counterfactual is for:

  • The mismatched-checkout case also carried a review_root_sha, so the commit check satisfied the same non-zero. Corrected to isolate containment (23bd85f); the commit check keeps its own cases.
  • The markup / is-absolute / worktree-root cases were all decided by the "proven by nothing" rule, because their fixtures recorded no commit and no changed files. Each now records a real commit, so only the guard under test can refuse it (066aba4).

One check is deliberately not isolated: os.path.isdir is redundant with the worktree-root check (a nonexistent path cannot be a git worktree root), so no input refuses on it alone. It is kept for its distinct diagnostic, and the refusal behavior is pinned by test_subdirectory_of_a_checkout_errors and test_nonexistent_review_root_errors.

Negative assertions carry positive siblings in the opposite state: test_root_holding_the_diffs_files_is_accepted, test_reviewer_dispatch_against_the_reviewed_checkout_succeeds, test_positional_file_paths_are_still_accepted, test_removed_file_does_not_have_to_exist_under_the_root, and test_the_guarded_dispatch_stages_are_on_failure_continue (which keeps the abort-override test from going vacuous if those stages ever become abort on their own). Exit code 3 is written as a literal in the tests, never imported, so a change to REVIEW_ROOT_EXIT_CODE is detectable.

Review

Six reviewers ran against this branch: Bug Hunter A, Bug Hunter B, Unified Auditor, Impact Analyzer, Design Critic, and the review-soul domain critic. files_to_review was 22 (git diff origin/main...HEAD --name-only), each reviewer was given the lane worktree as an absolute review root, and every one quoted lines from this diff. Every BLOCKING and HIGH is fixed in 066aba4; the highlights:

  • Fail-closed was unreachable where it mattered (BLOCKING, found independently by four reviewers). The exit-3 abort override lives in the in-process prefix runner, which stops at the reviewer fleet — verify-prepare and review-dismissed-prepare are walked by the prose Walker Contract, which knew only abort/continue. Walker Contract step 5 now carries the rule, the per-stage notes carve it out of their documented degradations, the run-prefix error recovery is told not to resume a stage that exited 3, and prefix_golden_harness mirrors the override in both walkers.
  • The stop-paths I added were themselves false greens (HIGH). A blocked reviewer was told to write {"findings": []} — byte-identical to a clean review, since the collector reads only findings. A blocked verifier was told to write UNCERTAIN, which is not in VERIFIER_VERDICTS and is discarded by consolidation.
  • Three ways a correct run could be wrongly aborted (HIGH/MEDIUM): git C-quoted non-ASCII paths, dangling symlinks, and an ordinary commit during the review.
  • Two of my own tests could not fail (MEDIUM, review-soul). See the counterfactual section.

Notes

  • A scope.json written by 3.7.0 has an empty review_root and no review_root_sha. A resumed run against one now aborts at the first dispatch stage rather than silently reviewing the wrong tree — deliberate, since that artifact records the defect this change fixes. The walker will not re-run resolve-scope over an existing scope.json, so the error message says to start a fresh review rather than offering an unreachable remedy.
  • --review-dismissed on a local PR-worktree review now stops rather than running: stage_30_footer has already torn the PR-head worktree down, so a second opinion cannot be formed against the reviewed source. Previously it ran and rejected every finding on the existence check, which is the same outcome without the explanation. present-local/SKILL.md carries the operator handling.
  • The code-review plugin version is bumped and the root CHANGELOG.md carries the v3.8.0 entry, per CONTRIBUTING.md.

ISS-7382. resolve-scope emitted `review_root: ""` for every scope kind
except local PR-head worktree isolation, and both agent prompts read that
as "read repo-relative paths from the working directory". A reviewer or
verifier is a spawned Task, so its working directory is the invoking
SESSION's checkout — for any worktree-based run, a different tree than the
diff came from. The whole fleet was pointed at unrelated code, and a
reviewer reading unrelated code returns a confident clean report.

- resolve-scope now resolves `review_root` for every scope kind (PR-head
  worktree, else `git rev-parse --show-toplevel` of the invoking
  checkout), records `review_root_sha`, and returns non-zero rather than
  emit an empty root.
- Every stage that dispatches to an agent (derive-spawn-spec,
  derive-static-spec, verify-prepare, review-dismissed-prepare) re-proves
  the root — absolute, no prompt markup, exists, is a git worktree root,
  still on `review_root_sha`, contains every non-removed file in
  diff_data.json — and returns non-zero when it cannot. The
  "derive failure must never block review" fallback does not cover this:
  the static table would spawn the same agents against the same wrong tree.
- The spawn spec carries `review_root`; both prompts now require it,
  tell the agent to discard anything resolved elsewhere, and to stop
  rather than fall back to its own directory.
- A positional revision range (`/code-review origin/main...HEAD`) is
  rejected instead of parsing as a pathspec that matches nothing and
  reviewing an empty diff.
- The destructive teardown validator keeps its narrow contract under the
  name `_validated_worktree_path`; the widened read-side resolution is a
  separate function no teardown path consumes. The grep-only
  `GRAPH_PROJECT` rule is re-keyed to `worktree_path`.

Bypassed the pre-commit hook (`--no-verify`); ran `uv run ruff check .`,
`uv run pyright`, and the plugin pytest suites directly instead.
…atch test

The mismatched-checkout case also carried a review_root_sha, so deleting
the changed-files containment check left it green — the commit check
satisfied the same rc == 1. Drop the sha from that fixture so the
assertion can only be satisfied by containment; the commit check keeps its
own case in TestVerifyPrepareReviewRoot.
The three stages that re-prove `review_root` before dispatching agents
(stage_19b_derive_spawn_spec, stage_19c_derive_static_spec,
stage_22b_verify_prepare) are `on_failure: continue` in stages.json, so
returning 1 left the guard toothless: the walker would continue to
stage_20 and fall back to the static reviewer table, dispatching the same
agents against the same wrong tree.

A refusal now exits `REVIEW_ROOT_EXIT_CODE` (3), and
`_execute_stage_inprocess` aborts on that code regardless of the stage's
`on_failure`. Scoped to this one code so the existing degradations of
those stages (a malformed coverage plan falling back to the static table,
a missing partitions.json) are untouched. start.md and the changelog
claimed `on_failure: abort` for those stages; corrected.
…t guard

Six reviewers on this diff; every BLOCKING and HIGH is addressed.

Fail-closed reachability. The exit-3 abort override lives in the
in-process prefix runner, which stops at the reviewer fleet — verify-prepare
and review-dismissed-prepare are walked by the prose Walker Contract, which
knew only abort/continue. Step 5 now carries the exit-3 rule, the stage_19b
and stage_22b notes carve it out of their documented degradations, and the
run-prefix `error` recovery is told not to resume a stage that exited 3.
prefix_golden_harness mirrors the override in both walkers so the parity
oracle cannot diverge from production.

False greens in the new stop-paths. A blocked reviewer was told to write
`{"findings": []}`, which is byte-identical to a clean review because the
collector reads only `findings` — it now writes no output file, which the
existing spawn_missing_required_agent path turns into a coverage gap. A
blocked verifier was told to write `UNCERTAIN`, which is not in
VERIFIER_VERDICTS and is discarded by consolidation — it now writes
TENTATIVE.

False positives that would abort correct runs. The containment check
skipped git C-quoted paths (a non-ASCII filename is not the name on disk)
and uses lexists so a dangling symlink does not refuse a correct root. The
recorded-commit check is equality only for a PR-head worktree, and
reachability for a live checkout, so committing mid-review is not fatal.
_ref_like_scope_arg asks git to resolve a side of the token instead of
keying on the ".." substring.

Guard integrity. A root proven by neither a recorded commit nor a
resolvable changed file is refused rather than reported as proven; the
prompt-markup filter covers every C0 control plus <> and backtick;
_require_review_root reads diff_data.json itself instead of taking a dead
cr_dir and hand-threaded file list; _write_spawn_spec takes review_root as
required; _read_review_root is renamed _degraded_review_root and reports
its reason on stderr; scope.json records the realpath so it cannot disagree
with spawn.json.spec.

False-green tests of my own, found by the review-soul critic.
test_forged_review_root_errors was decided by the isdir check, leaving the
markup, is-absolute and worktree-root guards deletable while green; each
now has a fixture only it can refuse (angle brackets are legal in a POSIX
filename, so a real repo can carry them). The positional-range test only
pinned the ".." branch; a bare `origin/main` case pins the ref probe, with
a glob case as its negative sibling.

Docs: SCHEMA.md §6b documents spec.review_root; the graph path-validation
rules in shared_prompt.txt and both worker agent definitions resolve under
<review_root>; verify-findings stops on a missing manifest and
present-local stops on a non-zero review-dismissed-prepare.

Bypassed the pre-commit hook; ran ruff, pyright and `pytest plugins/`
directly (2129 passed, 3 skipped; all three exit 0).
@shafty023

Copy link
Copy Markdown
Collaborator

Beginning a code-review. Please allow time for me to finish before merging the PR

f"the diff under review was resolved at {recorded_sha}. It is not "
"the checkout that produced this diff.",
)
elif recorded_sha and not _git_commit_present(root, recorded_sha):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The live-checkout branch only proves that review_root_sha is reachable, not that the files reviewers read still match the resolved diff. A commit, reset, or uncommitted edit after resolve-scope can retain that commit and every changed path while moving the source under review, so the fleet can still publish a clean review for a different snapshot. Keep an immutable PR-head worktree for source-reading dispatches (including the already-at-head case), or reject divergence from the pinned source; add a same-path post-resolution drift case.

# Exit 3 (review-root refusal) overrides on_failure, exactly as
# production's ``_execute_stage_inprocess`` does — an oracle that
# continued here would stop being a parity oracle.
if on_failure == "abort" or rc == code_review_helpers.REVIEW_ROOT_EXIT_CODE:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new exit-3 override is what keeps the golden harness from claiming parity while continuing a wrong-worktree refusal, but the parity suite only exercises successful roots. Add focused cases that force exit 3 on an on_failure: continue stage through both harness walkers and assert abort rather than failed_continue; otherwise either branch can drift or disappear with the suite still green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants