feat(hooks): prompt gate — let the model decide when recall is loud#535
Conversation
the per-prompt hook injected a block on every prompt, so a coding
command like "fix the failing test" spent its reply opener announcing a
memory search nobody asked for. three cases now, decided by a
deterministic llm-free classifier (no added latency):
chatter with no informative tokens ("ok thanks", "which one is better?")
injects nothing. retrieval ORs every query token, so those used to match
claims on "one" and inject noise on turns that wanted none — this half
is ported from the unmerged #518, rebased onto phase 3 so the two stop
fighting over hooks.py.
"do work" imperatives get a smaller background pack (3 items / 700
chars) and NO reply contract: no "From vouch memory:" opener, no
blockquote ritual, cite an id inline only where it was actually relied
on — and nothing at all when the kb has no match. recall can still
inform the work (conventions, architecture) without taking the turn
over.
lookups — questions, what/why/how, anything not imperative — keep the
full visible-recall behaviour including the honest "Nothing in vouch on
this." on a miss. silence there would be indistinguishable from vouch
being broken, which is the thing 1.5.0 set out to fix.
the imperative test reads past politeness lead-ins ("please fix", "ok
now refactor", "can you run"), and a prompt made only of discourse
tokens ("continue", "go ahead") counts as work, not a question. new kbs
get the gate on; existing kbs behave exactly as before until they add
retrieval.prompt_gate.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughPrompt gating filters prompt tokens, classifies task and question phrasing, and injects conditional retrieval instructions. Starter configuration, legacy fallback behavior, tests, live evaluation, and changelog documentation were updated. ChangesPrompt-gated retrieval
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ClaudeCode
participant PromptHook
participant ContextPack
participant MemoryInstruction
ClaudeCode->>PromptHook: submit prompt
PromptHook->>ContextPack: filter tokens and retrieve context
ContextPack-->>PromptHook: context body or miss
PromptHook->>MemoryInstruction: render conditional guidance
MemoryInstruction-->>ClaudeCode: inject instruction or empty string
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
docs
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/vouch/hooks.py`:
- Around line 88-91: Update the token-filtering condition in the prompt
tokenization loop to stop excluding all-digit tokens via tok.isdigit(). Keep the
existing len(tok) < 2 check so single-digit numeric noise remains filtered,
while retaining multi-digit IDs, error codes, and port numbers for retrieval.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1e16c6bf-b5be-4a1a-abe5-a1f327ef5edb
📒 Files selected for processing (4)
CHANGELOG.mdsrc/vouch/hooks.pysrc/vouch/storage.pytests/test_hooks.py
the first cut of the prompt gate enumerated "action verbs" and defaulted everything else to a loud "From vouch memory:" block. that fails open: action verbs are an unbounded class, so any verb not on the list (vendorize, dockerize, spin-up, knock-out, yarn) fell through to loud recall — the exact noise the gate exists to stop. so stop making the hook guess. when the gate is on it now hands the host model ONE conditional instruction and lets the model — which already reads the prompt — choose per turn: a QUESTION the items answer opens with "From vouch memory:" and quotes them; a TASK (fix/build/change/run anything, known verb or not) uses them silently as background with no banner, citing an id inline only where relied on; irrelevant items are ignored. on a miss the same judgment applies — "Nothing in vouch on this." for a question, silence for a task. chatter with no informative tokens still injects nothing. no per-turn model call, no latency — the decision rides in instruction text the model already processes, and it generalizes to any phrasing because it no longer depends on recognizing the verb. the legacy unconditional block is kept intact behind the gate flag, so existing KBs are byte-for-byte unchanged until they add retrieval.prompt_gate; new KBs get delegation via the starter config. compliance was MEASURED before shipping, not assumed — real claude -p across haiku / sonnet / opus, offline and against the real context-hook pipeline. coding tasks were never wrongly announced on any tier (the goal). questions are announced by every tier in the real pipeline; the only misses were sonnet under-announcing when a test harness fed a KB-claiming block into an empty dir with hooks off, which it correctly distrusted — resolved in-environment. tests/test_hooks.py pins the injected block; tests/test_prompt_gate_live.py is an opt-in live probe (skips without an authenticated claude cli).
github-actions
Bot
added
size: L
plind-junior
changed the title
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/vouch/hooks.py`:
- Around line 300-309: Update _retrieve_body to propagate build_context_pack
exceptions instead of returning the same empty-body result as a genuine miss,
and correct its docstring to describe the actual successful return contract. In
_gated_injection and _legacy_injection, catch retrieval exceptions and return no
injection, while preserving the existing honest “nothing relevant” messaging
only when retrieval succeeds with an empty result.
In `@tests/test_prompt_gate_live.py`:
- Around line 68-76: Update _ask_model to detect a nonzero subprocess return
code and raise an error that includes stderr, while preserving the existing
stripped stdout return for successful invocations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f5d3f595-fdc8-4157-aeab-22214a02e842
📒 Files selected for processing (4)
CHANGELOG.mdsrc/vouch/hooks.pytests/test_hooks.pytests/test_prompt_gate_live.py
🚧 Files skipped from review as they are similar to previous changes (1)
- CHANGELOG.md
| def _ask_model(block: str, prompt: str, workdir: Path) -> str: | ||
| payload = f"{block}\n\n{prompt}" if block else prompt | ||
| res = subprocess.run( | ||
| ["claude", "-p", "--settings", '{"hooks":{}}', "--model", MODEL, | ||
| "--disallowed-tools", "Bash", "Read", "Edit", "Write", "Glob", | ||
| "Grep", "WebFetch", "WebSearch", "Task"], | ||
| input=payload, capture_output=True, text=True, timeout=180, cwd=str(workdir), | ||
| ) | ||
| return (res.stdout or "").strip() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does the current claude-code CLI's --disallowedTools flag accept multiple space-separated positional values, or must they be passed as one comma/space-separated string argument?
💡 Result:
To pass multiple values to the --disallowedTools flag in the claude-code CLI, you should use a single comma-separated string argument [1]. While some legacy or inconsistent documentation examples have historically shown space-separated quoted strings (e.g., --disallowedTools "ToolA" "ToolB") [2][3], this syntax is technically invalid in most shell environments. In practice, using such syntax often results in the shell passing only the first argument to the flag and treating subsequent quoted strings as separate, invalid positional arguments [1]. To ensure your configuration is parsed correctly, you should consolidate your list into one string using commas as delimiters, such as: claude --disallowedTools "Bash,Edit,Read" This comma-separated format is the recommended approach to avoid command failures and delimiter ambiguity [1].
Citations:
- 1: [DOCS] Inconsistent and potentially broken CLI argument syntax for
--toolsand--disallowedToolsflags anthropics/claude-code#19366 - 2: https://code.claude.com/docs/en/cli
- 3: https://code.claude.com/docs/en/cli-reference
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n## tests/test_prompt_gate_live.py\n'
nl -ba tests/test_prompt_gate_live.py | sed -n '1,180p'
printf '\n## claude invocations in repo\n'
rg -n --hidden --no-ignore-vcs '\bclaude\b|--disallowedTools|--disallowed-tools' .Repository: vouchdev/vouch
Length of output: 227
🌐 Web query:
claude-code CLI reference disallowedTools camelCase comma-separated string exit code nonzero subprocess CLI docs
💡 Result:
In Claude Code, the --disallowedTools CLI flag is used to restrict the tools available to the agent [1][2]. Regarding your specific points: - Tool Restriction Syntax: While documentation has historically shown both space-separated and comma-separated patterns, standard shell practice and recent community feedback emphasize that passing multiple rules as a single comma-separated string or as separate arguments can be shell-dependent [3]. Official documentation lists examples such as "Bash(git log *)" "Bash(git diff *)" "Edit", but users have reported that using a single comma-separated string (e.g., "Bash,Edit,Read") is often cleaner and more reliable for command-line parsing [3][2]. - Non-Zero Exit Codes: - General Execution: If Claude Code encounters errors while running (e.g., exceeding the 10MB limit for piped stdin), it exits with a non-zero status [4][5]. - Hooks: In the context of hooks, exit code 2 is treated as a blocking error (preventing the action), while exit code 1 is generally treated as a non-blocking error, except for WorktreeCreate where any non-zero exit code aborts the operation [6]. - Process Interruption: If interrupted by a signal like SIGTERM, Claude Code typically exits with code 143 [4][5]. For the most up-to-date and reliable syntax, refer to the tool-specific permission rule format, which uses ToolName(specifier) syntax for all restriction flags [7][8].
Citations:
- 1: https://code.claude.com/docs/en/cli
- 2: https://code.claude.com/docs/en/cli-reference
- 3: [DOCS] Inconsistent and potentially broken CLI argument syntax for
--toolsand--disallowedToolsflags anthropics/claude-code#19366 - 4: https://code.claude.com/docs/en/headless
- 5: https://code.claude.com/docs/en/headless.md
- 6: https://code.claude.com/docs/en/hooks
- 7: https://code.claude.com/docs/en/tools-reference.md
- 8: https://code.claude.com/docs/en/tools-reference
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n## tests/test_prompt_gate_live.py\n'
cat -n tests/test_prompt_gate_live.py | sed -n '1,180p'
printf '\n## claude invocations and denylist flags in repo\n'
rg -n --hidden --no-ignore-vcs '\bclaude\b|--disallowedTools|--disallowed-tools|Bash,Read,Edit|Bash" "Read' .Repository: vouchdev/vouch
Length of output: 50371
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path("tests/test_prompt_gate_live.py")
text = p.read_text()
lines = text.splitlines()
for start in (1, 45, 60, 68, 76, 90, 110, 130):
end = min(len(lines), start + 20)
print(f"\n## lines {start}-{end}")
for i in range(start, end + 1):
if i <= len(lines):
print(f"{i:4d}: {lines[i-1]}")
PY
printf '\n## exact claude/tool flag occurrences in tests/test_prompt_gate_live.py\n'
grep -nE 'claude|disallowed|returncode|stdout|stderr' tests/test_prompt_gate_live.py || trueRepository: vouchdev/vouch
Length of output: 6339
🌐 Web query:
site:code.claude.com claude code CLI --disallowedTools tool names comma separated space separated current docs
💡 Result:
In the Claude Code CLI, the --disallowedTools flag accepts a list of deny rules to remove specific tools from Claude's context or to restrict their capabilities [1][2]. You should provide the tool names (and optional specifications) as a space-separated list enclosed in quotes, where each individual rule may contain its own internal spacing or specific syntax [1][2][3]. Usage Example: claude --disallowedTools "Edit" "Bash(rm )" "mcp" Key points regarding the format and behavior: 1. Syntax: Each rule follows the format ToolName(specifier) [4][5]. 2. Bare Tool Names: A bare name (e.g., "Edit") removes the entire tool from Claude's context [1][2]. 3. Scoped Rules: A scoped rule (e.g., "Bash(rm )") leaves the tool available but denies matching calls [1][3]. 4. Wildcards: You can use "" to match all tools (e.g., "" removes all tools, "mcp" removes all MCP tools) [1][6]. 5. Precedence: The --disallowedTools flag adds restrictions beyond what managed settings define, and if a tool is denied at any level, it cannot be overridden by other settings [7][8]. For more details on specific rule formats for different tools, refer to the official Claude Code documentation on tools reference [4][5].
Citations:
- 1: https://code.claude.com/docs/en/cli-reference
- 2: https://code.claude.com/docs/en/cli
- 3: https://code.claude.com/docs/en/cli-reference.md
- 4: https://code.claude.com/docs/en/tools-reference
- 5: https://code.claude.com/docs/en/tools
- 6: https://code.claude.com/docs/en/sub-agents
- 7: https://code.claude.com/docs/en/permissions
- 8: https://code.claude.com/docs/en/permissions.md
🌐 Web query:
Claude Code CLI --disallowed-tools alias --disallowedTools kebab-case camelCase accepted
💡 Result:
The Claude Code CLI supports both camelCase (--disallowedTools) and kebab-case (--disallowed-tools) variants for the disallowed tools flag [1][2]. The CLI parser is designed to accept and process both formats [2], ensuring consistency regardless of which casing convention you use. Regarding usage, the flag accepts a comma- or space-separated list of tool names [3][4]. While official documentation sometimes displays examples using multiple quoted blocks (e.g., "Bash(git log:*)" "Bash(git diff:*)") [5][4], this syntax can be problematic in some shells [5]. A more robust and widely recommended approach is to provide a single comma-separated string (e.g., --disallowed-tools "Bash(git:*),Edit") [5]. When a tool is specified in the --disallowedTools (or --disallowed-tools) list, it is removed from the model's context entirely, causing Claude to act as if that tool does not exist [1][4]. If both --allowedTools and --disallowedTools are used, the deny list generally takes precedence [6][7].
Citations:
- 1: https://deepwiki.com/zebbern/claude-code-guide/8.2-tool-access-control
- 2: https://github.com/anthropics/claude-code-action/blob/6cad158a/base-action/src/parse-sdk-options.ts
- 3: https://jackdog668-claude-code.mintlify.app/commands/overview
- 4: https://code.claude.com/docs/en/cli-reference
- 5: [DOCS] Inconsistent and potentially broken CLI argument syntax for
--toolsand--disallowedToolsflags anthropics/claude-code#19366 - 6: https://code.claude.com/docs/en/permissions
- 7: https://code.claude.com/docs/en/tools-reference
Surface claude -p failures instead of counting them as misses tests/test_prompt_gate_live.py:68-76 — _ask_model should raise on nonzero exit (and include stderr) so auth/timeouts/bad CLI invocations don’t get misclassified as model behavior.
🧰 Tools
🪛 ast-grep (0.44.1)
[error] 69-74: Command coming from incoming request
Context: subprocess.run(
["claude", "-p", "--settings", '{"hooks":{}}', "--model", MODEL,
"--disallowed-tools", "Bash", "Read", "Edit", "Write", "Glob",
"Grep", "WebFetch", "WebSearch", "Task"],
input=payload, capture_output=True, text=True, timeout=180, cwd=str(workdir),
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_prompt_gate_live.py` around lines 68 - 76, Update _ask_model to
detect a nonzero subprocess return code and raise an error that includes stderr,
while preserving the existing stripped stdout return for successful invocations.
…n a retrieval error two coderabbit findings on the prompt-gate pr, both real: the informative-token filter dropped every all-digit token, losing issue ids, error codes (404/500), and ports (8080) — exactly the terms a technical query needs. the len < 2 check already drops single-digit noise, so only tok.isdigit() had to go. _retrieve_body returned (None, "") on a build_context_pack exception, which both injection paths saw as a genuine empty result — so a broken index made the model falsely assert "Nothing in vouch on this." it now raises, and the hook dispatch catches any retrieval/render error and injects NOTHING, restoring the pre-gate fail-safe. a true empty search (no exception) still gets the honest miss message.
1 participant
recall used to inject an unconditional "open with From vouch memory:" block on every prompt, so a coding command like "fix the failing test" spent its reply opener announcing a memory search nobody asked for.
stacked on #534 (phase 3) — both touch
hooks.py, so this branches offfeat/personal-kb. merge #534 first and this shrinks to its own files.the approach: don't make the hook guess — let the model decide
the first cut (commit
ec1c440) enumerated "action verbs" and defaulted everything else to loud recall. that fails open: action verbs are an unbounded class, so any verb not on the list (vendorize,dockerize,spin up,knock out,yarn build) fell through to the banner — the exact noise the gate exists to stop. commit8d07646replaces it.when the gate is on, the hook now hands the host model one conditional instruction and lets the model — which already reads the prompt — choose per turn:
[id]inline only where relied on, no bannerno per-turn model call, no latency — the decision is instruction text the model already processes. it generalizes to any phrasing or language because it no longer depends on recognizing the verb. the legacy unconditional block is kept intact behind the gate flag, so existing KBs are byte-for-byte unchanged until they add
retrieval.prompt_gate; new KBs get delegation via the starter config.why not a rule / a confidence threshold / an LLM call
compliance was measured, not assumed
real
claude -pacross haiku / sonnet / opus, offline and against the realvouch context-hookpipeline:tests/test_hooks.pypins the injected block deterministically;tests/test_prompt_gate_live.pyis an opt-in live probe (skips without an authenticatedclaudeCLI, like the openclaw live gate).gate green: 1625 passed, mypy clean, ruff clean.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes