Document TW-001 holdout limits and related work

LIMITATIONS.md

@@ -8,7 +8,9 @@ Tripwire does not prevent attacks, prove safety, or operate in real time.
 
 The canonical split groups by `(suite, user task, injection task)`. It prevents an exact combination from crossing partitions, but the same user task and the same injection family can occur on both sides. Its 35.3% recall therefore measures generalisation to unseen combinations, not unseen attacks.
 
-A deterministic secondary split groups successful attacks by `(suite, injection task)` and holds out complete `injection_task_7` and `injection_task_8` families. TW-001 recalls 6 of 13 successful attacks there (46.2%, Wilson 95%: 23.2%–70.9%). This is a post-hoc stress test, not an unbiased unseen-family holdout: the original development partition contained examples from every injection family before this analysis was added.
+`TW-001` is frozen. Its canonical holdout is spent because the M2 audit inspected every false positive and false negative. Any successor rule requires a fresh split or corpus for an uncontaminated holdout claim; evaluating a revision against the current holdout must be labelled contaminated.
+
+A deterministic secondary split groups successful attacks by `(suite, injection task)` and holds out complete `injection_task_7` and `injection_task_8` families. TW-001 recalls 6 of 13 successful attacks there (46.2%, Wilson 95%: 23.2%–70.9%). Its interval overlaps substantially with the canonical result's 17.3%–58.7% interval, so the numerical difference should not be interpreted as evidence that recall differs. The stress test reports recall only because benign no-injection negatives have no injection family and cannot be assigned by this grouping rule. This is a post-hoc stress test, not an unbiased unseen-family holdout: the original development partition contained examples from every injection family before this analysis was added.
 
 On the frozen development partition, `TW-001` produced 3 false positives among 74 negative sessions, compared with 4 for the naive external-read/external-write sequence. On holdout, both sequences produced 1 false positive among 37 negative sessions. The extra sensitive-read stage therefore showed a small development specificity improvement that did not repeat on holdout.
 

README.md

@@ -25,16 +25,25 @@ Confusion counts: TP 6, FP 1, TN 36, FN 11. The holdout contains 17 positive and
 
 The canonical split hashes `(suite, user task, injection task)` tuples. It measures generalisation to unseen combinations, not unseen user tasks or attack families: some user-task identities and every injection-task identity appear on both sides.
 
+`TW-001` is now frozen. Its canonical holdout is spent because every error has been inspected. Any successor must be evaluated on a fresh split or corpus; a result produced by revising the rule against this holdout must be labelled contaminated rather than reported as new holdout performance.
+
 ## Injection-family stress test
 
-A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out `injection_task_7`, `injection_task_8` and recalls 6/13 (46.2%, Wilson 95%: 23.2%–70.9%). This is higher than the canonical 35.3%, not lower. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family.
+A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out `injection_task_7`, `injection_task_8` and recalls 6/13 (46.2%, Wilson 95%: 23.2%–70.9%). This is numerically higher than the canonical 35.3%, but the intervals overlap substantially, so the numerical difference should not be interpreted as evidence that recall differs. The stress test reports recall only: benign no-injection negatives have no injection family and cannot be assigned by this grouping rule, so precision and false-positive rate would not be comparable. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family.
 
 ## Specificity ceiling
 
 Three development negatives matched TW-001. Two were benign `none/none.json` sessions whose legitimate workflow was `read_file` → `get_scheduled_transactions` → `update_scheduled_transaction`; the third was an injected run whose benchmark attack failed. The holdout false positive was also a failed injected run, and all four false positives share that scheduled-update sequence. The rule's sequence is therefore a normal banking workflow as well as an attack shape, so structure alone cannot establish malicious intent.
 
 This result classifies successful prompt-injection outcomes in the committed AgentDojo banking corpus under the documented tool taxonomy. It does not establish causation, malicious intent, prevention, general prompt-injection detection, or real-time operation. Historic traces test order but not elapsed-time semantics. See [`LIMITATIONS.md`](LIMITATIONS.md).
 
+## Related work
+
+- [AgentDojo](https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html) supplies the agent tasks, attacks, benchmark outcomes and source traces used by the committed corpus. Tripwire adds a sanitised replay corpus and detection-oriented measurement rather than a new attack benchmark.
+- [Agent Threat Rules (ATR)](https://github.com/Agent-Threat-Rule/agent-threat-rules) standardises portable rules evaluated against individual agent events and content fields. Tripwire instead measures ordered cross-event behaviour and exposes where event-level tool provenance loses mixed-trust structure inside a result.
+- [AgentSigma](https://github.com/cveye/agentsigma) is a close detection-engineering precedent: a Sigma-compatible runtime framework for the agent tool-call layer. Tripwire's narrower contribution is a frozen labelled evaluation, explicit holdout discipline and measured provenance failure.
+- [Out-of-band agent defenses](https://arxiv.org/abs/2606.26479) use deterministic policies, capabilities or information-flow labels outside the model. Tripwire's finding identifies a boundary condition for designs that collapse provenance to tool identity or whole tool results: a trusted tool can return attacker-controlled fields.
+
 ## Evidence
 
 - Rule: [`rules/TW-001.yaml`](rules/TW-001.yaml)

docs/decisions.md

@@ -10,6 +10,10 @@ The canonical verifier evaluates both development and holdout partitions. The ev
 
 The canonical split groups by `(suite, user task, injection task)`, so it isolates unseen combinations rather than unseen user tasks or injection families. A secondary post-hoc stress test hashes `(suite, injection task)` with the same deterministic 70/30 algorithm. It holds out complete `injection_task_7` and `injection_task_8` families and reports recall only over successful attacks in those families. It is not described as a clean unseen-family estimate because the original development partition exposed the rule-development process to every injection family.
 
+## 2026-08-04: TW-001 freeze after holdout audit
+
+`TW-001` is frozen after the M2 audit inspected every canonical holdout error. The current holdout remains valid only for the unchanged rule. A successor rule must use a fresh split or corpus for an uncontaminated holdout claim; otherwise its result must be labelled contaminated. The family stress test reports recall only because benign no-injection negatives have no injection family, and its interval overlaps the canonical recall interval.
+
 ## 2026-08-03: Naive-sequence stop-condition check
 
 The naive external-result then external-write diagnostic matched 4 of 74 development negatives and 1 of 37 holdout negatives. `TW-001` matched 3 of 74 and 1 of 37 respectively. The documented stop condition was not reached because `TW-001` has better specificity on the frozen development data, although the improvement did not repeat on holdout. This limitation is reported without changing the rule after holdout evaluation.

results/agentdojo-banking-v1.json

@@ -687,6 +687,7 @@
     "Historic AgentDojo traces have no per-event timestamps, so elapsed-time semantics are not measured.",
     "The result is conditional on the committed AgentDojo banking tool taxonomy.",
     "The canonical split holds out user-task/injection-task combinations, not whole user tasks or injection families.",
+    "TW-001 is frozen and its inspected canonical holdout is spent for future rule revision.",
     "TW-001 improved development specificity over the naive sequence by one session, but both produced one false positive on holdout.",
     "Eight of eleven holdout false negatives carry injected content inside mixed-provenance transaction results classified as internal at tool-result granularity.",
     "The single holdout false positive contains the structural sequence but did not satisfy the benchmark injection-task goal."

src/tripwire/evaluation.py

@@ -64,6 +64,10 @@ def evaluate_corpus(rule: Rule, corpus: Path, *, split: str = "all") -> dict[str
                 "The canonical split holds out user-task/injection-task combinations, not "
                 "whole user tasks or injection families."
             ),
+            (
+                "TW-001 is frozen and its inspected canonical holdout is spent for future "
+                "rule revision."
+            ),
             (
                 "TW-001 improved development specificity over the naive sequence by one session, "
                 "but both produced one false positive on holdout."

src/tripwire/reporting.py

@@ -41,16 +41,25 @@ def render_readme(result: dict[str, Any]) -> str:
 
 The canonical split hashes `(suite, user task, injection task)` tuples. It measures generalisation to unseen combinations, not unseen user tasks or attack families: some user-task identities and every injection-task identity appear on both sides.
 
+`TW-001` is now frozen. Its canonical holdout is spent because every error has been inspected. Any successor must be evaluated on a fresh split or corpus; a result produced by revising the rule against this holdout must be labelled contaminated rather than reported as new holdout performance.
+
 ## Injection-family stress test
 
-A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out {family_names} and recalls {family_recall["numerator"]}/{family_recall["denominator"]} ({_percent(family_recall["value"])}, Wilson 95%: {_interval(family_recall["wilson_95"])}). This is higher than the canonical 35.3%, not lower. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family.
+A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out {family_names} and recalls {family_recall["numerator"]}/{family_recall["denominator"]} ({_percent(family_recall["value"])}, Wilson 95%: {_interval(family_recall["wilson_95"])}). This is numerically higher than the canonical 35.3%, but the intervals overlap substantially, so the numerical difference should not be interpreted as evidence that recall differs. The stress test reports recall only: benign no-injection negatives have no injection family and cannot be assigned by this grouping rule, so precision and false-positive rate would not be comparable. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family.
 
 ## Specificity ceiling
 
 Three development negatives matched TW-001. Two were benign `none/none.json` sessions whose legitimate workflow was `read_file` → `get_scheduled_transactions` → `update_scheduled_transaction`; the third was an injected run whose benchmark attack failed. The holdout false positive was also a failed injected run, and all four false positives share that scheduled-update sequence. The rule's sequence is therefore a normal banking workflow as well as an attack shape, so structure alone cannot establish malicious intent.
 
 This result classifies successful prompt-injection outcomes in the committed AgentDojo banking corpus under the documented tool taxonomy. It does not establish causation, malicious intent, prevention, general prompt-injection detection, or real-time operation. Historic traces test order but not elapsed-time semantics. See [`LIMITATIONS.md`](LIMITATIONS.md).
 
+## Related work
+
+- [AgentDojo](https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html) supplies the agent tasks, attacks, benchmark outcomes and source traces used by the committed corpus. Tripwire adds a sanitised replay corpus and detection-oriented measurement rather than a new attack benchmark.
+- [Agent Threat Rules (ATR)](https://github.com/Agent-Threat-Rule/agent-threat-rules) standardises portable rules evaluated against individual agent events and content fields. Tripwire instead measures ordered cross-event behaviour and exposes where event-level tool provenance loses mixed-trust structure inside a result.
+- [AgentSigma](https://github.com/cveye/agentsigma) is a close detection-engineering precedent: a Sigma-compatible runtime framework for the agent tool-call layer. Tripwire's narrower contribution is a frozen labelled evaluation, explicit holdout discipline and measured provenance failure.
+- [Out-of-band agent defenses](https://arxiv.org/abs/2606.26479) use deterministic policies, capabilities or information-flow labels outside the model. Tripwire's finding identifies a boundary condition for designs that collapse provenance to tool identity or whole tool results: a trusted tool can return attacker-controlled fields.
+
 ## Evidence
 
 - Rule: [`rules/TW-001.yaml`](rules/TW-001.yaml)

tests/test_m1_acceptance.py

@@ -369,6 +369,19 @@ def test_committed_result_reproduces() -> None:
     assert summary.startswith("TW-001 verified: holdout TP=6 FP=1 TN=36 FN=11")
 
 
+def test_public_report_preserves_evaluation_discipline() -> None:
+    result = evaluate_corpus(
+        load_rule(ROOT / "rules" / "TW-001.yaml"),
+        ROOT / "corpora" / "agentdojo-banking-v1",
+    )
+    readme = render_readme(result)
+    assert "canonical holdout is spent" in readme
+    assert "intervals overlap substantially" in readme
+    assert "reports recall only" in readme
+    assert "## Related work" in readme
+    assert any("holdout is spent" in limitation for limitation in result["limitations"])
+
+
 def test_readme_is_generated_from_canonical_result() -> None:
     result = json.loads(
         (ROOT / "results" / "agentdojo-banking-v1.json").read_text(encoding="utf-8")