From b3503d20b6763921f654ba6a4bc8773c601b1798 Mon Sep 17 00:00:00 2001 From: Jonathan Asare <294263306+jonathanjasare@users.noreply.github.com> Date: Tue, 4 Aug 2026 00:57:57 +0100 Subject: [PATCH] Document holdout evaluation limits --- LIMITATIONS.md | 4 +++- README.md | 11 ++++++++++- docs/decisions.md | 4 ++++ results/agentdojo-banking-v1.json | 1 + src/tripwire/evaluation.py | 4 ++++ src/tripwire/reporting.py | 11 ++++++++++- tests/test_m1_acceptance.py | 13 +++++++++++++ 7 files changed, 45 insertions(+), 3 deletions(-) diff --git a/LIMITATIONS.md b/LIMITATIONS.md index 14a6b15..ba09ebe 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -8,7 +8,9 @@ Tripwire does not prevent attacks, prove safety, or operate in real time. The canonical split groups by `(suite, user task, injection task)`. It prevents an exact combination from crossing partitions, but the same user task and the same injection family can occur on both sides. Its 35.3% recall therefore measures generalisation to unseen combinations, not unseen attacks. -A deterministic secondary split groups successful attacks by `(suite, injection task)` and holds out complete `injection_task_7` and `injection_task_8` families. TW-001 recalls 6 of 13 successful attacks there (46.2%, Wilson 95%: 23.2%–70.9%). This is a post-hoc stress test, not an unbiased unseen-family holdout: the original development partition contained examples from every injection family before this analysis was added. +`TW-001` is frozen. Its canonical holdout is spent because the M2 audit inspected every false positive and false negative. Any successor rule requires a fresh split or corpus for an uncontaminated holdout claim; evaluating a revision against the current holdout must be labelled contaminated. + +A deterministic secondary split groups successful attacks by `(suite, injection task)` and holds out complete `injection_task_7` and `injection_task_8` families. TW-001 recalls 6 of 13 successful attacks there (46.2%, Wilson 95%: 23.2%–70.9%). Its interval overlaps substantially with the canonical result's 17.3%–58.7% interval, so the numerical difference should not be interpreted as evidence that recall differs. The stress test reports recall only because benign no-injection negatives have no injection family and cannot be assigned by this grouping rule. This is a post-hoc stress test, not an unbiased unseen-family holdout: the original development partition contained examples from every injection family before this analysis was added. On the frozen development partition, `TW-001` produced 3 false positives among 74 negative sessions, compared with 4 for the naive external-read/external-write sequence. On holdout, both sequences produced 1 false positive among 37 negative sessions. The extra sensitive-read stage therefore showed a small development specificity improvement that did not repeat on holdout. diff --git a/README.md b/README.md index 33024f9..8895248 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,11 @@ Confusion counts: TP 6, FP 1, TN 36, FN 11. The holdout contains 17 positive and The canonical split hashes `(suite, user task, injection task)` tuples. It measures generalisation to unseen combinations, not unseen user tasks or attack families: some user-task identities and every injection-task identity appear on both sides. +`TW-001` is now frozen. Its canonical holdout is spent because every error has been inspected. Any successor must be evaluated on a fresh split or corpus; a result produced by revising the rule against this holdout must be labelled contaminated rather than reported as new holdout performance. + ## Injection-family stress test -A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out `injection_task_7`, `injection_task_8` and recalls 6/13 (46.2%, Wilson 95%: 23.2%–70.9%). This is higher than the canonical 35.3%, not lower. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family. +A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out `injection_task_7`, `injection_task_8` and recalls 6/13 (46.2%, Wilson 95%: 23.2%–70.9%). This is numerically higher than the canonical 35.3%, but the intervals overlap substantially, so the numerical difference should not be interpreted as evidence that recall differs. The stress test reports recall only: benign no-injection negatives have no injection family and cannot be assigned by this grouping rule, so precision and false-positive rate would not be comparable. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family. ## Specificity ceiling @@ -35,6 +37,13 @@ Three development negatives matched TW-001. Two were benign `none/none.json` ses This result classifies successful prompt-injection outcomes in the committed AgentDojo banking corpus under the documented tool taxonomy. It does not establish causation, malicious intent, prevention, general prompt-injection detection, or real-time operation. Historic traces test order but not elapsed-time semantics. See [`LIMITATIONS.md`](LIMITATIONS.md). +## Related work + +- [AgentDojo](https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html) supplies the agent tasks, attacks, benchmark outcomes and source traces used by the committed corpus. Tripwire adds a sanitised replay corpus and detection-oriented measurement rather than a new attack benchmark. +- [Agent Threat Rules (ATR)](https://github.com/Agent-Threat-Rule/agent-threat-rules) standardises portable rules evaluated against individual agent events and content fields. Tripwire instead measures ordered cross-event behaviour and exposes where event-level tool provenance loses mixed-trust structure inside a result. +- [AgentSigma](https://github.com/cveye/agentsigma) is a close detection-engineering precedent: a Sigma-compatible runtime framework for the agent tool-call layer. Tripwire's narrower contribution is a frozen labelled evaluation, explicit holdout discipline and measured provenance failure. +- [Out-of-band agent defenses](https://arxiv.org/abs/2606.26479) use deterministic policies, capabilities or information-flow labels outside the model. Tripwire's finding identifies a boundary condition for designs that collapse provenance to tool identity or whole tool results: a trusted tool can return attacker-controlled fields. + ## Evidence - Rule: [`rules/TW-001.yaml`](rules/TW-001.yaml) diff --git a/docs/decisions.md b/docs/decisions.md index c4cb365..a41c908 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -10,6 +10,10 @@ The canonical verifier evaluates both development and holdout partitions. The ev The canonical split groups by `(suite, user task, injection task)`, so it isolates unseen combinations rather than unseen user tasks or injection families. A secondary post-hoc stress test hashes `(suite, injection task)` with the same deterministic 70/30 algorithm. It holds out complete `injection_task_7` and `injection_task_8` families and reports recall only over successful attacks in those families. It is not described as a clean unseen-family estimate because the original development partition exposed the rule-development process to every injection family. +## 2026-08-04: TW-001 freeze after holdout audit + +`TW-001` is frozen after the M2 audit inspected every canonical holdout error. The current holdout remains valid only for the unchanged rule. A successor rule must use a fresh split or corpus for an uncontaminated holdout claim; otherwise its result must be labelled contaminated. The family stress test reports recall only because benign no-injection negatives have no injection family, and its interval overlaps the canonical recall interval. + ## 2026-08-03: Naive-sequence stop-condition check The naive external-result then external-write diagnostic matched 4 of 74 development negatives and 1 of 37 holdout negatives. `TW-001` matched 3 of 74 and 1 of 37 respectively. The documented stop condition was not reached because `TW-001` has better specificity on the frozen development data, although the improvement did not repeat on holdout. This limitation is reported without changing the rule after holdout evaluation. diff --git a/results/agentdojo-banking-v1.json b/results/agentdojo-banking-v1.json index e84c650..9415ea6 100644 --- a/results/agentdojo-banking-v1.json +++ b/results/agentdojo-banking-v1.json @@ -687,6 +687,7 @@ "Historic AgentDojo traces have no per-event timestamps, so elapsed-time semantics are not measured.", "The result is conditional on the committed AgentDojo banking tool taxonomy.", "The canonical split holds out user-task/injection-task combinations, not whole user tasks or injection families.", + "TW-001 is frozen and its inspected canonical holdout is spent for future rule revision.", "TW-001 improved development specificity over the naive sequence by one session, but both produced one false positive on holdout.", "Eight of eleven holdout false negatives carry injected content inside mixed-provenance transaction results classified as internal at tool-result granularity.", "The single holdout false positive contains the structural sequence but did not satisfy the benchmark injection-task goal." diff --git a/src/tripwire/evaluation.py b/src/tripwire/evaluation.py index c1e8fa1..cf81e80 100644 --- a/src/tripwire/evaluation.py +++ b/src/tripwire/evaluation.py @@ -64,6 +64,10 @@ def evaluate_corpus(rule: Rule, corpus: Path, *, split: str = "all") -> dict[str "The canonical split holds out user-task/injection-task combinations, not " "whole user tasks or injection families." ), + ( + "TW-001 is frozen and its inspected canonical holdout is spent for future " + "rule revision." + ), ( "TW-001 improved development specificity over the naive sequence by one session, " "but both produced one false positive on holdout." diff --git a/src/tripwire/reporting.py b/src/tripwire/reporting.py index b32ffb7..aa94e0e 100644 --- a/src/tripwire/reporting.py +++ b/src/tripwire/reporting.py @@ -41,9 +41,11 @@ def render_readme(result: dict[str, Any]) -> str: The canonical split hashes `(suite, user task, injection task)` tuples. It measures generalisation to unseen combinations, not unseen user tasks or attack families: some user-task identities and every injection-task identity appear on both sides. +`TW-001` is now frozen. Its canonical holdout is spent because every error has been inspected. Any successor must be evaluated on a fresh split or corpus; a result produced by revising the rule against this holdout must be labelled contaminated rather than reported as new holdout performance. + ## Injection-family stress test -A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out {family_names} and recalls {family_recall["numerator"]}/{family_recall["denominator"]} ({_percent(family_recall["value"])}, Wilson 95%: {_interval(family_recall["wilson_95"])}). This is higher than the canonical 35.3%, not lower. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family. +A secondary deterministic split hashes `(suite, injection task)` so complete injection families stay together. It holds out {family_names} and recalls {family_recall["numerator"]}/{family_recall["denominator"]} ({_percent(family_recall["value"])}, Wilson 95%: {_interval(family_recall["wilson_95"])}). This is numerically higher than the canonical 35.3%, but the intervals overlap substantially, so the numerical difference should not be interpreted as evidence that recall differs. The stress test reports recall only: benign no-injection negatives have no injection family and cannot be assigned by this grouping rule, so precision and false-positive rate would not be comparable. It is a post-hoc stress test rather than an unbiased unseen-attack estimate because the original development partition exposed TW-001's development process to examples from every injection family. ## Specificity ceiling @@ -51,6 +53,13 @@ def render_readme(result: dict[str, Any]) -> str: This result classifies successful prompt-injection outcomes in the committed AgentDojo banking corpus under the documented tool taxonomy. It does not establish causation, malicious intent, prevention, general prompt-injection detection, or real-time operation. Historic traces test order but not elapsed-time semantics. See [`LIMITATIONS.md`](LIMITATIONS.md). +## Related work + +- [AgentDojo](https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html) supplies the agent tasks, attacks, benchmark outcomes and source traces used by the committed corpus. Tripwire adds a sanitised replay corpus and detection-oriented measurement rather than a new attack benchmark. +- [Agent Threat Rules (ATR)](https://github.com/Agent-Threat-Rule/agent-threat-rules) standardises portable rules evaluated against individual agent events and content fields. Tripwire instead measures ordered cross-event behaviour and exposes where event-level tool provenance loses mixed-trust structure inside a result. +- [AgentSigma](https://github.com/cveye/agentsigma) is a close detection-engineering precedent: a Sigma-compatible runtime framework for the agent tool-call layer. Tripwire's narrower contribution is a frozen labelled evaluation, explicit holdout discipline and measured provenance failure. +- [Out-of-band agent defenses](https://arxiv.org/abs/2606.26479) use deterministic policies, capabilities or information-flow labels outside the model. Tripwire's finding identifies a boundary condition for designs that collapse provenance to tool identity or whole tool results: a trusted tool can return attacker-controlled fields. + ## Evidence - Rule: [`rules/TW-001.yaml`](rules/TW-001.yaml) diff --git a/tests/test_m1_acceptance.py b/tests/test_m1_acceptance.py index 4e28ab9..9ffaabd 100644 --- a/tests/test_m1_acceptance.py +++ b/tests/test_m1_acceptance.py @@ -369,6 +369,19 @@ def test_committed_result_reproduces() -> None: assert summary.startswith("TW-001 verified: holdout TP=6 FP=1 TN=36 FN=11") +def test_public_report_preserves_evaluation_discipline() -> None: + result = evaluate_corpus( + load_rule(ROOT / "rules" / "TW-001.yaml"), + ROOT / "corpora" / "agentdojo-banking-v1", + ) + readme = render_readme(result) + assert "canonical holdout is spent" in readme + assert "intervals overlap substantially" in readme + assert "reports recall only" in readme + assert "## Related work" in readme + assert any("holdout is spent" in limitation for limitation in result["limitations"]) + + def test_readme_is_generated_from_canonical_result() -> None: result = json.loads( (ROOT / "results" / "agentdojo-banking-v1.json").read_text(encoding="utf-8")