Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,15 @@ on:
push:
pull_request:

permissions:
contents: read

jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
with:
version: "0.11.28"
- run: uv sync --frozen
Expand All @@ -18,4 +21,3 @@ jobs:
- run: uv run pytest
- run: uv build
- run: uv run tripwire verify

6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,12 @@ Tripwire evaluates deterministic sequence rules over AI-agent tool-call traces.

The main finding is a provenance failure, not a new attack signature. Eight of the eleven canonical holdout false negatives received attacker-controlled strings inside transaction records returned by banking tools classified as internal. Tool-result-level provenance is too coarse when a trusted tool returns mixed-trust records; item-level provenance is required to distinguish the injected content without distrusting the entire result.

Tripwire is a research prototype and reproducibility artifact. It is not affiliated with Fortra's Tripwire products. GitHub issues may be used to report reproducibility defects; no support or release cadence is promised.

## Reproduce the committed result

Prerequisites: Python 3.11–3.13 and [uv](https://docs.astral.sh/uv/getting-started/installation/).

```console
uv sync --frozen
uv run tripwire verify
Expand Down Expand Up @@ -41,7 +45,7 @@ This result classifies successful prompt-injection outcomes in the committed Age

- [AgentDojo](https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html) supplies the agent tasks, attacks, benchmark outcomes and source traces used by the committed corpus. Tripwire adds a sanitised replay corpus and detection-oriented measurement rather than a new attack benchmark.
- [Agent Threat Rules (ATR)](https://github.com/Agent-Threat-Rule/agent-threat-rules) standardises portable detections over agent events and content fields.
- Cross-event agent detection is established prior art, not Tripwire's novelty. [AgentSigma](https://github.com/cveye/agentsigma) applies Sigma-compatible runtime detection at the tool-call layer; [AgentShield's sigma-ai](https://github.com/agentshield-ai/sigma-ai) documents temporal correlation for sequential agent events; [AIDR Sigma](https://github.com/netzilo/aidr-sigma) describes a persistent per-session behaviour graph for multi-step patterns; and [RSigma](https://github.com/timescale/rsigma) provides in-process stateful correlation and OTLP ingest. RSigma is also a candidate substrate for any future streaming backend and should be evaluated before Tripwire implements those capabilities itself.
- Cross-event agent detection is established prior art, not Tripwire's novelty. [AgentSigma](https://medium.com/@dasgupta.pratip/agentsigma-v0-1-building-the-detection-framework-i-wrote-about-3f6e14c194c5) applies Sigma-compatible runtime detection at the tool-call layer; [AgentShield's sigma-ai](https://github.com/agentshield-ai/sigma-ai) documents temporal correlation for sequential agent events; [AIDR Sigma](https://github.com/netzilo/aidr-sigma) describes a persistent per-session behaviour graph for multi-step patterns; and [RSigma](https://github.com/timescale/rsigma) provides in-process stateful correlation and OTLP ingest. RSigma is also a candidate substrate for any future streaming backend and should be evaluated before Tripwire implements those capabilities itself.
- Tripwire's contribution here is the measurement: a sanitised labelled AgentDojo corpus, reproducible metrics, a frozen holdout and the provenance-granularity finding that resulted from error analysis.
- [Out-of-band agent defenses](https://arxiv.org/abs/2606.26479) use deterministic policies, capabilities or information-flow labels outside the model. Tripwire's finding identifies a boundary condition for designs that collapse provenance to tool identity or whole tool results: a trusted tool can return attacker-controlled fields.

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ tripwire = "tripwire.cli:main"
dev = [
"build==1.2.2.post1",
"mypy==1.15.0",
"pytest==8.3.5",
"pytest==9.0.3",
"ruff==0.9.10",
"types-PyYAML==6.0.12.20250516",
]
Expand Down
6 changes: 5 additions & 1 deletion src/tripwire/reporting.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,12 @@ def render_readme(result: dict[str, Any]) -> str:

The main finding is a provenance failure, not a new attack signature. Eight of the eleven canonical holdout false negatives received attacker-controlled strings inside transaction records returned by banking tools classified as internal. Tool-result-level provenance is too coarse when a trusted tool returns mixed-trust records; item-level provenance is required to distinguish the injected content without distrusting the entire result.

Tripwire is a research prototype and reproducibility artifact. It is not affiliated with Fortra's Tripwire products. GitHub issues may be used to report reproducibility defects; no support or release cadence is promised.

## Reproduce the committed result

Prerequisites: Python 3.11–3.13 and [uv](https://docs.astral.sh/uv/getting-started/installation/).

```console
uv sync --frozen
uv run tripwire verify
Expand Down Expand Up @@ -57,7 +61,7 @@ def render_readme(result: dict[str, Any]) -> str:

- [AgentDojo](https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html) supplies the agent tasks, attacks, benchmark outcomes and source traces used by the committed corpus. Tripwire adds a sanitised replay corpus and detection-oriented measurement rather than a new attack benchmark.
- [Agent Threat Rules (ATR)](https://github.com/Agent-Threat-Rule/agent-threat-rules) standardises portable detections over agent events and content fields.
- Cross-event agent detection is established prior art, not Tripwire's novelty. [AgentSigma](https://github.com/cveye/agentsigma) applies Sigma-compatible runtime detection at the tool-call layer; [AgentShield's sigma-ai](https://github.com/agentshield-ai/sigma-ai) documents temporal correlation for sequential agent events; [AIDR Sigma](https://github.com/netzilo/aidr-sigma) describes a persistent per-session behaviour graph for multi-step patterns; and [RSigma](https://github.com/timescale/rsigma) provides in-process stateful correlation and OTLP ingest. RSigma is also a candidate substrate for any future streaming backend and should be evaluated before Tripwire implements those capabilities itself.
- Cross-event agent detection is established prior art, not Tripwire's novelty. [AgentSigma](https://medium.com/@dasgupta.pratip/agentsigma-v0-1-building-the-detection-framework-i-wrote-about-3f6e14c194c5) applies Sigma-compatible runtime detection at the tool-call layer; [AgentShield's sigma-ai](https://github.com/agentshield-ai/sigma-ai) documents temporal correlation for sequential agent events; [AIDR Sigma](https://github.com/netzilo/aidr-sigma) describes a persistent per-session behaviour graph for multi-step patterns; and [RSigma](https://github.com/timescale/rsigma) provides in-process stateful correlation and OTLP ingest. RSigma is also a candidate substrate for any future streaming backend and should be evaluated before Tripwire implements those capabilities itself.
- Tripwire's contribution here is the measurement: a sanitised labelled AgentDojo corpus, reproducible metrics, a frozen holdout and the provenance-granularity finding that resulted from error analysis.
- [Out-of-band agent defenses](https://arxiv.org/abs/2606.26479) use deterministic policies, capabilities or information-flow labels outside the model. Tripwire's finding identifies a boundary condition for designs that collapse provenance to tool identity or whole tool results: a trusted tool can return attacker-controlled fields.

Expand Down
3 changes: 3 additions & 0 deletions tests/test_m1_acceptance.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,10 +380,13 @@ def test_public_report_preserves_evaluation_discipline() -> None:
assert "reports recall only" in readme
assert "## Related work" in readme
assert "Cross-event agent detection is established prior art" in readme
assert "medium.com/@dasgupta.pratip/agentsigma-v0-1" in readme
assert "github.com/cveye/agentsigma" not in readme
assert "github.com/agentshield-ai/sigma-ai" in readme
assert "github.com/netzilo/aidr-sigma" in readme
assert "github.com/timescale/rsigma" in readme
assert "Tripwire's contribution here is the measurement" in readme
assert "It is not affiliated with Fortra's Tripwire products" in readme
assert any("holdout is spent" in limitation for limitation in result["limitations"])


Expand Down
18 changes: 14 additions & 4 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading