Add an external-content prompt-injection sentinel

[aebrer]

Summary

Add an optional External Content Sentinel that inspects untrusted tool output for likely prompt injection before that content enters the parent model context. The first increment should cover web_fetch and web_search, reuse the Dispatch Arbiter's bounded headless/tool-less call pattern where appropriate, and be explicitly presented as detection and defense-in-depth rather than a security boundary.

The immediate goal is visibility: surface suspicious content loudly enough that users and the parent agent can react, while collecting evidence about false positives, false negatives, latency, and useful policy boundaries before attempting automatic enforcement.

Current behavior

• web_search returns externally controlled titles, URLs, and snippets directly to the parent context.
• web_fetch fetches a page, converts HTML to extracted text, truncates it, and returns that text directly to the parent context.
• Extension tool_result handlers can inspect and modify these results before final tool-result message emission, but there is no built-in prompt-injection classification or external-content provenance marker.
• The main model is expected to distinguish trusted instructions from untrusted content in a shared token stream. Existing secret scrubbing, sensitive-file guards, SSRF checks, and nested-context trust controls address adjacent risks but do not detect indirect prompt injection in ordinary tool output.

Proposed behavior

Introduce a global-only, opt-in External Content Sentinel with progressive operating modes:

1. Observe — classify external content and persist safe decision metadata, without changing the result or interrupting the user.
2. Warn — additionally show a host-generated UI warning and add a host-generated warning marker to the tool result.
3. Quarantine — withhold suspicious content from the parent until the user explicitly releases it.
4. Gate — treat suspicious external content as taint and require independent approval before subsequent consequential or outbound actions.

Only the first increment needs to ship initially. It should establish the classification contract, observability, configuration, tests, and realistic evaluation fixtures without claiming reliable prevention.

Recommended first increment

Implement observe and warn modes for built-in web_fetch and web_search results:

• Use a configured exact guard model in a bounded, headless, tool-less call.
• Pass the user intent, source URL/query, tool identity, and the exact text that would otherwise enter the parent context.
• Require strict host-validated structured output with a small fixed verdict schema.
• Keep raw guard prompts, responses, and reasoning out of parent/child transcripts and normal persisted context.
• Persist or emit only safe host-generated metadata: tool call ID, source, guard model, verdict, timestamp, content hash, operating mode, and safe failure code.
• Secret-scrub content before remote guard inference and secret-scrub surfaced errors.
• In warn mode, render the warning from host-owned strings rather than forwarding guard-generated prose or quoted payloads into the parent context.
• Make failure behavior explicit: observe/warn may fail open only with a loud diagnostic; future quarantine/gate modes must fail closed.
• Add regression fixtures including overt injections, obfuscated instructions, benign security documentation, prompt-engineering examples, shell-command documentation, and the UA-cloaked TCRF-style payload pattern.

A temporary extension using tool_result is a viable prototype and measurement harness. A first-class implementation may be preferable once the classification contract is validated.

Acceptance criteria for the first increment

• A global-only setting enables/disables the sentinel and selects an exact available guard model.
• The setting exposes at least observe and warn modes and cannot be enabled or weakened by project settings.
• Every successful non-guidance web_fetch and populated web_search result is classified before entering the parent context.
• The guard has no tools and receives bounded, secret-scrubbed input.
• Guard output is strictly schema-validated; malformed output, timeout, auth, provider, and unsupported-model failures are surfaced loudly.
• Warn mode produces a clear TUI/dashboard/RPC-visible security event and a host-generated parent-context marker without reproducing attacker-controlled instructions.
• Observe mode does not modify the parent-visible tool content.
• Safe decision metadata is available in session/RPC observability without persisting raw guard prompts, responses, or reasoning.
• Tests cover clean results, suspicious results, malformed verdicts, provider failures, truncation, cached fetches, parallel tool execution, and secret scrubbing.
• Documentation states that detection is probabilistic, web-only coverage is incomplete, and the feature is not authorization or sandboxing.
• Root README, coding-agent README, settings documentation, and relevant RPC/JSON/dashboard documentation remain consistent if the feature is first-class rather than extension-only.

Technical notes

Relevant implementation points

• packages/coding-agent/src/core/tools/web.ts
  • web_search formats externally controlled search results.
  • web_fetch currently calls stripHtmlToText() before the result reaches extension tool_result hooks.
• packages/coding-agent/src/core/dispatch-arbiter.ts
  • Provides a useful precedent for global-only configuration, bounded packages, strict JSON, no-tools inference, secret scrubbing, timeout handling, host validation, safe observability, and loud failures.
• packages/coding-agent/docs/extensions.md
  • Documents that tool_result runs before final result emission and can modify content, making it suitable for an extension prototype.

Raw HTML versus extracted text

The parent currently sees extracted text, so scanning that exact text covers the immediate dreb prompt-injection surface. However, BrowseSafe is trained on raw HTML and reports degraded performance on extracted text. Supporting that model as designed may require a pre-extraction hook in web.ts, or passing both raw HTML and final extracted text to a core sentinel while ensuring raw content never leaks into normal context or persistence.

Parallel execution

Sibling tool calls from one assistant message are proposed before their results execute. Therefore content returned by a web tool cannot influence sibling calls from that same assistant message; a verdict is available before the next model turn can propose an injection-influenced action. This gives a clean future interception point for gate mode.

External actions are broader than writes

Future gate mode must not classify only edit, write, and destructive shell commands as risky. A compromised agent can exfiltrate through a generated web_search query, web_fetch URL, gh invocation, network-capable shell command, subagent task, publish operation, or other outward communication.

Extensibility

Hard-coding only two built-in tool names is acceptable for the first experiment but not a complete trust model. A future design should let built-in, SDK, and extension tools declare external-content provenance and action effects so policy can cover curl, gh, MCP/custom tools, downloaded files, repository content, and other ingress paths.

Risks and limitations

• False negatives: adversaries only need one formulation the guard misses; a probabilistic verdict must not become authorization.
• False positives and alert fatigue: coding and security documentation frequently contains instruction-like text and destructive-command examples.
• Guard injection: the detector processes the same adversarial content and can itself be manipulated.
• Model correlation: using the same model family for the parent and guard may reproduce shared blind spots; configuration should permit architectural diversity.
• Latency and cost: unlike browser-agent designs that run a guard in parallel with action generation, a tool_result interceptor adds classification latency before the next parent turn.
• Long content: chunking and aggregation strategy affect both misses and false positives; boundary-spanning attacks need tests.
• Incomplete ingress coverage: guarding only built-in web tools leaves bash, gh, cloned repositories, local files, custom tools, and MCP responses untreated.
• Warning-channel weakness: a warning in the same model context as malicious content is useful telemetry, not isolation.
• Security wording: UI and docs must avoid claims such as “safe,” “blocked,” or “protected” unless deterministic enforcement justifies them.

Open questions

• Should the first implementation be an extension experiment or first-class core infrastructure?
• Should the verdict be binary, categorical, or include an untrusted confidence score? A confidence value must not be treated as calibrated authorization.
• Which model should be the documented default, if any? General LLM judges and specialized classifiers have materially different recall, context, latency, and deployment requirements.
• Should warn mode prepend a marker to content, attach typed metadata, inject a separate host message, or combine these mechanisms?
• How long should a future taint state persist: one model turn, until the next user message, until explicit clearance, or based on data-flow provenance?
• How should non-interactive modes behave when quarantine or approval is introduced?
• Should raw HTML be made available only to the sentinel, and how should it be bounded, cached, and excluded from session persistence?

Scope assessment

• Observe/warn prototype: medium. The extension API already provides the interception point, but model invocation, settings, structured verdicts, observability, and cross-mode UI need careful integration.
• First-class observe/warn feature: medium-to-large because it spans settings, core tool execution, session/RPC events, dashboard/TUI controls, tests, and documentation.
• Quarantine/gate with provenance-aware action policy: large architectural follow-up. It requires deterministic action classification, approval semantics, taint lifetime/data flow, and broad tool coverage.

Keep the initial issue implementation focused on trustworthy observability and empirical evaluation. Do not fold full CaMeL-style capability tracking or sandboxing into the first increment.

Research context

• WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agents — dedicated parallel guard plus action gateway and user override.
• BrowseSafe model and paper — released raw-HTML classifier with chunking guidance and published precision/recall tradeoffs.
• CaMeL: Defeating Prompt Injections by Design — capability and data-flow enforcement rather than model-only authorization.
• Design Patterns for Securing LLM Agents against Prompt Injections — argues that agents exposed to untrusted content must be structurally constrained before consequential actions.
• OWASP AI Agent Security Cheat Sheet — recommends separate validation calls as one defense layer while warning against model-only authorization.

[m-aebrer]

Superseded by #460 — the quarantine-boundary architecture replaces the classifier-based sentinel concept. Rationale from the author: "This boundary IS the sentinel... a possible followup issue would be to add some kind of lightweight classifier, but this isn't that, and those are adversarially attackable anyway and thus security theater IMO." If a lightweight classifier on retriever output is ever revisited, it should be filed as a new follow-up issue against the retriever chokepoint.