test(aiguard): cover multi-turn sensitive data redaction scenarios

[avara1986]

Description

Jira: APPSEC-69387 (epic APPSEC-68806)
RFC: AI Guard Sensitive Data Redaction (v1.2) and AI Guard Sensitive Data Redaction. Backend (v1.1)

Test-only change. No behaviour change, no production code touched.

AI Guard attack analysis targets the latest logical message, but SDS scans every model-visible string in the complete provider-bound context. Because request redaction is copy-on-write, the customer-owned list may still hold an original sensitive value from an earlier turn, so a single /evaluate response can carry redaction_replacements for any mix of historical and latest messages, and every path is local to the array it was returned for.

The existing corpus and tests only covered single-response shapes where the replacements were adjacent or targeted the last message, so these requirements were untested:

• one response with replacements on non-adjacent historical and latest messages;
• a leak that lives only in the history, with a benign latest message;
• sensitive data in the system and assistant roles alongside the user's;
• anything that spans more than one /evaluate call.

This PR closes those gaps. Everything passed against the current ddtrace/aiguard/_redaction.py unchanged — this is coverage for previously untested RFC requirements, not a fix.

Corpus: apply/conversation/* in tests/aiguard/api/redaction_scenarios.json (73 → 80 cases)

Each case is driven twice by the existing runners: once through redact_messages, once end to end through evaluate().

Case	Shape
apply/conversation/history-and-latest	The RFC multi-turn example: [system, user(original SSN), assistant("Your SSN is <REDACTED>."), user(new email)] with replacements on messages[1] and messages[3]. Asserts the already-redacted assistant turn and the system prompt are untouched.
apply/conversation/every-role	System, user and assistant all carry sensitive data and are all replaced: no role is exempt.
apply/conversation/history-only	Only an earlier assistant answer leaked; the latest message is benign. Fails outright for any tracer that scopes redaction to the last message.
apply/conversation/several-turns	6 messages / 3 turns: system prompt, two earlier turns and the latest message replaced, benign turn left alone.
apply/conversation/tool-turn-in-history	A completed tool round trip stays in history: function.arguments and the role:tool result replaced alongside the latest message.
apply/conversation/content-parts-across-turns	Multimodal turn in history: the text part of the earlier message and the latest message replaced, image_url.url not.
mixed/conversation/history-applied-latest-skipped	A partial failure on the latest message must not drop the history's redaction.

Sequential tests in tests/aiguard/api/test_redaction.py

The corpus is single-response by construction, so these cover what it cannot express:

• test_history_is_redacted_again_on_every_turn — two consecutive evaluate() calls on the same customer-owned list. Asserts the caller's list still holds the original SSN after turn 1 (copy-on-write), that turn 2's outgoing payload still carries the original history (the service needs the raw text to compute a replacement), and that turn 2's result and meta struct redact both the historical SSN and the new email.
• test_paths_are_not_reused_across_turns — the application rebuilds the context so the sensitive message moves to index 0 and a benign one takes over index 1, the index turn 1 redacted. Asserts index 1 is not touched. Regression guard for the RFC's "MUST NOT cache or reuse message indexes"; it holds today only because redact_messages is stateless.
• test_redacted_is_reported_per_turn — ai_guard.redacted and the redacted telemetry tag go true then false across turns, so a redacting turn does not mark the next one.

Testing

scripts/run-tests --venv f63a4f0 (aiguard::ai_guard_api, Python 3.12): 184 passed. The 17 tests matching the new work (14 corpus runs + 3 sequential tests) were also run in isolation and pass.

scripts/lint fmt and scripts/lint spelling are clean on both files. scripts/lint typing reports one pre-existing no-any-return in tests/aiguard/utils.py:27, which this PR does not touch.

Risks

None. Test-only, additive: no existing case or assertion was modified, and no production code was touched.

Additional Notes

• No release note: test-only, so this carries changelog/no-changelog.
• Follow-up, not in this PR: the equivalent scenarios are still missing from system-tests. DataDog/system-tests#7385 has REDACT_MIXED (historical + latest at messages[0]/messages[2]), but no non-adjacent messages[1]+messages[3] case, no already-redacted-assistant-in-history shape, and no sequential turn test — its weblog endpoint takes one message list per POST. Porting history-and-latest and history-only over needs a cassette regeneration via utils/scripts/gen_redaction_cassettes.py.
• The two corpora are hand-maintained in parallel (80 cases here keyed apply/…/skip/… with a redaction_replacements field; 25 there keyed REDACT_*/SKIP_* with a replacements field). Worth deciding whether one should be generated from the other to guarantee cross-language parity.

🤖 Generated with Claude Code

[datadog-datadog-prod-us1]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3d4ceeb | Docs | View more details | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-08-26 13:21:21

Comparing candidate commit 2e2802d in PR branch avara1986/aiguard-redaction-multi-turn-tests with baseline commit 2803c38 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 8 performance regressions! Performance is the same for 576 metrics, 10 unstable metrics, 2 known flaky benchmarks, 16 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.280µs; +2.447µs] or [+12.214%; +13.107%]

scenario:iastaspects-add_aspect

• 🟥 execution_time [+16.478µs; +19.662µs] or [+16.400%; +19.570%]

scenario:iastaspects-join_aspect

• 🟥 execution_time [+45.584µs; +49.988µs] or [+21.354%; +23.416%]

scenario:iastaspects-title_aspect

• 🟥 execution_time [+68.882µs; +73.374µs] or [+25.931%; +27.621%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+133.610µs; +141.981µs] or [+32.623%; +34.667%]

scenario:iastaspectssplit-rsplit_aspect

• 🟥 execution_time [+19.821µs; +24.448µs] or [+13.922%; +17.172%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+421.499ns; +467.278ns] or [+15.670%; +17.372%]

scenario:tracer-small

• 🟥 execution_time [+26.571µs; +28.343µs] or [+8.119%; +8.660%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-659.115ns; +825.985ns] or [-5.980%; +7.494%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-34.292ns; +31.823ns] or [-5.599%; +5.196%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1672.629ns; +1655.052ns] or [-9.776%; +9.674%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1104.192ns; +1356.810ns] or [-8.603%; +10.571%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-326.418ns; +331.222ns] or [-8.870%; +9.001%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-278.563ns; +230.038ns] or [-9.494%; +7.840%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-90.819ns; +55.636ns] or [-7.897%; +4.838%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-3489.960ns; +4423.925ns] or [-8.678%; +11.000%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-775.181ns; +755.906ns] or [-9.611%; +9.372%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-57.213µs; +61.337µs] or [-9.151%; +9.811%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:iastaspects-ljust_noaspect

• 🟥 execution_time [+58.299µs; +63.601µs] or [+20.262%; +22.104%]

scenario:span-start

• 🟥 execution_time [+1.391ms; +1.537ms] or [+9.436%; +10.425%]

Known flaky benchmarks without significant changes:

• scenario:errortrackingflasksqli-baseline
• scenario:flasksimple-iast-get
• scenario:iastaspects-casefold_aspect
• scenario:iastaspects-casefold_noaspect
• scenario:iastaspects-index_aspect
• scenario:iastaspects-lower_aspect
• scenario:iastaspects-replace_aspect
• scenario:iastaspects-swapcase_aspect
• scenario:iastaspects-title_noaspect
• scenario:iastaspects-translate_aspect
• scenario:iastaspects-translate_noaspect
• scenario:iastaspects-upper_noaspect
• scenario:packagespackageforrootmodulemapping-cache_off
• scenario:packagespackageforrootmodulemapping-cache_on
• scenario:sethttpmeta-all-enabled
• scenario:telemetryaddmetric-record-100-metrics

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

tests/aiguard/api/redaction_scenarios.json                              @DataDog/asm-python
tests/aiguard/api/test_redaction.py                                     @DataDog/asm-python

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 240 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 240 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=135)
ddtrace.aiguard._api_client -×-> ddtrace.trace  (product:aiguard -> product:tracing, score=133)
ddtrace.internal.ci_visibility.api._base -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=133)
ddtrace.llmobs._integrations.base -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.appsec._listeners -×-> ddtrace.trace  (product:appsec -> product:tracing, score=133)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json