feat(traces): introduce DD_TRACE_FILTERING_RULES by bwoebi · Pull Request #19851 · DataDog/dd-trace-py

bwoebi · 2026-08-25T12:22:02Z

This works just like sampling rules, serving as a way to fully drop a trace, but does not sample it.

cit-pr-commenter-54b7da · 2026-08-25T12:22:19Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/_trace/filter_rule.py                                           @DataDog/apm-sdk-capabilities-python
ddtrace/_trace/processor/__init__.py                                    @DataDog/apm-sdk-capabilities-python
ddtrace/_trace/sampling_rule.py                                         @DataDog/apm-sdk-capabilities-python
ddtrace/internal/sampling.py                                            @DataDog/apm-sdk-capabilities-python
ddtrace/internal/settings/_config.py                                    @DataDog/python-guild @DataDog/apm-sdk-capabilities-python
ddtrace/internal/settings/_supported_configurations.py                  @DataDog/apm-python
docs/configuration.rst                                                  @DataDog/python-guild
releasenotes/notes/add-trace-filtering-rules-73d81c9f2e662781.yaml      @DataDog/apm-python
supported-configurations.json                                           @DataDog/apm-python
tests/tracer/test_filter_rule.py                                        @DataDog/apm-sdk-capabilities-python
tests/tracer/test_processors.py                                         @DataDog/apm-sdk-capabilities-python
tests/utils.py                                                          @DataDog/python-guild

cit-pr-commenter-54b7da · 2026-08-25T12:22:35Z

Dependency direction analysis

📈 Existing violations got worse

31 pre-existing violation(s) increased in severity (e.g. their target became more depended-on, or got pulled into an import cycle), though the edge itself isn't new:

Show violations that got worse (showing 5 of 31 highest severity)

ddtrace.internal.opentelemetry.context -×-> ddtrace._trace.span  (product:opentelemetry -> product:tracing, score=56, +1 vs base)
ddtrace.internal.span_bus -×-> ddtrace._trace.span  (internal-core -> product:tracing, score=58, +1 vs base)
ddtrace.internal.ci_visibility.context -×-> ddtrace._trace.span  (product:ci_visibility -> product:tracing, score=56, +1 vs base)
ddtrace.appsec._handlers -×-> ddtrace._trace.span  (product:appsec -> product:tracing, score=56, +1 vs base)
ddtrace.llmobs._integrations.bedrock_agents -×-> ddtrace._trace.span  (product:llmobs -> product:tracing, score=56, +1 vs base)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

⚠️ Existing dependency direction violations

There are 240 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 240 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=135)
ddtrace.internal.test_visibility.api -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=133)
ddtrace.llmobs._integrations.bedrock -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.profiling.scheduler -×-> ddtrace.trace  (product:profiling -> product:tracing, score=133)
ddtrace.llmobs._integrations.vllm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

cit-pr-commenter-54b7da · 2026-08-25T12:22:41Z

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

datadog-prod-us1-5 · 2026-08-25T12:22:55Z

✨ Unblock PR with BitsAI

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 6 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-py | validate_supported_configurations_v2_local_file — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f47854f | Docs | View more details | Give us feedback!}

pr-commenter · 2026-08-25T12:56:49Z

Benchmarks

Benchmark execution time: 2026-08-27 16:12:44

Comparing candidate commit f47854f in PR branch bob/trace-filter with baseline commit 89fbfa5 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 6 performance regressions! Performance is the same for 580 metrics, 10 unstable metrics, 18 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-tracecontext_headers

🟥 execution_time [+2.473µs; +2.700µs] or [+7.814%; +8.531%]

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.452µs; +2.592µs] or [+15.085%; +15.950%]

scenario:iastaspects-repr_aspect

🟥 execution_time [+38.727µs; +45.237µs] or [+10.739%; +12.544%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+113.485µs; +119.613µs] or [+28.648%; +30.196%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+607.807ns; +645.391ns] or [+23.125%; +24.555%]

scenario:tracer-small

🟥 execution_time [+32.697µs; +34.480µs] or [+10.213%; +10.770%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

unstable execution_time [-694.567ns; +786.560ns] or [-6.293%; +7.127%]

scenario:coreapiscenario-core_dispatch_1_listener

unstable execution_time [-37.917ns; +28.135ns] or [-6.201%; +4.601%]

scenario:coreapiscenario-core_dispatch_50_listeners

unstable execution_time [-1712.886ns; +1586.363ns] or [-10.041%; +9.299%]

scenario:coreapiscenario-core_dispatch_exception_listeners

unstable execution_time [-1248.032ns; +1252.205ns] or [-9.592%; +9.624%]

scenario:coreapiscenario-core_dispatch_listeners

unstable execution_time [-354.374ns; +296.073ns] or [-9.647%; +8.060%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

unstable execution_time [-276.030ns; +240.366ns] or [-9.424%; +8.207%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

unstable execution_time [+8.144ns; +157.583ns] or [+0.719%; +13.912%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

unstable execution_time [-3805.093ns; +4128.608ns] or [-9.364%; +10.160%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

unstable execution_time [-641.923ns; +897.571ns] or [-7.988%; +11.170%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

unstable execution_time [-56.686µs; +61.950µs] or [-9.073%; +9.916%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

scenario:errortrackingflasksqli-baseline
scenario:flasksimple-iast-get
scenario:iastaspects-casefold_aspect
scenario:iastaspects-casefold_noaspect
scenario:iastaspects-index_aspect
scenario:iastaspects-ljust_noaspect
scenario:iastaspects-lower_aspect
scenario:iastaspects-replace_aspect
scenario:iastaspects-swapcase_aspect
scenario:iastaspects-title_noaspect
scenario:iastaspects-translate_aspect
scenario:iastaspects-translate_noaspect
scenario:iastaspects-upper_noaspect
scenario:packagespackageforrootmodulemapping-cache_off
scenario:packagespackageforrootmodulemapping-cache_on
scenario:sethttpmeta-all-enabled
scenario:span-start
scenario:telemetryaddmetric-record-100-metrics

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01a70924b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-27T13:44:55Z

+            rule_kwargs = dict(rule)
+            filter_rate = rule_kwargs.pop("filter_rate", 1.0)
+            filtering_rules.append(FilterRule(filter_rate=filter_rate, **rule_kwargs))
+    except (JSONDecodeError, ValueError, TypeError):


Handle malformed rule fields without aborting startup

When DD_TRACE_FILTERING_RULES contains valid JSON but a field has the wrong shape—for example, [{"tags":["x"]}]—SamplingRule.__init__ calls tags.items() and raises AttributeError, which this handler does not catch. Since SpanAggregator.__init__ parses these rules while the global tracer is constructed, such a configuration prevents ddtrace.trace from importing instead of logging and disabling the invalid rules.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-27T13:44:55Z

+    """
+    Definition of a filtering rule used to fully drop trace chunks matching a pattern.
+
+    Reuses :class:`SamplingRule`'s glob matching and probability handling.


Use plain prose in this private docstring

This module is private and is not rendered by Sphinx, but the new docstring uses the reStructuredText :class: role. The repository convention requires non-rendered private docstrings to use plain editor-readable prose without rST inline markup, so this should refer to SamplingRule directly.

AGENTS.md reference: AGENTS.md:L43-L50

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-27T13:44:55Z

+        if not trace or not self.filtering_rules:
+            return trace
+        chunk_root = trace[0]
+        matched_rule = _get_highest_precedence_rule_matching(chunk_root._local_root, self.filtering_rules)


Prevent partial flushes from bypassing final-root filters

With partial flushing enabled by default, a trace can export one or more child-span chunks while its local root is still unfinished, so matching here observes incomplete root metadata. For example, Django assigns the request's resolved resource later in ddtrace/contrib/internal/django/utils.py:237-241; if 300 child spans finish first, a rule for that resource passes the early chunk and only drops the final root chunk. The resulting trace is therefore still partially sent to the Agent and included in metrics despite the documented full-drop behavior.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-27T13:44:55Z

        # perf: Process spans outside of the span aggregator lock
        spans = finished
        for tp in chain(
+            [self.filtering_processor],


Route LLMObs events before discarding matching traces

When LLMObs is enabled and a filtering rule matches a trace containing an LLM span, this processor returns None and breaks the chain before self.llmobs_processor runs. LLMObsProcessor.process_trace in ddtrace/llmobs/_processor.py:47-97 is the sole owner that either leaves the event on a transmitted APM trace or enqueues it to the LLMObs writer; here the APM trace is discarded and the fallback enqueue never occurs, so the LLMObs event is lost entirely.

Useful? React with 👍 / 👎.

This works just like sampling rules, serving as a way to fully drop a trace, but does not sample it.

bwoebi force-pushed the bob/trace-filter branch from 2929906 to cf6fa36 Compare August 25, 2026 12:27

bwoebi marked this pull request as ready for review August 27, 2026 13:36

bwoebi requested review from a team as code owners August 27, 2026 13:36

bwoebi requested a review from mabdinur August 27, 2026 13:36

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

bwoebi changed the title ~~Introduce DD_TRACE_FILTERING_RULES~~ feat(traces): introduce DD_TRACE_FILTERING_RULES Aug 27, 2026

bwoebi added 3 commits August 27, 2026 17:37

Introduce DD_TRACE_FILTERING_RULES

70ea1f8

This works just like sampling rules, serving as a way to fully drop a trace, but does not sample it.

Add missing tests

4917dbd

Handle malformed filtering json

f47854f

bwoebi force-pushed the bob/trace-filter branch from 01a7092 to f47854f Compare August 27, 2026 15:37

Uh oh!

Conversation

bwoebi commented Aug 25, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

📈 Existing violations got worse

⚠️ Existing dependency direction violations

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

datadog-prod-us1-5 Bot commented Aug 25, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:httppropagationextract-tracecontext_headers

scenario:httppropagationinject-ids_only

scenario:iastaspects-repr_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

Known flaky benchmarks

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cit-pr-commenter-54b7da Bot commented Aug 25, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 25, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 25, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented Aug 25, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

pr-commenter Bot commented Aug 25, 2026 •

edited

Loading