Skip to content

ci(py-315): withhold cp315 wheels from the PyPI upload - #19880

Draft
vlad-scherbich wants to merge 2 commits into
mainfrom
vlad/no-cp315-to-pypi
Draft

ci(py-315): withhold cp315 wheels from the PyPI upload#19880
vlad-scherbich wants to merge 2 commits into
mainfrom
vlad/no-cp315-to-pypi

Conversation

@vlad-scherbich

@vlad-scherbich vlad-scherbich commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Description

release_pypi_prod uploads with the unfiltered glob pywheels/ddtrace-*, so every cp315 wheel the pipeline builds is published to PyPI. This replaces the glob with a POSIX-shell loop that skips *cp315*, so cp315 wheels stop reaching PyPI. The sdist and every other wheel are unaffected.

This is not hypothetical. cp315 wheels have been built since cp315-cp315 joined the matrix in .gitlab/package.yml (allow_failure, per the TODO citing #17849), and 122 of them are on PyPI today, across all 31 releases from 4.11.0rc1 through 4.14.0. They are ABI-broken rather than merely untested: CPython inserted uintptr_t last_profiled_frame_seq into PyThreadState immediately above c_profilefunc between the v3.15.0b1 and v3.15.0b4 tags, shifting every subsequent field, and these wheels were compiled against a pre-shift beta.

requires-python = ">=3.9,<3.15" is not a sufficient guard on its own. pip honours it — pip 26.2.1 on a 3.15.0a6 environment refuses ddtrace==4.14.0rc4 outright — but uv 0.10.9 resolves and installs the cp315 wheel into that same environment with no error or warning, even though the wheel's own METADATA says Requires-Python: <3.15,>=3.9. Withholding the upload is the guard that holds for every installer.

Deliberately unchanged: the S3 upload (.upload_wheels_base) and the private prerelease index (.patch_wheel_versions_base, datadog/pypi-private-prereleases) still receive cp315, so internal distribution and custom customer builds keep working. Only PyPI is filtered.

twine check --strict moves from pywheels/* onto the same filtered list, so a malformed cp315 wheel from an allow_failure build cannot fail a release whose supported wheels are all fine. That loses no coverage: release_pypi_prod depends on "ddtrace package", which carries no serverless artifacts, so the two globs were already the same set. The filter is marked TODO(py-315) and comes out when 3.15 becomes a supported target.

Testing

The loop is POSIX shell inside release_pypi_prod, which runs only on release tags, so it was exercised by hand against a fixture pywheels/ of cp313 and cp314 manylinux/musllinux wheels, cp315 manylinux/musllinux wheels, and an sdist. cp315 wheels are skipped and everything else retained. It fails closed: when filtering leaves nothing, the job exits 1 with [ERROR] no PyPI-eligible distributions found in pywheels/ -- refusing to upload, instead of letting twine fail with an argparse usage error that never says why the list was empty. The list is printed before uploading, so the job log records what was published.

Risks

Low, and bounded to the release job. cp315 wheels stop appearing on PyPI for new releases; nothing changes for supported interpreters.

Out of scope: the 122 already-published files, where yank vs. delete is a release-engineering decision (PEP 592 yanking may be per-release rather than per-file, and deletion is irreversible); making .gitlab/validate-ddtrace-package.py fail on unrecognised interpreter tags rather than warn, which is why its PYTHON_TAGS stopping at cp314 let the gate pass; and #19861, which separately enables cp315 manylinux wheel builds.

Additional Notes

changelog/no-changelog: CI-only, with no user-facing behaviour change on any supported interpreter. The title is ci( rather than fix/feat, so scripts/check-releasenotes honours the label.

The release_pypi job uploaded every artifact matching pywheels/ddtrace-*,
which includes the cp315 wheels that "build linux" produces for internal S3
and the private prerelease index. Nothing prevented a Python 3.15 artifact
from reaching PyPI other than the Requires-Python bound baked into the wheel
metadata, which is a property of the artifact rather than a release guard.

Filter cp315 out of the upload list and refuse to upload at all if the
resulting list is empty, so a broken filter fails the job instead of
silently uploading everything or nothing. twine check --strict still runs
over the whole of pywheels/, so cp315 artifacts remain validated. The S3
and adms prerelease paths are unchanged: those are the intended consumers.
@vlad-scherbich vlad-scherbich added the changelog/no-changelog A changelog entry is not required for this PR. label Aug 27, 2026
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 6 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 42ff65d | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Aug 27, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-27 11:39:00

Comparing candidate commit 42ff65d in PR branch vlad/no-cp315-to-pypi with baseline commit 741fda8 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 5 performance regressions! Performance is the same for 581 metrics, 10 unstable metrics, 2 known flaky benchmarks, 16 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+2.342µs; +2.444µs] or [+14.369%; +14.995%]

scenario:iastaspects-rstrip_aspect

  • 🟥 execution_time [+63.088µs; +70.274µs] or [+15.995%; +17.817%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+114.716µs; +120.621µs] or [+27.640%; +29.063%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+544.411ns; +587.189ns] or [+20.346%; +21.945%]

scenario:tracer-small

  • 🟥 execution_time [+32.311µs; +35.241µs] or [+10.063%; +10.976%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-748.391ns; +737.185ns] or [-6.751%; +6.650%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-31.996ns; +34.352ns] or [-5.269%; +5.657%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1619.133ns; +1681.804ns] or [-9.528%; +9.897%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1377.583ns; +1161.934ns] or [-10.485%; +8.844%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-324.284ns; +324.431ns] or [-8.885%; +8.889%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-249.964ns; +254.469ns] or [-8.639%; +8.795%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-82.240ns; +62.931ns] or [-6.823%; +5.221%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-3955.572ns; +4035.013ns] or [-9.590%; +9.783%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-767.404ns; +772.559ns] or [-9.362%; +9.425%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-62.788µs; +54.960µs] or [-10.052%; +8.798%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:iastaspects-swapcase_aspect

  • 🟥 execution_time [+40.364µs; +47.057µs] or [+14.695%; +17.132%]

scenario:span-start

  • 🟥 execution_time [+1.160ms; +1.333ms] or [+8.183%; +9.403%]

Known flaky benchmarks without significant changes:

  • scenario:errortrackingflasksqli-baseline
  • scenario:flasksimple-iast-get
  • scenario:iastaspects-casefold_aspect
  • scenario:iastaspects-casefold_noaspect
  • scenario:iastaspects-index_aspect
  • scenario:iastaspects-ljust_noaspect
  • scenario:iastaspects-lower_aspect
  • scenario:iastaspects-replace_aspect
  • scenario:iastaspects-title_noaspect
  • scenario:iastaspects-translate_aspect
  • scenario:iastaspects-translate_noaspect
  • scenario:iastaspects-upper_noaspect
  • scenario:packagespackageforrootmodulemapping-cache_off
  • scenario:packagespackageforrootmodulemapping-cache_on
  • scenario:sethttpmeta-all-enabled
  • scenario:telemetryaddmetric-record-100-metrics

…is uploaded

The comment used DEV(py-315), a marker form this repo has never used; main has
three TODO(py-315) anchors and only a bare "# DEV:" convention.

Build the upload list before "twine check" so both steps see the same
distributions. --strict promotes warnings to errors, so checking the unfiltered
directory let a malformed cp315 wheel from an allow_failure build fail a release
whose supported wheels were all fine. release_pypi_prod depends on "ddtrace
package", which carries no serverless artifacts, so the narrowed check loses no
coverage beyond cp315.
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.gitlab/release.yml                                                     @DataDog/python-guild @DataDog/apm-core-python

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 240 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 240 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=135)
ddtrace.internal.opentelemetry.context -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=133)
ddtrace.llmobs._integrations.openai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.appsec._contrib.flask -×-> ddtrace.trace  (product:appsec -> product:tracing, score=133)
ddtrace.internal.openfeature._span_enrichment -×-> ddtrace.trace  (product:openfeature -> product:tracing, score=133)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the GitLab release pipeline so release_pypi_prod no longer uploads cp315 wheels to PyPI, preventing publication of wheels for an unsupported (and potentially ABI-incompatible) Python 3.15 target while leaving internal distribution paths unchanged.

Changes:

  • Builds an explicit upload list from pywheels/ddtrace-* and filters out any *cp315* artifacts before both twine check --strict and twine upload.
  • Fails the release job early with a clear error if filtering results in an empty upload set, and logs the exact distributions that will be published.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants