ci(py-315): unblock cp315 manylinux wheels by constraining Cython below 3.3 - #19861
Draft
vlad-scherbich wants to merge 14 commits into
Draft
ci(py-315): unblock cp315 manylinux wheels by constraining Cython below 3.3#19861vlad-scherbich wants to merge 14 commits into
vlad-scherbich wants to merge 14 commits into
Conversation
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 986086e | Docs | View more details | Give us feedback! |
vlad-scherbich
added
the
changelog/no-changelog
vlad-scherbich
added a commit
that referenced
this pull request
Aug 25, 2026
The dd mirror pypa tag 2026.05.13-1 still ships cp315 as 3.15.0b1 and the native build SIGSEGVs; use quay.io 2026.08.24-1 until the mirror catches up. Restore musllinux cp315 allow_failure, sync requirements.csv, and add a reno.
Codeowners resolved asResolved from the full PR diff against |
Dependency direction analysis
|
Circular import analysis
|
BenchmarksBenchmark execution time: 2026-08-26 22:33:19 Comparing candidate commit 986086e in PR branch Found 0 performance improvements and 5 performance regressions! Performance is the same for 580 metrics, 10 unstable metrics, 1 known flaky benchmarks, 17 flaky benchmarks without significant changes.
|
vlad-scherbich
force-pushed
the
vlad/py315-manylinux-wheels
branch
2 times, most recently
from
August 25, 2026 20:17
38d6d87 to
55d77a2
Compare
vlad-scherbich
changed the title
Root cause: stale manylinux2014 derived images (d2b8243 / May 2026) fail cp315 wheel builds while rebuilt musllinux images (233089d) succeed, leaving S3 with musllinux-only cp315 wheels that prof-python-3.15 glibc images cannot install. - Route cp315 manylinux2014 builds through the pypa 2026.05.13-1 mirror until derived manylinux images are rebuilt - Split the build-linux matrix so cp315 skips the stale derived manylinux tag - Lift requires-python to <3.16 and bytecode>=0.18.1 for 3.15 (PyPI 3.15 classifier deferred until the package is PyPI-ready) - Add cp315 to ddtrace package validation and drop cp315 allow_failure
The dd mirror pypa tag 2026.05.13-1 still ships cp315 as 3.15.0b1 and the native build SIGSEGVs; use quay.io 2026.08.24-1 until the mirror catches up. Restore musllinux cp315 allow_failure, sync requirements.csv, and add a reno.
GitLab third-party-registry policy blocks quay.io. Revert to the mirrored pypa manylinux2014 2026.05.13-1 image and rebuild cp315 as 3.15.0rc1 in before_script because the mirror still ships 3.15.0b1, which SIGSEGVs during the native wheel build. Drop ensure-cp315-rc1.sh once DataDog/images mirrors >= 2026.08.04-1.
The runtime manylinux2014 image ships build-cpython.sh but not cosign, which is only copied into pypa's intermediate build_cpython stage.
finalize-one.sh bootstraps pip for the newly built interpreter by running /usr/local/bin/cpython3.15, a shim that execs /opt/python/cp315-cp315/bin/python, and then ends with a plain "ln -s" into /opt/python. Deleting that directory up front broke the pip bootstrap; leaving it in place would break the symlink step. Free only the /opt/python name, repoint the shim at the stale prefix for the bootstrap, and assert the result is rc/final so a wheel can never be built against 3.15.0b1. Also install the extra manylinux build deps best-effort: EPEL-7 aarch64 has no libzstd-devel, and zstd-sys vendors its own zstd source, so a missing package must not abort the job. Includes temporary stdout/NDJSON instrumentation for CI diagnosis.
"ddtrace package" demanded cp315 macOS and Windows wheels that no job produces, because cp315 went into PYTHON_TAGS and that list is a full cross-product over BASE_PLATFORMS. Require cp315 only on manylinux, and tolerate musllinux cp315 either way since it is still allow_failure: requiring it would have let a best-effort build fail the pipeline. Also drop the debug scaffolding this investigation left behind. setup.py is back to its upstream state, and the cp315 rebuild scripts keep the comments explaining the missing headers in the runtime image without the NDJSON probes. The now-unreferenced AARCH64_IMAGES and X86_64_IMAGES anchors go too.
openssl-devel and openssl-static were never doing anything here. manylinux's own install-build-packages.sh already lists openssl-devel in its yum COMPILE_DEPS, and build-openssl.sh then calls manylinux_pkg_remove openssl-devel before building OpenSSL into /opt/_internal/openssl-3.5 -- so we installed it twice and it was uninstalled either way. Nothing links the system openssl. src/native/Cargo.lock has no openssl-sys or native-tls; TLS is rustls with the ring backend, and the only openssl-named crate is openssl-probe, which is pure Rust CA-path discovery. CPython links the image's own /opt/_internal/openssl-*, which ensure-cp315-rc1.sh restores headers for explicitly. Split from the autotools/libzstd removal so a revert can be partial.
With openssl gone the script installed autoconf, automake, libtool and libzstd, and the image already provides all of them. manylinux builds autoconf, automake and libtool from source at image build time and copies the result into / (install-autoconf.sh, install-automake.sh and install-libtool.sh all end in `cp -rlf /manylinux-rootfs/* /`), so the yum packages were shadowed by newer versions already on PATH. libzstd is not needed either: the dd-trace-py manylinux2014_aarch64 Dockerfile never installed it, and aarch64 manylinux wheels have been building green all along. zstd reaches the native crate transitively via zstd-sys, which compiles its own vendored C source. Epistemic note: the openssl removal in the previous commit is verified from upstream script contents, and the autotools claim is verified from those install scripts. libzstd resting on "aarch64 was always green" is the weakest link here, so if this commit is what breaks CI, libzstd is the first thing to look at.
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
3.3.0 is the first *stable* release with compiled cp315 wheels; 3.3.0b1 shipped them too, but uv skips pre-releases by default. Also attribute the segfault to the wheel being built against the 3.15.0rc1 PyThreadState layout rather than implying 3.15.0b1 is broken, scope the pure-Python fallback to the manylinux2014 images it actually applies to, and name the pypa manylinux tag that first ships 3.15.0rc1 so the exit condition is checkable.
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1 (pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so rather than letting rc1 read as the version that introduced it. The wheel is still described as built against the rc1 layout, and the exit condition stays at >= 3.15.0rc1 deliberately. Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
vlad-scherbich
force-pushed
the
vlad/py315-manylinux-wheels
branch
from
August 26, 2026 21:40
dfc217e to
a0e7c01
Compare
requires-python widens from <3.15 to <3.16, which ships in published package metadata: pip on Python 3.15 stops refusing the install and starts attempting a source build. Add a release note so that lands with notice rather than as a silent change in failure mode. Also correct the validator comment on musllinux cp315. The package.yml allow_failure rule keys on PYTHON_TAG alone, so it covers manylinux cp315 identically and cannot be the reason musllinux is optional; the real reason is a policy choice about which platforms must produce cp315 wheels. Make the Phase 4 wheel-count breakdown add up to the expected total, and record what the bytecode 0.18.1 floor actually is.
vlad-scherbich
removed
the
changelog/no-changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributes to #17816.
mainalready runscp315-cp315in the sharedbuild linuxmatrix on bothmanylinux and musllinux under
allow_failure, and the manylinux half has segfaulted since 2026-08-22, sono cp315 manylinux2014 wheel reaches the internal S3 index. This PR makes those already-existing jobs
succeed — it adds no build job and leaves
.gitlab/package.yml,allow_failureincluded, identical tomain. The substance is two lines in[build-system] requires:Cython 3.3.0's cp315 manylinux wheel is compiled against a
PyThreadStatelayout that gained a member in3.15.0b4 (python/cpython#151614); the mirror's
manylinux2014 images still ship 3.15.0b1, which lacks it, so that wheel segfaults the moment
setup.pyimports
Cython.Build. No Cython below 3.3 publishes acp315-tagged wheel, so 3.15 resolves thepure-Python
py3-none-anywheel instead, which has no ABI to mismatch. This is a wheel/interpretermismatch, not a CPython defect: b1 is not broken and rc1 fixes no bug. musllinux was never affected,
resolving Cython's limited-API
cp39-abi3-musllinux_1_2_*wheel, which cannot seePyThreadStatemembers.Also in the diff: a
bytecode>=0.18.1row for 3.15 withpython_version<'3.15'bounds on the 3.13/3.14rows, mirrored into both
requirements.csvfiles; andvalidate-ddtrace-package.pylearning about cp315without requiring it — every cp315 build leg is
allow_failure, whileddtrace packagegatesrelease.requires-pythonstays>=3.9,<3.15, so nothing here advertises 3.15 to users.Exit condition: drop the pin once the image mirror ships a cp315 interpreter >= 3.15.0rc1 — pypa
manylinux
2026.08.04-1is the first such tag, andDataDog/images#11355 is in flight for it. Bump theimage and
prof-correctness'spython:3.15.0b1pin together.Testing
All 8 cp315 build jobs green with manylinux wheels uploaded to S3, on the unmodified b1 image (logs show
Using CPython 3.15.0b1 interpreter, no in-job interpreter rebuild). Each job'stest_wheelstep installedits own cp315 wheel and ran
tests/smoke_test.pyagainst it, so that green is not vacuous.Risks
Low and scoped to 3.15 by the marker; cp39-cp314 still resolve
cython==3.3.0unchanged. Any future cp315build dependency that publishes a compiled manylinux wheel re-breaks this the same way.
Additional Notes
Out of scope: flipping cp315
allow_failure, now that a pipeline is green; and filtering cp315 out of.gitlab/release.yml's upload glob, which is why 4.13.0 and 4.14.0 carry cp315 wheels on PyPI today.