ci(py-315): unblock cp315 wheel builds on the stock manylinux image - #19865
Closed
vlad-scherbich wants to merge 9 commits into
Closed
ci(py-315): unblock cp315 wheel builds on the stock manylinux image#19865vlad-scherbich wants to merge 9 commits into
vlad-scherbich wants to merge 9 commits into
Conversation
Root cause: stale manylinux2014 derived images (d2b8243 / May 2026) fail cp315 wheel builds while rebuilt musllinux images (233089d) succeed, leaving S3 with musllinux-only cp315 wheels that prof-python-3.15 glibc images cannot install. - Route cp315 manylinux2014 builds through the pypa 2026.05.13-1 mirror until derived manylinux images are rebuilt - Split the build-linux matrix so cp315 skips the stale derived manylinux tag - Lift requires-python to <3.16 and bytecode>=0.18.1 for 3.15 (PyPI 3.15 classifier deferred until the package is PyPI-ready) - Add cp315 to ddtrace package validation and drop cp315 allow_failure
The dd mirror pypa tag 2026.05.13-1 still ships cp315 as 3.15.0b1 and the native build SIGSEGVs; use quay.io 2026.08.24-1 until the mirror catches up. Restore musllinux cp315 allow_failure, sync requirements.csv, and add a reno.
GitLab third-party-registry policy blocks quay.io. Revert to the mirrored pypa manylinux2014 2026.05.13-1 image and rebuild cp315 as 3.15.0rc1 in before_script because the mirror still ships 3.15.0b1, which SIGSEGVs during the native wheel build. Drop ensure-cp315-rc1.sh once DataDog/images mirrors >= 2026.08.04-1.
The runtime manylinux2014 image ships build-cpython.sh but not cosign, which is only copied into pypa's intermediate build_cpython stage.
finalize-one.sh bootstraps pip for the newly built interpreter by running /usr/local/bin/cpython3.15, a shim that execs /opt/python/cp315-cp315/bin/python, and then ends with a plain "ln -s" into /opt/python. Deleting that directory up front broke the pip bootstrap; leaving it in place would break the symlink step. Free only the /opt/python name, repoint the shim at the stale prefix for the bootstrap, and assert the result is rc/final so a wheel can never be built against 3.15.0b1. Also install the extra manylinux build deps best-effort: EPEL-7 aarch64 has no libzstd-devel, and zstd-sys vendors its own zstd source, so a missing package must not abort the job. Includes temporary stdout/NDJSON instrumentation for CI diagnosis.
"ddtrace package" demanded cp315 macOS and Windows wheels that no job produces, because cp315 went into PYTHON_TAGS and that list is a full cross-product over BASE_PLATFORMS. Require cp315 only on manylinux, and tolerate musllinux cp315 either way since it is still allow_failure: requiring it would have let a best-effort build fail the pipeline. Also drop the debug scaffolding this investigation left behind. setup.py is back to its upstream state, and the cp315 rebuild scripts keep the comments explaining the missing headers in the runtime image without the NDJSON probes. The now-unreferenced AARCH64_IMAGES and X86_64_IMAGES anchors go too.
Experiment to check whether ensure-cp315-rc1.sh was ever necessary. Its
premise was that the mirrored pypa 2026.05.13-1 base ships 3.15.0b1 and
that b1 SIGSEGVs the native wheel build. The first half holds; the second
looks contaminated.
DataDog/images dd-trace-py/{manylinux2014,musllinux_1_2}_{x86_64,aarch64}
Dockerfiles, read at the SHAs baked into the image tags in use (d2b8243
and 233089d), are all FROM pypa 2026.05.13-1. Upstream added CPython 3.15
on 2026-05-07 as b1 and bumped to b2 on 2026-06-03, so every one of those
images ships b1 -- and musllinux cp315 wheels build on it today. Meanwhile
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 was set from the first commit of
this branch until the most recent one, spanning every commit that reported
the manylinux SIGSEGV. Forcing abi3 is what removed PyContextVar_* on
musl; on glibc it plausibly crashed instead of failing to link.
So: restore package.yml to its state on main, which already builds cp315
on manylinux and musllinux with allow_failure, and drop both helper
scripts. install-manylinux-build-deps.sh is redundant regardless -- the
derived Dockerfiles already yum install autoconf, automake, libtool,
openssl-devel, openssl-static and (on x86_64) libzstd.
The validator requires no cp315 wheel here, since every cp315 build is
allow_failure, so "ddtrace package" cannot cascade and the signal comes
from the build job alone. A missing cp39-cp314 wheel still fails.
If cp315 manylinux2014 wheels appear, the 209 lines of rebuild machinery
were working around a self-inflicted crash and stay deleted. If it
SIGSEGVs, revert this commit.
|
vlad-scherbich
added
the
changelog/no-changelog
vlad-scherbich
changed the title
BenchmarksBenchmark execution time: 2026-08-26 01:15:45 Comparing candidate commit 4d6ed15 in PR branch Found 0 performance improvements and 8 performance regressions! Performance is the same for 577 metrics, 10 unstable metrics, 1 known flaky benchmarks, 17 flaky benchmarks without significant changes.
|
Contributor
Author
|
Closing in favor of #19861 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
mainalready builds cp315 wheels —cp315-cp315is in.PYTHON_TAGSunderallow_failure: true— but the jobs can't produce anything because the packagemetadata refuses 3.15. This is the smallest change that lets them run.
requires-python:>=3.9,<3.15→>=3.9,<3.16. The PyPI 3.15 classifier isdeliberately not added; that waits on chore: wrapping context support for Python 3.15 #17849.
bytecode>=0.18.1,<1forpython_version>='3.15', with the 3.13/3.14 rowsbounded below 3.15. Applied to
pyproject.tomland bothrequirements.csvmirrors; the
lintdependency group's pin is split the same way souv run --only-group lintstill resolves on 3.15.validate-ddtrace-package.py: cp315 joinsPYTHON_TAGSbut is required on noplatform, and cp315 Linux wheels are absorbed as optional rather than flagged
unexpected. Every cp315 build is
allow_failure, so requiring one would let abest-effort failure hard-fail
ddtrace packageand mask the signal.No cp315 build machinery is added;
.gitlab/package.ymlis byte-identical tomain.Why (context, not part of this diff)
Does building cp315 manylinux2014 wheels need an in-job CPython rebuild? #19861 is
the arm that assumes yes, via a 184-line
ensure-cp315-rc1.sh. This is the arm thatassumes no.
Verified: the image's
manylinux-interpreterscannot upgrade an already-presentCPython, and
python_versions.jsonhas no CPython at all, so a rebuild is the onlyin-image option. All four derived images are
FROMpypa2026.05.13-1, which ships3.15.0b1 — and musllinux cp315 wheels build on that same b1 base today. The mirror
has nothing newer than
2026.05.13-1, so a newer base isn't available without aDataDog/imageschange.Inferred, and the point of this PR: the manylinux SIGSEGV that motivated the rebuild
was observed while
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1was set — introduced in9ce758c134, unset in55d77a285d, a window containing both crash reports(
589c9eb129,ac417a9f41). Forcing abi3 is what compiled outPyContextVar_*andbroke the musl link; a glibc crash is a plausible second symptom of the same cause.
Plausible, not proven. This PR decides it.
Reading the result
Watch
build linuxforPYTHON_TAG: cp315-cp315. Success means #19861's rebuildmachinery was working around a self-inflicted crash. Another SIGSEGV means the b1
theory was right, and #19861 becomes the path forward.
ddtrace packageshould staygreen either way.
Validator change verified locally: no cp315 → pass; all four cp315 Linux wheels →
pass with zero unexpected warnings; a cp314 wheel removed → fails with
Missing wheels: 1, so the tolerance doesn't mask real regressions.Notes
requires-pythonis the only change reaching beyond CI: a built package installsunder 3.15 while the stack is incomplete, though nothing advertises 3.15. The
bytecodemarkers are mutually exclusive, so 3.9–3.14 resolution is unchanged.This branch shares its first seven commits with #19861 and then reverts the rebuild
machinery, so the Commits tab shows
ensure-cp315-rc1.shadded and deleted. Reviewgit diff main...HEAD, and squash-merge.