From 978872914c09b2070ff42cae35357646b9b6f0d0 Mon Sep 17 00:00:00 2001 From: Ramakrishna Prabhu Date: Mon, 24 Aug 2026 13:51:23 -0500 Subject: [PATCH] ci: make a hanging pytest identify itself The 'pytest cuopt' step has been hitting its step time limit on 13.3.0 amd64 with 140 of 142 tests complete -- two tests start and never finish. When the outer 'timeout' fires, pytest is killed before it can report, and the conda path runs without -v, so the output is bare progress dots and the responsible tests cannot be identified. Raising the limit does not help: 30m and 45m both end at 140/142, so these are hangs rather than slow tests. Adds three diagnostics to the shared runner, so both the conda and wheel paths get them: - -v names each test as it is dispatched. - faulthandler_timeout dumps the stack of any test still running after FAULTHANDLER_TIMEOUT seconds (default 600), so a stuck test identifies itself while the step is still alive. - --durations=25 surfaces tests approaching the limit. None of these kill a test; they only make it visible. Verified against a deliberately hanging test under xdist: the test is named at dispatch and faulthandler prints its stack at the threshold. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Ramakrishna Prabhu --- ci/run_cuopt_pytests.sh | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/ci/run_cuopt_pytests.sh b/ci/run_cuopt_pytests.sh index c3b047c888..54961ba334 100755 --- a/ci/run_cuopt_pytests.sh +++ b/ci/run_cuopt_pytests.sh @@ -33,12 +33,23 @@ done export PYTHONPATH="${SCRIPT_DIR}/utils:${PYTHONPATH:-}" rc=0 +# A test that never returns takes the whole step down when the outer 'timeout' +# fires, and pytest is killed before it can say which test was running. -v names +# each test as it is dispatched, and faulthandler_timeout dumps the stack of any +# test still running after FAULTHANDLER_TIMEOUT seconds, so a stuck test +# identifies itself while the step is still alive. Neither kills the test; they +# only make it visible. +FAULTHANDLER_TIMEOUT=${FAULTHANDLER_TIMEOUT:-600} +PYTEST_DIAG_ARGS=(-v -o "faulthandler_timeout=${FAULTHANDLER_TIMEOUT}" --durations=25) + if [ "${IS_NIGHTLY}" = "nightly" ]; then - pytest -s --cache-clear --reruns 2 --reruns-delay 5 -p cuopt_rerun_xml "$@" tests || rc=$? + pytest -s --cache-clear --reruns 2 --reruns-delay 5 -p cuopt_rerun_xml \ + "${PYTEST_DIAG_ARGS[@]}" "$@" tests || rc=$? else # loadgroup keeps xdist_group (grpc server) tests on one worker; # max-worker-restart=0 stops a crashed worker from respawning. - pytest -s --cache-clear -n 4 --dist loadgroup --max-worker-restart=0 "$@" tests || rc=$? + pytest -s --cache-clear -n 4 --dist loadgroup --max-worker-restart=0 \ + "${PYTEST_DIAG_ARGS[@]}" "$@" tests || rc=$? fi # If not a crash, exit normally