test: demonstrate cancellation can't free blocking sync code (2 failing tests) - #4599
Open
neubig wants to merge 1 commit into
Open
test: demonstrate cancellation can't free blocking sync code (2 failing tests)#4599neubig wants to merge 1 commit into
neubig wants to merge 1 commit into
Conversation
Two failing tests showing the same root cause: asyncio cancellation (CancelledError) cannot be delivered to tasks blocked in synchronous code (C-level locks, blocking I/O, futexes). The cleanup code tries to cancel but gives up after a tiny timeout, leaving zombie threads. Test 1: AsyncExecutor.close() hangs forever when a task is blocked in a worker thread (anyio.to_thread). close() waits indefinitely for the task to finish on its own. Test 2: bubus EventBus handler timeout fires after 1s and logs the error, but the handler's worker thread (via run_in_executor) stays alive forever because cancelling the asyncio Future doesn't cancel the underlying thread. The finally block waits only 0.1s then abandons the task. In production, both accumulate zombie threads until the thread pool is exhausted and new conversation creation stalls.
Contributor
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Collaborator
|
👋 This PR needs a couple of things fixed before OpenHands can review it:
Push an update once this is addressed and this check re-runs automatically. This is an automated check - no AI was used to generate this comment. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two failing tests that demonstrate the same underlying problem: asyncio cancellation (
CancelledError) cannot be delivered to tasks blocked in synchronous code (C-level locks, blocking I/O, futexes). The cleanup code tries to cancel but gives up after a tiny timeout, leaving zombie threads that accumulate until the process is killed.Tests
Test 1:
test_async_executor_close_hangs_on_blocking_taskDemonstrates that
AsyncExecutor.close()hangs forever when a task is blocked in a worker thread. Without the fix,close()waits indefinitely for the task to finish on its own.Result on main: FAILS after 5s —
close()hangs.Addresses #4597 (duplicate of #4546, with a cleaner test).
Test 2:
test_bubus_timeout_does_not_free_blocked_handler_threadDemonstrates that
bubusEventBus handler timeout fires after 1s and logs the error, but the handler's worker thread (viarun_in_executor) stays alive forever. Thefinallyblock waits only 0.1s for cancellation to take effect, then abandons the task.Result on main: FAILS — 1 zombie thread remains.
Addresses #4598.
Production impact
Both bugs manifested on a self-hosted agent-server: conversations that used the browser tool accumulated zombie threads (21 futex-stuck threads across 11 conversations) until the thread pool was exhausted and new conversation creation stalled. The server looked healthy (
/healthreturned 200) but was wedged.Run
Both tests should fail on
main. They should pass once #4597 and #4598 are fixed.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:4be3a8f-pythonRun
All tags pushed for this build
About Multi-Architecture Support
4be3a8f-python) is a multi-arch manifest supporting both amd64 and arm644be3a8f-python-amd64) are also available if needed