fix #179 - #196
Open
thesues wants to merge 2 commits into
Open
Conversation
thesues
force-pushed
the
fix
branch
9 times, most recently
from
January 8, 2026 19:43
b9b196b to
faa12a2
Compare
poll_handle_ was only uv_poll_stop()'ed, never uv_close()'ed, and it lives
inside the Client object which on_close deleted right away. uv_poll_stop does
not take the handle out of loop->handle_queue - only uv_close does, in
uv__finish_close. So every rdma connection left the loop with a queue node
pointing into freed memory, and the next uv__queue_insert_tail wrote through
it, corrupting the heap. The abort surfaced much later at an unrelated free(),
which is why it only showed up occasionally and only for rdma clients.
libuv is explicit about this: uv_close "MUST be called on each handle before
memory is released. Moreover, the memory can only be released in close_cb or
after it has returned."
Give the client a single teardown path instead:
- Client::close() is the only way to tear a client down, and it is
idempotent. on_read/on_write/on_chunk_write/on_head_write all call it now.
This also fixes the double uv_close in on_write: libuv flushes pending
write requests with UV_ECANCELED while the handle is closing, so on_write
is called with a negative status during a normal teardown and used to
close the handle a second time.
- close() closes every handle the client owns, and the last close callback
to run deletes the client, so both handles outlive the loop's use of them.
- the tcp handle is embedded in the client as well now, no reason to keep it
on the heap. This also plugs the leak in the uv_accept failure path, which
closed the handle with a NULL callback and never freed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Callbacks like on_write and on_read can be invoked before on_close. Since they may also call uv_close internally,
this could result in uv_close being called twice.
Update: the actual root cause of #179
The double close above is real, but it is not what makes the server abort with
free(): invalid pointeron RDMA connections. That one is a missinguv_close.A client owns two libuv handles: the tcp handle and, after the rdma exchange,
poll_handle_for the completion channel.poll_handle_was onlyuv_poll_stop()ed in~Client(), neveruv_close()ed — and it lives insidethe Client object that
on_closedeletes.uv_poll_stoponly stops the io watcher (uv__io_stop+uv__handle_stop); itdoes not take the handle out of
loop->handle_queue. That only happens inuv__finish_close, i.e. viauv_close. So every closed RDMA connection left theloop with an intrusive queue node pointing into freed memory, and the next
uv__queue_insert_tailwrote through it, corrupting glibc's freelist. The abortthen surfaced at some unrelated
free()much later — which is why it onlyreproduces occasionally, only for RDMA clients, and why the stack trace lands in
~Client()(the victim, not the culprit).libuv is explicit about this:
What this adds
Client::close()is now the only teardown path, and it is idempotent.on_read/on_write/on_chunk_write/on_head_writeall call it, whichalso removes the need for per-callsite
uv_is_closingguards. This is whatfixes the
on_writedouble close: libuv flushes pending write requests withUV_ECANCELEDwhile the handle is closing, soon_writeruns with a negativestatus during a perfectly normal teardown and used to close the handle again.
close()closes every handle the client owns; the last close callback torun deletes the client. Both handles therefore outlive the loop's use of them.
the close callbacks there is no reason to keep it on the heap. This also plugs
the leak in the
uv_acceptfailure path, which closed the handle with a NULLcallback and never freed it.
Testing
Built and smoke tested: TCP connect/disconnect plus read/write round trips of
100B / 4K / 64K / 256K all pass, and each connection logs
free client resourcesexactly once.The RDMA path is not covered by this testing — no RDMA device was available.
Please exercise repeated connect/disconnect on a machine with an IB card
(ideally under ASAN or
MALLOC_CHECK_=3) before merging.🤖 Generated with Claude Code