Skip to content

fix #179 - #196

Open
thesues wants to merge 2 commits into
bytedance:mainfrom
thesues:fix
Open

fix #179#196
thesues wants to merge 2 commits into
bytedance:mainfrom
thesues:fix

Conversation

@thesues

@thesues thesues commented Jan 5, 2026

Copy link
Copy Markdown
Collaborator

Callbacks like on_write and on_read can be invoked before on_close. Since they may also call uv_close internally,
this could result in uv_close being called twice.

Update: the actual root cause of #179

The double close above is real, but it is not what makes the server abort with
free(): invalid pointer on RDMA connections. That one is a missing uv_close.

A client owns two libuv handles: the tcp handle and, after the rdma exchange,
poll_handle_ for the completion channel. poll_handle_ was only
uv_poll_stop()ed in ~Client(), never uv_close()ed — and it lives inside
the Client object that on_close deletes.

uv_poll_stop only stops the io watcher (uv__io_stop + uv__handle_stop); it
does not take the handle out of loop->handle_queue. That only happens in
uv__finish_close, i.e. via uv_close. So every closed RDMA connection left the
loop with an intrusive queue node pointing into freed memory, and the next
uv__queue_insert_tail wrote through it, corrupting glibc's freelist. The abort
then surfaced at some unrelated free() much later — which is why it only
reproduces occasionally, only for RDMA clients, and why the stack trace lands in
~Client() (the victim, not the culprit).

libuv is explicit about this:

uv_close: This MUST be called on each handle before memory is released.
Moreover, the memory can only be released in close_cb or after it has returned.

What this adds

  • Client::close() is now the only teardown path, and it is idempotent.
    on_read / on_write / on_chunk_write / on_head_write all call it, which
    also removes the need for per-callsite uv_is_closing guards. This is what
    fixes the on_write double close: libuv flushes pending write requests with
    UV_ECANCELED while the handle is closing, so on_write runs with a negative
    status during a perfectly normal teardown and used to close the handle again.
  • close() closes every handle the client owns; the last close callback to
    run deletes the client. Both handles therefore outlive the loop's use of them.
  • The tcp handle is embedded in the Client now too — with the lifetime driven by
    the close callbacks there is no reason to keep it on the heap. This also plugs
    the leak in the uv_accept failure path, which closed the handle with a NULL
    callback and never freed it.

Testing

Built and smoke tested: TCP connect/disconnect plus read/write round trips of
100B / 4K / 64K / 256K all pass, and each connection logs free client resources exactly once.

The RDMA path is not covered by this testing — no RDMA device was available.
Please exercise repeated connect/disconnect on a machine with an IB card
(ideally under ASAN or MALLOC_CHECK_=3) before merging.

🤖 Generated with Claude Code

poll_handle_ was only uv_poll_stop()'ed, never uv_close()'ed, and it lives
inside the Client object which on_close deleted right away. uv_poll_stop does
not take the handle out of loop->handle_queue - only uv_close does, in
uv__finish_close. So every rdma connection left the loop with a queue node
pointing into freed memory, and the next uv__queue_insert_tail wrote through
it, corrupting the heap. The abort surfaced much later at an unrelated free(),
which is why it only showed up occasionally and only for rdma clients.

libuv is explicit about this: uv_close "MUST be called on each handle before
memory is released. Moreover, the memory can only be released in close_cb or
after it has returned."

Give the client a single teardown path instead:

  - Client::close() is the only way to tear a client down, and it is
    idempotent. on_read/on_write/on_chunk_write/on_head_write all call it now.
    This also fixes the double uv_close in on_write: libuv flushes pending
    write requests with UV_ECANCELED while the handle is closing, so on_write
    is called with a negative status during a normal teardown and used to
    close the handle a second time.
  - close() closes every handle the client owns, and the last close callback
    to run deletes the client, so both handles outlive the loop's use of them.
  - the tcp handle is embedded in the client as well now, no reason to keep it
    on the heap. This also plugs the leak in the uv_accept failure path, which
    closed the handle with a NULL callback and never freed it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant