Answer the sender's closing handshake before dropping the socket - #32
Merged
Conversation
A sender could intermittently see `Connection reset by peer` after a transfer that actually succeeded: the receiver had confirmed the full byte count and written correct output, but the sender's last socket operation reported an error. The upload receive task broke out of its loop as soon as the sender sent `complete`, which dropped `ws_receiver`. The socket stayed open after that, because the send task still owed the sender a `transfer_complete` status and a `Close` frame. A WebSocket close is a two-way handshake, so the peer answered that `Close` with one of its own — into a socket the relay had stopped reading. Closing a socket with unread data queued sends RST rather than FIN, and the peer reports a reset. Whether the sender saw it was a race: it returns as soon as it reads `transfer_complete`, so the error surfaced only when its next socket operation touched the connection after the reset arrived. The fix cannot live in the send task, which owns the sink and has nothing to await on. Instead the receive task records that the sender finished cleanly and stays alive through teardown: it waits for the send task to exit, which it detects through the event receiver being dropped, then reads briefly until the peer's `Close` arrives. The wait is deliberately two stages. A single short deadline would not work, because the send task cannot write `transfer_complete` until the receiver has finished writing the file, which is unbounded work. Both stages are bounded, so a peer that stops answering cannot hold the task and its per-IP connection slot open. Evidence and the full diagnosis are in the pull request; the plan document lands separately in #31. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Marks the relay teardown plan done now that its documents are on main, and writes down what the work actually produced rather than what it predicted. Three things are recorded because they change what a reader should believe: - the drain needed two stages, not the single short deadline the plan assumed, because the send task cannot write `transfer_complete` until the receiver has finished writing the file; - the defect does not reproduce through separate CLI processes, so the first evidence harness measured nothing, and running it only against the fix would have produced a clean result that meant nothing; - the assertions #30 removed stay out for a measured reason, and the download socket carries the same latent shape without a user-visible symptom, now tracked as its own unscheduled item. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the relay teardown reset recorded as a follow-up in #30.
The defect
A sender could intermittently see
Connection reset by peerafter a transfer that actually succeeded — the receiver had confirmed the full byte count and written correct output.The upload receive task broke out of its loop as soon as the sender sent
complete, droppingws_receiver. The socket stayed open after that, because the send task still owed the sender atransfer_completestatus and aCloseframe. A WebSocket close is a two-way handshake, so the peer answered thatClose— into a socket the relay had stopped reading. Closing a socket with unread data queued sends RST rather than FIN, and the peer reports a reset.Whether the user saw it was a race: the sender returns as soon as it reads
transfer_complete, so the error surfaced only when its next socket operation touched the connection after the reset arrived.The fix
It cannot live in the send task — that half owns the sink and has nothing to await on. Instead the receive task records that the sender finished cleanly and stays alive through teardown: it waits for the send task to exit (detected through the event receiver being dropped), then reads briefly until the peer's
Closearrives.The wait is deliberately two stages. A single short deadline does not work, because the send task cannot write
transfer_completeuntil the receiver has finished writing the file — unbounded work. Both stages are bounded, so a peer that stops answering cannot hold the task and its per-IP connection slot open. The newWS_CLOSE_DRAIN_TIMEOUT_SECSbounds only the second stage.Evidence
The defect does not reproduce through separate CLI processes — the sender exits before touching the socket again. It reproduces in the in-process test harness, with two transfers sharing one relay, which is the shape the tests had before #30.
Restoring that pre-#30 test file and running it repeatedly:
main(unfixed)Failure text on
main, for the record:That matches the ~16% #30 reported.
Why the removed assertions stay out
#30 removed the completion-handshake assertions from two tests and gave each transfer its own relay. I tried restoring the assertions and measured the result: with per-transfer relay isolation they never fail on unfixed
maineither (0/30). The isolation removes the contention that triggers the race, so those assertions would guard nothing.The shape that does catch it — two transfers sharing one relay — detects at roughly 10% per run. That is too weak to serve as a CI guard and would reintroduce exactly the flakiness #30 removed. So the tests are unchanged here, and the reproduction stays a manual recipe rather than a CI assertion.
If a stronger guard is wanted later, it should be a targeted test that drives the close handshake directly rather than a probabilistic end-to-end race.
Verification
scripts/check-secrets.sh,cargo fmt --check,cargo clippy --workspace --all-targets --all-features -D warningsall pass.cargo test --workspace --all-targets: 56 passed, 0 failed.The plan document for this work is in #31 and its evidence checkboxes should be ticked once that merges.
🤖 Generated with Claude Code