Remote I/O egress controls and retryable transport failures - #1052
Draft
karthikeyann wants to merge 2 commits into
Draft
Remote I/O egress controls and retryable transport failures#1052karthikeyann wants to merge 2 commits into
karthikeyann wants to merge 2 commits into
Conversation
…ccounting fix Reading S3 over HTTPS at multi-hundred-Gbps exposed several gaps in the remote path. Measured on a g7e.48xlarge, four ENA cards, against s3://rapids-tpch. Retry accounting (a correctness bug). CurlHandle::perform() retries curl_easy_perform() without rolling back what the write callback already accepted, so a transfer that times out part way through resumes with the destination offset still advanced, overruns the caller's buffer and returns CURLE_WRITE_ERROR, which is not retryable. A recoverable timeout therefore became a hard failure, reported as the misleading "maybe the server doesn't support file ranges?". With 512 MiB ranges under four-card load this killed two of four processes on every attempt. CurlHandle now takes an on-retry hook and RemoteHandle::read uses it to reset the callback context. Streaming-store receive, KVIKIO_REMOTE_IO_NT_COPY. libcurl delivers one buffer at a time, 16 KiB by default, so every memcpy in the write callback is far below glibc's non-temporal threshold and uses ordinary stores. An ordinary store to a line not in cache fetches it first, so with a destination far larger than last-level cache each delivered byte costs two DRAM accesses and the receive path becomes memory-bandwidth-bound rather than CPU-bound. Streaming stores skip the fetch: 669 -> 874 Gbps delivered into pinned host memory, +29%, reproduced. Off by default because it is the wrong choice for a small destination that stays in cache, which is exactly the device-memory bounce buffer. CURLOPT_INTERFACE, KVIKIO_REMOTE_IO_INTERFACE. The if!<name> form binds with SO_BINDTODEVICE, the only way to choose an egress NIC from inside the process when a host has several cards on one subnet. Without it the kernel picks by route metric and every connection leaves through one card. CURLOPT_DNS_SHUFFLE_ADDRESSES, KVIKIO_REMOTE_IO_DNS_SHUFFLE. S3 publishes many front-end addresses but libcurl connects to the first that answers, so a process otherwise piles every connection onto one endpoint. CURLOPT_BUFFERSIZE, KVIKIO_REMOTE_IO_BUFFER_SIZE, default unchanged. Raising it looks right and measures worse, because libcurl allocates this buffer per easy handle and there is one handle per in-flight sub-range: at 4096-way concurrency that is 64 MiB at 16 KiB against 4 GiB at 1 MiB, and the smaller total stays in L3. Measured 862 Gbps at 16 KiB, 745 at 256 KiB, 727 at 1 MiB. Exposed because the trade-off inverts at low concurrency. KVIKIO_REMOTE_IO_KTLS sets SSL_OP_ENABLE_KTLS. It does not currently engage through libcurl, which installs its own BIO while OpenSSL requires the BIO under the SSL object to advertise kTLS on a real socket fd; TlsRxSw stays at zero. Kept off by default and left in place for when libcurl gains support. KVIKIO_REMOTE_IO_DISCARD accounts for bytes without copying them, to separate transport cost from destination cost. Benchmarking only: the accounting stays honest but the destination holds garbage. It measured the destination write at 31% of achievable throughput, which is what motivated the streaming-store path. All options are read once into function-local statics, because a handle is constructed per sub-range transfer and getenv would otherwise sit on the hot path.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Two commits, both driven by measurements while reading a 3 TB TPCH dataset from S3 into GPU
memory on a
g7e.48xlarge(4 ENA cards, 8 GPUs) through Velox/Presto.1.
feat(remote): egress controls, streaming-store receive, and a retry-accounting fixUnchanged from
karthikeyann:s3-max-bandwidth, rebased onto currentmain. AddsKVIKIO_REMOTE_IO_INTERFACE(CURLOPT_INTERFACE,if!<name>/SO_BINDTODEVICE),KVIKIO_REMOTE_IO_DNS_SHUFFLE,KVIKIO_REMOTE_IO_NT_COPY,KVIKIO_REMOTE_IO_BUFFER_SIZE,KVIKIO_REMOTE_IO_KTLS,KVIKIO_REMOTE_IO_DISCARD, and fixes retry accounting so a partiallycompleted range transfer rolls the write-callback context back before being retried.
Independent confirmation of the egress control's value, measured end to end through Presto on
SF3000 TPCH: with all four cards on one subnet the kernel picks by route metric, so every
connection left through a single NIC.
KVIKIO_REMOTE_IO_INTERFACE[95.8, 0, 0, 0]Gbps[68.8, 45.4, 81.4, 51.0]GbpsOne deployment note that cost some time to find: the interface name must exist in the process's
network namespace, so a containerised reader needs host networking. Without it
SO_BINDTODEVICEfails and the failure surfaces from
UrlParser::parseas a misleadingurl.cpp: Unsupported URL schemeon the originals3://URL, becauseencode_special_chars_in_path()parses withoutCURLU_NON_SUPPORT_SCHEMEand outside anytry/catch. Worth improving separately.
2.
fix(remote): retry transport-level curl failures (resets, DNS, partial body)HttpRetryPolicy::is_retryable()currently accepts onlyCURLE_OPERATION_TIMEDOUT, with aTODOasking for more candidates. Every other transport failure is fatal, so a single dropped packet fails
an entire multi-TB table scan. Two failure modes hit repeatedly at these rates:
CURLE_COULDNT_RESOLVE_HOST— the remote path resolves per request with no shared DNS cache, so at512 concurrent requests x 8 processes the query rate exceeds the AWS VPC resolver's 1024 packets/s
per-ENI allowance.
CURLE_RECV_ERROR— S3 recycles connections; observed 6-7 times per 22-query run.Both were fatal, and in our stack the throwing worker then crashed, so one lost packet cascaded into
unrelated query failures. Measured impact: TPCH runs went from 17/22 and 14/22 queries passing (a
different set failing each time) to 22/22 in a single run.
This now also retries
CURLE_COULDNT_CONNECT,CURLE_SEND_ERROR,CURLE_PARTIAL_FILEandCURLE_GOT_NOTHING, and stops reportingGot HTTP code 0for failures that never carried an HTTPstatus. It is only safe on top of commit 1's retry-accounting fix:
perform(on_retry)rolls thecallback context back, so a retried partial transfer cannot corrupt the destination buffer.
Not included
KVIKIO_REMOTE_IO_KTLSis retained from commit 1 but confirmed still inert: setting it left/proc/net/tls_statTlsRxSwunchanged, consistent with libcurl installing its own BIO whileOpenSSL requires the BIO under the
SSLobject to advertise kTLS on a real socket fd.Open questions for reviewers
encode_special_chars_in_path()passCURLU_NON_SUPPORT_SCHEMEso endpoint-probe failuresstop masquerading as scheme errors?