Skip to content

S3FileIoOptions.direct_io now supports download as well#750

Merged
TingDaoK merged 4 commits into
mainfrom
o_direct
Jun 24, 2026
Merged

S3FileIoOptions.direct_io now supports download as well#750
TingDaoK merged 4 commits into
mainfrom
o_direct

Conversation

@TingDaoK

@TingDaoK TingDaoK commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

The S3FileIoOptions class docstring and direct_io field documentation were outdated — the class said "only applies when send_filepath is set" and the field only mentioned upload.

Updated documentation to reflect current behavior:

  • Class docstring: applies to both send_filepath (upload) and recv_filepath (download)
  • direct_io field: documents non-Linux transparent fallback, upload hard-fail on unaligned part_size, download best-effort fallback conditions
  • Fixed RST formatting (nested bullets caused Sphinx warnings)

Highlight: With this change, download 5TiB file to a RAID0 disk with 150Gbps throughput, we are able to reach 109 Gb/s with 400GiB memory usage. Meanwhile same configurations and resource usage without direct_io write can only reach 15 Gb/s due to kernel limitation


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@TingDaoK TingDaoK changed the title O direct Update S3FileIoOptions.direct_io documentation for download support Jun 24, 2026
@TingDaoK TingDaoK changed the title Update S3FileIoOptions.direct_io documentation for download support S3FileIoOptions.direct_io now supports download as well Jun 24, 2026
@TingDaoK TingDaoK merged commit c8c605c into main Jun 24, 2026
145 of 146 checks passed
@TingDaoK TingDaoK deleted the o_direct branch June 24, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants