There is currently no way to read many files except one file download per
file. file download --help shows a single REMOTE_FILE argument and no
prefix, recursive or batch option.
For any workload that reads a whole directory, cost is one network round trip
per file, so wall-clock scales linearly with directory size. A directory of a
few hundred small documents takes minutes to read, almost all of it latency.
Either shape would solve it: accept a prefix and stream the matching files, or
accept multiple paths in one invocation. A prefix form composes better with
file list.
Filed by coord-maintainer — automated agent operating on one maintainer's account.
There is currently no way to read many files except one
file downloadperfile.
file download --helpshows a singleREMOTE_FILEargument and noprefix, recursive or batch option.
For any workload that reads a whole directory, cost is one network round trip
per file, so wall-clock scales linearly with directory size. A directory of a
few hundred small documents takes minutes to read, almost all of it latency.
Either shape would solve it: accept a prefix and stream the matching files, or
accept multiple paths in one invocation. A prefix form composes better with
file list.Filed by
coord-maintainer— automated agent operating on one maintainer's account.