LastModifiedDownloader.download() fetches with a hard-coded AbortSignal.timeout(300_000). That signal bounds the whole transfer, not just the wait for headers: on Node 24 the body stream is aborted at the deadline, pipeline() rejects with a TimeoutError, the partial file is removed and the error propagates. DownloadOptions offers no way to change it.
Five minutes is fine for the distributions the downloader was written for, but geonames-rdf’s allCountries.zip is ~420 MB and alternateNamesV2.zip ~200 MB, so the download fails whenever sustained throughput drops below ~1.3 MB/s. A recent run fetched allCountries.zip in about 40 s, so this is latent, but on a throttled day the LDE conversion fails where curl in the shell script, which has no limit, succeeds. And since the distributions carry no lastModified, there is no resume: the next run starts from zero.
Suggested fix
Either of:
- A
timeout (or signal) in DownloadOptions, so a caller who knows the size can raise it, with the current 300 000 ms as the default.
- An idle timeout instead of a whole-transfer budget: abort when no bytes have arrived for N seconds, which catches a stalled server without penalising a large file that is still flowing. This is the better semantics and needs no option.
Found while porting geonames-rdf (netwerk-digitaal-erfgoed/geonames-rdf#50), see #782.
LastModifiedDownloader.download()fetches with a hard-codedAbortSignal.timeout(300_000). That signal bounds the whole transfer, not just the wait for headers: on Node 24 the body stream is aborted at the deadline,pipeline()rejects with aTimeoutError, the partial file is removed and the error propagates.DownloadOptionsoffers no way to change it.Five minutes is fine for the distributions the downloader was written for, but geonames-rdf’s
allCountries.zipis ~420 MB andalternateNamesV2.zip~200 MB, so the download fails whenever sustained throughput drops below ~1.3 MB/s. A recent run fetchedallCountries.zipin about 40 s, so this is latent, but on a throttled day the LDE conversion fails wherecurlin the shell script, which has no limit, succeeds. And since the distributions carry nolastModified, there is no resume: the next run starts from zero.Suggested fix
Either of:
timeout(orsignal) inDownloadOptions, so a caller who knows the size can raise it, with the current 300 000 ms as the default.Found while porting geonames-rdf (netwerk-digitaal-erfgoed/geonames-rdf#50), see #782.