chunk() names its files <name>-<index>.<ext> with String(index).padStart(4, '0'), so indices 0 to 9999 are four digits and index 10000 onwards is five. removeChunksOf(), which clears an earlier call’s chunks before a re-run, matches ^<name>-\d{4}<ext>$ – four digits exactly – so chunks past the 10,000th are never removed.
The consequence is the one the removal exists to prevent: a re-run with a larger chunk size, or over a smaller input, leaves a longer run’s tail in place for something to pick up. It only triggers with more than 10,000 chunks (rows in the tens of thousands over a 13.5M-row input, say), so it is latent for the default sizes, but the fix is one character.
A caller who rediscovers chunks by name rather than keeping the paths chunk() returned has the same bug on its side; geonames-rdf did (netwerk-digitaal-erfgoed/geonames-rdf#50) and is fixing its regex too. The docs could say that chunk()’s return value is the list to keep.
Suggested fix
\d{4,} in removeChunksOf(). Optionally a test that chunks a 10,001-row input at 1 row a chunk and re-runs it.
chunk()names its files<name>-<index>.<ext>withString(index).padStart(4, '0'), so indices 0 to 9999 are four digits and index 10000 onwards is five.removeChunksOf(), which clears an earlier call’s chunks before a re-run, matches^<name>-\d{4}<ext>$– four digits exactly – so chunks past the 10,000th are never removed.The consequence is the one the removal exists to prevent: a re-run with a larger chunk size, or over a smaller input, leaves a longer run’s tail in place for something to pick up. It only triggers with more than 10,000 chunks (rows in the tens of thousands over a 13.5M-row input, say), so it is latent for the default sizes, but the fix is one character.
A caller who rediscovers chunks by name rather than keeping the paths
chunk()returned has the same bug on its side; geonames-rdf did (netwerk-digitaal-erfgoed/geonames-rdf#50) and is fixing its regex too. The docs could say thatchunk()’s return value is the list to keep.Suggested fix
\d{4,}inremoveChunksOf(). Optionally a test that chunks a 10,001-row input at 1 row a chunk and re-runs it.