#22700 adds metadata prefetching to cudf-polars. It implements it as a distinct stage that runs after prefetching and before IR execution in a rapidsmpf streaming network.
While metadata prefetching should be relatively fast compared to IR execution, the barrier created by prefetching isn't optimal. It waits for all metadata prefetching to complete before any of the Scan nodes can begin. But a Scan node really only requires the prefetched metadata for the paths it reads, not all the paths in the entire query.
We can fix this by removing the prefetching barrier and doing the
TomAugspurger@18913f5 is out of date now, but has a POC for how this might be done. At a high level:
- traverse the IR graph, grouping the Scan nodes by
paths (we want to read each parquet metadata exactly once)
- Create actors responsible for each group of parquet metadata
- Feed the metadata into the Actors performing the Scans
#22700 adds metadata prefetching to cudf-polars. It implements it as a distinct stage that runs after prefetching and before IR execution in a rapidsmpf streaming network.
While metadata prefetching should be relatively fast compared to IR execution, the barrier created by prefetching isn't optimal. It waits for all metadata prefetching to complete before any of the Scan nodes can begin. But a Scan node really only requires the prefetched metadata for the
pathsit reads, not all the paths in the entire query.We can fix this by removing the prefetching barrier and doing the
TomAugspurger@18913f5 is out of date now, but has a POC for how this might be done. At a high level:
paths(we want to read each parquet metadata exactly once)