When querying a Parquet file, the following query shows a ~2x performance drop at commit cbb4109 compared to the commit just prior. On my Intel-based Macbook, the drop is from ~1.2 seconds to ~2.4 seconds.
$ aws s3 cp --quiet s3://brim-sampledata/mgbench/bench3.parquet . &&
super -version &&
hyperfine "super -c \"from 'bench3.parquet' | where event_type=='power' and device_floor > 2 | count()\""
Version: v0.1.0-23-gcbb41094c
Benchmark 1: super -c "from 'bench3.parquet' | where event_type=='power' and device_floor > 2 | count()"
Time (mean ± σ): 2.496 s ± 0.168 s [User: 23.684 s, System: 0.961 s]
Range (min … max): 2.209 s … 2.711 s 10 runs
$ super -version &&
hyperfine "super -c \"from 'bench3.parquet' | where event_type=='power' and device_floor > 2 | count()\""
Version: v0.1.0-22-g1bced4402
Benchmark 1: super -c "from 'bench3.parquet' | where event_type=='power' and device_floor > 2 | count()"
Time (mean ± σ): 1.179 s ± 0.038 s [User: 10.735 s, System: 0.432 s]
Range (min … max): 1.126 s … 1.244 s 10 runs
Details
Repro is with super commit cbb4109, which is associated with the merge of #6633. The query above is a simplification of the mgbench bench3/q2 query where the symptom was first noticed.
After performing the binary search to spot where the perf changed, I used Claude to help create the simplified repro query above and offer its take on the root cause and ideas for addressing it, which is captured in a Gist titled "Performance regression: nullable Parquet columns became union{T,null}, costing ~1.5–2x on scans, filters, and aggregations".
When querying a Parquet file, the following query shows a ~2x performance drop at commit cbb4109 compared to the commit just prior. On my Intel-based Macbook, the drop is from ~1.2 seconds to ~2.4 seconds.
Details
Repro is with super commit cbb4109, which is associated with the merge of #6633. The query above is a simplification of the mgbench bench3/q2 query where the symptom was first noticed.
After performing the binary search to spot where the perf changed, I used Claude to help create the simplified repro query above and offer its take on the root cause and ideas for addressing it, which is captured in a Gist titled "Performance regression: nullable Parquet columns became union{T,null}, costing ~1.5–2x on scans, filters, and aggregations".