Speed up multi-state spatial binding - #653
Open
dshkol wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've been putting together a continuously-improving agent skill for performance optimizations for R packages, particularly ones like this that have workflows requiring large extracts from APIs and assembling them. We've implemented some really significant speeds up in other packages and I tested against and benchmarked some optimization improvements against tidycensus. This is one PR, but there's a few more I can submit later -- I didn't want to send too many at once.
The change here is a really simple, adding a helper that binds sf objects in one go rather than sequentially. Speed up on large spatial downloads are meaningful, especially when working with tracts.
Benchmarks are against real-data extracts and unit tests are already added.
Summary
reduce(rbind)accumulation of multi-statesfresults with one-pass row bindingas_tibble()/st_as_sf()normalizationsfregression test; keep the official-geometry benchmark harness localReal-world benchmark
The benchmark uses official Census geometry, including 2023 national cartographic-boundary tracts from tigris. The largest workload combines those tract geometries with actual 2023 ACS 5-year B01001 estimates/MOEs for six variables, matching a tidy multi-state
get_acs(..., geometry = TRUE)output. Final normalizedsfresults are checked withidentical()before timing.Times are randomized-order medians after warm-up. Allocations are cumulative
Rprofmem()totals, not peak memory. Tidy national timing ranges were 27.340–27.525 seconds for baseline and 0.482–0.485 seconds for this PR.Cold national geometry download took 4.250 seconds, and capture of the 52 state/DC/Puerto Rico ACS fixtures took 147.426 seconds. Those network measurements are reported separately and are not counted as code speedup because live tigris caching and API transfer vary independently.
The county workload saves only 0.142 seconds; this change is motivated by the 6.7–26.9 second absolute savings at national tract scale.
The benchmark harness, fixture cache, and full reproduction notes are retained locally and intentionally excluded from git.
Validation
devtools::test(): 84 passed, 0 failed, 0 warningsdevtools::check(error_on = "warning"): 0 errors, 0 warnings, 0 notessfattributes are byte-for-byte identical at every benchmark scale