Very large speedup to Census data retrieval post-processing with col type scans and vectorized comparisons

[dshkol]

This PR can work independently or additively with #653. The speed benchmarks in each are for the fixes individually, however combined results are additive in internal testing. The change is not complex, basically a lookup on what cols are characters and using a vectorized function to do the check instead, but the speed gains in large datasets are huge.

In this original:

for (i in seq_along(variables)) {
    dat2[dat2 == variables[i]] <- names(variables)[i]
  }

For every requested variable, the expression dat2 == variables[i] compares that value against every cell in the result, even numeric estimate and margin-of-error columns which are a waste. The subsequent data-frame subassignment can also copy substantial portions of the object on every iteration.

The replacement first identifies character columns by inspecting each column’s class:

character_columns <- which(vapply(data, is.character, logical(1)))

the helper then uses vectorized match() lookups only on those char columns:

for (column in character_columns) {
    index <- match(data[[column]], original)
    data[[column]][matched] <- replacement[index[matched]]
  }

In the original, lets say you have 50 variables, 4M rows, and 5 cols that is 1bn cell-level comparisons. The replacement does the same with about 12M vectorized comparisons instead.

Like the other PR, this was flagged and recommended by a run of the r-pkg-opt skill that's a collection of best practices for performant R code in different R styles and use cases, and then verified by me. Model used here is Sol 5.6. I've noticed that these types of packages that process huge amounts of wide tabular data + spatial data seem to have a fair bit of room to get faster.

AI content below

---

Summary

• replace repeated full-data-frame Census sentinel scans with one type-aware pass
• replace named-variable full-table recode loops with a single match-based pass while preserving sequential alias behavior
• share the same post-processing helpers across ACS, decennial Census, and population estimates
• add regression tests; keep the real-ACS benchmark harness local

Real-world benchmark

The benchmark uses actual 2023 ACS 5-year tract extracts for all 48 estimate/MOE pairs in B01001. Raw responses are captured once and replayed through origin/master (5461f03) and this branch so network variance is excluded. Outputs are checked with identical() before timing.

Scale	Tracts	Output rows	Reps	Baseline	PR	Saved	Speedup	Allocations, baseline → PR
King County, WA	495	23,760	7	0.558 s	0.024 s	0.534 s	23.3x	233.0 → 30.7 MB
California	9,129	438,192	5	9.898 s	0.671 s	9.227 s	14.8x	4,258.7 → 529.4 MB
United States + Puerto Rico	85,381	4,098,288	3	90.646 s	4.719 s	85.927 s	19.2x	40,340.7 → 5,459.6 MB

Times are randomized-order medians after warm-up. Allocations are cumulative Rprofmem() totals, not peak memory. The national timing ranges were 90.553–91.077 seconds for baseline and 4.713–4.724 seconds for this PR.

The fixture capture itself took 2.289 seconds for King County, 14.212 seconds for California, and 147.426 seconds across the 52 state/DC/Puerto Rico responses. Those API times are reported separately and are not counted as speedup.

The benchmark harness, fixture cache, and full reproduction notes are retained locally and intentionally excluded from git.

A direct randomized-order stress test on the 4,098,288-row national result compared the retained explicit-loop helper with the functional formulation: 0.201 vs. 0.204 seconds. The loop controls only requested-variable and character-column selection; match() still processes every response row at once. The explicit form is both clearer about sequential alias behavior and marginally faster.

Validation

• devtools::test(): 88 passed, 0 failed, 0 warnings
• devtools::check(error_on = "warning"): 0 errors, 0 warnings, 0 notes
• baseline and PR public get_acs() results are byte-for-byte identical at all three benchmark scales