v0.3.8 - #15
Open
mountainMath wants to merge 9 commits into
Open
Conversation
Rewrite the correspondence table computation and fix a family of arguments that were accepted but never used. Performance: - replace the row-by-row union-find in get_tongfen_correspondence with a vectorised hook and pointer jumping connected components pass. Verified to produce byte-identical TongfenID and TongfenUID on BC/ON DA and DB correspondence tables. Dissemination blocks for Ontario go from 437s to 1.9s, the whole country from hours to 8s - the "statcan" method no longer downloads census geometries, which it never looked at, and get_tongfen_ca_census only downloads the geometry of the base_geo dataset - summarize_geometry_by_group passes single geometry groups through instead of sending them to st_union - collapse_unique_by_row replaces the row-wise apply in check_tongfen_areas and aggregate_correspondences Behaviour: - get_tongfen_ca_census now passes base_geo, na.rm, tolerance, crs and data_transform on instead of dropping them. base_geo = NULL now returns data without geographic information as documented - drop the area_mismatch_cutoff argument, which never had any effect - get_tongfen_correspondence_ca_census gained a crs argument, default 3347 - NA geographic identifiers no longer merge unrelated regions - fix crash when tongfen-ing census tracts across non-adjacent censuses - fix the deprecated get_tongfen_census_* wrappers erroring out on their default geo_format=NA - fix infinite loop guard and a GEOID10 typo in the US correspondence Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Expose `get_tongfen_correspondence_us_census` so US correspondence tables can be built without also fetching the data, and extend the tract correspondence back to the 1990 census via the Census Bureau 1990-to-2000 tract relationship files. The 1990 census stays out of `get_tongfen_us_census` and `valid_us_census_datasets`: the Census Bureau retired the 1990 API endpoint, so tidycensus cannot fetch the data. 1990 data has to be obtained elsewhere and combined with the correspondence table via `tongfen_aggregate`. Chaining the relationship files is now a loop over consecutive censuses instead of nested special cases, which verifies byte-identical to the previous implementation for all combinations of the 2000, 2010 and 2020 censuses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The only county subdivision source so far was the 2000 to 2010 comparability file, and `datasets` was ignored at that level, so asking for 2020 built a 2000/2010 correspondence and failed downstream with "Did not find matching geographic identifiers". Add the 2010 to 2020 relationship file and run county subdivisions through the same chaining loop as census tracts. Unlike the comparability file that file is a geometric overlay: for Rhode Island 84 of its 124 rows are slivers along boundaries that shifted slightly, three of them reaching into Massachusetts, and chaining them collapses the state's 40 subdivisions into 8 common geographies. Cut them by the share of area two subdivisions have in common, exposed as `min_area_share`. At the default of 0.01 Rhode Island comes out as 40 one-to-one matches, which is what it should be, its towns did not change between 2010 and 2020. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every US relationship file is a geometric overlay listing each overlap between two censuses, including boundaries that only shifted slightly. Chaining those merged unrelated regions into one common geography: Vermont's 179 tracts came out as 42 common geographies across the 2000 and 2010 censuses, its 187 tracts as 26 across 2010 and 2020. Apply the same cutoff used for county subdivisions to all four links, computed from whatever areas the file carries: summed part areas for 1990 to 2000, tract areas for 2000 to 2010, block areas summed up per tract for 2010 to 2020. With `min_area_share = 0` the result is identical to before, at the 0.01 default Rhode Island tracts across 2010 and 2020 give 198 common geographies instead of 60 and Vermont 151 instead of 26. No region is ever dropped, if all of its parts are slivers its largest part is kept. Rereading the 2010 to 2020 file also fixes its column types: it has 18 columns but was read with 15 type specs, so TRACT_2020 was parsed as a number and lost its leading zeros. 246 of Rhode Island's 250 2020 tract identifiers came out malformed and could not join against the identifiers tidycensus returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tidycensus defaults the 2020 census to the PL 94-171 redistricting file, which carries almost none of the variables people tongfen across censuses. There was no way to reach the DHC file through get_tongfen_us_census, so 2010 to 2020 comparisons of anything but population counts were impossible. sumfile takes a single value for all censuses or one named by dataset, since a 2000 to 2020 chain needs a different file per census. NULL keeps tidycensus' own defaults, so existing calls are unaffected. Extends the US vignette with a 2010 to 2020 example on the same variables and the same Bay Area map, plus notes on min_area_share and on reaching back to 1990 through the correspondence layer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
perf: speed up area-weighted interpolation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release branch for v0.3.8.
Breaking changes
get_tongfen_ca_censusnow honours itsbase_geo,na.rm,tolerance,crsanddata_transformarguments, all of which were silently ignored. Most visibly, the documented defaultbase_geo = NULLnow returns data without geographic information, where previously the geography of the first dataset came back regardless.area_mismatch_cutofffromget_tongfen_ca_censusandget_tongfen_correspondence_ca_census, it never had any effect.check_tongfen_areasis the way to inspect area mismatches.Performance
check_tongfen_areas/aggregate_correspondencesgot faster.US census
New
get_tongfen_correspondence_us_census, for correspondence tables without also fetching the data.Tract correspondence reaches back to the 1990 census (
dec1990). Correspondence layer only, the Census Bureau retired the 1990 API endpoint so the data itself has to come from elsewhere (NHGIS) and be combined viatongfen_aggregate.County subdivisions can be matched across the 2010 and 2020 censuses, previously only 2000 and 2010.
Correspondence tables no longer chain regions together over slivers. The relationship files are geometric overlays listing every intersection, including boundaries that only shifted slightly, and matching those up merged unrelated regions into one common geography. New
min_area_shareargument, default0.01, applied in memory at parse time so cached files stay byte-identical andmin_area_share = 0reproduces the old behaviour. No region is ever dropped: if all of a region's parts are slivers, its largest part is kept.Common geographies at cutoff 0 / 0.01 / 0.05, region counts constant across all three:
The default is calibrated on Rhode Island, whose towns provably did not change between 2010 and 2020:
0.01is the value that recovers exactly its 40 one-to-one county subdivision matches.Bug fix uncovered by the above: the 2010→2020 tract relationship file has 18 columns but was read with a 15-character
col_typesstring, soTRACT_2020was parsed as a number and lost its leading zeros (051000→51000). Those malformed GEOIDs cannot join against tidycensus, so most tracts were silently dropped from any result tongfen'd into 2020. 246 of Rhode Island's 250 and 53 of Vermont's 193 2020 tract identifiers were affected.Other fixes
get_tongfen_correspondence_ca_censusgained acrsargument for the spatial intersections, default3347.get_tongfen_census_ct,get_tongfen_census_daandget_tongfen_ca_census_ct_from_daerroring out withgeo_format=NA.Checks
R CMD check --as-cran: 1 NOTE (HTML Tidy toolchain artifact, not package content). All tests pass.🤖 Generated with Claude Code