Skip to content

v0.3.8 - #15

Open
mountainMath wants to merge 9 commits into
mainfrom
v0.3.8
Open

v0.3.8#15
mountainMath wants to merge 9 commits into
mainfrom
v0.3.8

Conversation

@mountainMath

Copy link
Copy Markdown
Owner

Release branch for v0.3.8.

Breaking changes

  • get_tongfen_ca_census now honours its base_geo, na.rm, tolerance, crs and data_transform arguments, all of which were silently ignored. Most visibly, the documented default base_geo = NULL now returns data without geographic information, where previously the geography of the first dataset came back regardless.
  • Removed area_mismatch_cutoff from get_tongfen_ca_census and get_tongfen_correspondence_ca_census, it never had any effect. check_tongfen_areas is the way to inspect area mismatches.

Performance

  • Correspondence tables are built via a vectorised connected components pass instead of a row-by-row union-find. Dissemination blocks for a large province go from minutes to seconds.
  • The "statcan" method no longer downloads census geometries it does not use, dissolving skips regions that don't need merging, and check_tongfen_areas / aggregate_correspondences got faster.

US census

  • New get_tongfen_correspondence_us_census, for correspondence tables without also fetching the data.

  • Tract correspondence reaches back to the 1990 census (dec1990). Correspondence layer only, the Census Bureau retired the 1990 API endpoint so the data itself has to come from elsewhere (NHGIS) and be combined via tongfen_aggregate.

  • County subdivisions can be matched across the 2010 and 2020 censuses, previously only 2000 and 2010.

  • Correspondence tables no longer chain regions together over slivers. The relationship files are geometric overlays listing every intersection, including boundaries that only shifted slightly, and matching those up merged unrelated regions into one common geography. New min_area_share argument, default 0.01, applied in memory at parse time so cached files stay byte-identical and min_area_share = 0 reproduces the old behaviour. No region is ever dropped: if all of a region's parts are slivers, its largest part is kept.

    Common geographies at cutoff 0 / 0.01 / 0.05, region counts constant across all three:

    regions 0 0.01 0.05
    RI 1990→2000 tract 232 130 179 200
    RI 2000→2010 tract 234 132 192 207
    RI 2010→2020 tract 248 60 198 234
    RI 2010→2020 cousub 40 8 40 40
    VT 1990→2000 tract 181 106 152 166
    VT 2000→2010 tract 179 42 162 174
    VT 2010→2020 tract 187 26 151 168
    VT 2010→2020 cousub 255 62 238 252

    The default is calibrated on Rhode Island, whose towns provably did not change between 2010 and 2020: 0.01 is the value that recovers exactly its 40 one-to-one county subdivision matches.

  • Bug fix uncovered by the above: the 2010→2020 tract relationship file has 18 columns but was read with a 15-character col_types string, so TRACT_2020 was parsed as a number and lost its leading zeros (05100051000). Those malformed GEOIDs cannot join against tidycensus, so most tracts were silently dropped from any result tongfen'd into 2020. 246 of Rhode Island's 250 and 53 of Vermont's 193 2020 tract identifiers were affected.

Other fixes

  • get_tongfen_correspondence_ca_census gained a crs argument for the spatial intersections, default 3347.
  • Missing geographic identifiers no longer merge unrelated regions into one common geography.
  • Fixed a crash when tongfen-ing census tracts across non-adjacent censuses.
  • Fixed get_tongfen_census_ct, get_tongfen_census_da and get_tongfen_ca_census_ct_from_da erroring out with geo_format=NA.
  • US county subdivision data errors out up front on censuses it can't be matched across, instead of failing with "Did not find matching geographic identifiers" after downloading the comparability file and the census data.

Checks

R CMD check --as-cran: 1 NOTE (HTML Tidy toolchain artifact, not package content). All tests pass.

🤖 Generated with Claude Code

mountainMath and others added 9 commits August 10, 2026 19:14
Rewrite the correspondence table computation and fix a family of
arguments that were accepted but never used.

Performance:
- replace the row-by-row union-find in get_tongfen_correspondence with a
  vectorised hook and pointer jumping connected components pass. Verified
  to produce byte-identical TongfenID and TongfenUID on BC/ON DA and DB
  correspondence tables. Dissemination blocks for Ontario go from 437s to
  1.9s, the whole country from hours to 8s
- the "statcan" method no longer downloads census geometries, which it
  never looked at, and get_tongfen_ca_census only downloads the geometry
  of the base_geo dataset
- summarize_geometry_by_group passes single geometry groups through
  instead of sending them to st_union
- collapse_unique_by_row replaces the row-wise apply in
  check_tongfen_areas and aggregate_correspondences

Behaviour:
- get_tongfen_ca_census now passes base_geo, na.rm, tolerance, crs and
  data_transform on instead of dropping them. base_geo = NULL now returns
  data without geographic information as documented
- drop the area_mismatch_cutoff argument, which never had any effect
- get_tongfen_correspondence_ca_census gained a crs argument, default 3347
- NA geographic identifiers no longer merge unrelated regions
- fix crash when tongfen-ing census tracts across non-adjacent censuses
- fix the deprecated get_tongfen_census_* wrappers erroring out on their
  default geo_format=NA
- fix infinite loop guard and a GEOID10 typo in the US correspondence

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Expose `get_tongfen_correspondence_us_census` so US correspondence tables can be
built without also fetching the data, and extend the tract correspondence back to
the 1990 census via the Census Bureau 1990-to-2000 tract relationship files.

The 1990 census stays out of `get_tongfen_us_census` and
`valid_us_census_datasets`: the Census Bureau retired the 1990 API endpoint, so
tidycensus cannot fetch the data. 1990 data has to be obtained elsewhere and
combined with the correspondence table via `tongfen_aggregate`.

Chaining the relationship files is now a loop over consecutive censuses instead
of nested special cases, which verifies byte-identical to the previous
implementation for all combinations of the 2000, 2010 and 2020 censuses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The only county subdivision source so far was the 2000 to 2010 comparability
file, and `datasets` was ignored at that level, so asking for 2020 built a
2000/2010 correspondence and failed downstream with "Did not find matching
geographic identifiers". Add the 2010 to 2020 relationship file and run county
subdivisions through the same chaining loop as census tracts.

Unlike the comparability file that file is a geometric overlay: for Rhode Island
84 of its 124 rows are slivers along boundaries that shifted slightly, three of
them reaching into Massachusetts, and chaining them collapses the state's 40
subdivisions into 8 common geographies. Cut them by the share of area two
subdivisions have in common, exposed as `min_area_share`. At the default of 0.01
Rhode Island comes out as 40 one-to-one matches, which is what it should be, its
towns did not change between 2010 and 2020.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every US relationship file is a geometric overlay listing each overlap between
two censuses, including boundaries that only shifted slightly. Chaining those
merged unrelated regions into one common geography: Vermont's 179 tracts came
out as 42 common geographies across the 2000 and 2010 censuses, its 187 tracts
as 26 across 2010 and 2020.

Apply the same cutoff used for county subdivisions to all four links, computed
from whatever areas the file carries: summed part areas for 1990 to 2000, tract
areas for 2000 to 2010, block areas summed up per tract for 2010 to 2020. With
`min_area_share = 0` the result is identical to before, at the 0.01 default
Rhode Island tracts across 2010 and 2020 give 198 common geographies instead of
60 and Vermont 151 instead of 26. No region is ever dropped, if all of its parts
are slivers its largest part is kept.

Rereading the 2010 to 2020 file also fixes its column types: it has 18 columns
but was read with 15 type specs, so TRACT_2020 was parsed as a number and lost
its leading zeros. 246 of Rhode Island's 250 2020 tract identifiers came out
malformed and could not join against the identifiers tidycensus returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tidycensus defaults the 2020 census to the PL 94-171 redistricting file,
which carries almost none of the variables people tongfen across censuses.
There was no way to reach the DHC file through get_tongfen_us_census, so
2010 to 2020 comparisons of anything but population counts were impossible.

sumfile takes a single value for all censuses or one named by dataset,
since a 2000 to 2020 chain needs a different file per census. NULL keeps
tidycensus' own defaults, so existing calls are unaffected.

Extends the US vignette with a 2010 to 2020 example on the same variables
and the same Bay Area map, plus notes on min_area_share and on reaching
back to 1990 through the correspondence layer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
perf: speed up area-weighted interpolation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants