From 2afb815591143fcf25dbabbe20c775fc24f81bf6 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 19:14:50 -0700 Subject: [PATCH 1/8] v0.3.8: vectorised correspondence tables and honour ignored arguments Rewrite the correspondence table computation and fix a family of arguments that were accepted but never used. Performance: - replace the row-by-row union-find in get_tongfen_correspondence with a vectorised hook and pointer jumping connected components pass. Verified to produce byte-identical TongfenID and TongfenUID on BC/ON DA and DB correspondence tables. Dissemination blocks for Ontario go from 437s to 1.9s, the whole country from hours to 8s - the "statcan" method no longer downloads census geometries, which it never looked at, and get_tongfen_ca_census only downloads the geometry of the base_geo dataset - summarize_geometry_by_group passes single geometry groups through instead of sending them to st_union - collapse_unique_by_row replaces the row-wise apply in check_tongfen_areas and aggregate_correspondences Behaviour: - get_tongfen_ca_census now passes base_geo, na.rm, tolerance, crs and data_transform on instead of dropping them. base_geo = NULL now returns data without geographic information as documented - drop the area_mismatch_cutoff argument, which never had any effect - get_tongfen_correspondence_ca_census gained a crs argument, default 3347 - NA geographic identifiers no longer merge unrelated regions - fix crash when tongfen-ing census tracts across non-adjacent censuses - fix the deprecated get_tongfen_census_* wrappers erroring out on their default geo_format=NA - fix infinite loop guard and a GEOID10 typo in the US correspondence Co-Authored-By: Claude Opus 5 --- DESCRIPTION | 2 +- NEWS.md | 26 ++ R/helpers.R | 252 +++++++++++++------- R/tongfen.R | 41 ++-- R/tongfen_ca.R | 85 ++++--- R/tongfen_ca_deprecated.R | 6 +- R/tongfen_ca_estimate.R | 3 +- R/tongfen_us.R | 2 +- cran-comments.md | 24 +- man/check_tongfen_areas.Rd | 4 + man/get_tongfen_ca_census.Rd | 11 +- man/get_tongfen_correspondence_ca_census.Rd | 12 +- man/tongfen_aggregate.Rd | 11 +- tests/testthat/test-aggregate.R | 81 +++++++ tests/testthat/test-helpers.R | 138 +++++++++++ vignettes/tongfen_ca.Rmd | 4 +- 16 files changed, 534 insertions(+), 168 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index f9aa3c6..c014060 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,7 +1,7 @@ Package: tongfen Type: Package Title: Make Data Based on Different Geographies Comparable -Version: 0.3.7 +Version: 0.3.8 Authors@R: c( person("Jens", "von Bergmann", email = "jens@mountainmath.ca", role = c("aut", "cre"), comment = "creator and maintainer")) Description: Several functions to allow comparisons of data across different geographies, in particular for Canadian census data from different censuses. diff --git a/NEWS.md b/NEWS.md index 3b64630..1d80c50 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,29 @@ +# tongfen v.0.3.8 +## Breaking changes +- `get_tongfen_ca_census` now honours its `base_geo`, `na.rm`, `tolerance`, `crs` and + `data_transform` arguments, all of which were silently ignored. Most visibly, the + documented default `base_geo = NULL` now returns data without geographic information, + where previously the geography of the first dataset was returned. Pass `base_geo` to + get an `sf` object back +- removed the `area_mismatch_cutoff` argument from `get_tongfen_ca_census` and + `get_tongfen_correspondence_ca_census`, it never had any effect. Use `check_tongfen_areas` + to inspect area mismatches, keeping in mind that geographies for different years are + simplified independently and differ in how water features are cut out +## Major changes +- correspondence tables are now built via a vectorised connected components pass instead of + a row-by-row union-find, which makes tongfen on large geographies dramatically faster + (dissemination blocks for a large province: minutes down to seconds) +- the "statcan" method no longer downloads census geometries it does not use +- dissolving geometries skips regions that don't need to be merged +## Minor changes +- `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial + intersections, default is `3347` (Statistics Canada Lambert) +- missing geographic identifiers no longer merge unrelated regions into one common geography +- fix crash when tongfen-ing census tracts across non-adjacent censuses +- fix `get_tongfen_census_ct`, `get_tongfen_census_da` and `get_tongfen_ca_census_ct_from_da` + erroring out when called with `geo_format=NA` +- faster `check_tongfen_areas` and `aggregate_correspondences` + # tongfen v.0.3.7 ## Major changes - accommodate factors in proportional_reaggregate diff --git a/R/helpers.R b/R/helpers.R index a59078b..2d6e332 100644 --- a/R/helpers.R +++ b/R/helpers.R @@ -24,97 +24,116 @@ inner_join_tongfen_correspondence <- function(data,correspondence,link){ -get_tongfen_correspondence <- function(dd){ - hs <- names(dd)[!grepl("TongfenMethod",names(dd))] - index = 1 - ddd<- dd %>% - mutate(TongfenID=!!as.name(hs[index])) - - while (index% nrow > 0) { - ddd<- ddd %>% - mutate(TongfenID=coalesce(.data$TongfenID,paste0(index,"_",!!as.name(hs[index])))) - index <- index + 1 +# Connected components of the "rows sharing an identifier value" graph. +# +# Each row of `dd` is a node, two rows are adjacent if they agree on the value of +# at least one identifier column. Rather than materialising the (potentially huge) +# edge set, each row points at a parent row and we alternate between hooking every +# row onto the smallest parent in each of its identifier groups and pointer +# jumping to collapse the resulting chains. This computes connected components in +# a logarithmic number of fully vectorised passes and never allocates more than a +# handful of vectors of length `nrow(dd)`. +# +# `codes` is a list of integer vectors, one per identifier column, with 0 marking +# missing values (which must not link rows). +connected_components <- function(codes, n) { + comp <- seq_len(n) + repeat { + changed <- FALSE + # hook each row onto the smallest parent among the rows sharing one of its + # identifier values + for (cd in codes) { + keep <- cd > 0L + if (!any(keep)) next + k <- cd[keep] + current <- comp[keep] + o <- order(k, current, method = "radix") + ko <- k[o] + first <- c(TRUE, ko[-1L] != ko[-length(ko)]) + group_min <- integer(max(k)) + group_min[ko[first]] <- current[o][first] + new <- group_min[k] + if (any(new != current)) { + comp[keep] <- new + changed <- TRUE + } + } + # pointer jumping, halves the depth of every chain per pass + repeat { + jumped <- comp[comp] + if (all(jumped == comp)) break + comp <- jumped + changed <- TRUE + } + if (!changed) break } + comp +} - # Optimized connected components using union-find approach - # Build a mapping of all unique IDs to their component root - # This is much faster than repeated group_by operations - - # Create union-find parent mapping - all_ids <- unique(ddd$TongfenID) - parent <- setNames(all_ids, all_ids) - - # Find root with path compression - find_root <- function(x) { - if (parent[x] == x) return(x) - parent[x] <<- find_root(parent[x]) # Path compression - return(parent[x]) - } +# smallest value of `values` within each group of `comp`, returned as a lookup +# vector indexed by component +group_first_sorted <- function(comp, values, k) { + o <- order(comp, values, method = "radix") + co <- comp[o] + first <- c(TRUE, co[-1L] != co[-length(co)]) + out <- character(k) + out[co[first]] <- values[o][first] + out +} - # Union two components - union_ids <- function(x, y) { - root_x <- find_root(x) - root_y <- find_root(y) - if (root_x != root_y) { - # Always attach to the smaller ID (alphabetically) - if (root_x < root_y) { - parent[root_y] <<- root_x - } else { - parent[root_x] <<- root_y - } - } +get_tongfen_correspondence <- function(dd){ + hs <- names(dd)[!grepl("TongfenMethod",names(dd))] + n <- nrow(dd) + if (n == 0) { + return(dd %>% mutate(TongfenID=character(0),TongfenUID=character(0)) %>% ungroup()) } - # For each identifier column, union all IDs that share the same identifier value - for (nn in hs) { - id_groups <- ddd %>% - select(identifier = !!as.name(nn), "TongfenID") %>% - distinct() %>% - group_by(.data$identifier) %>% - summarise(ids = list(.data$TongfenID), .groups = "drop") - - for (i in seq_len(nrow(id_groups))) { - ids_in_group <- id_groups$ids[[i]] - if (length(ids_in_group) > 1) { - # Union all pairs in this group - for (j in 2:length(ids_in_group)) { - union_ids(ids_in_group[1], ids_in_group[j]) - } - } + values <- lapply(hs, function(nn) as.character(dd[[nn]])) + names(values) <- hs + + # dense integer codes per column, 0 for NA so that missing identifiers never + # link two rows together + codes <- lapply(values, function(v) { + match(v, c(NA_character_, sort(unique(v[!is.na(v)])))) - 1L + }) + + comp <- connected_components(codes, n) + k <- max(comp) + + # label each row by its first available identifier, later columns are prefixed + # by their position so that identifiers from different columns cannot collide + base <- values[[1]] + if (anyNA(base) && length(hs) > 1) { + for (i in seq(2, length(hs))) { + missing <- is.na(base) + if (!any(missing)) break + v <- values[[i]][missing] + base[missing] <- ifelse(is.na(v), NA_character_, paste0(i, "_", v)) } } - - # Apply the final mapping - find root for each ID - final_mapping <- setNames( - vapply(all_ids, find_root, character(1), USE.NAMES = FALSE), - all_ids - ) - - # Map all TongfenIDs to their roots - ddd <- ddd %>% - mutate(TongfenID = final_mapping[.data$TongfenID]) - - # Vectorized UID generation - # Pre-compute all the grouped values at once - uid_parts <- ddd %>% - group_by(.data$TongfenID) %>% - summarise( - across( - all_of(hs), - ~paste0(cur_column(), ":", paste0(sort(unique(.x)), collapse = ",")), - .names = "uid_{.col}" - ), - .groups = "drop" - ) %>% - mutate( - TongfenUID = do.call(paste, c(select(., starts_with("uid_")), sep = " ")) - ) %>% - select("TongfenID", "TongfenUID") - - # Join the UIDs back - ddd %>% - left_join(uid_parts, by = "TongfenID") %>% - ungroup() + if (anyNA(base)) base[is.na(base)] <- paste0("row_", which(is.na(base))) + + # TongfenID is the smallest row label in the component + dd$TongfenID <- group_first_sorted(comp, base, k)[comp] + + # TongfenUID enumerates all identifiers making up the component + uid_parts <- lapply(hs, function(nn) { + v <- values[[nn]] + ok <- !is.na(v) + cc <- comp[ok] + vv <- v[ok] + o <- order(cc, vv, method = "radix") + cc <- cc[o] + vv <- vv[o] + keep <- c(TRUE, cc[-1L] != cc[-length(cc)] | vv[-1L] != vv[-length(vv)]) + collapsed <- vapply(split(vv[keep], cc[keep]), paste0, character(1), collapse = ",") + out <- character(k) + out[as.integer(names(collapsed))] <- paste0(nn, ":", collapsed) + out + }) + dd$TongfenUID <- do.call(paste, c(uid_parts, list(sep = " ")))[comp] + + dd %>% ungroup() } assert <- function (expr, error) { @@ -122,18 +141,67 @@ assert <- function (expr, error) { } +# Dissolve the geometries of `data` by `grouping_var`, the geometric equivalent +# of `summarize()`. Groups holding a single geometry - the bulk of the groups +# when tongfen-ing fine geographies - are passed through directly instead of +# being sent through `st_union()`, which is where most of the time in dissolving +# large geographies goes. +summarize_geometry_by_group <- function(data,grouping_var){ + geo_column <- attr(data,"sf_column") + keys <- data %>% ungroup() %>% sf::st_drop_geometry() %>% select(all_of(grouping_var)) + key <- do.call(paste,c(unname(as.list(keys)),list(sep="\x1f"))) + u <- unique(key) + u <- u[order(u,method="radix")] + index <- match(key,u) + geometry <- data[[geo_column]] + + counts <- tabulate(index,nbins=length(u)) + # first row belonging to each group, groups are in sorted key order + first_row <- integer(length(u)) + first_row[rev(index)] <- rev(seq_along(index)) + + result <- vector("list",length(u)) + singles <- counts==1L + result[singles] <- geometry[first_row[singles]] + multi <- which(counts>1L) + if (length(multi)>0) { + rows <- split(seq_along(index)[index %in% multi],index[index %in% multi]) + result[as.integer(names(rows))] <- lapply(rows,function(i) + suppressMessages(sf::st_union(geometry[i]))[[1]]) + } + + out <- keys[first_row,,drop=FALSE] + out[[geo_column]] <- sf::st_sfc(result,crs=sf::st_crs(geometry)) %>% + sf::st_cast("MULTIPOLYGON") + sf::st_sf(out,sf_column_name=geo_column) +} + + +# row-wise `paste0(unique(...),collapse=", ")` over a set of columns. Values are +# drawn from a small vocabulary, so the collapse is only computed once per +# distinct combination rather than once per row. +collapse_unique_by_row <- function(data,columns){ + sep <- "\x1f" + key <- do.call(paste,c(unname(as.list(data[columns])),list(sep=sep))) + u <- unique(key) + collapsed <- vapply(strsplit(u,sep,fixed=TRUE), + function(x) paste0(unique(x),collapse=", "), + character(1)) + unname(collapsed[match(key,u)]) +} + aggregate_correspondences <- function(correspondences){ clean_correspondence_names <- function(correspondence) { correspondence %>% select(!matches("Tongfen") | matches("TongfenMethod")) } - # compute full correspondence - # order by length to speed up the process - lengths <- correspondences %>% lapply(nrow) %>% unlist %>% rank(ties.method = "first") + # compute full correspondence, smallest table first to keep intermediate + # join results as small as possible + index_order <- correspondences %>% lapply(nrow) %>% unlist() %>% order() - correspondence <- correspondences[[lengths[1]]] %>% + correspondence <- correspondences[[index_order[1]]] %>% clean_correspondence_names() - if (length(correspondences)>1) for (index in lengths[-1]) { + if (length(correspondences)>1) for (index in index_order[-1]) { c <- correspondences[[index]] %>% clean_correspondence_names() match_columns <- intersect(names(correspondence),names(c)) @@ -143,9 +211,9 @@ aggregate_correspondences <- function(correspondences){ } method_columns <- names(correspondence)[grepl("TongfenMethod",names(correspondence))] - correspondence$M <- apply(correspondence[,method_columns],1,function(d)paste0(unique(d),collapse = ", ")) - correspondence %>% select(-method_columns) %>% - rename(TongfenMethod=.data$M) + correspondence$M <- collapse_unique_by_row(correspondence,method_columns) + correspondence %>% select(-all_of(method_columns)) %>% + rename(TongfenMethod="M") } diff --git a/R/tongfen.R b/R/tongfen.R index 79a6e9c..f3ccac3 100644 --- a/R/tongfen.R +++ b/R/tongfen.R @@ -141,11 +141,7 @@ aggregate_data_with_meta <- function(data,meta,geo=FALSE,na.rm=TRUE,quiet=FALSE) meta <- meta %>% bind_rows(tibble(variable=base_variables,type="Base")) if ("sf" %in% class(data)) { - geo_column=attr(data,"sf_column") - data <- left_join(data %>% - select(c(geo_column,grouping_var)) %>% - summarize(!!geo_column:=suppressMessages(sf::st_union(!!as.name(geo_column))) %>% - sf::st_cast("MULTIPOLYGON")), + data <- left_join(summarize_geometry_by_group(data,grouping_var), data %>% sf::st_set_geometry(NULL) %>% summarize_at(meta$variable,sum,na.rm=na.rm), @@ -198,6 +194,8 @@ rename_with_meta <- function(data,meta,ds=NULL){ #' @param base_geo identifier for which data element to base the final geography on, #' uses the first data element if `NULL` (default), #' expects that `base_geo` is an element of `names(data)`. +#' @param na.rm logical, determines how NA values should be treated when aggregating variables, +#' default is `TRUE` #' @return aggregated dataset of class sf if base_geo is not NULL and data is of type sf or tibble otherwise. #' @export #' @@ -214,7 +212,7 @@ rename_with_meta <- function(data,meta,ds=NULL){ #' result <- tongfen_aggregate(list(geo1 %>% rename(GeoUIDCA06=GeoUID), #' geo2 %>% rename(GeoUIDCA16=GeoUID)),correspondence,meta) #'} -tongfen_aggregate <- function(data,correspondence,meta=NULL, base_geo = NULL){ +tongfen_aggregate <- function(data,correspondence,meta=NULL, base_geo = NULL, na.rm = TRUE){ data <- ensure_names(data) nn <- names(data) if (is.null(base_geo)) base_geo <- nn[1] @@ -240,14 +238,10 @@ tongfen_aggregate <- function(data,correspondence,meta=NULL, base_geo = NULL){ by=match_column) %>% group_by(.data$TongfenID,.data$TongfenUID) if (!is.null(meta)) { - d <- d %>% aggregate_data_with_meta(meta) + d <- d %>% aggregate_data_with_meta(meta,na.rm=na.rm) } else { if ("sf" %in% class(d)) { - geo_column=attr(d,"sf_column") - d <- d %>% summarize(!!geo_column:=suppressMessages(sf::st_union(!!as.name(geo_column))) %>% - sf::st_cast("MULTIPOLYGON"), - .groups="drop") - + d <- summarize_geometry_by_group(d,c("TongfenID","TongfenUID")) } else { d <- d %>% summarize(.groups="drop") } @@ -261,7 +255,7 @@ tongfen_aggregate <- function(data,correspondence,meta=NULL, base_geo = NULL){ for (ds in nn[nn!=base_geo]) { aggregated_data <- aggregated_data %>% inner_join(data_new[[ds]] %>% - select(-.data$TongfenUID) %>% + select(-"TongfenUID") %>% rename_with_meta(meta,ds), by="TongfenID") } @@ -351,7 +345,7 @@ proportional_reaggregate <- function(data,parent_data,geo_match,categories,base= na_base <- "...na_base" while (na_base %in% names(data)) { - na_base <- paste0(na_base) + na_base <- paste0("...",na_base) } @@ -615,6 +609,10 @@ estimate_tongfen_correspondence <- function(data, #' Sanity check for areas of estimated tongfen correspondence. This is useful if for example the total extent #' of geo1 and geo2 differ and there are regions at the edges with large difference in overlap. #' +#' The result is a diagnostic, not a pass/fail test. Geographies for different years are +#' simplified independently and differ in how water features are cut out, so a sizable area +#' mismatch does not by itself mean the regions were matched up incorrectly. +#' #' @param data alist of geogrpahic data of class sf #' @param correspondence Correspondence table with columns the unique geographic identifiers for each of the #' geographies and the TongfenID (and optionally TongfenUID and TongfenMethod) @@ -668,14 +666,15 @@ check_tongfen_areas <- function(data,correspondence) { purrr::reduce(full_join,by="TongfenID") method_columns <- names(summary_data)[grepl("TongfenMethod",names(summary_data))] - summary_data$M <- apply(summary_data[,method_columns],1,function(d)paste0(unique(d),collapse = ", ")) + summary_data$M <- collapse_unique_by_row(summary_data,method_columns) + summary_data <- summary_data %>% + select(-all_of(method_columns)) %>% + rename(TongfenMethod="M") + + area_columns <- names(summary_data)[grepl("^area_",names(summary_data))] + areas <- lapply(area_columns,function(cc) as.numeric(summary_data[[cc]])) summary_data %>% - select(-method_columns) %>% - rename(TongfenMethod=.data$M) %>% - mutate(maxa=apply(select(.,matches("^area_")),1,max), - mina=apply(select(.,matches("^area_")),1,min)) %>% - mutate(max_log_ratio=log(.data$maxa/.data$mina)) %>% - select(-.data$maxa,-.data$mina) + mutate(max_log_ratio=log(do.call(pmax,areas)/do.call(pmin,areas))) } diff --git a/R/tongfen_ca.R b/R/tongfen_ca.R index d212812..def0d44 100644 --- a/R/tongfen_ca.R +++ b/R/tongfen_ca.R @@ -198,7 +198,7 @@ get_single_correspondence_ca_census_for <- function(year,level=c("DA","DB"),refr tmp=tempfile() utils::download.file(url,tmp) exdir=file.path(tempdir(),paste0("correspondence_",year,"_",level)) - if (dir.exists(exdir)) file.remove(exdir,recursive=TRUE) + if (dir.exists(exdir)) unlink(exdir,recursive=TRUE) dir.create(exdir,showWarnings = FALSE) utils::unzip(tmp,exdir=exdir) file=dir(exdir,"\\.txt|\\.csv") @@ -250,10 +250,12 @@ get_single_correspondence_ca_census_for <- function(year,level=c("DA","DB"),refr #' this method only works for "DB", "DA" and "CT" levels. #' * "estimate" uses `estimate_tongfen_correspondence` to build up the common geography from scratch based on geographies. #' * "identifier" assumes regions with identical geographic identifier are identical, and builds up the the correspondence for regions with unmatched geographic identifiers. -#' @param tolerance tolerance for `estimate_tongen_correspondence` in metres, default value is 50 metres. -#' @param area_mismatch_cutoff discard areas returned by `estimate_tongfen_correspondence` with area mismatch (log ratio) greater than cutoff. +#' @param tolerance tolerance for `estimate_tongen_correspondence` in metres, default value is 50 metres, +#' only used when method is 'estimate' or 'identifier' #' @param quiet suppress download progress output, default is `FALSE` #' @param refresh optional character, refresh data cache for this call, (default `FALSE`) +#' @param crs CRS to use for the spatial intersections if method is 'identifier' or +#' 'estimate', default is `3347` (Statistics Canada Lambert) #' @return dataframe with the multi-census correspondence file #' @export #' @@ -264,8 +266,8 @@ get_single_correspondence_ca_census_for <- function(year,level=c("DA","DB"),refr #' regions=list(CMA="59933"),level='CT') #'} get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="CT", method="statcan", - tolerance = 50, area_mismatch_cutoff = 0.1, - quiet = FALSE, refresh = FALSE) { + tolerance = 50, + quiet = FALSE, refresh = FALSE, crs = 3347) { geo_datasets <- normalize_datasets(geo_datasets) if (method=="statcan") { @@ -282,12 +284,21 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C use_cache <- !refresh - data <- lapply(geo_datasets,function(g_ds){ - cancensus::get_census(dataset=g_ds, regions=regions, level=level, geo_format='sf', - labels="short", quiet=quiet, use_cache = use_cache) %>% - mutate(!!paste0("GeoUID",g_ds):=.data$GeoUID) - }) %>% - setNames(geo_datasets) + # the "statcan" method only ever looks at geographic identifiers, only the + # geometry based methods need to download the (potentially very large) geometries + geo_format <- if (method=="statcan") NA else 'sf' + + if (method=="statcan" && level=="CT") { + # correspondence is built from DA level links below, the CT level data is not needed + data <- list() + } else { + data <- lapply(geo_datasets,function(g_ds){ + cancensus::get_census(dataset=g_ds, regions=regions, level=level, geo_format=geo_format, + labels="short", quiet=quiet, use_cache = use_cache) %>% + mutate(!!paste0("GeoUID",g_ds):=.data$GeoUID) + }) %>% + setNames(geo_datasets) + } if (method=="statcan") { statcan_level <- level @@ -296,10 +307,12 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C years<-as.integer(geo_years) all_geo_years=seq(min(years),max(years),5) all_geo_datasets <- geo_dataset_for_years(all_geo_years) - for (g_ds in setdiff(all_geo_datasets,geo_datasets)) { - data[[g_ds]] <- cancensus::get_census(dataset=g_ds, regions=regions, level=level, geo_format='sf', - labels="short", quiet=quiet, use_cache = use_cache) %>% - mutate(!!paste0("GeoUID",g_ds):=.data$GeoUID) + if (level!="CT") { + for (g_ds in setdiff(all_geo_datasets,geo_datasets)) { + data[[g_ds]] <- cancensus::get_census(dataset=g_ds, regions=regions, level=level, geo_format=geo_format, + labels="short", quiet=quiet, use_cache = use_cache) %>% + mutate(!!paste0("GeoUID",g_ds):=.data$GeoUID) + } } prefix=paste0(statcan_level,"UID") @@ -321,7 +334,7 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C base_column <- paste0(prefix,year) match_column <- paste0("GeoUID",ds) data[[ds]] %>% - st_set_geometry(NULL) %>% + sf::st_drop_geometry() %>% select_at(match_column) %>% mutate(!!base_column:=!!as.name(match_column)) }) %>% @@ -340,7 +353,7 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C ds2 <- all_geo_datasets[all_geo_years==previous_year] if (!is.null(ds1) && length(ds1)>0) { match_column <- intersect(names(c),names(c_links[[ds1]])) - if (!is.null(match_column)) { + if (length(match_column)>0) { c <- c %>% inner_join(c_links[[ds1]],by=match_column) %>% select(-all_of(match_column)) %>% @@ -349,7 +362,7 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C } if (!is.null(ds2) && length(ds2)>0) { match_column <- intersect(names(c),names(c_links[[ds2]])) - if (!is.null(match_column)) { + if (length(match_column)>0) { c <- c %>% inner_join(c_links[[ds2]],by=match_column) %>% select(-all_of(match_column)) %>% @@ -369,8 +382,8 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C correspondence <- estimate_tongfen_correspondence(data, geo_identifiers, method = method, - tolerance=200, - computation_crs=3347) + tolerance=tolerance, + computation_crs=crs) } correspondence @@ -394,16 +407,15 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C #' * "estimate" uses `estimate_tongfen_correspondence` to build up the common geography from scratch based on geographies. #' * "identifier" assumes regions with identical geographic identifier are identical, and builds up the the correspondence for regions with unmatched geographic identifiers. #' @param base_geo base census year to build up common geography from, `NULL` (the default) to not return -#' any geographi data -#' @param na.rm logical, determines how NA values should be treated when aggregating variables +#' any geographic data +#' @param na.rm logical, determines how NA values should be treated when aggregating variables, +#' default is `FALSE` #' @param tolerance tolerance for `estimate_tongen_correspondence` in metres, default value is 50 metres, #' only used when method is 'estimate' or 'identifier' -#' @param area_mismatch_cutoff discard areas returned by `estimate_tongfen_correspondence` with area mismatch (log ratio) greater than cutoff, -#' only used when method is 'estimate' or 'identifier' #' @param quiet suppress download progress output, default is `FALSE` #' @param refresh optional character, refresh data cache for this call, (default `FALSE`) #' @param crs optional CRS to transform data to, and use for spatial intersections if method is -#' 'identifier' or 'estimate' +#' 'identifier' or 'estimate', defaults to `3347` (Statistics Canada Lambert) for the intersections #' @param data_transform optional transform function to be applied to census data after being returned from cancensus #' @return dataframe with variables on common geography #' @export @@ -423,7 +435,6 @@ get_tongfen_correspondence_ca_census <- function(geo_datasets, regions, level="C get_tongfen_ca_census <- function(regions,meta,level="CT",method="statcan", base_geo=NULL,na.rm=FALSE, tolerance = 50, - area_mismatch_cutoff = 0.1, quiet=FALSE, refresh=FALSE, crs=NULL, @@ -432,6 +443,13 @@ get_tongfen_ca_census <- function(regions,meta,level="CT",method="statcan", geo_datasets <- meta$geo_dataset %>% unique() %>% sort() + if (!is.null(base_geo)) { + base_geo <- normalize_datasets(base_geo) + assert(length(base_geo)==1,"base_geo has to be a single dataset") + assert(base_geo %in% geo_datasets, + paste0("base_geo has to be one of the datasets ",paste0(geo_datasets,collapse=", "))) + } + meta <- meta %>% add_census_ca_base_variables() data <- lapply(geo_datasets,function(g_ds){ @@ -440,13 +458,15 @@ get_tongfen_ca_census <- function(regions,meta,level="CT",method="statcan", .data$type != "Base") %>% pull(.data$variable) %>% as.character() + # only the base geography is returned, no need to download geometries for the others + geo_format <- if (!is.null(base_geo) && g_ds==base_geo) 'sf' else NA c <- cancensus::get_census(dataset=g_ds, regions=regions, vectors=vectors, - level=level, geo_format='sf', + level=level, geo_format=geo_format, labels="short", quiet=quiet, use_cache = use_cache) %>% mutate(!!paste0("GeoUID",g_ds):=.data$GeoUID) - if (!is.null(crs)) c <- c %>% sf::st_transform(crs) - c + if (!is.null(crs) && !is.null(base_geo) && g_ds==base_geo) c <- c %>% sf::st_transform(crs) + c %>% data_transform() }) %>% setNames(geo_datasets) @@ -460,10 +480,11 @@ get_tongfen_ca_census <- function(regions,meta,level="CT",method="statcan", level = level, method = method, tolerance = tolerance, - area_mismatch_cutoff = area_mismatch_cutoff, quiet = quiet, - refresh = refresh) - aggregated_data <- tongfen_aggregate(data,correspondence,meta) + refresh = refresh, + crs = crs %||% 3347) + aggregated_data <- tongfen_aggregate(data,correspondence,meta, + base_geo=base_geo,na.rm=na.rm) } aggregated_data %>% rename_with_meta(meta) diff --git a/R/tongfen_ca_deprecated.R b/R/tongfen_ca_deprecated.R index 5d5e77f..40c4d75 100644 --- a/R/tongfen_ca_deprecated.R +++ b/R/tongfen_ca_deprecated.R @@ -23,7 +23,7 @@ get_tongfen_census_ct <- function(regions, lifecycle::deprecate_warn("0.2.0", "get_tongfen_census_ct()", "get_tongfen_ca_census()") #warning("This method is deprecated, use `get_tongfen_ca_census(regions,vectors,level='CT', method = 'identifier', tolerance = 500, ...)` instead") meta <- meta_for_ca_census_vectors(vectors) - base_geo <- ifelse(is.na(geo_format),NULL,meta$geo_dataset %>% unique %>% sort %>% first ) + base_geo <- if (is.na(geo_format)) NULL else meta$geo_dataset %>% unique() %>% sort() %>% first() get_tongfen_ca_census(regions = regions, meta = meta, @@ -55,7 +55,7 @@ get_tongfen_census_ct <- function(regions, get_tongfen_census_da <- function(regions,vectors,geo_format=NA,use_cache=TRUE,na.rm=TRUE,quiet=TRUE) { lifecycle::deprecate_warn("0.2.0", "get_tongfen_census_da()", "get_tongfen_ca_census()") meta <- meta_for_ca_census_vectors(vectors) - base_geo <- ifelse(is.na(geo_format),NULL,meta$geo_dataset %>% unique %>% sort %>% first ) + base_geo <- if (is.na(geo_format)) NULL else meta$geo_dataset %>% unique() %>% sort() %>% first() get_tongfen_ca_census(regions = regions, meta = meta, @@ -87,7 +87,7 @@ get_tongfen_census_da <- function(regions,vectors,geo_format=NA,use_cache=TRUE,n get_tongfen_ca_census_ct_from_da <- function(regions,vectors,geo_format=NA,use_cache=TRUE,na.rm=TRUE,quiet=TRUE) { lifecycle::deprecate_warn("0.2.0", "get_tongfen_census_da()", "get_tongfen_census_ca()") meta <- meta_for_ca_census_vectors(vectors) - base_geo <- ifelse(is.na(geo_format),NULL,meta$geo_dataset %>% unique %>% sort %>% first ) + base_geo <- if (is.na(geo_format)) NULL else meta$geo_dataset %>% unique() %>% sort() %>% first() get_tongfen_ca_census(regions = regions, meta = meta, diff --git a/R/tongfen_ca_estimate.R b/R/tongfen_ca_estimate.R index 90cd31c..badd6b5 100644 --- a/R/tongfen_ca_estimate.R +++ b/R/tongfen_ca_estimate.R @@ -67,7 +67,8 @@ tongfen_estimate_ca_census <- function(geometry, meta, level, # So maybe a function like get_tongfen_correspondence_from_seed census_data <- get_tongfen_ca_census(regions = regions, meta = meta, - level = level, na.rm = na.rm, quiet = quiet) %>% + level = level, base_geo = datasets, + na.rm = na.rm, quiet = quiet) %>% sf::st_transform(st_crs(geometry)) if (!is.null(downsample_level)){ diff --git a/R/tongfen_us.R b/R/tongfen_us.R index 4b27553..4101170 100644 --- a/R/tongfen_us.R +++ b/R/tongfen_us.R @@ -75,7 +75,7 @@ get_us_ct_correspondence <- function(state, datasets, select(matches("^GEOID\\d{2}$")) c <- full_join(c,c2,by="GEOID10") } - if (!("dec2010" %in% datasets)) c <- c %>% select(-.data$GEOIOD10) + if (!("dec2010" %in% datasets)) c <- c %>% select(-.data$GEOID10) c <- c %>% unique } else if ("dec2020" %in% datasets) { c<-get_us_ct_correspondence_2020(state,cache_path=cache_path) %>% diff --git a/cran-comments.md b/cran-comments.md index 4459ea3..c288a43 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -1,3 +1,20 @@ +# tongfen v.0.3.8 +## Breaking changes +- `get_tongfen_ca_census` now honours its `base_geo`, `na.rm`, `tolerance`, `crs` and + `data_transform` arguments, all of which were silently ignored +- removed the `area_mismatch_cutoff` argument from `get_tongfen_ca_census` and + `get_tongfen_correspondence_ca_census`, it never had any effect +## Major changes +- correspondence tables are now built via a vectorised connected components pass instead of + a row-by-row union-find, making tongfen on large geographies dramatically faster +- the "statcan" method no longer downloads census geometries it does not use +- dissolving geometries skips regions that don't need to be merged +## Minor changes +- `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections +- missing geographic identifiers no longer merge unrelated regions into one common geography +- fix crash when tongfen-ing census tracts across non-adjacent censuses +- several fixes to the deprecated `get_tongfen_census_*` functions + # tongfen v.0.3.7 ## Major changes - accommodate factors in proportional_reaggregate @@ -26,7 +43,10 @@ # Submission - v.0.3 # Test environments -* local R installation, R 4.0.2 -* GitHub action release +* local macOS installation, R 4.6.0 +* GitHub actions (windows-latest, macOS-latest, ubuntu-latest) on release, devel and oldrel + +# R CMD check results +0 errors | 0 warnings | 0 notes diff --git a/man/check_tongfen_areas.Rd b/man/check_tongfen_areas.Rd index 7257baa..d03414a 100644 --- a/man/check_tongfen_areas.Rd +++ b/man/check_tongfen_areas.Rd @@ -23,6 +23,10 @@ of the areas. Sanity check for areas of estimated tongfen correspondence. This is useful if for example the total extent of geo1 and geo2 differ and there are regions at the edges with large difference in overlap. + +The result is a diagnostic, not a pass/fail test. Geographies for different years are +simplified independently and differ in how water features are cut out, so a sizable area +mismatch does not by itself mean the regions were matched up incorrectly. } \examples{ # Estimate a common geography for 2006 and 2016 dissemination areas in the City of Vancouver diff --git a/man/get_tongfen_ca_census.Rd b/man/get_tongfen_ca_census.Rd index 5a4f214..b5d388c 100644 --- a/man/get_tongfen_ca_census.Rd +++ b/man/get_tongfen_ca_census.Rd @@ -12,7 +12,6 @@ get_tongfen_ca_census( base_geo = NULL, na.rm = FALSE, tolerance = 50, - area_mismatch_cutoff = 0.1, quiet = FALSE, refresh = FALSE, crs = NULL, @@ -34,22 +33,20 @@ this method only works for "DB", "DA" and "CT" levels. * "identifier" assumes regions with identical geographic identifier are identical, and builds up the the correspondence for regions with unmatched geographic identifiers.} \item{base_geo}{base census year to build up common geography from, `NULL` (the default) to not return -any geographi data} +any geographic data} -\item{na.rm}{logical, determines how NA values should be treated when aggregating variables} +\item{na.rm}{logical, determines how NA values should be treated when aggregating variables, +default is `FALSE`} \item{tolerance}{tolerance for `estimate_tongen_correspondence` in metres, default value is 50 metres, only used when method is 'estimate' or 'identifier'} -\item{area_mismatch_cutoff}{discard areas returned by `estimate_tongfen_correspondence` with area mismatch (log ratio) greater than cutoff, -only used when method is 'estimate' or 'identifier'} - \item{quiet}{suppress download progress output, default is `FALSE`} \item{refresh}{optional character, refresh data cache for this call, (default `FALSE`)} \item{crs}{optional CRS to transform data to, and use for spatial intersections if method is -'identifier' or 'estimate'} +'identifier' or 'estimate', defaults to `3347` (Statistics Canada Lambert) for the intersections} \item{data_transform}{optional transform function to be applied to census data after being returned from cancensus} } diff --git a/man/get_tongfen_correspondence_ca_census.Rd b/man/get_tongfen_correspondence_ca_census.Rd index 2c3e1f4..ad8d6c8 100644 --- a/man/get_tongfen_correspondence_ca_census.Rd +++ b/man/get_tongfen_correspondence_ca_census.Rd @@ -10,9 +10,9 @@ get_tongfen_correspondence_ca_census( level = "CT", method = "statcan", tolerance = 50, - area_mismatch_cutoff = 0.1, quiet = FALSE, - refresh = FALSE + refresh = FALSE, + crs = 3347 ) } \arguments{ @@ -28,13 +28,15 @@ this method only works for "DB", "DA" and "CT" levels. * "estimate" uses `estimate_tongfen_correspondence` to build up the common geography from scratch based on geographies. * "identifier" assumes regions with identical geographic identifier are identical, and builds up the the correspondence for regions with unmatched geographic identifiers.} -\item{tolerance}{tolerance for `estimate_tongen_correspondence` in metres, default value is 50 metres.} - -\item{area_mismatch_cutoff}{discard areas returned by `estimate_tongfen_correspondence` with area mismatch (log ratio) greater than cutoff.} +\item{tolerance}{tolerance for `estimate_tongen_correspondence` in metres, default value is 50 metres, +only used when method is 'estimate' or 'identifier'} \item{quiet}{suppress download progress output, default is `FALSE`} \item{refresh}{optional character, refresh data cache for this call, (default `FALSE`)} + +\item{crs}{CRS to use for the spatial intersections if method is 'identifier' or +'estimate', default is `3347` (Statistics Canada Lambert)} } \value{ dataframe with the multi-census correspondence file diff --git a/man/tongfen_aggregate.Rd b/man/tongfen_aggregate.Rd index d2a90f3..e54da9d 100644 --- a/man/tongfen_aggregate.Rd +++ b/man/tongfen_aggregate.Rd @@ -4,7 +4,13 @@ \alias{tongfen_aggregate} \title{Perform tongfen according to correspondence} \usage{ -tongfen_aggregate(data, correspondence, meta = NULL, base_geo = NULL) +tongfen_aggregate( + data, + correspondence, + meta = NULL, + base_geo = NULL, + na.rm = TRUE +) } \arguments{ \item{data}{list of datasets to be aggregated} @@ -16,6 +22,9 @@ tongfen_aggregate(data, correspondence, meta = NULL, base_geo = NULL) \item{base_geo}{identifier for which data element to base the final geography on, uses the first data element if `NULL` (default), expects that `base_geo` is an element of `names(data)`.} + +\item{na.rm}{logical, determines how NA values should be treated when aggregating variables, +default is `TRUE`} } \value{ aggregated dataset of class sf if base_geo is not NULL and data is of type sf or tibble otherwise. diff --git a/tests/testthat/test-aggregate.R b/tests/testthat/test-aggregate.R index d7439db..378192e 100644 --- a/tests/testthat/test-aggregate.R +++ b/tests/testthat/test-aggregate.R @@ -153,3 +153,84 @@ test_that("pre_scale is a no-op when there are no Average/Median variables", { result <- tongfen:::pre_scale(data, meta, quiet = TRUE) expect_equal(result, data) }) + +# ── tongfen_aggregate ───────────────────────────────────────────────────────── + +make_tongfen_data <- function(pop_a = c(100L, 200L, 50L)) { + correspondence <- tibble( + GeoUIDa = c("a1", "a2", "a3"), + GeoUIDb = c("b1", "b1", "b2"), + TongfenID = c("a1", "a1", "a3"), + TongfenUID = c("u1", "u1", "u2") + ) + data <- list( + A = tibble(GeoUIDa = c("a1", "a2", "a3"), pop_a = pop_a), + B = tibble(GeoUIDb = c("b1", "b2"), pop_b = c(300L, 50L)) + ) + list(data = data, correspondence = correspondence) +} + +test_that("tongfen_aggregate: aggregates both datasets onto the common geography", { + d <- make_tongfen_data() + meta <- bind_rows( + make_meta("pop_a", "Additive") %>% mutate(dataset = "A", geo_dataset = "A"), + make_meta("pop_b", "Additive") %>% mutate(dataset = "B", geo_dataset = "B") + ) + result <- tongfen_aggregate(d$data, d$correspondence, meta) + expect_equal(nrow(result), 2L) + expect_equal(result %>% filter(.data$TongfenID == "a1") %>% pull(.data$pop_a), 300L) + expect_equal(result %>% filter(.data$TongfenID == "a1") %>% pull(.data$pop_b), 300L) +}) + +test_that("tongfen_aggregate: na.rm is passed through to the aggregation", { + d <- make_tongfen_data(pop_a = c(100L, NA_integer_, 50L)) + meta <- bind_rows( + make_meta("pop_a", "Additive") %>% mutate(dataset = "A", geo_dataset = "A"), + make_meta("pop_b", "Additive") %>% mutate(dataset = "B", geo_dataset = "B") + ) + kept <- tongfen_aggregate(d$data, d$correspondence, meta, na.rm = FALSE) + expect_true(is.na(kept %>% filter(.data$TongfenID == "a1") %>% pull(.data$pop_a))) + + dropped <- tongfen_aggregate(d$data, d$correspondence, meta, na.rm = TRUE) + expect_equal(dropped %>% filter(.data$TongfenID == "a1") %>% pull(.data$pop_a), 100L) +}) + +test_that("tongfen_aggregate: base_geo determines which geometry is returned", { + skip_if_not_installed("sf") + square <- function(x, y) { + sf::st_polygon(list(cbind(c(x, x + 1, x + 1, x, x), + c(y, y, y + 1, y + 1, y)))) + } + correspondence <- tibble( + GeoUIDa = c("a1", "a2"), + GeoUIDb = c("b1", "b1"), + TongfenID = c("a1", "a1"), + TongfenUID = c("u1", "u1") + ) + data <- list( + A = sf::st_sf(GeoUIDa = c("a1", "a2"), pop_a = c(100L, 200L), + geometry = sf::st_sfc(square(0, 0), square(1, 0), crs = 3347)), + B = sf::st_sf(GeoUIDb = "b1", pop_b = 300L, + geometry = sf::st_sfc(square(0, 0), crs = 3347)) + ) + meta <- bind_rows( + make_meta("pop_a", "Additive") %>% mutate(dataset = "A", geo_dataset = "A"), + make_meta("pop_b", "Additive") %>% mutate(dataset = "B", geo_dataset = "B") + ) + # base geography A is the union of the two squares, base geography B is one square + result_a <- tongfen_aggregate(data, correspondence, meta, base_geo = "A") + result_b <- tongfen_aggregate(data, correspondence, meta, base_geo = "B") + expect_s3_class(result_a, "sf") + expect_equal(as.numeric(sf::st_area(result_a)), 2) + expect_equal(as.numeric(sf::st_area(result_b)), 1) +}) + +test_that("tongfen_aggregate: returns a plain tibble when no dataset has geometry", { + d <- make_tongfen_data() + meta <- bind_rows( + make_meta("pop_a", "Additive") %>% mutate(dataset = "A", geo_dataset = "A"), + make_meta("pop_b", "Additive") %>% mutate(dataset = "B", geo_dataset = "B") + ) + result <- tongfen_aggregate(d$data, d$correspondence, meta) + expect_false("sf" %in% class(result)) +}) diff --git a/tests/testthat/test-helpers.R b/tests/testthat/test-helpers.R index 0bde447..2914fd9 100644 --- a/tests/testthat/test-helpers.R +++ b/tests/testthat/test-helpers.R @@ -101,3 +101,141 @@ test_that("get_tongfen_correspondence: three-column input supported", { result <- tongfen:::get_tongfen_correspondence(dd) expect_equal(n_distinct(result$TongfenID), 2L) }) + +test_that("get_tongfen_correspondence: NA identifiers do not link rows", { + # rows that only share a missing identifier belong to separate components + dd <- make_correspondence( + geo_a = c("A1", "A2", "A3"), + geo_b = c(NA, NA, "B3") + ) + result <- tongfen:::get_tongfen_correspondence(dd) + expect_equal(n_distinct(result$TongfenID), 3L) + # a missing identifier is not listed in the UID either + expect_false(any(grepl("NA", result$TongfenUID))) +}) + +test_that("get_tongfen_correspondence: rows are labelled from later columns when first is NA", { + dd <- make_correspondence( + geo_a = c(NA, "A2"), + geo_b = c("B1", "B2") + ) + result <- tongfen:::get_tongfen_correspondence(dd) + expect_equal(n_distinct(result$TongfenID), 2L) + expect_equal(result$TongfenID[[1]], "2_B1") +}) + +test_that("get_tongfen_correspondence: TongfenID is unnamed", { + dd <- make_correspondence(geo_a = c("A1", "A2"), geo_b = c("B1", "B2")) + result <- tongfen:::get_tongfen_correspondence(dd) + expect_null(names(result$TongfenID)) +}) + +test_that("get_tongfen_correspondence: TongfenMethod column is not treated as an identifier", { + dd <- make_correspondence( + geo_a = c("A1", "A2"), + geo_b = c("B1", "B2"), + TongfenMethod = c("statcan", "statcan") + ) + result <- tongfen:::get_tongfen_correspondence(dd) + expect_equal(n_distinct(result$TongfenID), 2L) + expect_false(any(grepl("TongfenMethod", result$TongfenUID))) +}) + +test_that("get_tongfen_correspondence: resolves a long chain of linked rows", { + # zig-zag chain: every row links to the next through alternating columns, the + # worst case for label propagation without pointer jumping + n <- 20000 + i <- seq_len(n) + dd <- make_correspondence( + geo_a = paste0("x", ceiling(i / 2)), + geo_b = paste0("y", ceiling((i + 1) / 2)) + ) + result <- tongfen:::get_tongfen_correspondence(dd) + expect_equal(n_distinct(result$TongfenID), 1L) + expect_equal(nrow(result), n) +}) + +# ── collapse_unique_by_row ──────────────────────────────────────────────────── + +test_that("collapse_unique_by_row: collapses distinct values per row", { + d <- tibble(a = c("x", "x", "y"), b = c("x", "y", "y")) + expect_equal(tongfen:::collapse_unique_by_row(d, c("a", "b")), + c("x", "x, y", "y")) +}) + +test_that("collapse_unique_by_row: works for a single column", { + d <- tibble(a = c("x", "y")) + expect_equal(tongfen:::collapse_unique_by_row(d, "a"), c("x", "y")) +}) + +# ── aggregate_correspondences ───────────────────────────────────────────────── + +test_that("aggregate_correspondences: joins on shared identifier and merges methods", { + cl <- list( + tibble(A = c("a1", "a2"), B = c("b1", "b2"), TongfenMethod = "statcan"), + tibble(B = c("b1", "b2"), C = c("c1", "c2"), TongfenMethod = "estimate") + ) + result <- tongfen:::aggregate_correspondences(cl) + expect_equal(sort(names(result)), c("A", "B", "C", "TongfenMethod")) + expect_equal(nrow(result), 2L) + expect_equal(unique(result$TongfenMethod), "statcan, estimate") +}) + +test_that("aggregate_correspondences: uses every input correspondence exactly once", { + cl <- list( + tibble(A = c("a1", "a2", "a3"), B = c("b1", "b2", "b3"), TongfenMethod = "statcan"), + tibble(B = c("b1", "b2"), C = c("c1", "c2"), TongfenMethod = "statcan"), + tibble(C = c("c1", "c2", "c3", "c4"), D = c("d1", "d2", "d3", "d4"), TongfenMethod = "statcan") + ) + result <- tongfen:::aggregate_correspondences(cl) + expect_equal(sort(names(result)), c("A", "B", "C", "D", "TongfenMethod")) + expect_equal(nrow(result), 2L) +}) + +# ── summarize_geometry_by_group ─────────────────────────────────────────────── + +test_that("summarize_geometry_by_group: matches grouped st_union", { + skip_if_not_installed("sf") + square <- function(x, y) { + sf::st_polygon(list(cbind(c(x, x + 1, x + 1, x, x), + c(y, y, y + 1, y + 1, y)))) + } + # groups: "a" two adjacent squares, "b" a single square, "c" two disjoint squares + d <- sf::st_sf( + grp = c("a", "a", "b", "c", "c"), + geometry = sf::st_sfc(square(0, 0), square(1, 0), square(5, 5), + square(10, 10), square(20, 20), crs = 3347) + ) + + expected <- d %>% + group_by(.data$grp) %>% + summarize(geometry = suppressMessages(sf::st_union(.data$geometry)) %>% + sf::st_cast("MULTIPOLYGON"), .groups = "drop") + result <- tongfen:::summarize_geometry_by_group(d %>% group_by(.data$grp), "grp") + + expect_equal(result$grp, expected$grp) + expect_equal(as.numeric(sf::st_area(result)), as.numeric(sf::st_area(expected))) + expect_true(all(vapply(seq_len(nrow(result)), + function(i) sf::st_equals(result$geometry[i], expected$geometry[i], + sparse = FALSE)[1, 1], + logical(1)))) + expect_equal(sf::st_crs(result), sf::st_crs(d)) + expect_false(dplyr::is_grouped_df(result)) +}) + +test_that("summarize_geometry_by_group: supports multiple grouping columns", { + skip_if_not_installed("sf") + square <- function(x, y) { + sf::st_polygon(list(cbind(c(x, x + 1, x + 1, x, x), + c(y, y, y + 1, y + 1, y)))) + } + d <- sf::st_sf( + TongfenID = c("a", "a", "b"), + TongfenUID = c("u1", "u1", "u2"), + geometry = sf::st_sfc(square(0, 0), square(1, 0), square(5, 5), crs = 3347) + ) + result <- tongfen:::summarize_geometry_by_group(d, c("TongfenID", "TongfenUID")) + expect_equal(nrow(result), 2L) + expect_equal(result$TongfenID, c("a", "b")) + expect_equal(result$TongfenUID, c("u1", "u2")) +}) diff --git a/vignettes/tongfen_ca.Rmd b/vignettes/tongfen_ca.Rmd index 8cbf90e..33668f6 100644 --- a/vignettes/tongfen_ca.Rmd +++ b/vignettes/tongfen_ca.Rmd @@ -91,8 +91,8 @@ variables <- c("2016_0-14"="v_CA16_4", "2011_0-4"="v_CA11F_8","2011_5-9"="v_CA11F_11","2011_10-14"="v_CA11F_14") meta <- meta_for_ca_census_vectors(variables) %>% bind_rows(meta_for_additive_variables(c("CA11","CA16"),"Population")) -children_data <- get_tongfen_ca_census(regions = vsb_regions, meta = meta, - level="DA", quiet = TRUE) %>% +children_data <- get_tongfen_ca_census(regions = vsb_regions, meta = meta, + level="DA", base_geo = "CA16", quiet = TRUE) %>% mutate(`2011_0-14`=purrr::reduce(select(sf::st_set_geometry(.,NULL), starts_with("2011_")), `+`)) %>% mutate(change=`2016_0-14`/Population_CA16-`2011_0-14`/Population_CA11) From 123a613b0301be8272a3359cf7feac10ed903b32 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 19:17:30 -0700 Subject: [PATCH 2/8] note absence of reverse dependencies in cran-comments Co-Authored-By: Claude Opus 5 --- cran-comments.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cran-comments.md b/cran-comments.md index c288a43..8679f72 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -49,4 +49,6 @@ # R CMD check results 0 errors | 0 warnings | 0 notes +There are no reverse dependencies. + From 05e52d34d7b0d43327d1f8b2d362a82f288698b5 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 20:54:02 -0700 Subject: [PATCH 3/8] feat(us): add 1990 census to the correspondence layer Expose `get_tongfen_correspondence_us_census` so US correspondence tables can be built without also fetching the data, and extend the tract correspondence back to the 1990 census via the Census Bureau 1990-to-2000 tract relationship files. The 1990 census stays out of `get_tongfen_us_census` and `valid_us_census_datasets`: the Census Bureau retired the 1990 API endpoint, so tidycensus cannot fetch the data. 1990 data has to be obtained elsewhere and combined with the correspondence table via `tongfen_aggregate`. Chaining the relationship files is now a loop over consecutive censuses instead of nested special cases, which verifies byte-identical to the previous implementation for all combinations of the 2000, 2010 and 2020 censuses. Co-Authored-By: Claude Opus 5 --- NAMESPACE | 1 + NEWS.md | 8 + R/tongfen_us.R | 189 +++++++++++++++----- cran-comments.md | 2 + man/get_tongfen_correspondence_us_census.Rd | 52 ++++++ man/get_tongfen_us_census.Rd | 5 + tests/testthat/test-us-correspondence.R | 43 +++++ 7 files changed, 260 insertions(+), 40 deletions(-) create mode 100644 man/get_tongfen_correspondence_us_census.Rd create mode 100644 tests/testthat/test-us-correspondence.R diff --git a/NAMESPACE b/NAMESPACE index 863178c..1ba1cc6 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -8,6 +8,7 @@ export(get_tongfen_ca_census_ct_from_da) export(get_tongfen_census_ct) export(get_tongfen_census_da) export(get_tongfen_correspondence_ca_census) +export(get_tongfen_correspondence_us_census) export(get_tongfen_us_census) export(meta_for_additive_variables) export(meta_for_ca_census_vectors) diff --git a/NEWS.md b/NEWS.md index 1d80c50..a4a2f18 100644 --- a/NEWS.md +++ b/NEWS.md @@ -15,6 +15,11 @@ (dissemination blocks for a large province: minutes down to seconds) - the "statcan" method no longer downloads census geometries it does not use - dissolving geometries skips regions that don't need to be merged +- new `get_tongfen_correspondence_us_census` to get correspondence tables for US census + geographies without also fetching the data +- US census tract correspondence tables now reach back to the 1990 census (`dec1990`). The + Census Bureau has retired the 1990 API endpoint, so 1990 data itself has to be brought in + separately, for example from NHGIS, and combined via `tongfen_aggregate` ## Minor changes - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections, default is `3347` (Statistics Canada Lambert) @@ -22,6 +27,9 @@ - fix crash when tongfen-ing census tracts across non-adjacent censuses - fix `get_tongfen_census_ct`, `get_tongfen_census_da` and `get_tongfen_ca_census_ct_from_da` erroring out when called with `geo_format=NA` +- US county subdivision data now errors out up front on censuses it can't be matched across, + instead of failing with "Did not find matching geographic identifiers" after downloading + the comparability file and the census data - faster `check_tongfen_areas` and `aggregate_correspondences` # tongfen v.0.3.7 diff --git a/R/tongfen_us.R b/R/tongfen_us.R index 4101170..9f892cc 100644 --- a/R/tongfen_us.R +++ b/R/tongfen_us.R @@ -1,27 +1,39 @@ fips_code_for_state <- function(s){ tidycensus::fips_codes %>% filter(.data$state==s | .data$state_code==s) %>% - select(.data$state,.data$state_code) %>% + select("state","state_code") %>% unique() } +# census tract vintages the correspondence layer can bridge, in chronological order, +# together with the name of the GEOID column identifying tracts of that vintage +us_ct_geoid_columns <- c(dec1990 = "GEOID90", + dec2000 = "GEOID00", + dec2010 = "GEOID10", + dec2020 = "GEOID20") + +# `year` is the later of the two censuses the relationship file links get_us_ct_correspondence_path <- function(state,year){ - if (year=="2010") { - states <- fips_code_for_state(state) - if (nrow(states)!= 1) { - stop(paste0("Could not determine state: ",state)) - } + states <- fips_code_for_state(state) + if (nrow(states)!= 1) { + stop(paste0("Could not determine state: ",state)) + } + if (year=="2000") { + path <- paste0("https://www2.census.gov/geo/relfiles/tract/", + tolower(states$state),"/", + tolower(states$state), + states$state_code,"pop.txt") + } else if (year=="2010") { path <- paste0("https://www2.census.gov/geo/docs/maps-data/data/rel/trf_txt/", tolower(states$state), states$state_code,"trf.txt") } else if (year=="2020") { - states <- fips_code_for_state(state) - if (nrow(states)!= 1) { - stop(paste0("Could not determine state: ",state)) - } path <- paste0("https://www2.census.gov/geo/docs/maps-data/data/rel2020/t10t20/TAB2010_TAB2020_ST", states$state_code,".zip") + } else { + stop(paste0("No census tract relationship file available for ",year)) } + path } get_us_ct_correspondence_2020 <- function(state,cache_path=getOption("tongfen.cache_path")) { @@ -61,27 +73,66 @@ get_us_ct_correspondence_2010 <- function(state,cache_path=getOption("tongfen.ca col_types = "cccciiccccccciicccnnnnnnnnnnnn") } +# the 1990 to 2000 relationship files are fixed width, the "pop" variant is the +# complete one, listing every tract rather than only the ones that changed +get_us_ct_correspondence_2000 <- function(state,cache_path=getOption("tongfen.cache_path")){ + path <- get_us_ct_correspondence_path(state,"2000") + cache_path = file.path(cache_path %||% tempdir(),"us_data") + local_path <- file.path(cache_path,basename(path)) + if (!file.exists(local_path)) { + if (!dir.exists(cache_path)) dir.create(cache_path) + utils::download.file(path,local_path,quiet=TRUE) + } + readr::read_fwf(local_path, + readr::fwf_cols(STATE90=c(1,2),COUNTY90=c(3,5),TRACT90BASE=c(6,9), + TRACT90SUF=c(10,11),PART90=c(12,12),POP90TRACT=c(13,21), + PCT90=c(22,25),STATE00=c(26,27),COUNTY00=c(28,30), + TRACT00BASE=c(31,34),TRACT00SUF=c(35,36),PART00=c(37,37), + POP00TRACT=c(38,46),PCT00=c(47,50),POPPART=c(51,59), + AREALAND=c(60,73),STAB=c(74,75),COUNTYNAME=c(76,135)), + col_types=readr::cols(.default="c")) %>% + mutate(GEOID90=paste0(.data$STATE90,.data$COUNTY90,.data$TRACT90BASE, + coalesce(.data$TRACT90SUF,"00")), + GEOID00=paste0(.data$STATE00,.data$COUNTY00,.data$TRACT00BASE, + coalesce(.data$TRACT00SUF,"00"))) %>% + select("GEOID90","GEOID00") %>% + unique() +} + +# stitch relationship files for consecutive censuses into one table spanning all +# requested censuses, dropping the vintages that only served as stepping stones +join_us_ct_correspondence <- function(links, datasets){ + c <- links[[1]] + for (l in links[-1]) c <- full_join(c,l,by=intersect(names(c),names(l))) + c %>% + select(all_of(unname(us_ct_geoid_columns[datasets]))) %>% + unique() +} + get_us_ct_correspondence <- function(state, datasets, cache_path=getOption("tongfen.cache_path")){ - c <- NULL - if (setdiff(datasets,c("dec2000","dec2010","dec2020")) %>% length() > 0) { - stop("Invalid census years, can only match censuses 2000 through 2020") + years <- names(us_ct_geoid_columns) + invalid_datasets <- setdiff(datasets,years) + if (length(invalid_datasets) > 0) { + stop(paste0("Invalid census years ",paste0(invalid_datasets,collapse=", "), + ", can only match censuses ",paste0(years,collapse=", "))) } - if ("dec2000" %in% datasets) { - c<-get_us_ct_correspondence_2010(state,cache_path=cache_path) %>% - select(matches("^GEOID\\d{2}$")) - if ("dec2020" %in% datasets) { - c2 <- get_us_ct_correspondence_2020(state,cache_path=cache_path) %>% - select(matches("^GEOID\\d{2}$")) - c <- full_join(c,c2,by="GEOID10") - } - if (!("dec2010" %in% datasets)) c <- c %>% select(-.data$GEOID10) - c <- c %>% unique - } else if ("dec2020" %in% datasets) { - c<-get_us_ct_correspondence_2020(state,cache_path=cache_path) %>% - select(matches("^GEOID\\d{2}$")) - } else stop("Invalid census years, can only match censuses 2000 through 2020") - c + datasets <- intersect(years,datasets) + if (length(datasets) < 2) { + stop("Need at least two censuses to build a correspondence table.") + } + # censuses in between the requested ones still have to be traversed, there are no + # relationship files skipping a census + span <- years[seq(match(datasets[1],years),match(utils::tail(datasets,1),years))] + links <- utils::head(span,-1) %>% + lapply(function(year){ + link <- switch(year, + dec1990 = get_us_ct_correspondence_2000(state,cache_path=cache_path), + dec2000 = get_us_ct_correspondence_2010(state,cache_path=cache_path), + dec2010 = get_us_ct_correspondence_2020(state,cache_path=cache_path)) + link %>% select(matches("^GEOID\\d{2}$")) %>% unique() + }) + join_us_ct_correspondence(links,datasets) } get_us_county_subdivision_correspondence <- function(cache_path=getOption("tongfen.cache_path")){ @@ -99,6 +150,68 @@ get_us_county_subdivision_correspondence <- function(cache_path=getOption("tongf } +#' Get correspondence table for US census geographies +#' +#' @description +#' \lifecycle{maturing} +#' +#' Builds a correspondence table matching US census geographies across censuses, based on the +#' relationship files published by the US Census Bureau. Censuses that aren't requested but sit +#' in between two that are get traversed on the way, the Census Bureau only publishes +#' relationship files between consecutive censuses. +#' +#' The correspondence layer reaches back one census further than +#' \code{\link{get_tongfen_us_census}}. The 1990 census is available as `dec1990` here, but the +#' Census Bureau has retired the 1990 API endpoint, so 1990 data has to be brought in by other +#' means, for example from NHGIS via the ipumsr package, and handed to +#' \code{\link{tongfen_aggregate}} together with this correspondence table. +#' +#' @param datasets vector of censuses to match up, valid values are `dec1990`, `dec2000`, +#' `dec2010` and `dec2020` for census tracts, `dec2000` and `dec2010` for county subdivisions. +#' At least two censuses are needed. +#' @param regions list with regions to query the correspondence for. At this stage, the only +#' valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))` +#' @param level aggregation level, at this stage the only valid levels are 'tract' and +#' 'county subdivision'. +#' @param cache_path optional path to cache the relationship files in, defaults to the +#' `tongfen.cache_path` option and falls back to a temporary directory +#' @return tibble with one row per census geography, a GEOID column for each requested census, +#' and the common geography identified by `TongfenID` and `TongfenUID`. +#' @export +#' +#' @examples +#' # Match up census tracts for the 1990 and 2000 censuses in Rhode Island +#' \dontrun{ +#' correspondence <- get_tongfen_correspondence_us_census(datasets = c("dec1990","dec2000"), +#' regions = list(state="RI")) +#'} +get_tongfen_correspondence_us_census <- function(datasets, regions, level='tract', + cache_path=getOption("tongfen.cache_path")){ + assert(level %in% c('tract','county subdivision'), + "Only census tracts and county subdivisions are supported right now.") + if (level=="county subdivision") { + invalid_datasets <- setdiff(datasets,c("dec2000","dec2010")) + assert(length(invalid_datasets)==0, + paste0("County subdivisions can only be matched between the 2000 and 2010 censuses, got: ", + paste0(invalid_datasets,collapse=", "))) + } + + regions$state %>% + lapply(function(state){ + if (level=='tract') { + get_us_ct_correspondence(state,datasets,cache_path=cache_path) + } else { + fips <- fips_code_for_state(state)$state_code + get_us_county_subdivision_correspondence(cache_path=cache_path) %>% + filter(.data$STATEFP10==fips) %>% + select("GEOID00","GEOID10") + } + }) %>% + bind_rows() %>% + get_tongfen_correspondence() +} + + valid_us_census_datasets <- c( dec2000 = "US decentennial census 2000", dec2010 = "US decentennial census 2010", @@ -113,6 +226,11 @@ valid_us_census_datasets <- c( #' This wraps data acquisition via the tidycensus package and tongfen on a common geography into #' a single convenience function. #' +#' Data is only available for the 2000, 2010 and 2020 censuses, the Census Bureau has retired the +#' 1990 API endpoint. To tongfen 1990 data, obtain it elsewhere and combine it with a +#' correspondence table from \code{\link{get_tongfen_correspondence_us_census}} via +#' \code{\link{tongfen_aggregate}}. +#' #' @param regions list with regions to query the data for. At this stage, the only #' valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))`` #' @param meta metadata for variables to retrieve @@ -150,18 +268,9 @@ get_tongfen_us_census <- function(regions,meta,level='tract',survey="census", assert(survey %in% c('census'),"Only census surveys are supported right now.") regions$state %>% lapply(function(state){ - if (level=='tract') { - correspondence <- get_us_ct_correspondence(state,datasets) - } else if (level=="county subdivision") { - fips <- fips_code_for_state(state)$state_code - correspondence <- get_us_county_subdivision_correspondence() %>% - filter(.data$STATEFP10==fips) %>% - select(.data$GEOID00,.data$GEOID10) - } else { - stop("Ooops, should have caught this earler.") - } - correspondence <- correspondence %>% - get_tongfen_correspondence() + correspondence <- get_tongfen_correspondence_us_census(datasets = datasets, + regions = list(state=state), + level = level) data <- datasets %>% lapply(function(ds){ diff --git a/cran-comments.md b/cran-comments.md index 8679f72..74037ca 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -9,6 +9,8 @@ a row-by-row union-find, making tongfen on large geographies dramatically faster - the "statcan" method no longer downloads census geometries it does not use - dissolving geometries skips regions that don't need to be merged +- new `get_tongfen_correspondence_us_census`, US correspondence tables now reach back to the + 1990 census ## Minor changes - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections - missing geographic identifiers no longer merge unrelated regions into one common geography diff --git a/man/get_tongfen_correspondence_us_census.Rd b/man/get_tongfen_correspondence_us_census.Rd new file mode 100644 index 0000000..c7e7159 --- /dev/null +++ b/man/get_tongfen_correspondence_us_census.Rd @@ -0,0 +1,52 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/tongfen_us.R +\name{get_tongfen_correspondence_us_census} +\alias{get_tongfen_correspondence_us_census} +\title{Get correspondence table for US census geographies} +\usage{ +get_tongfen_correspondence_us_census( + datasets, + regions, + level = "tract", + cache_path = getOption("tongfen.cache_path") +) +} +\arguments{ +\item{datasets}{vector of censuses to match up, valid values are `dec1990`, `dec2000`, +`dec2010` and `dec2020` for census tracts, `dec2000` and `dec2010` for county subdivisions. +At least two censuses are needed.} + +\item{regions}{list with regions to query the correspondence for. At this stage, the only +valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))`} + +\item{level}{aggregation level, at this stage the only valid levels are 'tract' and +'county subdivision'.} + +\item{cache_path}{optional path to cache the relationship files in, defaults to the +`tongfen.cache_path` option and falls back to a temporary directory} +} +\value{ +tibble with one row per census geography, a GEOID column for each requested census, +and the common geography identified by `TongfenID` and `TongfenUID`. +} +\description{ +\lifecycle{maturing} + +Builds a correspondence table matching US census geographies across censuses, based on the +relationship files published by the US Census Bureau. Censuses that aren't requested but sit +in between two that are get traversed on the way, the Census Bureau only publishes +relationship files between consecutive censuses. + +The correspondence layer reaches back one census further than +\code{\link{get_tongfen_us_census}}. The 1990 census is available as `dec1990` here, but the +Census Bureau has retired the 1990 API endpoint, so 1990 data has to be brought in by other +means, for example from NHGIS via the ipumsr package, and handed to +\code{\link{tongfen_aggregate}} together with this correspondence table. +} +\examples{ +# Match up census tracts for the 1990 and 2000 censuses in Rhode Island +\dontrun{ +correspondence <- get_tongfen_correspondence_us_census(datasets = c("dec1990","dec2000"), + regions = list(state="RI")) +} +} diff --git a/man/get_tongfen_us_census.Rd b/man/get_tongfen_us_census.Rd index 2592220..86ed907 100644 --- a/man/get_tongfen_us_census.Rd +++ b/man/get_tongfen_us_census.Rd @@ -32,6 +32,11 @@ sf object with (wide form) census variables with census year as suffix (separate This wraps data acquisition via the tidycensus package and tongfen on a common geography into a single convenience function. + +Data is only available for the 2000, 2010 and 2020 censuses, the Census Bureau has retired the +1990 API endpoint. To tongfen 1990 data, obtain it elsewhere and combine it with a +correspondence table from \code{\link{get_tongfen_correspondence_us_census}} via +\code{\link{tongfen_aggregate}}. } \examples{ # Get US census data on population and households for 2000 and 2010 censuses on a uniform geography diff --git a/tests/testthat/test-us-correspondence.R b/tests/testthat/test-us-correspondence.R new file mode 100644 index 0000000..01540d4 --- /dev/null +++ b/tests/testthat/test-us-correspondence.R @@ -0,0 +1,43 @@ +test_that("us tract correspondence links chain across censuses", { + l9000 <- tibble::tibble(GEOID90=c("a","b","c"), + GEOID00=c("A","A","C")) + l0010 <- tibble::tibble(GEOID00=c("A","C","D"), + GEOID10=c("1","2","3")) + l1020 <- tibble::tibble(GEOID10=c("1","2","3"), + GEOID20=c("X","X","Y")) + + all <- tongfen:::join_us_ct_correspondence(list(l9000,l0010,l1020), + c("dec1990","dec2000","dec2010","dec2020")) + expect_equal(names(all),c("GEOID90","GEOID00","GEOID10","GEOID20")) + # tract D has no 1990 predecessor but is kept + expect_equal(nrow(all),4) + expect_true(all(c("a","b","c") %in% all$GEOID90)) + expect_true(is.na(all$GEOID90[all$GEOID00=="D"])) + + # censuses that only serve as stepping stones get dropped + ends <- tongfen:::join_us_ct_correspondence(list(l9000,l0010,l1020), + c("dec1990","dec2020")) + expect_equal(names(ends),c("GEOID90","GEOID20")) + expect_equal(sort(ends$GEOID20[!is.na(ends$GEOID90)]),c("X","X","X")) + + short <- tongfen:::join_us_ct_correspondence(list(l9000),c("dec1990","dec2000")) + expect_equal(nrow(short),3) +}) + +test_that("us tract correspondence validates the requested censuses", { + expect_error(tongfen:::get_us_ct_correspondence("RI",c("dec1980","dec1990")),"dec1980") + expect_error(tongfen:::get_us_ct_correspondence("RI","dec1990"),"at least two") +}) + +test_that("us tract relationship file paths are built for all census pairs", { + skip_if_not_installed("tidycensus") + + expect_equal(tongfen:::get_us_ct_correspondence_path("RI","2000"), + "https://www2.census.gov/geo/relfiles/tract/ri/ri44pop.txt") + expect_equal(tongfen:::get_us_ct_correspondence_path("RI","2010"), + "https://www2.census.gov/geo/docs/maps-data/data/rel/trf_txt/ri44trf.txt") + expect_equal(tongfen:::get_us_ct_correspondence_path("RI","2020"), + paste0("https://www2.census.gov/geo/docs/maps-data/data/rel2020/t10t20/", + "TAB2010_TAB2020_ST44.zip")) + expect_error(tongfen:::get_us_ct_correspondence_path("RI","1990"),"1990") +}) From 30eba2a7cd2505a539ea62d69c960988a0992154 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 21:26:14 -0700 Subject: [PATCH 4/8] feat(us): match county subdivisions across the 2010 and 2020 censuses The only county subdivision source so far was the 2000 to 2010 comparability file, and `datasets` was ignored at that level, so asking for 2020 built a 2000/2010 correspondence and failed downstream with "Did not find matching geographic identifiers". Add the 2010 to 2020 relationship file and run county subdivisions through the same chaining loop as census tracts. Unlike the comparability file that file is a geometric overlay: for Rhode Island 84 of its 124 rows are slivers along boundaries that shifted slightly, three of them reaching into Massachusetts, and chaining them collapses the state's 40 subdivisions into 8 common geographies. Cut them by the share of area two subdivisions have in common, exposed as `min_area_share`. At the default of 0.01 Rhode Island comes out as 40 one-to-one matches, which is what it should be, its towns did not change between 2010 and 2020. Co-Authored-By: Claude Opus 5 --- NEWS.md | 4 + R/tongfen_us.R | 119 ++++++++++++++------ cran-comments.md | 4 +- man/get_tongfen_correspondence_us_census.Rd | 11 +- man/get_tongfen_us_census.Rd | 7 +- tests/testthat/test-us-correspondence.R | 26 ++++- 6 files changed, 130 insertions(+), 41 deletions(-) diff --git a/NEWS.md b/NEWS.md index a4a2f18..ae3a4e0 100644 --- a/NEWS.md +++ b/NEWS.md @@ -20,6 +20,10 @@ - US census tract correspondence tables now reach back to the 1990 census (`dec1990`). The Census Bureau has retired the 1990 API endpoint, so 1990 data itself has to be brought in separately, for example from NHGIS, and combined via `tongfen_aggregate` +- US county subdivisions can now be matched across the 2010 and 2020 censuses, previously only + the 2000 and 2010 censuses were available. The relationship file for these is a geometric + overlay, the new `min_area_share` argument controls how much area two subdivisions have to + have in common to count as related ## Minor changes - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections, default is `3347` (Statistics Canada Lambert) diff --git a/R/tongfen_us.R b/R/tongfen_us.R index 9f892cc..12a5024 100644 --- a/R/tongfen_us.R +++ b/R/tongfen_us.R @@ -5,12 +5,12 @@ fips_code_for_state <- function(s){ unique() } -# census tract vintages the correspondence layer can bridge, in chronological order, -# together with the name of the GEOID column identifying tracts of that vintage -us_ct_geoid_columns <- c(dec1990 = "GEOID90", - dec2000 = "GEOID00", - dec2010 = "GEOID10", - dec2020 = "GEOID20") +# census vintages the correspondence layer can bridge, in chronological order, together +# with the name of the GEOID column identifying geographies of that vintage +us_geoid_columns <- c(dec1990 = "GEOID90", + dec2000 = "GEOID00", + dec2010 = "GEOID10", + dec2020 = "GEOID20") # `year` is the later of the two censuses the relationship file links get_us_ct_correspondence_path <- function(state,year){ @@ -101,29 +101,33 @@ get_us_ct_correspondence_2000 <- function(state,cache_path=getOption("tongfen.ca # stitch relationship files for consecutive censuses into one table spanning all # requested censuses, dropping the vintages that only served as stepping stones -join_us_ct_correspondence <- function(links, datasets){ +join_us_correspondence <- function(links, datasets){ c <- links[[1]] for (l in links[-1]) c <- full_join(c,l,by=intersect(names(c),names(l))) c %>% - select(all_of(unname(us_ct_geoid_columns[datasets]))) %>% + select(all_of(unname(us_geoid_columns[datasets]))) %>% unique() } -get_us_ct_correspondence <- function(state, datasets, - cache_path=getOption("tongfen.cache_path")){ - years <- names(us_ct_geoid_columns) - invalid_datasets <- setdiff(datasets,years) +# the censuses that have to be traversed to get from the earliest to the latest requested +# one, the Census Bureau only publishes relationship files between consecutive censuses +us_correspondence_span <- function(datasets, available){ + invalid_datasets <- setdiff(datasets,available) if (length(invalid_datasets) > 0) { stop(paste0("Invalid census years ",paste0(invalid_datasets,collapse=", "), - ", can only match censuses ",paste0(years,collapse=", "))) + ", can only match censuses ",paste0(available,collapse=", "))) } - datasets <- intersect(years,datasets) + datasets <- intersect(available,datasets) if (length(datasets) < 2) { stop("Need at least two censuses to build a correspondence table.") } - # censuses in between the requested ones still have to be traversed, there are no - # relationship files skipping a census - span <- years[seq(match(datasets[1],years),match(utils::tail(datasets,1),years))] + available[seq(match(datasets[1],available),match(utils::tail(datasets,1),available))] +} + +get_us_ct_correspondence <- function(state, datasets, + cache_path=getOption("tongfen.cache_path")){ + available <- names(us_geoid_columns) + span <- us_correspondence_span(datasets,available) links <- utils::head(span,-1) %>% lapply(function(year){ link <- switch(year, @@ -132,9 +136,10 @@ get_us_ct_correspondence <- function(state, datasets, dec2010 = get_us_ct_correspondence_2020(state,cache_path=cache_path)) link %>% select(matches("^GEOID\\d{2}$")) %>% unique() }) - join_us_ct_correspondence(links,datasets) + join_us_correspondence(links,intersect(available,datasets)) } +# the 2000 to 2010 county subdivision comparability file, covering all states get_us_county_subdivision_correspondence <- function(cache_path=getOption("tongfen.cache_path")){ cache_path = file.path(cache_path %||% tempdir(),"us_data") file <- "Cousub_comparability.xlsx" @@ -149,6 +154,53 @@ get_us_county_subdivision_correspondence <- function(cache_path=getOption("tongf readxl::read_xlsx(local_path) } +# The 2010 to 2020 county subdivision relationship file, covering all states. Unlike the +# 2000 to 2010 comparability file this is a geometric overlay, most rows are slivers along +# boundaries that shifted slightly rather than actual relationships. Keeping them chains +# unrelated subdivisions into one common geography, so they get cut. A subdivision carved +# out of a larger one is only a small share of the old one but most of the new one, hence +# the share is taken over the larger of the two. +get_us_county_subdivision_correspondence_2020 <- function(min_area_share=0.01, + cache_path=getOption("tongfen.cache_path")){ + cache_path = file.path(cache_path %||% tempdir(),"us_data") + path <- paste0("https://www2.census.gov/geo/docs/maps-data/data/rel2020/cousub/", + "tab20_cousub20_cousub10_natl.txt") + local_path <- file.path(cache_path,basename(path)) + if (!file.exists(local_path)) { + if (!dir.exists(cache_path)) dir.create(cache_path) + utils::download.file(path,local_path,quiet=TRUE) + } + readr::read_delim(local_path,delim="|",progress=FALSE, + col_types=readr::cols_only(GEOID_COUSUB_10="c",GEOID_COUSUB_20="c", + AREALAND_COUSUB_10="d",AREAWATER_COUSUB_10="d", + AREALAND_COUSUB_20="d",AREAWATER_COUSUB_20="d", + AREALAND_PART="d",AREAWATER_PART="d")) %>% + mutate(area_part=.data$AREALAND_PART+.data$AREAWATER_PART) %>% + filter(pmax(.data$area_part/(.data$AREALAND_COUSUB_10+.data$AREAWATER_COUSUB_10), + .data$area_part/(.data$AREALAND_COUSUB_20+.data$AREAWATER_COUSUB_20)) + >= min_area_share) %>% + select(GEOID10="GEOID_COUSUB_10",GEOID20="GEOID_COUSUB_20") %>% + unique() +} + +get_us_county_subdivision_correspondence_for <- function(state, datasets, min_area_share=0.01, + cache_path=getOption("tongfen.cache_path")){ + available <- setdiff(names(us_geoid_columns),"dec1990") + span <- us_correspondence_span(datasets,available) + fips <- fips_code_for_state(state)$state_code + links <- utils::head(span,-1) %>% + lapply(function(year){ + link <- switch(year, + dec2000 = get_us_county_subdivision_correspondence(cache_path=cache_path) %>% + select("GEOID00","GEOID10"), + dec2010 = get_us_county_subdivision_correspondence_2020( + min_area_share=min_area_share,cache_path=cache_path)) + # both files are national, county subdivisions don't cross state lines + link %>% filter(substr(.data$GEOID10,1,2)==fips) %>% unique() + }) + join_us_correspondence(links,intersect(available,datasets)) +} + #' Get correspondence table for US census geographies #' @@ -167,12 +219,17 @@ get_us_county_subdivision_correspondence <- function(cache_path=getOption("tongf #' \code{\link{tongfen_aggregate}} together with this correspondence table. #' #' @param datasets vector of censuses to match up, valid values are `dec1990`, `dec2000`, -#' `dec2010` and `dec2020` for census tracts, `dec2000` and `dec2010` for county subdivisions. -#' At least two censuses are needed. +#' `dec2010` and `dec2020` for census tracts, `dec2000` through `dec2020` for county +#' subdivisions. At least two censuses are needed. #' @param regions list with regions to query the correspondence for. At this stage, the only #' valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))` #' @param level aggregation level, at this stage the only valid levels are 'tract' and #' 'county subdivision'. +#' @param min_area_share minimum share of area two geographies have to have in common to count +#' as related, only used when matching county subdivisions across the 2010 and 2020 censuses. +#' The Census Bureau relationship file for these is a geometric overlay, lowering this pulls in +#' slivers along boundaries that only shifted slightly and chains unrelated subdivisions into +#' one common geography. Default is `0.01`. #' @param cache_path optional path to cache the relationship files in, defaults to the #' `tongfen.cache_path` option and falls back to a temporary directory #' @return tibble with one row per census geography, a GEOID column for each requested census, @@ -186,25 +243,19 @@ get_us_county_subdivision_correspondence <- function(cache_path=getOption("tongf #' regions = list(state="RI")) #'} get_tongfen_correspondence_us_census <- function(datasets, regions, level='tract', + min_area_share=0.01, cache_path=getOption("tongfen.cache_path")){ assert(level %in% c('tract','county subdivision'), "Only census tracts and county subdivisions are supported right now.") - if (level=="county subdivision") { - invalid_datasets <- setdiff(datasets,c("dec2000","dec2010")) - assert(length(invalid_datasets)==0, - paste0("County subdivisions can only be matched between the 2000 and 2010 censuses, got: ", - paste0(invalid_datasets,collapse=", "))) - } regions$state %>% lapply(function(state){ if (level=='tract') { get_us_ct_correspondence(state,datasets,cache_path=cache_path) } else { - fips <- fips_code_for_state(state)$state_code - get_us_county_subdivision_correspondence(cache_path=cache_path) %>% - filter(.data$STATEFP10==fips) %>% - select("GEOID00","GEOID10") + get_us_county_subdivision_correspondence_for(state,datasets, + min_area_share=min_area_share, + cache_path=cache_path) } }) %>% bind_rows() %>% @@ -237,6 +288,9 @@ valid_us_census_datasets <- c( #' @param level aggregation level to return the data on. At this stage, the only valid levels are 'tract' and 'county subdivision'. #' @param survey survey to get data for, supported options is "census" #' @param base_geo census year to use as base geography, default is `2010`. +#' @param min_area_share minimum share of area two geographies have to have in common to count +#' as related, see \code{\link{get_tongfen_correspondence_us_census}}. Only used when matching +#' county subdivisions across the 2010 and 2020 censuses. #' @return sf object with (wide form) census variables with census year as suffix (separated by underdcore "_"). #' @export #' @@ -257,7 +311,7 @@ valid_us_census_datasets <- c( #' #'} get_tongfen_us_census <- function(regions,meta,level='tract',survey="census", - base_geo = NULL){ + base_geo = NULL, min_area_share = 0.01){ datasets <- meta$dataset %>% unique if (is.null(base_geo)) base_geo=datasets[1] @@ -270,7 +324,8 @@ get_tongfen_us_census <- function(regions,meta,level='tract',survey="census", regions$state %>% lapply(function(state){ correspondence <- get_tongfen_correspondence_us_census(datasets = datasets, regions = list(state=state), - level = level) + level = level, + min_area_share = min_area_share) data <- datasets %>% lapply(function(ds){ diff --git a/cran-comments.md b/cran-comments.md index 74037ca..396b141 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -9,8 +9,8 @@ a row-by-row union-find, making tongfen on large geographies dramatically faster - the "statcan" method no longer downloads census geometries it does not use - dissolving geometries skips regions that don't need to be merged -- new `get_tongfen_correspondence_us_census`, US correspondence tables now reach back to the - 1990 census +- new `get_tongfen_correspondence_us_census`, US tract correspondence tables now reach back to + the 1990 census and county subdivisions forward to the 2020 census ## Minor changes - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections - missing geographic identifiers no longer merge unrelated regions into one common geography diff --git a/man/get_tongfen_correspondence_us_census.Rd b/man/get_tongfen_correspondence_us_census.Rd index c7e7159..2999fe2 100644 --- a/man/get_tongfen_correspondence_us_census.Rd +++ b/man/get_tongfen_correspondence_us_census.Rd @@ -8,13 +8,14 @@ get_tongfen_correspondence_us_census( datasets, regions, level = "tract", + min_area_share = 0.01, cache_path = getOption("tongfen.cache_path") ) } \arguments{ \item{datasets}{vector of censuses to match up, valid values are `dec1990`, `dec2000`, -`dec2010` and `dec2020` for census tracts, `dec2000` and `dec2010` for county subdivisions. -At least two censuses are needed.} +`dec2010` and `dec2020` for census tracts, `dec2000` through `dec2020` for county +subdivisions. At least two censuses are needed.} \item{regions}{list with regions to query the correspondence for. At this stage, the only valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))`} @@ -22,6 +23,12 @@ valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))`} \item{level}{aggregation level, at this stage the only valid levels are 'tract' and 'county subdivision'.} +\item{min_area_share}{minimum share of area two geographies have to have in common to count +as related, only used when matching county subdivisions across the 2010 and 2020 censuses. +The Census Bureau relationship file for these is a geometric overlay, lowering this pulls in +slivers along boundaries that only shifted slightly and chains unrelated subdivisions into +one common geography. Default is `0.01`.} + \item{cache_path}{optional path to cache the relationship files in, defaults to the `tongfen.cache_path` option and falls back to a temporary directory} } diff --git a/man/get_tongfen_us_census.Rd b/man/get_tongfen_us_census.Rd index 86ed907..757ecbd 100644 --- a/man/get_tongfen_us_census.Rd +++ b/man/get_tongfen_us_census.Rd @@ -9,7 +9,8 @@ get_tongfen_us_census( meta, level = "tract", survey = "census", - base_geo = NULL + base_geo = NULL, + min_area_share = 0.01 ) } \arguments{ @@ -23,6 +24,10 @@ valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))``} \item{survey}{survey to get data for, supported options is "census"} \item{base_geo}{census year to use as base geography, default is `2010`.} + +\item{min_area_share}{minimum share of area two geographies have to have in common to count +as related, see \code{\link{get_tongfen_correspondence_us_census}}. Only used when matching +county subdivisions across the 2010 and 2020 censuses.} } \value{ sf object with (wide form) census variables with census year as suffix (separated by underdcore "_"). diff --git a/tests/testthat/test-us-correspondence.R b/tests/testthat/test-us-correspondence.R index 01540d4..864eef5 100644 --- a/tests/testthat/test-us-correspondence.R +++ b/tests/testthat/test-us-correspondence.R @@ -6,7 +6,7 @@ test_that("us tract correspondence links chain across censuses", { l1020 <- tibble::tibble(GEOID10=c("1","2","3"), GEOID20=c("X","X","Y")) - all <- tongfen:::join_us_ct_correspondence(list(l9000,l0010,l1020), + all <- tongfen:::join_us_correspondence(list(l9000,l0010,l1020), c("dec1990","dec2000","dec2010","dec2020")) expect_equal(names(all),c("GEOID90","GEOID00","GEOID10","GEOID20")) # tract D has no 1990 predecessor but is kept @@ -15,18 +15,36 @@ test_that("us tract correspondence links chain across censuses", { expect_true(is.na(all$GEOID90[all$GEOID00=="D"])) # censuses that only serve as stepping stones get dropped - ends <- tongfen:::join_us_ct_correspondence(list(l9000,l0010,l1020), + ends <- tongfen:::join_us_correspondence(list(l9000,l0010,l1020), c("dec1990","dec2020")) expect_equal(names(ends),c("GEOID90","GEOID20")) expect_equal(sort(ends$GEOID20[!is.na(ends$GEOID90)]),c("X","X","X")) - short <- tongfen:::join_us_ct_correspondence(list(l9000),c("dec1990","dec2000")) + short <- tongfen:::join_us_correspondence(list(l9000),c("dec1990","dec2000")) expect_equal(nrow(short),3) }) -test_that("us tract correspondence validates the requested censuses", { +test_that("us correspondence spans intermediate censuses", { + years <- names(tongfen:::us_geoid_columns) + + expect_equal(tongfen:::us_correspondence_span(c("dec1990","dec2020"),years),years) + expect_equal(tongfen:::us_correspondence_span(c("dec2000","dec2010"),years), + c("dec2000","dec2010")) + # order of the requested censuses does not matter + expect_equal(tongfen:::us_correspondence_span(c("dec2020","dec2000"),years), + c("dec2000","dec2010","dec2020")) + + expect_error(tongfen:::us_correspondence_span(c("dec1980","dec1990"),years),"dec1980") + expect_error(tongfen:::us_correspondence_span("dec1990",years),"at least two") + # county subdivisions have no 1990 relationship file + expect_error(tongfen:::us_correspondence_span(c("dec1990","dec2000"),years[-1]),"dec1990") +}) + +test_that("us correspondence validates the requested censuses", { expect_error(tongfen:::get_us_ct_correspondence("RI",c("dec1980","dec1990")),"dec1980") expect_error(tongfen:::get_us_ct_correspondence("RI","dec1990"),"at least two") + expect_error(tongfen:::get_us_county_subdivision_correspondence_for("RI",c("dec1990","dec2000")), + "dec1990") }) test_that("us tract relationship file paths are built for all census pairs", { From 75b48b6e1bdfb5834e0bdb55acfb3072e8150fc9 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 21:42:26 -0700 Subject: [PATCH 5/8] fix(us): cut slivers out of all correspondence links, fix 2020 tract ids Every US relationship file is a geometric overlay listing each overlap between two censuses, including boundaries that only shifted slightly. Chaining those merged unrelated regions into one common geography: Vermont's 179 tracts came out as 42 common geographies across the 2000 and 2010 censuses, its 187 tracts as 26 across 2010 and 2020. Apply the same cutoff used for county subdivisions to all four links, computed from whatever areas the file carries: summed part areas for 1990 to 2000, tract areas for 2000 to 2010, block areas summed up per tract for 2010 to 2020. With `min_area_share = 0` the result is identical to before, at the 0.01 default Rhode Island tracts across 2010 and 2020 give 198 common geographies instead of 60 and Vermont 151 instead of 26. No region is ever dropped, if all of its parts are slivers its largest part is kept. Rereading the 2010 to 2020 file also fixes its column types: it has 18 columns but was read with 15 type specs, so TRACT_2020 was parsed as a number and lost its leading zeros. 246 of Rhode Island's 250 2020 tract identifiers came out malformed and could not join against the identifiers tidycensus returns. Co-Authored-By: Claude Opus 5 --- NEWS.md | 14 +- R/tongfen_us.R | 139 +++++++++++++++----- cran-comments.md | 2 + man/get_tongfen_correspondence_us_census.Rd | 13 +- man/get_tongfen_us_census.Rd | 3 +- tests/testthat/test-us-correspondence.R | 26 ++++ 6 files changed, 152 insertions(+), 45 deletions(-) diff --git a/NEWS.md b/NEWS.md index ae3a4e0..adac2e1 100644 --- a/NEWS.md +++ b/NEWS.md @@ -21,9 +21,14 @@ Census Bureau has retired the 1990 API endpoint, so 1990 data itself has to be brought in separately, for example from NHGIS, and combined via `tongfen_aggregate` - US county subdivisions can now be matched across the 2010 and 2020 censuses, previously only - the 2000 and 2010 censuses were available. The relationship file for these is a geometric - overlay, the new `min_area_share` argument controls how much area two subdivisions have to - have in common to count as related + the 2000 and 2010 censuses were available +- US correspondence tables no longer chain regions together over slivers. The Census Bureau + relationship files list every geometric overlap, including boundaries that only shifted + slightly, and matching those up merged unrelated regions into one common geography. The new + `min_area_share` argument controls how much area two regions have to have in common to count + as related, default is `0.01`, and no region is ever dropped. This gives substantially finer + common geographies, for Rhode Island tracts across the 2010 and 2020 censuses 198 instead of + 60, for Vermont 151 instead of 26 ## Minor changes - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections, default is `3347` (Statistics Canada Lambert) @@ -31,6 +36,9 @@ - fix crash when tongfen-ing census tracts across non-adjacent censuses - fix `get_tongfen_census_ct`, `get_tongfen_census_da` and `get_tongfen_ca_census_ct_from_da` erroring out when called with `geo_format=NA` +- fix 2020 US census tract identifiers getting stripped of their leading zeros when read from + the relationship file, which silently dropped most tracts out of the result. For Rhode Island + 246 of 250 tracts were affected - US county subdivision data now errors out up front on censuses it can't be matched across, instead of failing with "Did not find matching geographic identifiers" after downloading the comparability file and the census data diff --git a/R/tongfen_us.R b/R/tongfen_us.R index 12a5024..f72d08c 100644 --- a/R/tongfen_us.R +++ b/R/tongfen_us.R @@ -12,6 +12,24 @@ us_geoid_columns <- c(dec1990 = "GEOID90", dec2010 = "GEOID10", dec2020 = "GEOID20") +# Relationship files list every geometric overlap between two censuses, including slivers +# along boundaries that only shifted slightly. Chaining those merges unrelated regions into +# one common geography, so they get cut. The share is taken over the larger of the two sides, +# a region carved out of a bigger one is only a small share of the old one but most of the +# new one. Regions never get dropped: if all parts of a region are slivers its largest part +# is kept, so every region still ends up in some common geography. +area_share <- function(part, total) ifelse(total > 0, part/total, 0) + +cut_correspondence_slivers <- function(d, share, min_area_share){ + keep <- share >= min_area_share + for (column in names(d)) { + g <- d[[column]] + largest <- share == unname(tapply(share, g, max)[g]) + keep <- keep | (!(g %in% g[keep]) & largest) + } + d[keep,,drop=FALSE] %>% unique() +} + # `year` is the later of the two censuses the relationship file links get_us_ct_correspondence_path <- function(state,year){ states <- fips_code_for_state(state) @@ -36,8 +54,10 @@ get_us_ct_correspondence_path <- function(state,year){ path } -get_us_ct_correspondence_2020 <- function(state,cache_path=getOption("tongfen.cache_path")) { - states <- fips_code_for_state(state) +# the 2010 to 2020 tract relationship file is block based, tract areas get summed up from +# the blocks they are made up of +get_us_ct_correspondence_2020 <- function(state,min_area_share=0.01, + cache_path=getOption("tongfen.cache_path")) { cache_path = file.path(cache_path %||% tempdir(),"us_data") path <- get_us_ct_correspondence_path(state,2020) @@ -46,14 +66,35 @@ get_us_ct_correspondence_2020 <- function(state,cache_path=getOption("tongfen.ca if (!dir.exists(cache_path)) dir.create(cache_path) utils::download.file(path,local_path,quiet = TRUE) } - readr::read_delim(local_path,delim="|", col_types = "cccccnncccnncnn") %>% + blocks <- readr::read_delim(local_path,delim="|",progress=FALSE, + col_types=readr::cols_only( + STATE_2010="c",COUNTY_2010="c",TRACT_2010="c",BLK_2010="c", + AREALAND_2010="d",AREAWATER_2010="d", + STATE_2020="c",COUNTY_2020="c",TRACT_2020="c",BLK_2020="c", + AREALAND_2020="d",AREAWATER_2020="d", + AREALAND_INT="d",AREAWATER_INT="d")) %>% mutate(GEOID10=paste0(.data$STATE_2010,.data$COUNTY_2010,.data$TRACT_2010), - GEOID20=paste0(.data$STATE_2020,.data$COUNTY_2020,.data$TRACT_2020)) %>% - select(.data$GEOID10,.data$GEOID20)%>% - unique + GEOID20=paste0(.data$STATE_2020,.data$COUNTY_2020,.data$TRACT_2020), + area10=.data$AREALAND_2010+.data$AREAWATER_2010, + area20=.data$AREALAND_2020+.data$AREAWATER_2020, + area_part=.data$AREALAND_INT+.data$AREAWATER_INT) + tracts10 <- blocks %>% select("GEOID10","BLK_2010","area10") %>% unique() %>% + group_by(.data$GEOID10) %>% summarize(area10=sum(.data$area10),.groups="drop") + tracts20 <- blocks %>% select("GEOID20","BLK_2020","area20") %>% unique() %>% + group_by(.data$GEOID20) %>% summarize(area20=sum(.data$area20),.groups="drop") + d <- blocks %>% + group_by(.data$GEOID10,.data$GEOID20) %>% + summarize(area_part=sum(.data$area_part),.groups="drop") %>% + left_join(tracts10,by="GEOID10") %>% + left_join(tracts20,by="GEOID20") + cut_correspondence_slivers(d %>% select("GEOID10","GEOID20"), + pmax(area_share(d$area_part,d$area10), + area_share(d$area_part,d$area20)), + min_area_share) } -get_us_ct_correspondence_2010 <- function(state,cache_path=getOption("tongfen.cache_path")){ +get_us_ct_correspondence_2010 <- function(state,min_area_share=0.01, + cache_path=getOption("tongfen.cache_path")){ path <- get_us_ct_correspondence_path(state,"2010") file <- basename(path) cache_path = file.path(cache_path %||% tempdir(),"us_data") @@ -62,7 +103,7 @@ get_us_ct_correspondence_2010 <- function(state,cache_path=getOption("tongfen.ca if (!dir.exists(cache_path)) dir.create(cache_path) utils::download.file(path,local_path,quiet=TRUE) } - d<-readr::read_csv(local_path, + d<-readr::read_csv(local_path,progress=FALSE, col_names=c("STATE00","COUNTY00","TRACT00","GEOID00", "POP00","HU00","PART00","AREA00","AREALAND00", "STATE10","COUNTY10","TRACT10","GEOID10", @@ -70,12 +111,23 @@ get_us_ct_correspondence_2010 <- function(state,cache_path=getOption("tongfen.ca "AREAPT","AREALANDPT","AREAPCT00PT", "AREALANDPCT00PT","AREAPCT10PT","AREALANDPCT10PT", "POP10PT","POPPCT00","POPPCT10","HU10PT","HUPCT00","HUPCT10"), - col_types = "cccciiccccccciicccnnnnnnnnnnnn") + col_types = "cccciiccccccciicccnnnnnnnnnnnn") %>% + group_by(.data$GEOID00,.data$GEOID10) %>% + summarize(area_part=sum(.data$AREAPT), + area00=max(as.numeric(.data$AREA00)), + area10=max(as.numeric(.data$AREA10)), + .groups="drop") + cut_correspondence_slivers(d %>% select("GEOID00","GEOID10"), + pmax(area_share(d$area_part,d$area00), + area_share(d$area_part,d$area10)), + min_area_share) } -# the 1990 to 2000 relationship files are fixed width, the "pop" variant is the -# complete one, listing every tract rather than only the ones that changed -get_us_ct_correspondence_2000 <- function(state,cache_path=getOption("tongfen.cache_path")){ +# the 1990 to 2000 relationship files are fixed width, the "pop" variant is the complete one, +# listing every tract rather than only the ones that changed. It only carries the land area of +# each part, tract areas get summed up from those +get_us_ct_correspondence_2000 <- function(state,min_area_share=0.01, + cache_path=getOption("tongfen.cache_path")){ path <- get_us_ct_correspondence_path(state,"2000") cache_path = file.path(cache_path %||% tempdir(),"us_data") local_path <- file.path(cache_path,basename(path)) @@ -83,20 +135,27 @@ get_us_ct_correspondence_2000 <- function(state,cache_path=getOption("tongfen.ca if (!dir.exists(cache_path)) dir.create(cache_path) utils::download.file(path,local_path,quiet=TRUE) } - readr::read_fwf(local_path, + d <- readr::read_fwf(local_path, readr::fwf_cols(STATE90=c(1,2),COUNTY90=c(3,5),TRACT90BASE=c(6,9), TRACT90SUF=c(10,11),PART90=c(12,12),POP90TRACT=c(13,21), PCT90=c(22,25),STATE00=c(26,27),COUNTY00=c(28,30), TRACT00BASE=c(31,34),TRACT00SUF=c(35,36),PART00=c(37,37), POP00TRACT=c(38,46),PCT00=c(47,50),POPPART=c(51,59), AREALAND=c(60,73),STAB=c(74,75),COUNTYNAME=c(76,135)), - col_types=readr::cols(.default="c")) %>% + col_types=readr::cols(.default="c"),progress=FALSE) %>% mutate(GEOID90=paste0(.data$STATE90,.data$COUNTY90,.data$TRACT90BASE, coalesce(.data$TRACT90SUF,"00")), GEOID00=paste0(.data$STATE00,.data$COUNTY00,.data$TRACT00BASE, coalesce(.data$TRACT00SUF,"00"))) %>% - select("GEOID90","GEOID00") %>% - unique() + group_by(.data$GEOID90,.data$GEOID00) %>% + summarize(area_part=sum(as.numeric(.data$AREALAND)),.groups="drop") %>% + group_by(.data$GEOID90) %>% mutate(area90=sum(.data$area_part)) %>% + group_by(.data$GEOID00) %>% mutate(area00=sum(.data$area_part)) %>% + ungroup() + cut_correspondence_slivers(d %>% select("GEOID90","GEOID00"), + pmax(area_share(d$area_part,d$area90), + area_share(d$area_part,d$area00)), + min_area_share) } # stitch relationship files for consecutive censuses into one table spanning all @@ -124,17 +183,17 @@ us_correspondence_span <- function(datasets, available){ available[seq(match(datasets[1],available),match(utils::tail(datasets,1),available))] } -get_us_ct_correspondence <- function(state, datasets, +get_us_ct_correspondence <- function(state, datasets, min_area_share=0.01, cache_path=getOption("tongfen.cache_path")){ available <- names(us_geoid_columns) span <- us_correspondence_span(datasets,available) links <- utils::head(span,-1) %>% lapply(function(year){ - link <- switch(year, - dec1990 = get_us_ct_correspondence_2000(state,cache_path=cache_path), - dec2000 = get_us_ct_correspondence_2010(state,cache_path=cache_path), - dec2010 = get_us_ct_correspondence_2020(state,cache_path=cache_path)) - link %>% select(matches("^GEOID\\d{2}$")) %>% unique() + f <- switch(year, + dec1990 = get_us_ct_correspondence_2000, + dec2000 = get_us_ct_correspondence_2010, + dec2010 = get_us_ct_correspondence_2020) + f(state,min_area_share=min_area_share,cache_path=cache_path) }) join_us_correspondence(links,intersect(available,datasets)) } @@ -170,17 +229,20 @@ get_us_county_subdivision_correspondence_2020 <- function(min_area_share=0.01, if (!dir.exists(cache_path)) dir.create(cache_path) utils::download.file(path,local_path,quiet=TRUE) } - readr::read_delim(local_path,delim="|",progress=FALSE, + d <- readr::read_delim(local_path,delim="|",progress=FALSE, col_types=readr::cols_only(GEOID_COUSUB_10="c",GEOID_COUSUB_20="c", AREALAND_COUSUB_10="d",AREAWATER_COUSUB_10="d", AREALAND_COUSUB_20="d",AREAWATER_COUSUB_20="d", AREALAND_PART="d",AREAWATER_PART="d")) %>% - mutate(area_part=.data$AREALAND_PART+.data$AREAWATER_PART) %>% - filter(pmax(.data$area_part/(.data$AREALAND_COUSUB_10+.data$AREAWATER_COUSUB_10), - .data$area_part/(.data$AREALAND_COUSUB_20+.data$AREAWATER_COUSUB_20)) - >= min_area_share) %>% - select(GEOID10="GEOID_COUSUB_10",GEOID20="GEOID_COUSUB_20") %>% - unique() + group_by(GEOID10=.data$GEOID_COUSUB_10,GEOID20=.data$GEOID_COUSUB_20) %>% + summarize(area_part=sum(.data$AREALAND_PART+.data$AREAWATER_PART), + area10=max(.data$AREALAND_COUSUB_10+.data$AREAWATER_COUSUB_10), + area20=max(.data$AREALAND_COUSUB_20+.data$AREAWATER_COUSUB_20), + .groups="drop") + cut_correspondence_slivers(d %>% select("GEOID10","GEOID20"), + pmax(area_share(d$area_part,d$area10), + area_share(d$area_part,d$area20)), + min_area_share) } get_us_county_subdivision_correspondence_for <- function(state, datasets, min_area_share=0.01, @@ -212,6 +274,10 @@ get_us_county_subdivision_correspondence_for <- function(state, datasets, min_ar #' in between two that are get traversed on the way, the Census Bureau only publishes #' relationship files between consecutive censuses. #' +#' The relationship files are geometric overlays that list every sliver along boundaries that +#' only shifted slightly. Those get cut via `min_area_share`, keeping them would chain +#' unrelated regions into one common geography. +#' #' The correspondence layer reaches back one census further than #' \code{\link{get_tongfen_us_census}}. The 1990 census is available as `dec1990` here, but the #' Census Bureau has retired the 1990 API endpoint, so 1990 data has to be brought in by other @@ -226,10 +292,11 @@ get_us_county_subdivision_correspondence_for <- function(state, datasets, min_ar #' @param level aggregation level, at this stage the only valid levels are 'tract' and #' 'county subdivision'. #' @param min_area_share minimum share of area two geographies have to have in common to count -#' as related, only used when matching county subdivisions across the 2010 and 2020 censuses. -#' The Census Bureau relationship file for these is a geometric overlay, lowering this pulls in -#' slivers along boundaries that only shifted slightly and chains unrelated subdivisions into -#' one common geography. Default is `0.01`. +#' as related, default is `0.01`. The Census Bureau relationship files list every geometric +#' overlap, lowering this pulls in slivers along boundaries that only shifted slightly and +#' chains unrelated regions into one common geography. Raising it gives finer common +#' geographies at the risk of separating regions that did change. No region is ever dropped, +#' if all of its parts are slivers its largest part is kept. #' @param cache_path optional path to cache the relationship files in, defaults to the #' `tongfen.cache_path` option and falls back to a temporary directory #' @return tibble with one row per census geography, a GEOID column for each requested census, @@ -251,7 +318,8 @@ get_tongfen_correspondence_us_census <- function(datasets, regions, level='tract regions$state %>% lapply(function(state){ if (level=='tract') { - get_us_ct_correspondence(state,datasets,cache_path=cache_path) + get_us_ct_correspondence(state,datasets,min_area_share=min_area_share, + cache_path=cache_path) } else { get_us_county_subdivision_correspondence_for(state,datasets, min_area_share=min_area_share, @@ -289,8 +357,7 @@ valid_us_census_datasets <- c( #' @param survey survey to get data for, supported options is "census" #' @param base_geo census year to use as base geography, default is `2010`. #' @param min_area_share minimum share of area two geographies have to have in common to count -#' as related, see \code{\link{get_tongfen_correspondence_us_census}}. Only used when matching -#' county subdivisions across the 2010 and 2020 censuses. +#' as related, default is `0.01`, see \code{\link{get_tongfen_correspondence_us_census}}. #' @return sf object with (wide form) census variables with census year as suffix (separated by underdcore "_"). #' @export #' diff --git a/cran-comments.md b/cran-comments.md index 396b141..aaa1961 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -11,6 +11,8 @@ - dissolving geometries skips regions that don't need to be merged - new `get_tongfen_correspondence_us_census`, US tract correspondence tables now reach back to the 1990 census and county subdivisions forward to the 2020 census +- US correspondence tables no longer chain regions together over slivers, and no longer strip + leading zeros off 2020 census tract identifiers ## Minor changes - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections - missing geographic identifiers no longer merge unrelated regions into one common geography diff --git a/man/get_tongfen_correspondence_us_census.Rd b/man/get_tongfen_correspondence_us_census.Rd index 2999fe2..71a3170 100644 --- a/man/get_tongfen_correspondence_us_census.Rd +++ b/man/get_tongfen_correspondence_us_census.Rd @@ -24,10 +24,11 @@ valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))`} 'county subdivision'.} \item{min_area_share}{minimum share of area two geographies have to have in common to count -as related, only used when matching county subdivisions across the 2010 and 2020 censuses. -The Census Bureau relationship file for these is a geometric overlay, lowering this pulls in -slivers along boundaries that only shifted slightly and chains unrelated subdivisions into -one common geography. Default is `0.01`.} +as related, default is `0.01`. The Census Bureau relationship files list every geometric +overlap, lowering this pulls in slivers along boundaries that only shifted slightly and +chains unrelated regions into one common geography. Raising it gives finer common +geographies at the risk of separating regions that did change. No region is ever dropped, +if all of its parts are slivers its largest part is kept.} \item{cache_path}{optional path to cache the relationship files in, defaults to the `tongfen.cache_path` option and falls back to a temporary directory} @@ -44,6 +45,10 @@ relationship files published by the US Census Bureau. Censuses that aren't reque in between two that are get traversed on the way, the Census Bureau only publishes relationship files between consecutive censuses. +The relationship files are geometric overlays that list every sliver along boundaries that +only shifted slightly. Those get cut via `min_area_share`, keeping them would chain +unrelated regions into one common geography. + The correspondence layer reaches back one census further than \code{\link{get_tongfen_us_census}}. The 1990 census is available as `dec1990` here, but the Census Bureau has retired the 1990 API endpoint, so 1990 data has to be brought in by other diff --git a/man/get_tongfen_us_census.Rd b/man/get_tongfen_us_census.Rd index 757ecbd..ed10055 100644 --- a/man/get_tongfen_us_census.Rd +++ b/man/get_tongfen_us_census.Rd @@ -26,8 +26,7 @@ valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))``} \item{base_geo}{census year to use as base geography, default is `2010`.} \item{min_area_share}{minimum share of area two geographies have to have in common to count -as related, see \code{\link{get_tongfen_correspondence_us_census}}. Only used when matching -county subdivisions across the 2010 and 2020 censuses.} +as related, default is `0.01`, see \code{\link{get_tongfen_correspondence_us_census}}.} } \value{ sf object with (wide form) census variables with census year as suffix (separated by underdcore "_"). diff --git a/tests/testthat/test-us-correspondence.R b/tests/testthat/test-us-correspondence.R index 864eef5..44fe0f8 100644 --- a/tests/testthat/test-us-correspondence.R +++ b/tests/testthat/test-us-correspondence.R @@ -24,6 +24,32 @@ test_that("us tract correspondence links chain across censuses", { expect_equal(nrow(short),3) }) +test_that("sliver cut drops slivers but keeps every region", { + d <- tibble::tibble(GEOID10=c("a","a","b","c"), + GEOID20=c("X","Y","Y","Z")) + k <- tongfen:::cut_correspondence_slivers(d,c(0.99,0.002,0.98,0.5),0.01) + + expect_equal(nrow(k),3) + # the sliver is gone + expect_false(any(k$GEOID10=="a" & k$GEOID20=="Y")) + # but no region on either side disappeared with it + expect_setequal(k$GEOID10,c("a","b","c")) + expect_setequal(k$GEOID20,c("X","Y","Z")) +}) + +test_that("sliver cut keeps the largest part of a region made up of slivers only", { + d <- tibble::tibble(GEOID10=c("a","a","b"),GEOID20=c("X","Y","Y")) + k <- tongfen:::cut_correspondence_slivers(d,c(0.004,0.001,0.99),0.01) + + # none of a's parts clears the cutoff, so its largest one is kept + expect_equal(nrow(k),2) + expect_equal(k$GEOID20[k$GEOID10=="a"],"X") +}) + +test_that("area shares are defined for regions without area", { + expect_equal(tongfen:::area_share(c(1,0),c(2,0)),c(0.5,0)) +}) + test_that("us correspondence spans intermediate censuses", { years <- names(tongfen:::us_geoid_columns) From 4c503301c035c0824264e8825b795275a3af0cb4 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 22:05:36 -0700 Subject: [PATCH 6/8] feat(us): let callers pick the census summary file tidycensus defaults the 2020 census to the PL 94-171 redistricting file, which carries almost none of the variables people tongfen across censuses. There was no way to reach the DHC file through get_tongfen_us_census, so 2010 to 2020 comparisons of anything but population counts were impossible. sumfile takes a single value for all censuses or one named by dataset, since a 2000 to 2020 chain needs a different file per census. NULL keeps tidycensus' own defaults, so existing calls are unaffected. Extends the US vignette with a 2010 to 2020 example on the same variables and the same Bay Area map, plus notes on min_area_share and on reaching back to 1990 through the correspondence layer. Co-Authored-By: Claude Opus 5 --- NEWS.md | 3 +++ R/tongfen_us.R | 29 ++++++++++++++++++++++++++++- cran-comments.md | 3 +++ man/get_tongfen_us_census.Rd | 8 +++++++- vignettes/tongfen_us.Rmd | 34 ++++++++++++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 2 deletions(-) diff --git a/NEWS.md b/NEWS.md index adac2e1..64e2c7a 100644 --- a/NEWS.md +++ b/NEWS.md @@ -30,6 +30,9 @@ common geographies, for Rhode Island tracts across the 2010 and 2020 censuses 198 instead of 60, for Vermont 151 instead of 26 ## Minor changes +- `get_tongfen_us_census` gained a `sumfile` argument, passed through to tidycensus, either a + single value for all censuses or a vector named by dataset. Without it 2020 data is read from + the PL 94-171 redistricting file, which carries almost no variables - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections, default is `3347` (Statistics Canada Lambert) - missing geographic identifiers no longer merge unrelated regions into one common geography diff --git a/R/tongfen_us.R b/R/tongfen_us.R index f72d08c..a54673a 100644 --- a/R/tongfen_us.R +++ b/R/tongfen_us.R @@ -337,6 +337,18 @@ valid_us_census_datasets <- c( dec2020 = "US decentennial census 2020" ) +# Censuses that published several summary files need to be told which one to read. tidycensus +# picks a default per census year, for 2020 that is the PL 94-171 redistricting file which only +# carries a handful of variables, most 2020 variables live in the DHC file. `sumfile` is either a +# single value for all censuses or a vector named by dataset, `NULL` leaves the choice to +# tidycensus. +sumfile_for_dataset <- function(sumfile, ds){ + if (is.null(sumfile)) return(NULL) + if (is.null(names(sumfile))) return(unname(sumfile)) + if (!(ds %in% names(sumfile))) return(NULL) + unname(sumfile[[ds]]) +} + #' Get US census data for 2000 and 2010 census on common census tract based geography #' #' @description @@ -358,6 +370,10 @@ valid_us_census_datasets <- c( #' @param base_geo census year to use as base geography, default is `2010`. #' @param min_area_share minimum share of area two geographies have to have in common to count #' as related, default is `0.01`, see \code{\link{get_tongfen_correspondence_us_census}}. +#' @param sumfile summary file to read the variables from, either a single value used for all +#' censuses or a vector named by dataset, for example `c(dec2010="sf1", dec2020="dhc")`. Default +#' is `NULL`, which leaves the choice to tidycensus. Note that tidycensus defaults the 2020 +#' census to the PL 94-171 redistricting file, most 2020 variables need `sumfile="dhc"`. #' @return sf object with (wide form) census variables with census year as suffix (separated by underdcore "_"). #' @export #' @@ -378,7 +394,7 @@ valid_us_census_datasets <- c( #' #'} get_tongfen_us_census <- function(regions,meta,level='tract',survey="census", - base_geo = NULL, min_area_share = 0.01){ + base_geo = NULL, min_area_share = 0.01, sumfile = NULL){ datasets <- meta$dataset %>% unique if (is.null(base_geo)) base_geo=datasets[1] @@ -387,6 +403,16 @@ get_tongfen_us_census <- function(regions,meta,level='tract',survey="census", assert(length(invalid_datasets)==0, paste0("Invalid datasets :",paste0(invalid_datasets,collapse = ", "))) assert(level %in% c('tract','county subdivision'),"Only census tracts and counties are supported right now.") assert(survey %in% c('census'),"Only census surveys are supported right now.") + if (!is.null(sumfile)) { + if (is.null(names(sumfile))) { + assert(length(sumfile)==1, + "sumfile has to be a single value or a vector named by dataset") + } else { + invalid_sumfiles <- setdiff(names(sumfile),datasets) + assert(length(invalid_sumfiles)==0, + paste0("Invalid datasets in sumfile: ",paste0(invalid_sumfiles,collapse=", "))) + } + } regions$state %>% lapply(function(state){ correspondence <- get_tongfen_correspondence_us_census(datasets = datasets, @@ -401,6 +427,7 @@ get_tongfen_us_census <- function(regions,meta,level='tract',survey="census", short_year <- substr(as.character(year),3,4) tidycensus::get_decennial(geography=level, state=state, county=regions$county, variables = m$variable, year = year, + sumfile = sumfile_for_dataset(sumfile,ds), geometry = base_geo==ds, output="wide") %>% rename(!!paste0("GEOID",short_year):=.data$GEOID) }) %>% diff --git a/cran-comments.md b/cran-comments.md index aaa1961..568f0c3 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -14,10 +14,13 @@ - US correspondence tables no longer chain regions together over slivers, and no longer strip leading zeros off 2020 census tract identifiers ## Minor changes +- `get_tongfen_us_census` gained a `sumfile` argument, passed through to tidycensus - `get_tongfen_correspondence_ca_census` gained a `crs` argument for the spatial intersections - missing geographic identifiers no longer merge unrelated regions into one common geography - fix crash when tongfen-ing census tracts across non-adjacent censuses +- US county subdivision data errors out up front on censuses it can't be matched across - several fixes to the deprecated `get_tongfen_census_*` functions +- faster `check_tongfen_areas` and `aggregate_correspondences` # tongfen v.0.3.7 ## Major changes diff --git a/man/get_tongfen_us_census.Rd b/man/get_tongfen_us_census.Rd index ed10055..d6235cd 100644 --- a/man/get_tongfen_us_census.Rd +++ b/man/get_tongfen_us_census.Rd @@ -10,7 +10,8 @@ get_tongfen_us_census( level = "tract", survey = "census", base_geo = NULL, - min_area_share = 0.01 + min_area_share = 0.01, + sumfile = NULL ) } \arguments{ @@ -27,6 +28,11 @@ valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))``} \item{min_area_share}{minimum share of area two geographies have to have in common to count as related, default is `0.01`, see \code{\link{get_tongfen_correspondence_us_census}}.} + +\item{sumfile}{summary file to read the variables from, either a single value used for all +censuses or a vector named by dataset, for example `c(dec2010="sf1", dec2020="dhc")`. Default +is `NULL`, which leaves the choice to tidycensus. Note that tidycensus defaults the 2020 +census to the PL 94-171 redistricting file, most 2020 variables need `sumfile="dhc"`.} } \value{ sf object with (wide form) census variables with census year as suffix (separated by underdcore "_"). diff --git a/vignettes/tongfen_us.Rmd b/vignettes/tongfen_us.Rmd index e1ef7c8..7b09a58 100644 --- a/vignettes/tongfen_us.Rmd +++ b/vignettes/tongfen_us.Rmd @@ -62,3 +62,37 @@ census_data %>% coord_sf(datum=NA,xlim=c(-122.6,-121.7),ylim=c(37.2,37.9)) ``` +## Bridging to the 2020 census + +The same works across the 2010 and 2020 censuses. Two things change. The 2020 census renamed the variables, *population in occupied housing units* is `H8_001N` and *households* is `H3_002N`. And those live in the Demographic and Housing Characteristics file, whereas tidycensus reads the PL 94-171 redistricting file for 2020 by default, so we point it at the right one via `sumfile`. It takes a single value for all censuses, or one named by dataset as we do here. + +```{r} +meta_2020 <- bind_rows( + meta_for_additive_variables("dec2010",c(population_2010="H011001", + households_2010="H013001")), + meta_for_additive_variables("dec2020",c(population_2020="H8_001N", + households_2020="H3_002N"))) +``` + +```{r results='hide'} +census_data_2020 <- get_tongfen_us_census(regions = list(state="CA"), meta=meta_2020, + level="tract", sumfile=c(dec2020="dhc")) %>% + mutate(change=population_2020/households_2020-population_2010/households_2010) +``` + +```{r} +census_data_2020 %>% + mutate(c=cut(change,c(-Inf,-0.5,-0.3,-0.2,-0.1,0,0.1,0.2,0.3,0.5,Inf))) %>% + ggplot() + + geom_sf(aes(fill=c), size=0.05) + + scale_fill_brewer(palette = "RdYlGn") + + labs(title="Bay area change in average household size 2010-2020", fill=NULL) + + coord_sf(datum=NA,xlim=c(-122.6,-121.7),ylim=c(37.2,37.9)) +``` + +## Notes on the common geography + +The Census Bureau relationship files these correspondences are built from are geometric overlays, they list every place two censuses' geographies intersect, including slivers along boundaries that only shifted by a few metres. Chaining those together merges regions that have nothing to do with each other, so `min_area_share` sets how much area two regions have to have in common before they count as related. The default of `0.01` works well, raising it gives finer common geographies at the risk of separating regions that genuinely did change. No region is ever dropped, if all of a region's parts fall below the cutoff its largest part is kept. + +Correspondence tables reach back one census further than the data does. The Census Bureau has retired the 1990 API endpoint, so tidycensus cannot fetch 1990 data, but `get_tongfen_correspondence_us_census` will match 1990 tracts up with later censuses. To use it, get the 1990 data elsewhere, for example from NHGIS via the ipumsr package, and hand it to `tongfen_aggregate` along with the correspondence table. + From 3ac3255e08184da575f20f2b8fe47794ddb0c5b5 Mon Sep 17 00:00:00 2001 From: Jens von Bergmann Date: Mon, 10 Aug 2026 22:40:35 -0700 Subject: [PATCH 7/8] regenerate docs --- docs/404.html | 6 +- docs/LICENSE-text.html | 4 +- docs/LICENSE.html | 4 +- docs/articles/index.html | 4 +- docs/articles/polling_districts.html | 6 +- docs/articles/tongfen-ca-estimate.html | 6 +- docs/articles/tongfen.html | 6 +- docs/articles/tongfen_ca.html | 12 +- docs/articles/tongfen_ca.md | 4 +- .../figure-html/unnamed-chunk-3-1.png | Bin 379093 -> 234588 bytes .../figure-html/unnamed-chunk-5-1.png | Bin 284296 -> 282594 bytes .../figure-html/unnamed-chunk-10-1.png | Bin 203940 -> 203879 bytes .../figure-html/unnamed-chunk-6-1.png | Bin 83616 -> 83589 bytes .../figure-html/unnamed-chunk-7-1.png | Bin 216641 -> 216590 bytes docs/articles/tongfen_us.html | 61 ++++++- docs/articles/tongfen_us.md | 59 +++++++ .../figure-html/unnamed-chunk-5-1.png | Bin 273278 -> 299517 bytes .../figure-html/unnamed-chunk-8-1.png | Bin 0 -> 301964 bytes docs/authors.html | 8 +- docs/authors.md | 4 +- .../bootstrap-5.3.1/bootstrap.bundle.min.js | 7 - .../bootstrap.bundle.min.js.map | 1 - docs/deps/bootstrap-5.3.1/bootstrap.min.css | 5 - ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3-UBGEe.woff2 | Bin 17628 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3CUBGEe.woff2 | Bin 4340 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3GUBGEe.woff2 | Bin 35532 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3KUBGEe.woff2 | Bin 26428 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3OUBGEe.woff2 | Bin 13040 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3iUBGEe.woff2 | Bin 22796 -> 0 bytes ...O7CnqEu92Fr1ME7kSn66aGLdTylUAMa3yUBA.woff2 | Bin 40128 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMawCUBGEe.woff2 | Bin 40688 -> 0 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMaxKUBGEe.woff2 | Bin 20408 -> 0 bytes .../bootstrap-5.3.8/bootstrap.bundle.min.js | 7 + .../bootstrap.bundle.min.js.map | 1 + docs/deps/bootstrap-5.3.8/bootstrap.min.css | 5 + .../font.css | 0 .../07d40e985ad7c747025dabb9f22142c4.woff2 | Bin .../fonts/1Ptug8zYS_SKggPNyC0ITw.woff2 | Bin .../fonts/1Ptug8zYS_SKggPNyCAIT5lu.woff2 | Bin .../fonts/1Ptug8zYS_SKggPNyCIIT5lu.woff2 | Bin .../fonts/1Ptug8zYS_SKggPNyCMIT5lu.woff2 | Bin .../fonts/1Ptug8zYS_SKggPNyCkIT5lu.woff2 | Bin .../1f5e011d6aae0d98fc0518e1a303e99a.woff2 | Bin .../fonts/4iCs6KVjbNBYlgoKcQ72j00.woff2 | Bin .../fonts/4iCs6KVjbNBYlgoKcg72j00.woff2 | Bin .../fonts/4iCs6KVjbNBYlgoKcw72j00.woff2 | Bin .../fonts/4iCs6KVjbNBYlgoKew72j00.woff2 | Bin .../fonts/4iCs6KVjbNBYlgoKfA72j00.woff2 | Bin .../fonts/4iCs6KVjbNBYlgoKfw72.woff2 | Bin .../fonts/4iCv6KVjbNBYlgoCxCvjs2yNL4U.woff2 | Bin .../fonts/4iCv6KVjbNBYlgoCxCvjsGyN.woff2 | Bin .../fonts/4iCv6KVjbNBYlgoCxCvjtGyNL4U.woff2 | Bin .../fonts/4iCv6KVjbNBYlgoCxCvjvGyNL4U.woff2 | Bin .../fonts/4iCv6KVjbNBYlgoCxCvjvWyNL4U.woff2 | Bin .../fonts/4iCv6KVjbNBYlgoCxCvjvmyNL4U.woff2 | Bin .../626330658504e338ee86aec8e957426b.woff2 | Bin ...K1dSBYKcSV-LCoeQqfX1RYOo3qPZ7jsDJT9g.woff2 | Bin ...K1dSBYKcSV-LCoeQqfX1RYOo3qPZ7ksDJT9g.woff2 | Bin .../6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7nsDI.woff2 | Bin ...K1dSBYKcSV-LCoeQqfX1RYOo3qPZ7osDJT9g.woff2 | Bin ...K1dSBYKcSV-LCoeQqfX1RYOo3qPZ7psDJT9g.woff2 | Bin ...K1dSBYKcSV-LCoeQqfX1RYOo3qPZ7qsDJT9g.woff2 | Bin ...K1dSBYKcSV-LCoeQqfX1RYOo3qPZ7rsDJT9g.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qN67lqDY.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qNK7lqDY.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qNa7lqDY.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qNq7lqDY.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qO67lqDY.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qOK7l.woff2 | Bin .../6xK3dSBYKcSV-LCoeQqfX1RYOo3qPK7lqDY.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3i54rwkxduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3i54rwlBduz8A.woff2 | Bin ...6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwlxdu.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3i54rwmBduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3i54rwmRduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3i54rwmhduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3i54rwmxduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ig4vwkxduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ig4vwlBduz8A.woff2 | Bin ...6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwlxdu.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ig4vwmBduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ig4vwmRduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ig4vwmhduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ig4vwmxduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ik4zwkxduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ik4zwlBduz8A.woff2 | Bin ...6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwlxdu.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ik4zwmBduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ik4zwmRduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ik4zwmhduz8A.woff2 | Bin ...ydSBYKcSV-LCoeQqfX1RYOo3ik4zwmxduz8A.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNBeudwk.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNReuQ.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNdeudwk.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNheudwk.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNleudwk.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNpeudwk.woff2 | Bin .../CSR54z1Qlv-GDxkbKVQ_dFsvWNteudwk.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fO0KTet_.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fO4KTet_.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fO8KTet_.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fOAKTQ.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fOMKTet_.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fOQKTet_.woff2 | Bin .../fonts/CSR64z1Qlv-GDxkbKVQ_fOwKTet_.woff2 | Bin ..._QiYsKILxRpg3hIP6sJ7fM7PqlONvQlMIXxw.woff2 | Bin .../HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvUlMI.woff2 | Bin ..._QiYsKILxRpg3hIP6sJ7fM7PqlONvXlMIXxw.woff2 | Bin ..._QiYsKILxRpg3hIP6sJ7fM7PqlONvYlMIXxw.woff2 | Bin ..._QiYsKILxRpg3hIP6sJ7fM7PqlONvZlMIXxw.woff2 | Bin ..._QiYsKILxRpg3hIP6sJ7fM7PqlONvalMIXxw.woff2 | Bin ..._QiYsKILxRpg3hIP6sJ7fM7PqlONvblMIXxw.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlM-vWjMY.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlMOvWjMY.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlMevWjMY.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlMuvWjMY.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlOevWjMY.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2 | Bin .../HI_SiYsKILxRpg3hIP6sJ7fM7PqlPuvWjMY.woff2 | Bin .../fonts/JTUSjIg1_i6t8kCHKm459W1hyzbi.woff2 | Bin .../fonts/JTUSjIg1_i6t8kCHKm459WRhyzbi.woff2 | Bin .../fonts/JTUSjIg1_i6t8kCHKm459WZhyzbi.woff2 | Bin .../fonts/JTUSjIg1_i6t8kCHKm459Wdhyzbi.woff2 | Bin .../fonts/JTUSjIg1_i6t8kCHKm459Wlhyw.woff2 | Bin ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3-UBGEe.woff2 | Bin 0 -> 17624 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3CUBGEe.woff2 | Bin 0 -> 4348 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3GUBGEe.woff2 | Bin 0 -> 36652 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3KUBGEe.woff2 | Bin 0 -> 29392 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3OUBGEe.woff2 | Bin 0 -> 14340 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMa3iUBGEe.woff2 | Bin 0 -> 23664 bytes ...O7CnqEu92Fr1ME7kSn66aGLdTylUAMa3yUBA.woff2 | Bin 0 -> 43136 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMawCUBGEe.woff2 | Bin 0 -> 41348 bytes ...CnqEu92Fr1ME7kSn66aGLdTylUAMaxKUBGEe.woff2 | Bin 0 -> 20556 bytes .../fonts/QGYpz_kZZAGCONcK2A4bGOj8mNhN.woff2 | Bin .../fonts/S6u8w4BMUTPHjxsAUi-qJCY.woff2 | Bin .../fonts/S6u8w4BMUTPHjxsAXC-q.woff2 | Bin .../fonts/S6u9w4BMUTPHh6UVSwaPGR_p.woff2 | Bin .../fonts/S6u9w4BMUTPHh6UVSwiPGQ.woff2 | Bin .../fonts/S6u9w4BMUTPHh7USSwaPGR_p.woff2 | Bin .../fonts/S6u9w4BMUTPHh7USSwiPGQ.woff2 | Bin .../fonts/S6uyw4BMUTPHjx4wXg.woff2 | Bin .../fonts/S6uyw4BMUTPHjxAwXjeu.woff2 | Bin ...73FwrK3iLTeHuS_nVMrMxCp50SjIa0ZL7SUc.woff2 | Bin ...UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa1ZL7.woff2 | Bin ...73FwrK3iLTeHuS_nVMrMxCp50SjIa1pL7SUc.woff2 | Bin ...73FwrK3iLTeHuS_nVMrMxCp50SjIa25L7SUc.woff2 | Bin ...73FwrK3iLTeHuS_nVMrMxCp50SjIa2JL7SUc.woff2 | Bin ...73FwrK3iLTeHuS_nVMrMxCp50SjIa2ZL7SUc.woff2 | Bin ...73FwrK3iLTeHuS_nVMrMxCp50SjIa2pL7SUc.woff2 | Bin .../fonts/XRXV3I6Li01BKofIMeaBXso.woff2 | Bin .../fonts/XRXV3I6Li01BKofINeaB.woff2 | Bin .../fonts/XRXV3I6Li01BKofIO-aBXso.woff2 | Bin .../fonts/XRXV3I6Li01BKofIOOaBXso.woff2 | Bin .../fonts/XRXV3I6Li01BKofIOuaBXso.woff2 | Bin .../c2f002b3a87d3f9bfeebb23d32cfd9f8.woff2 | Bin .../ee91700cdbf7ce16c054c2bb8946c736.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqW106F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWt06F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWtE6F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWtU6F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWtk6F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWu06F15M.woff2 | Bin ...126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWuU6F.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWuk6F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWvU6F15M.woff2 | Bin ...MiZpBA-UFUIcVXSCEkx2cmqvXlWqWxU6F15M.woff2 | Bin ...Gs126MiZpBA-UvWbX2vVnXBbObj2OVTS-muw.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTS2mu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTSCmu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTSGmu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTSKmu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTSOmu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTSumu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTSymu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTUGmu1aB.woff2 | Bin ...126MiZpBA-UvWbX2vVnXBbObj2OVTVOmu1aB.woff2 | Bin .../fonts/q5uGsou0JOdh94bfuQltOxU.woff2 | Bin .../fonts/q5uGsou0JOdh94bfvQlt.woff2 | Bin docs/deps/data-deps.txt | 4 +- docs/index.html | 6 +- docs/llms.txt | 2 + docs/news/index.html | 4 +- docs/pkgdown.yml | 4 +- .../add_census_ca_base_variables.html | 4 +- docs/reference/aggregate_data_with_meta.html | 4 +- docs/reference/check_tongfen_areas.html | 17 +- docs/reference/check_tongfen_areas.md | 5 + .../reference/check_tongfen_single_areas.html | 4 +- .../estimate_tongfen_correspondence.html | 4 +- ...stimate_tongfen_single_correspondence.html | 4 +- .../get_correspondence_ca_census_for.html | 4 +- ...t_single_correspondence_ca_census_for.html | 4 +- docs/reference/get_tongfen_ca_census.html | 17 +- docs/reference/get_tongfen_ca_census.md | 14 +- .../get_tongfen_ca_census_ct_from_da.html | 4 +- docs/reference/get_tongfen_census_ct.html | 4 +- docs/reference/get_tongfen_census_da.html | 4 +- .../get_tongfen_correspondence_ca_census.html | 20 ++- .../get_tongfen_correspondence_ca_census.md | 17 +- .../get_tongfen_correspondence_us_census.html | 162 ++++++++++++++++++ .../get_tongfen_correspondence_us_census.md | 86 ++++++++++ docs/reference/get_tongfen_us_census.html | 36 +++- docs/reference/get_tongfen_us_census.md | 26 ++- docs/reference/index.html | 11 +- docs/reference/index.md | 2 + .../meta_for_additive_variables.html | 4 +- .../reference/meta_for_ca_census_vectors.html | 4 +- docs/reference/proportional_reaggregate.html | 4 +- docs/reference/tongfen_aggregate.html | 17 +- docs/reference/tongfen_aggregate.md | 13 +- docs/reference/tongfen_ca_census_ct.html | 4 +- docs/reference/tongfen_estimate.html | 4 +- .../reference/tongfen_estimate_ca_census.html | 4 +- .../tongfen_tag_largest_overlap.html | 4 +- .../vancouver_elections_data_2015.html | 4 +- .../vancouver_elections_data_2019.html | 4 +- .../vancouver_elections_geos_2015.html | 4 +- .../vancouver_elections_geos_2019.html | 4 +- docs/search.json | 2 +- docs/sitemap.xml | 1 + 220 files changed, 605 insertions(+), 155 deletions(-) create mode 100644 docs/articles/tongfen_us_files/figure-html/unnamed-chunk-8-1.png delete mode 100644 docs/deps/bootstrap-5.3.1/bootstrap.bundle.min.js delete mode 100644 docs/deps/bootstrap-5.3.1/bootstrap.bundle.min.js.map delete mode 100644 docs/deps/bootstrap-5.3.1/bootstrap.min.css delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3-UBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3CUBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3GUBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3KUBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3OUBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3iUBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3yUBA.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMawCUBGEe.woff2 delete mode 100644 docs/deps/bootstrap-5.3.1/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMaxKUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/bootstrap.bundle.min.js create mode 100644 docs/deps/bootstrap-5.3.8/bootstrap.bundle.min.js.map create mode 100644 docs/deps/bootstrap-5.3.8/bootstrap.min.css rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/font.css (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/07d40e985ad7c747025dabb9f22142c4.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/1Ptug8zYS_SKggPNyC0ITw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/1Ptug8zYS_SKggPNyCAIT5lu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/1Ptug8zYS_SKggPNyCIIT5lu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/1Ptug8zYS_SKggPNyCMIT5lu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/1Ptug8zYS_SKggPNyCkIT5lu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/1f5e011d6aae0d98fc0518e1a303e99a.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCs6KVjbNBYlgoKcQ72j00.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCs6KVjbNBYlgoKcg72j00.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCs6KVjbNBYlgoKcw72j00.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCs6KVjbNBYlgoKew72j00.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCs6KVjbNBYlgoKfA72j00.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCs6KVjbNBYlgoKfw72.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCv6KVjbNBYlgoCxCvjs2yNL4U.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCv6KVjbNBYlgoCxCvjsGyN.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCv6KVjbNBYlgoCxCvjtGyNL4U.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCv6KVjbNBYlgoCxCvjvGyNL4U.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCv6KVjbNBYlgoCxCvjvWyNL4U.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/4iCv6KVjbNBYlgoCxCvjvmyNL4U.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/626330658504e338ee86aec8e957426b.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7jsDJT9g.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7ksDJT9g.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7nsDI.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7osDJT9g.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7psDJT9g.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7qsDJT9g.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK1dSBYKcSV-LCoeQqfX1RYOo3qPZ7rsDJT9g.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qN67lqDY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qNK7lqDY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qNa7lqDY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qNq7lqDY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qO67lqDY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qOK7l.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xK3dSBYKcSV-LCoeQqfX1RYOo3qPK7lqDY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwkxduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwlBduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwlxdu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwmBduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwmRduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwmhduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3i54rwmxduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwkxduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwlBduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwlxdu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwmBduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwmRduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwmhduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ig4vwmxduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwkxduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwlBduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwlxdu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwmBduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwmRduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwmhduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/6xKydSBYKcSV-LCoeQqfX1RYOo3ik4zwmxduz8A.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNBeudwk.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNReuQ.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNdeudwk.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNheudwk.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNleudwk.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNpeudwk.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR54z1Qlv-GDxkbKVQ_dFsvWNteudwk.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fO0KTet_.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fO4KTet_.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fO8KTet_.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fOAKTQ.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fOMKTet_.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fOQKTet_.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/CSR64z1Qlv-GDxkbKVQ_fOwKTet_.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvQlMIXxw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvUlMI.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvXlMIXxw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvYlMIXxw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvZlMIXxw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvalMIXxw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_QiYsKILxRpg3hIP6sJ7fM7PqlONvblMIXxw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlM-vWjMY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlMOvWjMY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlMevWjMY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlMuvWjMY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlOevWjMY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPevW.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/HI_SiYsKILxRpg3hIP6sJ7fM7PqlPuvWjMY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/JTUSjIg1_i6t8kCHKm459W1hyzbi.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/JTUSjIg1_i6t8kCHKm459WRhyzbi.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/JTUSjIg1_i6t8kCHKm459WZhyzbi.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/JTUSjIg1_i6t8kCHKm459Wdhyzbi.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/JTUSjIg1_i6t8kCHKm459Wlhyw.woff2 (100%) create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3-UBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3CUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3GUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3KUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3OUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3iUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMa3yUBA.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMawCUBGEe.woff2 create mode 100644 docs/deps/bootstrap-5.3.8/fonts/KFO7CnqEu92Fr1ME7kSn66aGLdTylUAMaxKUBGEe.woff2 rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/QGYpz_kZZAGCONcK2A4bGOj8mNhN.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6u8w4BMUTPHjxsAUi-qJCY.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6u8w4BMUTPHjxsAXC-q.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6u9w4BMUTPHh6UVSwaPGR_p.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6u9w4BMUTPHh6UVSwiPGQ.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6u9w4BMUTPHh7USSwaPGR_p.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6u9w4BMUTPHh7USSwiPGQ.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6uyw4BMUTPHjx4wXg.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/S6uyw4BMUTPHjxAwXjeu.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa0ZL7SUc.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa1ZL7.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa1pL7SUc.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa25L7SUc.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa2JL7SUc.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa2ZL7SUc.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/UcC73FwrK3iLTeHuS_nVMrMxCp50SjIa2pL7SUc.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/XRXV3I6Li01BKofIMeaBXso.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/XRXV3I6Li01BKofINeaB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/XRXV3I6Li01BKofIO-aBXso.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/XRXV3I6Li01BKofIOOaBXso.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/XRXV3I6Li01BKofIOuaBXso.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/c2f002b3a87d3f9bfeebb23d32cfd9f8.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/ee91700cdbf7ce16c054c2bb8946c736.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqW106F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWt06F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWtE6F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWtU6F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWtk6F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWu06F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWuU6F.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWuk6F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWvU6F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memtYaGs126MiZpBA-UFUIcVXSCEkx2cmqvXlWqWxU6F15M.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTS-muw.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTS2mu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTSCmu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTSGmu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTSKmu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTSOmu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTSumu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTSymu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTUGmu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTVOmu1aB.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/q5uGsou0JOdh94bfuQltOxU.woff2 (100%) rename docs/deps/{bootstrap-5.3.1 => bootstrap-5.3.8}/fonts/q5uGsou0JOdh94bfvQlt.woff2 (100%) create mode 100644 docs/reference/get_tongfen_correspondence_us_census.html create mode 100644 docs/reference/get_tongfen_correspondence_us_census.md diff --git a/docs/404.html b/docs/404.html index c868523..d0348fd 100644 --- a/docs/404.html +++ b/docs/404.html @@ -12,8 +12,8 @@ - - + + @@ -30,7 +30,7 @@ tongfen - 0.3.7 + 0.3.8 + + + + + +
+
+
+

Get correspondence table for US census geographies

+ Source: R/tongfen_us.R +
get_tongfen_correspondence_us_census.Rd
+
+ +
+

[Maturing]

+

Builds a correspondence table matching US census geographies across censuses, based on the +relationship files published by the US Census Bureau. Censuses that aren't requested but sit +in between two that are get traversed on the way, the Census Bureau only publishes +relationship files between consecutive censuses.

+

The relationship files are geometric overlays that list every sliver along boundaries that +only shifted slightly. Those get cut via `min_area_share`, keeping them would chain +unrelated regions into one common geography.

+

The correspondence layer reaches back one census further than +get_tongfen_us_census. The 1990 census is available as `dec1990` here, but the +Census Bureau has retired the 1990 API endpoint, so 1990 data has to be brought in by other +means, for example from NHGIS via the ipumsr package, and handed to +tongfen_aggregate together with this correspondence table.

+
+ +
+

Usage

+ 
get_tongfen_correspondence_us_census(
+  datasets,
+  regions,
+  level = "tract",
+  min_area_share = 0.01,
+  cache_path = getOption("tongfen.cache_path")
+)
+
+ +
+

Arguments

+ + +
datasets
+

vector of censuses to match up, valid values are `dec1990`, `dec2000`, +`dec2010` and `dec2020` for census tracts, `dec2000` through `dec2020` for county +subdivisions. At least two censuses are needed.

+ + +
regions
+

list with regions to query the correspondence for. At this stage, the only +valid list is a vector of states, i.e. `regions = list(state=c("CA","OR"))`

+ + +
level
+

aggregation level, at this stage the only valid levels are 'tract' and +'county subdivision'.

+ + +
min_area_share
+

minimum share of area two geographies have to have in common to count +as related, default is `0.01`. The Census Bureau relationship files list every geometric +overlap, lowering this pulls in slivers along boundaries that only shifted slightly and +chains unrelated regions into one common geography. Raising it gives finer common +geographies at the risk of separating regions that did change. No region is ever dropped, +if all of its parts are slivers its largest part is kept.

+ + +
cache_path
+

optional path to cache the relationship files in, defaults to the +`tongfen.cache_path` option and falls back to a temporary directory

+ +
+
+

Value

+

tibble with one row per census geography, a GEOID column for each requested census, +and the common geography identified by `TongfenID` and `TongfenUID`.

+
+ +
+

Examples

+ 
# Match up census tracts for the 1990 and 2000 censuses in Rhode Island
+if (FALSE) { # \dontrun{
+correspondence <- get_tongfen_correspondence_us_census(datasets = c("dec1990","dec2000"),
+                                                       regions = list(state="RI"))
+} # }
+
+
+
+ + +
+ + + + + + + diff --git a/docs/reference/get_tongfen_correspondence_us_census.md b/docs/reference/get_tongfen_correspondence_us_census.md new file mode 100644 index 0000000..aeaeede --- /dev/null +++ b/docs/reference/get_tongfen_correspondence_us_census.md @@ -0,0 +1,86 @@ +# Get correspondence table for US census geographies + +**\[maturing\]** + +Builds a correspondence table matching US census geographies across +censuses, based on the relationship files published by the US Census +Bureau. Censuses that aren't requested but sit in between two that are +get traversed on the way, the Census Bureau only publishes relationship +files between consecutive censuses. + +The relationship files are geometric overlays that list every sliver +along boundaries that only shifted slightly. Those get cut via +\`min_area_share\`, keeping them would chain unrelated regions into one +common geography. + +The correspondence layer reaches back one census further than +[`get_tongfen_us_census`](https://mountainmath.github.io/tongfen/reference/get_tongfen_us_census.md). +The 1990 census is available as \`dec1990\` here, but the Census Bureau +has retired the 1990 API endpoint, so 1990 data has to be brought in by +other means, for example from NHGIS via the ipumsr package, and handed +to +[`tongfen_aggregate`](https://mountainmath.github.io/tongfen/reference/tongfen_aggregate.md) +together with this correspondence table. + +## Usage + +``` r +get_tongfen_correspondence_us_census( + datasets, + regions, + level = "tract", + min_area_share = 0.01, + cache_path = getOption("tongfen.cache_path") +) +``` + +## Arguments + +- datasets: + + vector of censuses to match up, valid values are \`dec1990\`, + \`dec2000\`, \`dec2010\` and \`dec2020\` for census tracts, + \`dec2000\` through \`dec2020\` for county subdivisions. At least two + censuses are needed. + +- regions: + + list with regions to query the correspondence for. At this stage, the + only valid list is a vector of states, i.e. \`regions = + list(state=c("CA","OR"))\` + +- level: + + aggregation level, at this stage the only valid levels are 'tract' and + 'county subdivision'. + +- min_area_share: + + minimum share of area two geographies have to have in common to count + as related, default is \`0.01\`. The Census Bureau relationship files + list every geometric overlap, lowering this pulls in slivers along + boundaries that only shifted slightly and chains unrelated regions + into one common geography. Raising it gives finer common geographies + at the risk of separating regions that did change. No region is ever + dropped, if all of its parts are slivers its largest part is kept. + +- cache_path: + + optional path to cache the relationship files in, defaults to the + \`tongfen.cache_path\` option and falls back to a temporary directory + +## Value + +tibble with one row per census geography, a GEOID column for each +requested census, and the common geography identified by \`TongfenID\` +and \`TongfenUID\`. + +## Examples + +``` r +# Match up census tracts for the 1990 and 2000 censuses in Rhode Island +if (FALSE) { # \dontrun{ +correspondence <- get_tongfen_correspondence_us_census(datasets = c("dec1990","dec2000"), + regions = list(state="RI")) +} # } +``` diff --git a/docs/reference/get_tongfen_us_census.html b/docs/reference/get_tongfen_us_census.html index 8186ad9..57f67e0 100644 --- a/docs/reference/get_tongfen_us_census.html +++ b/docs/reference/get_tongfen_us_census.html @@ -1,9 +1,17 @@ -Get US census data for 2000 and 2010 census on common census tract based geography — get_tongfen_us_census • tongfenGet US census data for 2000 and 2010 census on common census tract based geography — get_tongfen_us_census • tongfen +a single convenience function. +Data is only available for the 2000, 2010 and 2020 censuses, the Census Bureau has retired the +1990 API endpoint. To tongfen 1990 data, obtain it elsewhere and combine it with a +correspondence table from get_tongfen_correspondence_us_census via +tongfen_aggregate."> Skip to contents @@ -11,7 +19,7 @@ tongfen - 0.3.7 + 0.3.8