Refactor labeling structure - #83
Open
diraol wants to merge 10 commits into
Open
Conversation
Add a versioned YAML schema for categorical labels and an internal API to load, validate, cache, compile, and apply configurations. Configurations preserve codes as strings, reject duplicate variables and codes, distinguish null unmatched values from explicit fallback labels, and compile all available mappings into one dplyr::mutate() while retaining character output. Add a population fixture and tests covering cache use, validation, unknown codes, missing values, binary fallback behavior, and absent columns. Add yaml to Imports and exempt YAML files from the repository tab-indentation whitespace rule because YAML syntax requires spaces. No existing add_labels_*() helper consumes the new API yet; this commit establishes the tested contract before migrating the published label maps.
Move the 2010 population mappings for urban/rural status (V1006), relationship to the household reference person (V0502), and sex (V0601) out of add_labels_population() into the installed YAML configuration. Load and compile these mappings through apply_label_config() before the remaining legacy mutations so available configured columns are updated together in one lazy dplyr mutate(). Add coverage for published label fidelity, unmatched values, character output, and Arrow query laziness, including an integration test for add_labels_population().
Extract all six categorical mappings from add_labels_families() into the installed families/2000-pt label configuration. Apply the configuration through one lazy mutation and cover mapped, unknown, and missing source values.
Extract the four 2010 mortality categorical mappings into the installed mortality label configuration and apply them through apply_label_config(). Add coverage for mapped, unknown, and missing values.
Extract all 2010 emigration categorical mappings, including the V3061 country map, into the installed emigration label configuration. Apply configured columns in one lazy mutation and retain the legacy first-match result for the duplicated V3061 code 8000826. Add coverage for mapped, unknown, and missing values.
Add the installed households/2000-pt configuration for all 2000 categorical labels and route 2000 household labeling through apply_label_config(). Keep the 2010 implementation unchanged in this commit so the annual migration remains independently runnable.
Extract all 2010 household categorical mappings into households/2010-pt and replace the remaining manual mutations with apply_label_config(). Represent the shared yes/no groups declaratively with unmatched: Não, preserving the legacy treatment of unknown and missing values for both supported years. Add regression coverage for the YAML yes/no mappings.
Contributor
Author
|
This is a BIG PR. I've worked on it to have meaningfull commits, so I'd suggest to evaluate the PR commit by commit. |
diraol
force-pushed
the
refactor/variable-metadata
branch
from
September 7, 2026 21:07
f4af4b8 to
7e401be
Compare
Extend label configuration loading with relative imports, named definitions, and use references that normalize to the existing mapping representation before validation and mutation compilation. Reject missing or unsafe imports, resolved paths outside the label root, duplicate definition names, unknown references, unsupported library fields, and circular imports. Centralize the shared urban/rural V1006 map in _shared/geography.yml and reuse it across population, households, mortality, and emigration configurations. Add fixtures and coverage for import expansion, collisions, path traversal, unknown definitions, and circular imports.
Add optional alias and description metadata to configured variables without changing physical dataset column names. Expose label_variable_metadata() as a catalog of configured aliases and descriptions and document the metadata-only YAML schema.
diraol
force-pushed
the
refactor/variable-metadata
branch
from
September 7, 2026 21:09
7e401be to
b458055
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Over here we are splitting code and configuration for label definition.
Mainly we are focusing on a shared code structure to evaluate and "apply" labels to the data objects, while keeping label definitions on configuration files (yaml files).
The shared code is a bit more efficient (one single mutate operation per "dataset" instead of one mutate per variable for each dataset).
The yaml based configuration for labels, besides being easier for humans to read (and contribute), it also allows the re-use of configurations (the same variable across many different cohorts can be defined only once and shared).
The proposed structure also allow us to give "alias" and "description" characteristics for the variables (besides its IDs).
Example:
tests/testthat/fixtures/label_imports/population/2010-pt.ymlNew features and metadata handling:
label_variable_metadata()function, which returns optional aliases and descriptions for variables as declared in the label configuration. This helps users interpret variable names without altering the data columns. (NEWS.md,NAMESPACE) [1] [2]Refactoring and code maintainability:
add_labels_emigration()andadd_labels_families()to use a configuration-based approach (load_label_config()andapply_label_config()), replacing large blocks of hardcoded label mappings. This reduces code duplication and improves maintainability. (R/add_labels_emigration.R,R/add_labels_families.R) [1] [2]Dependencies and configuration:
yamlpackage to the list of imports inDESCRIPTION, supporting the new configuration-driven labeling logic..gitattributesto clarify whitespace rules for YAML files, ensuring they are not affected by global indentation rules.