Protocol v0.4.1 and Phase 1 tasks 1.0 and 1.3

.gitignore

@@ -60,3 +60,6 @@ resources/
 manuscript/.quarto/
 
 **/*.quarto_ipynb
+
+# Protocol rendered Word output (docstyle)
+docs/protocol/output/

CLAUDE.md

@@ -40,6 +40,19 @@ quarto render
 
 # Render manuscript to Word (docstyle)
 quarto render manuscript/manuscript.qmd
+
+# Render protocol documents to Word (docstyle subproject docs/protocol/_quarto.yml;
+# output in docs/protocol/output/, gitignored)
+quarto render docs/protocol/full-protocol.qmd
+quarto render docs/protocol/study-summary.qmd
+```
+
+```r
+# Round-trip PI edits from Word back into the protocol QMD (save the edited
+# .docx under docs/protocol/source/ first). Check the citekeys in the diff:
+# harvest re-derives keys from Zotero URIs, and items with missing/duplicate
+# URIs in field-codes.json come back with wrong keys.
+docstyle::docx_to_qmd("docs/protocol/source/<edited>.docx", "docs/protocol/full-protocol.qmd")
 ```
 
 ## Developer context
@@ -78,6 +91,7 @@ The pipeline follows the DemPoRT-V2-dev pattern (`~/github/DemPoRT-V2-dev`); its
 |----------|---------|
 | [docs/protocol/full-protocol.qmd](docs/protocol/full-protocol.qmd) | Prespecified study protocol |
 | [docs/protocol/study-summary.qmd](docs/protocol/study-summary.qmd) | One-page protocol summary |
+| [docs/protocol/_quarto.yml](docs/protocol/_quarto.yml) | Docstyle (Word) config for the protocol documents; `_extensions` symlink and local `_docstyle/` sidecar because the docstyle tooling resolves both relative to the project directory. Uses `popcorn-base.css` + `pop-draft-manuscript.css` at the repo root |
 | [docs/workflow/](docs/workflow/) | Step QMDs — one per pipeline stage (Stages 1–8) |
 | [manuscript/manuscript.qmd](manuscript/manuscript.qmd) | Study manuscript (all numbers inline R from pipeline) |
 | [docs/how-to/](docs/how-to/) | Task-oriented guides |
@@ -110,15 +124,17 @@ python3 ~/github/cchsflow-docs/mcp-server/cli.py compare cchs2013_2014_p cchs201
 
 The `variableStart` worksheet column uses cchsflow notation: `cchs2001_p::SMKA_01A, cchs2007_2008_p::SMK_01A, [SMK_01A]` — `_p` = PUMF, `_m` = Master, `[VAR]` = fallback name.
 
-**Unified variables (preferred):** `age_first_cigarette`, `age_start_smoking`, `time_quit_smoking`
+**Unified variables (preferred):** `age_first_cigarette` (entry), `smoked_100_lifetime` (established-smoker criterion), `time_quit_smoking_complete` (cessation exit, 2003+), `age_start_smoking` and `time_quit_smoking_daily` (daily-smoking attributes)
 
 **Master-only continuous:** `SMK_01C`, `SMK_040`, `SMK_09C` / `SMK_06C` / `SMK_10C`
 
 **PUMF pseudo-continuous (midpoint imputed):** `SMKG01C_cont`, `SMKG040_cont`, `SMK_09A_cont` / `SMK_06A_cont` / `SMK_10A_cont`
 
 **Deprecated aliases:** `SMK_005` → `SMK_202`; `SMK_030` → `SMK_05D`
 
-**APC model variables (internal):** `age`, `cohort`, `period`, `init`, `weighting`, `ont_id`
+**APC model variables (internal):** `age`, `cohort`, `period`, `event`, `weight`
+
+**Value codes are configuration, not code.** Status groupings (`survey.smoking_status.<src>.ever_codes/current_codes/former_codes/never_code`), sex codes (`survey.sex.<src>.men_code/women_code`), the established-smoker criterion (`survey.established_smoker.<src>.yes_code`), and analytic thresholds (`apc.cessation_durability_years`, `apc.initiation_floor_age`) live in `config.yml` and are read with `survey_code()` / `initiation_floor()`. **Variable ranges (min/max) are not in config at all:** `survey.<key>.<src>.range: variable_details` points at the variable-details worksheet, and `details_range()` / `survey_range()` derive the range from the recoding rules for each cycle. Example of why: the PUMF groups quit duration to "3 or more years" from 2003 (largest midpoint 5), where the old config comments claimed a top-code of 15. Do not write literal codes or thresholds into R. Remaining exception, scheduled as plan task 2.6: literal variable names in `R/imputation.R`.
 
 ### cchsflow dependency
 

R/config-utils.R

@@ -22,6 +22,59 @@ survey_var <- function(cfg, key) {
 
 # Access a bound (min/max) for the active data source.
 # e.g. survey_bound(cfg, "age_first_cigarette", "min") → 13 (pumf) or 8 (master)
+# Access a value code for the active data source, e.g.
+# survey_code(cfg, "established_smoker", "yes_code") -> 1
+# (keys are named *_code because bare YAML keys such as `yes` parse as booleans)
+survey_code <- function(cfg, key, code) {
+  entry <- cfg$survey[[key]]
+  if (is.null(entry)) stop("survey_code: unknown key '", key, "'")
+  src <- cfg$data_source %||% "pumf"
+  src_entry <- entry[[src]]
+  if (is.null(src_entry)) stop("survey_code: no '", src, "' entry for key '", key, "'")
+  val <- if (is.list(src_entry)) src_entry[[code]] else NULL
+  if (is.null(val)) stop("survey_code: no code '", code, "' for key '", key, "' source '", src, "'")
+  val
+}
+
+# Database name for a survey-cycle code (1-based position in cfg$cchs_cycles).
+cycle_database <- function(cfg, cycle_code) {
+  code <- suppressWarnings(as.integer(as.character(cycle_code)))
+  out <- rep(NA_character_, length(code))
+  ok <- !is.na(code) & code >= 1 & code <= length(cfg$cchs_cycles)
+  out[ok] <- unlist(cfg$cchs_cycles)[code[ok]]
+  out
+}
+
+# Valid range of a survey variable for one database. config.yml declares
+# `range: variable_details` and the range is read from the worksheet rules
+# (details_range()); a literal min/max in config is honoured only as a legacy
+# fallback. Returns c(min, max), NA where the rules give no bound.
+survey_range <- function(cfg, key, database, variable_details_sheet) {
+  entry <- cfg$survey[[key]]
+  if (is.null(entry)) stop("survey_range: unknown key '", key, "'")
+  src <- cfg$data_source %||% "pumf"
+  src_entry <- entry[[src]]
+  if (is.null(src_entry) || !is.list(src_entry)) stop("survey_range: no '", src, "' entry for key '", key, "'")
+  if (identical(src_entry$range, "variable_details")) {
+    return(details_range(src_entry$var, database, variable_details_sheet))
+  }
+  if (!is.null(src_entry$min) || !is.null(src_entry$max)) {
+    return(c(min = src_entry$min %||% NA_real_, max = src_entry$max %||% NA_real_))
+  }
+  c(min = NA_real_, max = NA_real_)
+}
+
+# The initiation floor is an analysis decision (public issue #5), not a variable
+# range, so it lives under apc: in config, per data source.
+initiation_floor <- function(cfg) {
+  src <- cfg$data_source %||% "pumf"
+  val <- cfg$apc$initiation_floor_age[[src]]
+  if (is.null(val)) stop("cfg$apc$initiation_floor_age has no entry for source '", src, "'")
+  val
+}
+
+# Literal bounds in config are legacy; variable ranges come from the worksheet
+# through survey_range().
 survey_bound <- function(cfg, key, bound) {
   entry <- cfg$survey[[key]]
   if (is.null(entry)) stop("survey_bound: unknown key '", key, "'")

R/validate-coverage.R

@@ -154,3 +154,79 @@ validate_cycle_coverage <- function(variables_sheet,
 
   invisible(list(declared = declared_gaps, critical = critical_gaps))
 }
+
+
+#' Check that every derived study variable has all of its feeders in the sheet
+#'
+#' cchsflow derives a variable only when every input listed in its
+#' `DerivedVar::[...]` rule is present. A missing feeder makes rec_with_table()
+#' skip the derived variable without an error, and the gap is noticed only when a
+#' later stage looks for the column (in the first CI run of task 1.3, after 56
+#' minutes). This check reads the derivation rules from the variable-details
+#' sheet and stops at Stage 1 if a feeder is absent from the variables sheet.
+#'
+#' @param variables_sheet Study variables worksheet (data frame)
+#' @param variable_details_sheet Combined variable-details worksheet (data frame)
+#' @return Invisibly, a data frame of (variable, missing_feeder) pairs; stops if
+#'   any row exists
+check_feeder_closure <- function(variables_sheet, variable_details_sheet, databases = NULL) {
+  # `databases`: restrict the check to these database names (normally the study cycles,
+  # unlist(cfg$cchs_cycles)); NULL checks every database a derived rule names.
+  study <- unique(variables_sheet$variable)
+  det <- variable_details_sheet[, c("variable", "variableStart", "databaseStart")]
+  split_dbs <- function(x) trimws(strsplit(as.character(x), ",")[[1]])
+  det_dbs <- lapply(det$databaseStart, split_dbs)
+  rules <- which(det$variable %in% study & grepl("DerivedVar::\\[", det$variableStart))
+  gaps <- list()
+  for (i in rules) {
+    inner <- sub(".*DerivedVar::\\[([^]]*)\\].*", "\\1", det$variableStart[i])
+    feeders <- trimws(strsplit(inner, ",")[[1]])
+    dbs_i <- det_dbs[[i]]
+    if (!is.null(databases)) dbs_i <- intersect(dbs_i, databases)
+    for (db in dbs_i) {
+      for (f in feeders) {
+        in_study <- f %in% study
+        # A feeder only closes the chain for this database if the details sheet has a rule
+        # for it in the same database; a rule in another cycle does not count.
+        has_rule <- any(det$variable == f & vapply(det_dbs, function(x) db %in% x, logical(1)))
+        if (!in_study || !has_rule) {
+          gaps[[length(gaps) + 1]] <- data.frame(
+            variable = det$variable[i], missing_feeder = f, database = db,
+            reason = if (!in_study) "not in cshm-variables.csv" else "no variable-details rule for this database",
+            stringsAsFactors = FALSE
+          )
+        }
+      }
+    }
+  }
+  gaps <- if (length(gaps)) {
+    unique(do.call(rbind, gaps))
+  } else {
+    data.frame(
+      variable = character(0), missing_feeder = character(0),
+      database = character(0), reason = character(0)
+    )
+  }
+  not_in_study <- gaps[gaps$reason == "not in cshm-variables.csv", , drop = FALSE]
+  no_rule <- gaps[gaps$reason != "not in cshm-variables.csv", , drop = FALSE]
+  if (nrow(not_in_study) > 0) {
+    # Fixable in this repo: add the feeder as an intermediate row.
+    stop(
+      "Derived study variables with feeders missing from worksheets/cshm-variables.csv: ",
+      paste(unique(paste0(not_in_study$variable, " needs ", not_in_study$missing_feeder)), collapse = "; "),
+      ". Add the feeders as intermediate rows; cchsflow skips the derivation silently otherwise."
+    )
+  }
+  if (nrow(no_rule) > 0) {
+    # The derived rule names a database in which its feeder has no rule (the feeder is not in
+    # that cycle's file). cchsflow returns NA for the derived variable there. This is recorded
+    # as a warning so the gap is visible in the pipeline log and in the coverage_check target.
+    warning(
+      "Derived study variables whose feeders have no variable-details rule in a database they are derived for ",
+      "(the derived variable will be missing for that cycle): ",
+      paste(unique(paste0(no_rule$variable, " needs ", no_rule$missing_feeder, " in ", no_rule$database)), collapse = "; "),
+      call. = FALSE
+    )
+  }
+  invisible(gaps)
+}

R/variable-details-sheet-utils.R

@@ -39,3 +39,57 @@ get_variable_type <- function(variable, variable_details_sheet) {
 get_variable_rows <- function(variable, variable_details_sheet) {
   variable_details_sheet[variable_details_sheet$variable == variable, ]
 }
+
+
+#' Valid range of a variable for one database, from the variable-details rules
+#'
+#' The variable-details worksheet is the reference for minimum and maximum
+#' values; config.yml points at it (`range: variable_details`) rather than
+#' holding copies. The range is read from the recoding rules for `database`:
+#' a `copy` rule with `recStart` of the form `[lo, hi]` contributes lo and hi;
+#' a category-to-value rule contributes its numeric `recEnd` (the midpoint);
+#' a derived variable (`DerivedVar::[...]` / `Func::`) takes the union of its
+#' feeders' ranges. `NA::` and `else` rules are ignored. Ranges differ by
+#' database because top-codes and category boundaries differ by cycle.
+#'
+#' @param variable Variable name in the details sheet
+#' @param database Database name, e.g. "cchs2013_2014_p"
+#' @param variable_details_sheet Combined variable-details data frame
+#' @return Named numeric vector c(min, max); NA when the rules give no bound
+details_range <- function(variable, database, variable_details_sheet, .seen = character()) {
+  none <- c(min = NA_real_, max = NA_real_)
+  rows <- variable_details_sheet[variable_details_sheet$variable == variable, , drop = FALSE]
+  if (nrow(rows) == 0) {
+    return(none)
+  }
+  in_db <- vapply(strsplit(as.character(rows$databaseStart), ","), function(x) database %in% trimws(x), logical(1))
+  rows <- rows[in_db, , drop = FALSE]
+  if (nrow(rows) == 0) {
+    return(none)
+  }
+  vals <- numeric(0)
+  for (i in seq_len(nrow(rows))) {
+    rec_end <- trimws(as.character(rows$recEnd[i]))
+    rec_start <- trimws(as.character(rows$recStart[i]))
+    var_start <- as.character(rows$variableStart[i])
+    if (grepl("^Func::", rec_end) || grepl("DerivedVar::", var_start)) {
+      inner <- sub(".*DerivedVar::\\[([^]]*)\\].*", "\\1", var_start)
+      feeders <- setdiff(trimws(strsplit(inner, ",")[[1]]), c(.seen, variable))
+      for (f in feeders) {
+        r <- details_range(f, database, variable_details_sheet, c(.seen, variable))
+        vals <- c(vals, r)
+      }
+    } else if (rec_end == "copy") {
+      m <- regmatches(rec_start, regexec("^\\[\\s*(-?[0-9.]+)\\s*,\\s*(-?[0-9.]+)\\s*\\]$", rec_start))[[1]]
+      if (length(m) == 3) vals <- c(vals, as.numeric(m[2:3]))
+    } else {
+      num <- suppressWarnings(as.numeric(rec_end))
+      if (!is.na(num)) vals <- c(vals, num)
+    }
+  }
+  vals <- vals[is.finite(vals)]
+  if (length(vals) == 0) {
+    return(none)
+  }
+  c(min = min(vals), max = max(vals))
+}

README.md

@@ -36,7 +36,7 @@ Period effects are held constant beyond the observed data range:
 
 ### Mortality adjustment
 
-Ever-smokers have lower survival to survey date than never-smokers. Survival bias is corrected using MPoRT weights adjusted for age, smoking status, years since quitting, immigration, and sex. A sensitivity analysis uses the Peto constant mortality risk ratio, consistent with the original Holford et al. (2014) US implementation.
+Ever-smokers have lower survival to survey date than never-smokers, so smoking histories reconstructed from survivors under-represent smokers who died. The protocol specifies an MPoRT-based correction as the primary method and a Peto constant risk ratio as a sensitivity analysis (protocol section 3.4.5). **Neither is implemented yet.** The pipeline runs with `mortality_method: "none"`, and every current output is an estimate among respondents who survived to be surveyed, not a birth-cohort smoking history. A configured correction that leaves the weights unchanged stops the pipeline.
 
 ### Smoking status definitions
 
@@ -65,7 +65,8 @@ Variables are harmonized across CCHS cycles using the [cchsflow](https://github.
 
 - `age_first_cigarette` — age first smoked whole cigarette (Master: exact; PUMF: midpoint estimate)
 - `age_start_smoking` — age started smoking daily (Master: exact; PUMF: midpoint ±3 years)
-- `time_quit_smoking` — years since quit smoking
+- `time_quit_smoking_complete` — years since stopped smoking completely (the cessation event; 2003 onward)
+- `smoked_100_lifetime` — smoked 100 or more cigarettes (defines who is a smoker in the model)
 
 ## Pipeline
 

_targets.R

@@ -30,10 +30,11 @@ list(
 
   # Stage 0: Pre-flight validation — verify variable coverage before loading data
   # Returns gap report (declared + critical); warns or errors per cfg$strict_validation
-  tar_target(coverage_check,
+  tar_target(coverage_check, {
+    check_feeder_closure(variables_sheet, variable_details_sheet, databases = unlist(cfg$cchs_cycles))
     validate_cycle_coverage(variables_sheet, variable_details_sheet, cfg,
                             strict = cfg$strict_validation %||% FALSE)
-  ),
+  }),
 
   # Stage 2: Load and harmonize CCHS cycles
   # Combined harmonized cycles, study variables only (stored in the _targets/ store)
@@ -85,7 +86,7 @@ list(
   # Fitted on imputation 1 (protocol Appendix D documents the upgrade path
   # to per-imputation fits pooled by Rubin's rules).
   tar_target(apc_data,
-    prepare_apc_data(analysis_data$datasets[[1]], cfg)
+    prepare_apc_data(analysis_data$datasets[[1]], cfg, variable_details_sheet)
   ),
 
   # Stage 8: Fit APC models — four independent targets for parallel execution