Architecture deepening: six structural refactors (ADR-0011 → 0014)

.claude/skills/generate-database-options/SKILL.md

@@ -129,7 +129,7 @@ Use this table to guide refinements and answer user questions.
 ### Known limitations (tell the user if relevant)
 
 - ℹ️ **`global.loop`** (cross-datasource phase loops): **supported** — see
-  `wrapper.py::_resolve_signal_references`. Two signals per loop, each resolved via the
+  `plot_assembly.py::_resolve_signal_references`. Two signals per loop, each resolved via the
   3-mode chain (qualified `datasource::raw_name` → display name → raw name fallback).
   See `docs/user_guide/tutorial.md` → *Global Loops vs. Per-Source Loops*.
 - ℹ️ **Qualified signal references** (`"datasource::signal_name"`): supported in both

.claude/skills/new-datasource/SKILL.md

@@ -84,7 +84,7 @@ Create three files under `src/clinical_scope/datasource/sources/<datasource_name
 
 Critical contracts:
 - `_load` returns a `pd.DataFrame` with a sorted, deduplicated `DatetimeIndex` and numeric signal columns.
-- `_load` signature: `(cls, file_path: Path, path_output, **kwargs)` when `MULTI_FILE=False`, `(cls, file_path_list: list[Path], path_output, **kwargs)` when `MULTI_FILE=True`. The base class dispatches based on the option.
+- `_load` signature: `(cls, file_path: Path)` when `MULTI_FILE=False`, `(cls, file_path_list: list[Path])` when `MULTI_FILE=True`. The base class dispatches based on the option, and writes the parquet cache itself from the frame you return — `_load` must not save anything.
 - Empty data: return `pd.DataFrame(index=pd.DatetimeIndex([], name=cst.DATETIME_INDEX_NAME))` — never plain `pd.DataFrame()`, and never `tz=…`: `_load` output is the parquet cache, so it stays naive and `_format` localizes it ([ADR-0010](../../../docs/adr/0010-load-transcribes-format-interprets.md)).
 - No module-level `main()` is needed. The `@add_main_module(<module>)` decorator in `registry.py` finds the `DataSourceBase` subclass inside your module and binds its inherited `main` classmethod — your module only has to define the class.
 - Decorate `_load` with `@time_it` from `clinical_scope.datasource.timing`.

CHANGELOG.md

@@ -7,6 +7,12 @@ All notable changes to this project will be documented in this file.
 ## [Unreleased]
 
 ### Fixed
+- **A group in one data source no longer hides a same-named signal in another.** Grouping matched signals by name across the whole run, but a name like `HR`, `SpO2` or `ABP` is only unique *within* one source. So grouping `HR` under your monitor could silently drop the ventilator's `HR` from the page — no error, just a missing plot, and which one disappeared depended on the order the sources happened to load in. Groups now take the signals they actually name, and nothing else.
+
+- **A group that finds only one of its signals now keeps the group's name.** A group configured over four pressures but resolving to one used to be titled after that surviving signal — so the same configuration gave a differently-named panel depending on how much data a recording happened to contain. It is now always titled after the group.
+
+- **`loop`, `spectrogram` and `psd` written inside a data source's own section accept the same signal references as `global` ones.** They matched raw column names only; a display name (the `label` you configured) silently matched nothing. They now resolve display names too. A loop given other than exactly two signals is reported as a skipped plot instead of an unexplained error in the log.
+
 - **`trace_options` now applies to every datasource, not only `other::<stem>` files.** The block was accepted and validated in any `database_options` section, and the Excel sentinel row wrote it for any datasource, but only `other` ever read it — anywhere else it validated cleanly and did nothing. It now works everywhere it was already accepted.
 
   **What changes for you:** a configuration that already sets `trace_options` (or the Excel `trace_mode` / `line_width` / `opacity` / `marker_symbol` columns) on a device datasource starts taking effect, where before it was ignored. Where a datasource ships its own trace style, your block now wins key by key over it; keys you leave unset keep the shipped value. Nothing changes for a configuration that only styled `other::<stem>` files.
@@ -15,6 +21,10 @@ All notable changes to this project will be documented in this file.
 
   **What changes for you:** a code pasted without its leading `#` is accepted, a malformed one is flagged as you leave the field, and a colour that is not a valid six-digit hex now falls back to the default instead of being written into the annotation file as-is.
 
+- **A hand-edited `~/.clinical_scope/user_options.json` is now checked when it loads.** Settings were only validated as you typed them into the Settings modal, so a file edited by hand — or one holding a value from an older version — could carry a subplot height of `99999`, a palette that no longer exists, or a misspelled timezone, and reach the app unchecked. Each such value now falls back to its default and says which one it was in the log, and a setting stored under a name the app no longer knows is reported rather than dropped in silence.
+
+  **What changes for you:** the Settings modal also refuses a spectrogram colour range whose minimum is not below its maximum — both bounds snap back to their defaults as you save, instead of the pair being stored and quietly corrected at plot time.
+
 ---
 
 ## [1.1.0] — 2026-08-24

CLAUDE.md

@@ -16,8 +16,10 @@ CLI scripts (extract / inspect / visualize) and the Python API are documented in
 ```
 src/clinical_scope/
   wrapper.py            main pipeline — visualize / extract / inspect
+  plot_assembly.py      Signals + database_options → PlotGroups (grouping, derived plots)
   signal_container.py   Signal / PlotGroup / PlotModel data models
   constants.py          global constants + option schema classes
+  user_options.py       UserOptions schema as data: traversal, defaults, validate()
   datasource/
     base.py             DataSourceBase — find/load/format/extract/inspect template
     registry.py         registered sources (DataSource.AVAILABLE; keep Other last)
@@ -41,7 +43,9 @@ src/clinical_scope/
 - **Extract** (`wrapper.extract_patient` / `batch_extract` / `extract_datasource`, also `from clinical_scope import extract_datasource, extract_patient, batch_extract`) — stop at `format`, return DataFrame(s). `save_path`/`save_folder` write explicit output, independent of the per-patient `clinical_scope_output/` parquet cache (always written; reused when `quick_load` is set).
 - **Inspect** (`wrapper.inspect`) — stop at `format`, return `list[DataSourceInspection]` (columns, point counts, time ranges). `OtherDataSource.inspect()` returns **one entry per file** (`other::<stem>`); the wrapper handles single-or-list returns.
 
-**Signal references** in `grouped_fields` and `global.loop` resolve via a 3-mode lookup in `_resolve_signal_references`: qualified `datasource::raw_name` → display name → raw-name fallback.
+**Signal references** in `grouped_fields`, `loop`, `spectrogram` and `psd` resolve via a 3-mode lookup in `plot_assembly._resolve_signal_references`: qualified `datasource::raw_name` → display name → raw-name fallback.
+
+**Config scope is desugared once, and grouping joins on signal identity** ([ADR-0013](docs/adr/0013-signal-references-are-qualified-before-assembly.md)). `assemble_plot_groups` is called once, after the datasource loop, and its first step rewrites every per-datasource reference as a qualified global one — downstream, local scope does not exist. Both config spellings stay valid; the desugaring is a property of the code, not of the file format. A signal is left out of the default one-plot-per-signal pass only when a group took *that object*, never when something merely shares its `raw_name` (unique only within a datasource).
 
 `wrapper.main`/`inspect` call an optional `progress_callback(current, total, name)` between datasources, which drives the UI progress bar.
 
@@ -53,14 +57,16 @@ Registered in `datasource/registry.py` (`DataSource.AVAILABLE`); the canonical l
 
 **`_load` transcribes; `_format` interprets** ([ADR-0010](docs/adr/0010-load-transcribes-format-interprets.md)). `_load`'s output *is* the parquet cache, so it must be reproducible from the source file alone — no option resolved inside it. Mechanically: no `DATA_SOURCE_DEFAULT_TIMEZONE`, no `apply_timezone_to_dataframe` in any `_load`.
 
+**Datetime bounds are qualified at the boundary** ([ADR-0011](docs/adr/0011-datetime-bounds-are-qualified-at-the-boundary.md)). The UI turns naive form text into a tz-aware instant at Submit, using the user's `display_timezone` — *that* is what makes the Settings timezone govern the time window. The load path only ever localizes a bound that is still naive (script or hand-edited file), and does so with `cst.NAIVE_BOUND_TZ`, never a user option, so `extract_*` output does not depend on who is at the keyboard. `cst.NAIVE_BOUND_TZ` and `cst.DISPLAY_TIMEZONE` are separate literals on purpose; do not alias them.
+
 **Adding one**: use the `/new-datasource` skill — it is authoritative for the module layout, `options.py` constants, the loader, registration (Other stays last), example data, tests, snapshots, and the tutorial table.
 
 ## Config files
 
 Field-by-field reference is in the [tutorial](docs/user_guide/tutorial.md). The three tiers:
 - **`database_options`** (`.json` or `.xlsx`) — per-source signal config: `field_display`, `signals` (labels/units/colors), `grouped_fields`, `loop`; plus `global.grouped_fields`. Uploading one in the UI caches it to `~/.clinical_scope/last_database_options.json` (signal metadata only, no PHI).
 - **`patient_options`** (`.json`) — per-run settings: `data_folder`, `datetime_start`/`datetime_end`, `quick_load`, and per-source options (`time_shift`, `day`, …).
-- **`user_options`** (`~/.clinical_scope/user_options.json`) — the third tier: per-person app behaviour + display fallbacks, edited only in the Settings modal. **Never overrides `database_options`** ([ADR-0005](docs/adr/0005-user-options-are-fallbacks.md)). A new display setting = a `UserOptions` schema class (with `SECTION`) + a field on `DisplayFallbacks` (`signal_container.py`) + one read site; the carrier is threaded from `wrapper.main` down to both `Signal` and `PlotModel` construction, so no signature grows.
+- **`user_options`** (`~/.clinical_scope/user_options.json`) — the third tier: per-person app behaviour + display fallbacks, edited only in the Settings modal. **Never overrides `database_options`** ([ADR-0005](docs/adr/0005-user-options-are-fallbacks.md)). A new display setting = a `UserOptions` schema class (with `SECTION`) + a field on `DisplayFallbacks` (`signal_container.py`) + one read site; the carrier is threaded from `wrapper.main` down to both `Signal` and `PlotModel` construction, so no signature grows. Values are held to the schema by `user_options.validate()` at every boundary that accepts one, and only `dash_api` may read the file ([ADR-0014](docs/adr/0014-user-options-are-validated-at-the-boundary.md)).
 
 Reference configs: `example/demo_database/database_options.{xlsx,json}` — the canonical example, in both formats, runnable against `demo_patient/` and covering **every** datasource it ships. **The `.json` is generated from the `.xlsx`**; edit the spreadsheet and regenerate. `tests/unit/test_example_assets.py` enforces both the coverage and the parity, and prints the regeneration one-liner. `example/option_files/patient_options_example.json` covers the other tier, for library users who never launch the app and so never get an app-written one.
 

docs/adr/0004-validate-datetime-column-candidates.md

@@ -46,3 +46,7 @@ Migrated to the shared detector: `load_csv_with_datetime_index`, the new `load_p
 - **Harder / accepted trade-offs:** raising the parse threshold from `other`'s previous 50% to 90% means some previously-tolerated messy files (mostly-garbage timestamp column) now fail outright instead of loading with a partially-broken time axis — intentional, since that was never actually useful data. Per-datasource candidate-name priority is gone; if a real device's column repeatedly loses to a wrong candidate under the universal list, revisit.
 - **Explicitly deferred:** combining separate date + time columns into one datetime (no current datasource needs it). Epoch detection beyond nanoseconds (s/ms/µs) — revisit if a real datasource surfaces raw second/millisecond epoch timestamps.
 - **Not fully resolved, by design:** a file with several genuinely-plausible time columns (like the anesthesia record above) may land on any of the ones that survive validation + the uniqueness/`utc` tiebreak, not necessarily the single "best" one a human would pick. Per [0001](0001-diagnose-dont-resolve-patient-folders.md)'s precedent, deep disambiguation of a badly-overloaded file is left to the user (e.g. pre-pruning columns before use), not solved inside the detector.
+
+## Update — 2026-08-26
+
+The detector now has a file of its own, `io/time_axis.py`, holding this rule and nothing else. It exposes two adapters over the same tiers — `detect_time_axis_in_frame` for a loaded frame, `detect_time_axis_in_parquet` for a file read only by schema and bounded sample. They must pick the same column, so they stay in one module; `_is_numeric_pa_type`'s agreement with `pd.api.types.is_numeric_dtype` is the tripwire for that.

docs/adr/0009-other-stem-is-a-config-scope.md

@@ -26,11 +26,11 @@ So `other::waves` names a configuration scope — one file — carrying its own
 
 The asymmetry with other datasources is deliberate and is the point of recording this: **`other/` is the only source whose files are unrelated by construction.** Everywhere else, a folder is one recording. A contributor who reaches for `edf::<stem>` by analogy is generalising from the exception.
 
-Signal references resolve in three modes, in order — qualified `datasource::raw_name`, then display name, then bare raw name (`_resolve_signal_references`, `wrapper.py`).
+Signal references resolve in three modes, in order — qualified `datasource::raw_name`, then display name, then bare raw name (`_resolve_signal_references`, `plot_assembly.py`).
 
 ## Consequences
 
 - **Easier:** an `other/` folder can hold files from unrelated machines, each configured independently, without any of them earning a module. Cross-source `grouped_fields` and `loop` entries can address a specific file's column unambiguously.
-- **Harder / accepted trade-offs:** a file in `other/` named after a registered datasource makes a reference genuinely readable two ways — `other/servo_u.parquet` gives a scope whose qualified names collide with the real `servo_u` source. Both readings are legitimate, so this cannot be resolved by rule alone: the precedence order above decides, and `_warn_if_also_a_raw_name` (`wrapper.py:46-60`) logs the collision naming the losing signal and the spelling that reaches it. Silent shadowing was the alternative and was rejected.
+- **Harder / accepted trade-offs:** a file in `other/` named after a registered datasource makes a reference genuinely readable two ways — `other/servo_u.parquet` gives a scope whose qualified names collide with the real `servo_u` source. Both readings are legitimate, so this cannot be resolved by rule alone: the precedence order above decides, and `_warn_if_also_a_raw_name` (`plot_assembly.py`) logs the collision naming the losing signal and the spelling that reaches it. Silent shadowing was the alternative and was rejected.
 - **Also:** signals inside `other/` are named `<stem>::<column>` rather than bare column names, so configurations written against the old single-block form need their references rewritten. This is part of the [ADR-0008](0008-datasource-modules-need-format-specific-parsing.md) migration.
 - **Revisit if:** a second datasource appears whose folder genuinely holds unrelated recordings rather than chunks of one. At that point the scope mechanism generalises — but it should generalise to *that* source explicitly, not to all of them by default.

docs/adr/0010-load-transcribes-format-interprets.md

@@ -66,3 +66,17 @@ The rule holds unchanged. Two of the Consequences above have since been acted on
 **Deriving a column is not resolving an option.** EIT's `%Local N = Local N / Global` moved from `_format` into `_load`, so the percentages land in the cache and pruning can select them — a `field_display` naming `%Local 1*` previously matched nothing, the column not existing on disk. This is not a loosening. The test is whether the value could differ between two runs over the same source file: a ratio of two parsed columns could not, so it is transcription, the same category as `time_hours`, which `_parse_asc_table` has always derived inside `_load`. A timezone, a `day`, a `field_display` all could, and stay barred. `_format` no longer calls the helper at all: a cache written before this change has no `%` columns, and pruning would then select nothing for a `%Local N*` pattern — a back-fill there would work only when `Global` happened to be selected alongside. Rather than carry a guarantee that holds by luck, caches are treated as disposable: after an application update, un-tick *Re-use data if already loaded once* once, and the next run writes a complete cache.
 
 **The demo cache trades size for read width.** It grows 0.881 → 1.341 MB (four derived float columns, which compress poorly), while a configured read of it drops from all 71 columns to 9 of 75. The cost is paid once on the fresh load; the saving on every `quick_load` run after it.
+
+## Update — 2026-08-25
+
+The rule is now carried by the signature. `_load(file_path)` takes the file and nothing else: no `path_output`, no `database_options_specific`, no `**kwargs`. The parquet write moved into `DataSourceBase._load_raw_dataframe`, which saves whatever frame `_load` hands back, so a source cannot resolve an option inside `_load` for the simple reason that no option is in scope there. The two greppable rules still stand alongside a third — no reference to `DATA_SOURCE_DEFAULT_TIMEZONE`, no call to `apply_timezone_to_dataframe`, and now no configuration argument at all — because a module-level global is the one channel a signature cannot close.
+
+`tests/datasource/test_load_config_independence.py` changed kind rather than level. The rule used to be a behavioural property — call `_load` twice with two configs, assert the frames match — and it is now a property of the code's shape, which no amount of running it can observe: both runs take an identical path, and a module-level constant does not vary between them. Comparing the two written caches instead would assert something that cannot fail. So the file now reads each `_load` definition through the AST and makes two static checks, one per channel: that the signature is `(cls, file_path)` with no `*args`, `**kwargs` or keyword-only argument, and that the body references neither forbidden name. The second is this ADR's own greppable clause, executed rather than left to review — which is where it had always been. Reading the AST rather than the bound attribute is deliberate: `@time_it` is not `functools.wraps`ed, so introspecting `cls._load` describes the decorator's wrapper.
+
+The `configured_field_display` guard the Consequences flagged for removal is gone — the *guard*, not the parameter. The fresh-load branch that restored `field_display` for non-caching sources could never fire under this rule and has been deleted; the parameter survives on the quick-load branch, where it still prunes the cache read that serves `inspect(configured_columns_only=True)`.
+
+One small behaviour note: the base declines to write a cache for an empty frame, and does not create the output folder for one either. Four sources' early returns already skipped the save for that reason; the rule is now uniform and stated once.
+
+## Update — 2026-08-26
+
+The cache's provenance is now expressed by which function a caller reaches for. `io/file_utils.py` split along its concerns — `time_axis`, `parquet_pruning`, `discovery`, `column_patterns`, `export` — and the `index_is_time_axis` flag that carried this ADR's guarantee became a second front door: `read_cache_pruned` for a file we wrote, `read_parquet_pruned` for one we did not. "No other caller may claim that" was a comment at the call site; it is now the absence of any way to say it. The path cited in the Consequences above, `io/file_utils.py:568-571`, is today `_pruning_plan` in `io/parquet_pruning.py`.