From 56197fceefbfd89486b08027983e056c9e3512d6 Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Tue, 25 Aug 2026 16:39:29 +0200 Subject: [PATCH 1/9] Move parquet cache persistence into the datasource template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_load` now takes only the file path and returns a frame; DataSourceBase writes the cache from it, at one site instead of in eight sources by convention. This is the structural form of ADR-0010 — a source cannot resolve an option inside `_load` because no option is in scope there. Nine `**kwargs` + `noqa: ARG003` pairs go with it. The base declines to cache an empty frame, and does not create the output folder for one either. Four sources' early returns already skipped the save for that reason; the rule is now uniform and stated once. Removes the `configured_field_display` guard ADR-0010 flagged for removal: its fresh-load branch could never fire under the rule. The parameter survives on the quick-load branch, where it still prunes the cache read serving inspect(configured_columns_only=True). test_load_config_independence changes kind rather than level. Config independence is no longer a behavioural property — both runs of a load now take an identical path, and a module constant does not vary between them, so comparing two runs would assert something that cannot fail. It reads each `_load` through the AST instead and checks the rule's two channels: the signature takes only the file path, and the body references neither DATA_SOURCE_DEFAULT_TIMEZONE nor apply_timezone_to_dataframe. Snapshots unchanged: every source saved exactly the frame it returned. Co-Authored-By: Claude Opus 5 --- .claude/skills/new-datasource/SKILL.md | 2 +- ...0010-load-transcribes-format-interprets.md | 10 ++ src/clinical_scope/datasource/base.py | 53 ++++------- .../sources/edf/find_load_format.py | 14 +-- .../sources/eit/find_load_format.py | 7 +- .../fluxmed_parameters/find_load_format.py | 7 +- .../fluxmed_signals/find_load_format.py | 7 +- .../find_load_format.py | 9 +- .../mindray_respi_waves/find_load_format.py | 9 +- .../sources/mindray_scope/find_load_format.py | 9 +- .../sources/other/find_load_format.py | 4 +- .../sources/servo_u/find_load_format.py | 13 +-- tests/datasource/test_column_pruning.py | 21 +--- tests/datasource/test_edf.py | 10 +- tests/datasource/test_eit.py | 2 +- tests/datasource/test_fluxmed_parameters.py | 2 +- tests/datasource/test_fluxmed_signals.py | 4 +- .../test_load_config_independence.py | 95 ++++++++++++------- .../datasource/test_mindray_respi_numerics.py | 4 +- tests/datasource/test_mindray_respi_waves.py | 4 +- tests/datasource/test_mindray_scope.py | 4 +- tests/datasource/test_servo_u.py | 2 +- tests/unit/test_datasource_base_cache.py | 40 ++++++-- 23 files changed, 160 insertions(+), 172 deletions(-) diff --git a/.claude/skills/new-datasource/SKILL.md b/.claude/skills/new-datasource/SKILL.md index 5281e86..1c2b91b 100644 --- a/.claude/skills/new-datasource/SKILL.md +++ b/.claude/skills/new-datasource/SKILL.md @@ -84,7 +84,7 @@ Create three files under `src/clinical_scope/datasource/sources/)` decorator in `registry.py` finds the `DataSourceBase` subclass inside your module and binds its inherited `main` classmethod — your module only has to define the class. - Decorate `_load` with `@time_it` from `clinical_scope.datasource.timing`. diff --git a/docs/adr/0010-load-transcribes-format-interprets.md b/docs/adr/0010-load-transcribes-format-interprets.md index b52e671..05840fc 100644 --- a/docs/adr/0010-load-transcribes-format-interprets.md +++ b/docs/adr/0010-load-transcribes-format-interprets.md @@ -66,3 +66,13 @@ The rule holds unchanged. Two of the Consequences above have since been acted on **Deriving a column is not resolving an option.** EIT's `%Local N = Local N / Global` moved from `_format` into `_load`, so the percentages land in the cache and pruning can select them — a `field_display` naming `%Local 1*` previously matched nothing, the column not existing on disk. This is not a loosening. The test is whether the value could differ between two runs over the same source file: a ratio of two parsed columns could not, so it is transcription, the same category as `time_hours`, which `_parse_asc_table` has always derived inside `_load`. A timezone, a `day`, a `field_display` all could, and stay barred. `_format` no longer calls the helper at all: a cache written before this change has no `%` columns, and pruning would then select nothing for a `%Local N*` pattern — a back-fill there would work only when `Global` happened to be selected alongside. Rather than carry a guarantee that holds by luck, caches are treated as disposable: after an application update, un-tick *Re-use data if already loaded once* once, and the next run writes a complete cache. **The demo cache trades size for read width.** It grows 0.881 → 1.341 MB (four derived float columns, which compress poorly), while a configured read of it drops from all 71 columns to 9 of 75. The cost is paid once on the fresh load; the saving on every `quick_load` run after it. + +## Update — 2026-08-25 + +The rule is now carried by the signature. `_load(file_path)` takes the file and nothing else: no `path_output`, no `database_options_specific`, no `**kwargs`. The parquet write moved into `DataSourceBase._load_raw_dataframe`, which saves whatever frame `_load` hands back, so a source cannot resolve an option inside `_load` for the simple reason that no option is in scope there. The two greppable rules still stand alongside a third — no reference to `DATA_SOURCE_DEFAULT_TIMEZONE`, no call to `apply_timezone_to_dataframe`, and now no configuration argument at all — because a module-level global is the one channel a signature cannot close. + +`tests/datasource/test_load_config_independence.py` changed kind rather than level. The rule used to be a behavioural property — call `_load` twice with two configs, assert the frames match — and it is now a property of the code's shape, which no amount of running it can observe: both runs take an identical path, and a module-level constant does not vary between them. Comparing the two written caches instead would assert something that cannot fail. So the file now reads each `_load` definition through the AST and makes two static checks, one per channel: that the signature is `(cls, file_path)` with no `*args`, `**kwargs` or keyword-only argument, and that the body references neither forbidden name. The second is this ADR's own greppable clause, executed rather than left to review — which is where it had always been. Reading the AST rather than the bound attribute is deliberate: `@time_it` is not `functools.wraps`ed, so introspecting `cls._load` describes the decorator's wrapper. + +The `configured_field_display` guard the Consequences flagged for removal is gone — the *guard*, not the parameter. The fresh-load branch that restored `field_display` for non-caching sources could never fire under this rule and has been deleted; the parameter survives on the quick-load branch, where it still prunes the cache read that serves `inspect(configured_columns_only=True)`. + +One small behaviour note: the base declines to write a cache for an empty frame, and does not create the output folder for one either. Four sources' early returns already skipped the save for that reason; the rule is now uniform and stated once. diff --git a/src/clinical_scope/datasource/base.py b/src/clinical_scope/datasource/base.py index 4761978..e49178e 100644 --- a/src/clinical_scope/datasource/base.py +++ b/src/clinical_scope/datasource/base.py @@ -113,15 +113,13 @@ def _find(cls, folder_path: Path) -> list[Path] | Path | None: @classmethod @abstractmethod - def _load( - cls, file_path: Path | list[Path], path_output: Path | None, **kwargs - ) -> pd.DataFrame: + def _load(cls, file_path: Path | list[Path]) -> pd.DataFrame: """ - Load and parse raw data file(s) into a DataFrame. + Transcribe the raw data file(s) into a datetime-indexed DataFrame, and nothing more. - Args: - path_output: Path to save loaded DataFrame for quick loading, or none if no saving - needed + The base class writes the returned frame to the parquet cache, so per ADR-0010 no + option may be resolved here — a signature taking only the file path enforces it. + Interpretation (timezone, time shift, windowing) belongs in ``_format``. Returns: Loaded data, indexed by datetime. @@ -281,20 +279,15 @@ def _load_raw_dataframe( *apply_datetime_pushdown* set to ``False`` bypasses parquet row-pushdown (used by ``inspect()``, which needs the full raw file for date-range stats). - *configured_field_display* restores ``field_display`` into *database_options* for - reading — but only on branches where doing so is provably safe (a cache read, or a - fresh load this datasource never caches). It exists for - ``inspect(configured_columns_only=True)``: a narrowed ``field_display`` in front of a - fresh, cache-writing ``_load()`` must never reach it, since the cache it writes is - read back by every later run. ADR-0010 now forbids any ``_load()`` from resolving - ``field_display`` at all, so this is defence in depth rather than the only guard. + *configured_field_display* applies to the quick-load branch only: it restores a + narrowed ``field_display`` into *database_options* for the cache read that serves + ``inspect(configured_columns_only=True)``. Returns: (df, file_path_str, columns_pruned) on success, (None, None, False) if the file - was not found. ``columns_pruned`` reflects whether ``field_display`` actually - reached the read that ran — never inferred from the resulting frame's columns, - which can't tell "restricted at read time" apart from "the file only ever had - these columns" (a data-dependent coincidence, not pruning). + was not found. ``columns_pruned`` says whether the read that ran was structurally + restricted to ``field_display``, which only a cache read can be — a fresh + ``_load()`` takes no configuration and always parses every column. Raises exceptions for actual load errors. """ @@ -333,31 +326,19 @@ def _load_raw_dataframe( file_path_str = str(file_path[0]) if isinstance(file_path, list) else str(file_path) logger.info("🔍 [%s] Loading fresh data from: %s", cls.DATASOURCE_NAME, search_folder) - # write_cache=False means this _load() output is never cached, so honoring - # field_display here can't narrow a future full read. - # Sources that opt out of caching (ALLOW_QUICK_LOAD=False, e.g. other) are - # otherwise always on this branch and could never benefit from configured_columns_only. - load_column_options = database_options - if configured_field_display is not None and not write_cache: - load_column_options = { - **database_options, - cst.DatabaseOptions.FIELD_DISPLAY: configured_field_display, - } - df = cls._load( - file_path, - dataframe_path if write_cache else None, - database_options_specific=load_column_options, - patient_options=patient_options if apply_datetime_pushdown else None, - ) + df = cls._load(file_path) logger.info( "📥 [%s] Loaded: %d rows x %d columns.", cls.DATASOURCE_NAME, df.shape[0], df.shape[1], ) + # An empty frame would cache a "no data" state every later quick_load run reads back. + if write_cache and not df.empty: + cls._save_dataframe(df, dataframe_path) if not write_cache and cls.CREATE_SOURCE_SYMLINK: cls._create_source_symlink(file_path, dataframe_path.parent) - columns_pruned = bool(load_column_options.get(cst.DatabaseOptions.FIELD_DISPLAY)) + columns_pruned = False return df, file_path_str, columns_pruned @classmethod @@ -612,7 +593,7 @@ def extract( inspection metadata. Parquet caching inside ``clinical_scope_output/`` is always created automatically by - ``_load()`` inside ``_load_raw_dataframe()``. + ``_load_raw_dataframe()``. Args: patient_options: Patient-specific options (same as :meth:`main`). diff --git a/src/clinical_scope/datasource/sources/edf/find_load_format.py b/src/clinical_scope/datasource/sources/edf/find_load_format.py index 85d7964..5bc6062 100644 --- a/src/clinical_scope/datasource/sources/edf/find_load_format.py +++ b/src/clinical_scope/datasource/sources/edf/find_load_format.py @@ -1,6 +1,5 @@ import logging from pathlib import Path -from typing import Any import numpy as np import pandas as pd @@ -90,22 +89,13 @@ class EDFDataSource(DataSourceBase): @classmethod @time_it - def _load( - cls, - file_path_list: list[Path], - path_output: Path | None, - **kwargs: Any, # noqa: ARG003 - ) -> pd.DataFrame: + def _load(cls, file_path_list: list[Path]) -> pd.DataFrame: frames = [read_edf_file(file_path) for file_path in file_path_list] if not frames: return pd.DataFrame(index=pd.DatetimeIndex([], name=cst.DATETIME_INDEX_NAME)) df = frames[0] if len(frames) == 1 else pd.concat(frames) - df = deduplicate_then_sort_index(df) - - if path_output is not None: - cls._save_dataframe(df, path_output) - return df + return deduplicate_then_sort_index(df) @classmethod def _anchor_undated_recording(cls, df: pd.DataFrame, patient_options: dict) -> pd.DataFrame: diff --git a/src/clinical_scope/datasource/sources/eit/find_load_format.py b/src/clinical_scope/datasource/sources/eit/find_load_format.py index b2365e6..9a90014 100644 --- a/src/clinical_scope/datasource/sources/eit/find_load_format.py +++ b/src/clinical_scope/datasource/sources/eit/find_load_format.py @@ -279,7 +279,7 @@ class EITDataSource(DataSourceBase): @classmethod @time_it - def _load(cls, file_path_list: list[Path], path_output: Path | None, **kwargs) -> pd.DataFrame: # noqa: ARG003 + def _load(cls, file_path_list: list[Path]) -> pd.DataFrame: ( _list_metadata, _list_dynamic_images, @@ -289,10 +289,7 @@ def _load(cls, file_path_list: list[Path], path_output: Path | None, **kwargs) - ) = _parse_eit_asc_file_list(file_path_list) df = deduplicate_then_sort_index(df) - df = _add_columns_percentage_for_eit(df) - if path_output is not None: - cls._save_dataframe(df, path_output) - return df + return _add_columns_percentage_for_eit(df) @classmethod @time_it diff --git a/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py b/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py index 5319413..4bf4a6a 100644 --- a/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py +++ b/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py @@ -28,7 +28,7 @@ class FluxmedParametersDataSource(DataSourceBase): @classmethod @time_it - def _load(cls, file_path: Path, path_output: Path | None, **kwargs) -> pd.DataFrame: # noqa: ARG003 + def _load(cls, file_path: Path) -> pd.DataFrame: if file_path.suffix.lower() == ".parquet": df = load_parquet_with_datetime_index(file_path) elif file_path.suffix.lower() in [".txt", ".csv"]: @@ -107,7 +107,4 @@ def make_unique(columns: list[str]) -> list[str]: ) raise NotImplementedError(msg) - df = df[~df.index.duplicated(keep="first")] - if path_output is not None: - cls._save_dataframe(df, path_output) - return df + return df[~df.index.duplicated(keep="first")] diff --git a/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py b/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py index f0df36b..0ceb966 100644 --- a/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py +++ b/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py @@ -31,7 +31,7 @@ class FluxmedSignalsDataSource(DataSourceBase): @classmethod @time_it - def _load(cls, file_path: Path, path_output: Path | None, **kwargs) -> pd.DataFrame: # noqa: ARG003 + def _load(cls, file_path: Path) -> pd.DataFrame: if file_path.suffix.lower() == ".parquet": df = load_parquet_with_datetime_index(file_path) elif file_path.suffix.lower() in [".txt", ".csv"]: @@ -101,7 +101,4 @@ def _load(cls, file_path: Path, path_output: Path | None, **kwargs) -> pd.DataFr ) raise NotImplementedError(msg) - df = deduplicate_then_sort_index(df) - if path_output is not None: - cls._save_dataframe(df, path_output) - return df + return deduplicate_then_sort_index(df) diff --git a/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py b/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py index 833b7ff..845a74d 100644 --- a/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py +++ b/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py @@ -1,6 +1,5 @@ import logging from pathlib import Path -from typing import Any import pandas as pd @@ -20,7 +19,7 @@ class MindRayRespiNumericsDataSource(DataSourceBase): @classmethod @time_it - def _load(cls, file_path: Path, path_output: Path | None, **kwargs: Any) -> pd.DataFrame: # noqa: ARG003 + def _load(cls, file_path: Path) -> pd.DataFrame: """ Load and parse MindRay Respi Numerics data. @@ -55,8 +54,4 @@ def _load(cls, file_path: Path, path_output: Path | None, **kwargs: Any) -> pd.D df_pivoted.columns = df_pivoted.columns.get_level_values(0) df_pivoted.index = pd.to_datetime(df_pivoted.index) - df_pivoted = deduplicate_then_sort_index(df_pivoted) - - if path_output is not None: - cls._save_dataframe(df_pivoted, path_output) - return df_pivoted + return deduplicate_then_sort_index(df_pivoted) diff --git a/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py b/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py index 0b9cdd0..38becff 100644 --- a/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py +++ b/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py @@ -1,7 +1,6 @@ import ast import logging from pathlib import Path -from typing import Any import numpy as np import pandas as pd @@ -22,7 +21,7 @@ class MindRayRespiWavesDataSource(DataSourceBase): @classmethod @time_it - def _load(cls, file_path: Path, path_output: Path | None, **kwargs: Any) -> pd.DataFrame: # noqa: ARG003 + def _load(cls, file_path: Path) -> pd.DataFrame: """ Load and parse MindRay Respi Waves data. @@ -122,8 +121,4 @@ def _load(cls, file_path: Path, path_output: Path | None, **kwargs: Any) -> pd.D ) df_pivoted.columns = df_pivoted.columns.get_level_values(0) - df_pivoted = deduplicate_then_sort_index(df_pivoted) - - if path_output is not None: - cls._save_dataframe(df_pivoted, path_output) - return df_pivoted + return deduplicate_then_sort_index(df_pivoted) diff --git a/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py b/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py index 2d8cb0b..2ee7157 100644 --- a/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py +++ b/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py @@ -208,12 +208,7 @@ class MindRayScopeDataSource(DataSourceBase): @classmethod @time_it - def _load( - cls, - file_path_list: list[Path], - path_output: Path | None, - **kwargs: Any, # noqa: ARG003 - ) -> pd.DataFrame: + def _load(cls, file_path_list: list[Path]) -> pd.DataFrame: extension_preference = options_naming.FILE_EXTENSIONS file_dict = {} @@ -293,6 +288,4 @@ def _load( if optimize_storage_dtypes: df = _optimize_df_types(df) - if path_output is not None: - cls._save_dataframe(df, path_output) return df diff --git a/src/clinical_scope/datasource/sources/other/find_load_format.py b/src/clinical_scope/datasource/sources/other/find_load_format.py index 7e57ff5..3801c83 100644 --- a/src/clinical_scope/datasource/sources/other/find_load_format.py +++ b/src/clinical_scope/datasource/sources/other/find_load_format.py @@ -140,9 +140,7 @@ class OtherDataSource(DataSourceBase): OPTIONS_MODULE = options_naming @classmethod - def _load( - cls, file_path_list: Path | list[Path], path_output: Path | None, **kwargs - ) -> pd.DataFrame: + def _load(cls, file_path: Path | list[Path]) -> pd.DataFrame: """Not used — main() processes each file independently.""" msg = "OtherDataSource._load should not be called directly; use main() instead" raise NotImplementedError(msg) diff --git a/src/clinical_scope/datasource/sources/servo_u/find_load_format.py b/src/clinical_scope/datasource/sources/servo_u/find_load_format.py index bb9a964..abfa0ff 100644 --- a/src/clinical_scope/datasource/sources/servo_u/find_load_format.py +++ b/src/clinical_scope/datasource/sources/servo_u/find_load_format.py @@ -2,7 +2,6 @@ import re from datetime import datetime, timedelta from pathlib import Path -from typing import Any import pandas as pd @@ -140,12 +139,7 @@ class ServoUDataSource(DataSourceBase): @classmethod @time_it - def _load( - cls, - file_path_list: list[Path], - path_output: Path | None, - **kwargs: Any, # noqa: ARG003 - ) -> pd.DataFrame: + def _load(cls, file_path_list: list[Path]) -> pd.DataFrame: all_dfs = [] first_file_done = False start_time = None @@ -162,10 +156,7 @@ def _load( all_dfs.append(df_local) df = pd.concat(all_dfs) - df = deduplicate_then_sort_index(df) - if path_output is not None: - cls._save_dataframe(df, path_output) - return df + return deduplicate_then_sort_index(df) @classmethod @time_it diff --git a/tests/datasource/test_column_pruning.py b/tests/datasource/test_column_pruning.py index 566d862..8c5b9eb 100644 --- a/tests/datasource/test_column_pruning.py +++ b/tests/datasource/test_column_pruning.py @@ -406,7 +406,7 @@ def _wide_frame() -> pd.DataFrame: ) -def _make_source(load_calls: list | None = None) -> type: +def _make_source() -> type: """A datasource whose fresh `_load` returns the same wide frame the cache holds.""" class _FakeCachedSource(DataSourceBase): @@ -423,9 +423,7 @@ def _find(cls, folder_path: Path) -> Path: return folder_path / "raw_data.bin" @classmethod - def _load(cls, file_path, path_output, **kwargs): # noqa: ARG003 - if load_calls is not None: - load_calls.append(kwargs) + def _load(cls, file_path): # noqa: ARG003 return _wide_frame() return _FakeCachedSource @@ -449,8 +447,8 @@ class TestInspectConfiguredColumnsOnly: The opt-in trades the unconfigured-column rows for the memory — and nothing else. Rows stay unpruned in every case (inspect's `% retained` and raw date range are - comparisons against the *unwindowed* file), and the fresh-load path keeps seeing no - `field_display` at all, so the cache a first load writes is never narrowed by an inspect. + comparisons against the *unwindowed* file), and a fresh load reads every column — `_load` + takes no configuration, so an inspect can never narrow the cache a first load writes. """ DB_OPTIONS = {"field_display": ["HR", "SpO2"]} @@ -485,17 +483,6 @@ def test_flag_on_a_first_load_reports_every_column(self, tmp_path): assert {column.raw_name for column in result.columns} == {"HR", "SpO2", "RR"} assert result.columns_pruned is False - def test_fresh_load_never_sees_field_display(self, tmp_path): - # Regression guard: EIT's `_load` pre-filters on field_display and caches the result, - # so letting the flag reach a fresh load would write a narrowed cache. - (tmp_path / cst.FOLDER_NAME_OUTPUT).mkdir() - load_calls: list = [] - _make_source(load_calls).inspect( - _patient_options(tmp_path), self.DB_OPTIONS, configured_columns_only=True - ) - assert load_calls, "the fresh load path should have run" - assert "field_display" not in load_calls[0]["database_options_specific"] - @pytest.mark.parametrize("configured_columns_only", [False, True]) def test_rows_are_never_pruned(self, cached_patient, monkeypatch, configured_columns_only): # The window must cut only *after* the read, or "% retained" would always be 100%. diff --git a/tests/datasource/test_edf.py b/tests/datasource/test_edf.py index 7292e16..7a55a0f 100644 --- a/tests/datasource/test_edf.py +++ b/tests/datasource/test_edf.py @@ -22,7 +22,7 @@ def ds_folder(patient_full_path, edf_cls): def loaded_df(ds_folder, edf_cls): file_path = edf_cls._find(ds_folder) assert file_path is not None - return edf_cls._load(file_path, None) + return edf_cls._load(file_path) def _write_edf(path, start_datetime, sample_rate=4, seconds=2, labels=("chan A", "chan B")): @@ -172,7 +172,7 @@ class TestUndatedRecording: @pytest.fixture def undated_df(self, tmp_path, edf_cls): path = _write_edf(tmp_path / "undated.edf", datetime.datetime(1985, 1, 1, 9, 30)) # noqa: DTZ001 - return edf_cls._load([path], None) + return edf_cls._load([path]) @staticmethod def _first(df, edf_cls, recording_start=None): @@ -206,7 +206,7 @@ def test_an_already_parsed_date_still_keeps_the_files_time_of_day(self, undated_ def test_epoch_start_is_treated_as_undated(self, tmp_path, edf_cls): path = _write_edf(tmp_path / "epoch.edf", datetime.datetime(1970, 1, 1, 0, 0)) # noqa: DTZ001 - df = edf_cls._load([path], None) + df = edf_cls._load([path]) assert self._first(df, edf_cls, "2024-05-04 22:15:00") == pd.Timestamp( "2024-05-04 22:15:00", tz="Europe/Paris" ) @@ -215,7 +215,7 @@ def test_multi_file_spacing_survives_anchoring(self, tmp_path, edf_cls): """Anchoring shifts the whole recording, it does not collapse the gap between files.""" _write_edf(tmp_path / "a.edf", datetime.datetime(1985, 1, 1, 0, 0), seconds=2) # noqa: DTZ001 _write_edf(tmp_path / "b.edf", datetime.datetime(1985, 1, 1, 0, 1), seconds=2) # noqa: DTZ001 - df = edf_cls._load(sorted(tmp_path.glob("*.edf")), None) + df = edf_cls._load(sorted(tmp_path.glob("*.edf"))) formatted = edf_cls._format( df, {"data_folder": "", "edf": {"recording_start": "2024-05-04 10:00:00"}}, {} ) @@ -228,7 +228,7 @@ class TestDatedRecording: @pytest.fixture def dated_df(self, tmp_path, edf_cls): path = _write_edf(tmp_path / "dated.edf", datetime.datetime(2024, 3, 1, 7, 0)) # noqa: DTZ001 - return edf_cls._load([path], None) + return edf_cls._load([path]) def test_recording_start_is_ignored(self, dated_df, edf_cls): patient_options = {"data_folder": "", "edf": {"recording_start": "2024-05-04 22:15:00"}} diff --git a/tests/datasource/test_eit.py b/tests/datasource/test_eit.py index dbdcef4..7e7b991 100644 --- a/tests/datasource/test_eit.py +++ b/tests/datasource/test_eit.py @@ -16,7 +16,7 @@ def ds_folder(patient_full_path, eit_cls): def loaded_df(ds_folder, eit_cls): file_path = eit_cls._find(ds_folder) assert file_path is not None - return eit_cls._load(file_path, None) + return eit_cls._load(file_path) class TestFind: diff --git a/tests/datasource/test_fluxmed_parameters.py b/tests/datasource/test_fluxmed_parameters.py index 1f8be71..2411967 100644 --- a/tests/datasource/test_fluxmed_parameters.py +++ b/tests/datasource/test_fluxmed_parameters.py @@ -18,7 +18,7 @@ def ds_folder(patient_full_path, fluxmed_parameters_cls): def loaded_df(ds_folder, fluxmed_parameters_cls): file_path = fluxmed_parameters_cls._find(ds_folder) assert file_path is not None - return fluxmed_parameters_cls._load(file_path, None) + return fluxmed_parameters_cls._load(file_path) class TestFind: diff --git a/tests/datasource/test_fluxmed_signals.py b/tests/datasource/test_fluxmed_signals.py index 88aff55..95605fc 100644 --- a/tests/datasource/test_fluxmed_signals.py +++ b/tests/datasource/test_fluxmed_signals.py @@ -18,7 +18,7 @@ def ds_folder(patient_full_path, fluxmed_signals_cls): def loaded_df(ds_folder, fluxmed_signals_cls): file_path = fluxmed_signals_cls._find(ds_folder) assert file_path is not None - return fluxmed_signals_cls._load(file_path, None) + return fluxmed_signals_cls._load(file_path) class TestFind: @@ -71,7 +71,7 @@ def test_load_csv(self, patient_difficult_path, fluxmed_signals_cls): file_path = fluxmed_signals_cls._find(folder) if file_path is None: pytest.skip("No file found") - df = fluxmed_signals_cls._load(file_path, None) + df = fluxmed_signals_cls._load(file_path) assert isinstance(df, pd.DataFrame) assert isinstance(df.index, pd.DatetimeIndex) assert len(df) > 0 diff --git a/tests/datasource/test_load_config_independence.py b/tests/datasource/test_load_config_independence.py index 7951547..b99542b 100644 --- a/tests/datasource/test_load_config_independence.py +++ b/tests/datasource/test_load_config_independence.py @@ -1,10 +1,25 @@ """`_load` must transcribe the source file only — no option may reach the parquet cache. -See ADR-0010: whatever `_load` resolves is frozen into `clinical_scope_output/`, so a -later run with a different setting silently reads a cache built under the old one. +See ADR-0010: whatever a load resolves is frozen into `clinical_scope_output/`, so a later +run with a different setting silently reads a cache built under the old one. The base class +now writes that cache from whatever `_load` returns, which leaves the rule two channels, each +checked here: + +- *arguments* — closed by the signature. `_load(file_path)` has no option in scope at all. +- *module globals* — a signature cannot close these, so this is ADR-0010's own "mechanical + and greppable" clause, run as a test rather than left to review. + +Running a load twice under two configs would catch neither: both runs now take an identical +code path, and a module-level constant does not vary between them. + +Both checks read the definition as written, via the AST. `@time_it` is not `functools.wraps`ed, +so introspecting the bound attribute would describe the decorator's wrapper instead. """ -import pandas as pd +import ast +import inspect +from pathlib import Path + import pytest from clinical_scope.datasource.registry import DataSource @@ -14,42 +29,56 @@ entry.NAME for entry in DataSource.AVAILABLE if entry.DATASOURCE_CLASS.ALLOW_QUICK_LOAD ] -# A timezone no source defaults to, so an override that leaks is unmistakable. -OVERRIDE_TIMEZONE = "America/New_York" +# Referencing either inside `_load` means an option got frozen into the cache. +FORBIDDEN_IN_LOAD = frozenset({"DATA_SOURCE_DEFAULT_TIMEZONE", "apply_timezone_to_dataframe"}) + +def _load_definition(source_name: str) -> ast.FunctionDef: + """Return the `_load` definition of *source_name*'s datasource class, as written.""" + datasource_class = DataSource.get_subclass_by_name(source_name).DATASOURCE_CLASS + source_file = inspect.getsourcefile(datasource_class) + module = ast.parse(Path(source_file).read_text(encoding="utf-8")) -@pytest.fixture(scope="module") -def source_files(patient_full_path): - """{name: (cls, found_files)} for every caching source present in demo_patient.""" - found = {} - for name in CACHING_SOURCE_NAMES: - cls = DataSource.get_subclass_by_name(name).DATASOURCE_CLASS - folder = cls._find_folder(patient_full_path) - if folder is None: + for class_node in ast.walk(module): + if not isinstance(class_node, ast.ClassDef): continue - files = cls._find(folder) - if files is None: + if class_node.name != datasource_class.__name__: continue - found[name] = (cls, files) - return found + for node in class_node.body: + if isinstance(node, ast.FunctionDef) and node.name == "_load": + return node + pytest.fail(f"no `_load` definition found for '{source_name}' in {source_file}") @pytest.mark.parametrize("source_name", CACHING_SOURCE_NAMES) -def test_load_output_is_config_independent( - source_name, source_files, example_database_options -): - """Two configs, one file: `_load` must return the same frame from both.""" - if source_name not in source_files: - pytest.skip(f"'{source_name}' folder not found in demo_patient") - cls, files = source_files[source_name] - - bare_options = {} - configured_options = { - **example_database_options.get(source_name, {}), - "additional_informations": {"timezone": OVERRIDE_TIMEZONE}, - } +def test_load_takes_only_the_file_path(source_name): + """No option can be resolved from an argument that is not there.""" + node = _load_definition(source_name) + positional = [argument.arg for argument in node.args.args] + + assert len(positional) == 2, ( + f"'{source_name}'._load takes {positional}; only (cls, file_path) may be in scope" + ) + assert node.args.vararg is None and node.args.kwarg is None, ( + f"'{source_name}'._load accepts *args/**kwargs — an option can reach the cache through them" + ) + assert not node.args.kwonlyargs, ( + f"'{source_name}'._load takes keyword-only {[a.arg for a in node.args.kwonlyargs]}" + ) + - df_bare = cls._load(files, None, database_options_specific=bare_options) - df_configured = cls._load(files, None, database_options_specific=configured_options) +@pytest.mark.parametrize("source_name", CACHING_SOURCE_NAMES) +def test_load_resolves_no_option_from_module_scope(source_name): + """The channel the signature cannot close: a default reached for directly.""" + node = _load_definition(source_name) + referenced = { + inner.id if isinstance(inner, ast.Name) else inner.attr + for inner in ast.walk(node) + if isinstance(inner, (ast.Name, ast.Attribute)) + } - pd.testing.assert_frame_equal(df_bare, df_configured) + leaked = sorted(referenced & FORBIDDEN_IN_LOAD) + assert not leaked, ( + f"'{source_name}'._load references {leaked}: a default frozen into the cache is " + f"indistinguishable at read time from a user's choice frozen into it" + ) diff --git a/tests/datasource/test_mindray_respi_numerics.py b/tests/datasource/test_mindray_respi_numerics.py index 60657fb..68ce847 100644 --- a/tests/datasource/test_mindray_respi_numerics.py +++ b/tests/datasource/test_mindray_respi_numerics.py @@ -18,7 +18,7 @@ def ds_folder(patient_full_path, mindray_respi_numerics_cls): def loaded_df(ds_folder, mindray_respi_numerics_cls): file_path = mindray_respi_numerics_cls._find(ds_folder) assert file_path is not None - return mindray_respi_numerics_cls._load(file_path, None) + return mindray_respi_numerics_cls._load(file_path) class TestFind: @@ -67,7 +67,7 @@ def test_load_csv(self, patient_difficult_path, mindray_respi_numerics_cls): file_path = mindray_respi_numerics_cls._find(folder) if file_path is None: pytest.skip("No file found") - df = mindray_respi_numerics_cls._load(file_path, None) + df = mindray_respi_numerics_cls._load(file_path) assert isinstance(df, pd.DataFrame) assert isinstance(df.index, pd.DatetimeIndex) diff --git a/tests/datasource/test_mindray_respi_waves.py b/tests/datasource/test_mindray_respi_waves.py index 06977fa..4de075a 100644 --- a/tests/datasource/test_mindray_respi_waves.py +++ b/tests/datasource/test_mindray_respi_waves.py @@ -18,7 +18,7 @@ def ds_folder(patient_full_path, mindray_respi_waves_cls): def loaded_df(ds_folder, mindray_respi_waves_cls): file_path = mindray_respi_waves_cls._find(ds_folder) assert file_path is not None - return mindray_respi_waves_cls._load(file_path, None) + return mindray_respi_waves_cls._load(file_path) class TestFind: @@ -71,7 +71,7 @@ def test_load_csv(self, patient_difficult_path, mindray_respi_waves_cls): file_path = mindray_respi_waves_cls._find(folder) if file_path is None: pytest.skip("No file found") - df = mindray_respi_waves_cls._load(file_path, None) + df = mindray_respi_waves_cls._load(file_path) assert isinstance(df, pd.DataFrame) assert isinstance(df.index, pd.DatetimeIndex) diff --git a/tests/datasource/test_mindray_scope.py b/tests/datasource/test_mindray_scope.py index 3807a20..112c273 100644 --- a/tests/datasource/test_mindray_scope.py +++ b/tests/datasource/test_mindray_scope.py @@ -16,7 +16,7 @@ def ds_folder(patient_full_path, mindray_scope_cls): def loaded_df(ds_folder, mindray_scope_cls): file_path = mindray_scope_cls._find(ds_folder) assert file_path is not None - return mindray_scope_cls._load(file_path, None) + return mindray_scope_cls._load(file_path) class TestFind: @@ -73,7 +73,7 @@ def test_load_xml(self, patient_difficult_path, mindray_scope_cls): file_path = mindray_scope_cls._find(folder) if file_path is None: pytest.skip("No file found") - df = mindray_scope_cls._load(file_path, None) + df = mindray_scope_cls._load(file_path) assert isinstance(df, pd.DataFrame) assert isinstance(df.index, pd.DatetimeIndex) assert len(df) > 0 diff --git a/tests/datasource/test_servo_u.py b/tests/datasource/test_servo_u.py index 87ef8fd..6d45dcc 100644 --- a/tests/datasource/test_servo_u.py +++ b/tests/datasource/test_servo_u.py @@ -16,7 +16,7 @@ def ds_folder(patient_full_path, servo_u_cls): def loaded_df(ds_folder, servo_u_cls): file_path = servo_u_cls._find(ds_folder) assert file_path is not None - return servo_u_cls._load(file_path, None) + return servo_u_cls._load(file_path) class TestFind: diff --git a/tests/unit/test_datasource_base_cache.py b/tests/unit/test_datasource_base_cache.py index b1eb971..bf8aa7d 100644 --- a/tests/unit/test_datasource_base_cache.py +++ b/tests/unit/test_datasource_base_cache.py @@ -48,9 +48,8 @@ def _make_fake_source( Build a fresh ``DataSourceBase`` subclass per test. Each test gets its own class — no shared mutable state, no reset fixture needed. - The fake's ``_load`` mirrors the convention every real datasource follows: - save to ``path_output`` iff one was passed. That convention is the calling - contract under test, so the fake replicates it deliberately. + The fake's ``_load`` only returns a frame, like every real datasource: writing the + parquet cache is the base class's job, and that is precisely what these tests exercise. """ _source_file = source_file @@ -69,9 +68,7 @@ def _find(cls, folder_path: Path) -> Path: return _source_file if _source_file is not None else folder_path / "raw_data.bin" @classmethod - def _load(cls, file_path, path_output, **kwargs): # noqa: ARG003 - if path_output is not None: - cls._save_dataframe(fresh_df, path_output) + def _load(cls, file_path): # noqa: ARG003 return fresh_df return _FakeSource @@ -144,6 +141,37 @@ def test_quick_load_truth_table( ) +# --------------------------------------------------------------------------------------------------- +# The base class owns the write: `_load` transcribes and saves nothing itself (ADR-0010) +# --------------------------------------------------------------------------------------------------- +def test_base_writes_the_cache_a_load_never_saves(patient_folder: Path) -> None: + """A `_load` that only returns a frame still leaves a readable cache behind.""" + source = _make_fake_source(_df_v1()) + cache_path = patient_folder / cst.FOLDER_NAME_OUTPUT / "fake_source.parquet" + + df, _, _ = source._load_raw_dataframe( + _patient_options(patient_folder, quick_load=False), database_options={} + ) + + pd.testing.assert_frame_equal(df, _df_v1(), check_freq=False) + assert cache_path.is_file(), "the base class must write the cache for a saving-free _load" + pd.testing.assert_frame_equal(pd.read_parquet(cache_path), _df_v1(), check_freq=False) + + +def test_empty_frame_writes_neither_cache_nor_output_folder(tmp_path: Path) -> None: + """An empty frame is not worth a cache — nor the output folder it would be written into.""" + empty = pd.DataFrame({"col": []}, index=pd.DatetimeIndex([], tz="UTC")) + source = _make_fake_source(empty) + output_folder = tmp_path / cst.FOLDER_NAME_OUTPUT + + df, _, _ = source._load_raw_dataframe( + _patient_options(tmp_path, quick_load=False), database_options={} + ) + + assert df.empty + assert not output_folder.exists(), "an empty frame must not even create the output folder" + + # --------------------------------------------------------------------------------------------------- # Sanity: a datasource that opts out of caching must never produce a parquet, even when ticked # --------------------------------------------------------------------------------------------------- From 90a232e7bae85bd08983ab92c82065d22d45324c Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Tue, 25 Aug 2026 17:05:39 +0200 Subject: [PATCH 2/9] Name the load-path naive-bound timezone separately (ADR-0011) cst.DISPLAY_TIMEZONE was serving two unrelated concepts: the default of the display_timezone user option, and the timezone a tz-naive datetime_start/datetime_end is interpreted in on the load path. Reading the second through the name of the first makes the load path look like it wrongly ignores the user's setting -- a misreading an architecture review made, and then made again while implementing the fix. It does not ignore it. The UI qualifies its bounds at Submit, so the Settings timezone governs the window there; both load-path sites localize only when tzinfo is None, and are unreachable from the app. The constant is a fallback for bounds that never met a user: scripts, library calls, hand-edited files. Resolving a user option for those would make extract_* output depend on ~/.clinical_scope/user_options.json. Split out cst.NAIVE_BOUND_TZ as a separate literal, not an alias -- aliasing would let a change to the app's display default silently reinterpret every script's naive bounds, which is the coupling the ADR forbids. filter_data_by_timestamps' parameter follows the concept, and resolve_display_timezone gained a fallback so the load path's invalid-name branch cannot land on the display default. Load-path tests monkeypatch the new name; the one site still patching DISPLAY_TIMEZONE covers inspect()'s cosmetic date ranges, which is genuinely the other concept. Behaviour is unchanged -- both constants are "Europe/Paris". Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 2 + ...me-bounds-are-qualified-at-the-boundary.md | 37 ++++++++++++++++ src/clinical_scope/constants.py | 4 ++ src/clinical_scope/datasource/base.py | 16 +++---- .../datasource/formatting/timezone.py | 26 +++++++---- tests/datasource/test_column_pruning.py | 8 ++-- tests/datasource/test_datetime_pushdown.py | 44 +++++++++---------- 7 files changed, 94 insertions(+), 43 deletions(-) create mode 100644 docs/adr/0011-datetime-bounds-are-qualified-at-the-boundary.md diff --git a/CLAUDE.md b/CLAUDE.md index bd05afe..681d4c6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,6 +53,8 @@ Registered in `datasource/registry.py` (`DataSource.AVAILABLE`); the canonical l **`_load` transcribes; `_format` interprets** ([ADR-0010](docs/adr/0010-load-transcribes-format-interprets.md)). `_load`'s output *is* the parquet cache, so it must be reproducible from the source file alone — no option resolved inside it. Mechanically: no `DATA_SOURCE_DEFAULT_TIMEZONE`, no `apply_timezone_to_dataframe` in any `_load`. +**Datetime bounds are qualified at the boundary** ([ADR-0011](docs/adr/0011-datetime-bounds-are-qualified-at-the-boundary.md)). The UI turns naive form text into a tz-aware instant at Submit, using the user's `display_timezone` — *that* is what makes the Settings timezone govern the time window. The load path only ever localizes a bound that is still naive (script or hand-edited file), and does so with `cst.NAIVE_BOUND_TZ`, never a user option, so `extract_*` output does not depend on who is at the keyboard. `cst.NAIVE_BOUND_TZ` and `cst.DISPLAY_TIMEZONE` are separate literals on purpose; do not alias them. + **Adding one**: use the `/new-datasource` skill — it is authoritative for the module layout, `options.py` constants, the loader, registration (Other stays last), example data, tests, snapshots, and the tutorial table. ## Config files diff --git a/docs/adr/0011-datetime-bounds-are-qualified-at-the-boundary.md b/docs/adr/0011-datetime-bounds-are-qualified-at-the-boundary.md new file mode 100644 index 0000000..49e0893 --- /dev/null +++ b/docs/adr/0011-datetime-bounds-are-qualified-at-the-boundary.md @@ -0,0 +1,37 @@ +# 11. Datetime bounds are qualified at the boundary + +Date: 2026-08-25 + +## Status + +Accepted + +## Context + +`datetime_start` / `datetime_end` cut the window every pipeline honours. They are strings, and a naive one (`2004-09-15 08:20:00`) is not a moment in time until something says which clock it was read off. Either each consumer answers that from the reader's `display_timezone`, or the boundary that produced the bound answers it once. + +The code already did the latter and nowhere said so, while the naming argued for the former: `DISPLAY_TIMEZONE` served as both the default of the `display_timezone` user option and the timezone naive bounds are read in. An architecture review duly read the load path, concluded it was wrongly ignoring the user's setting, and proposed reversing the decision. + +What it missed is that the UI qualifies at Submit — `data_callbacks.py` bakes the Settings timezone into both fields before saving, and `rerender_datetime_on_timezone_change` rewrites the visible digits when that setting changes so the stored instant holds. Both load-path sites, `_pushdown_bounds._to_aware` and `filter_data_by_timestamps`, localize only when `tzinfo is None`, so their fallback is unreachable from the app entirely. + +It is reachable from the CLI scripts (none pass `user_options`), from `extract_datasource` / `batch_extract`, and from hand-edited files. Resolving a user option there would mean two colleagues running the same script over the same folder get different rows, because one of them once typed a different timezone into a GUI. + +## Decision + +**A bound is qualified once, at the boundary that produced it. No consumer re-qualifies an already-qualified bound; an unqualified one is read in a fixed constant, never in a user option.** + +- The UI is a boundary: naive form text becomes an instant at Submit, in the user's `display_timezone`. **This is what makes the Settings timezone govern the window** — not a load-path lookup. +- Scripts and library callers are boundaries too. One wanting a specific clock passes an aware bound. +- The load path is not a boundary. It localizes naive bounds in `cst.NAIVE_BOUND_TZ`, so `extract_*` output depends only on the folder and option files given to it. + +`NAIVE_BOUND_TZ` is a separate literal from `DISPLAY_TIMEZONE`, not an alias, though both are `"Europe/Paris"` — aliasing would let a change to the app's display default silently reinterpret every script's bounds, which is the coupling this ADR forbids. + +An explicit timezone argument on the load path was rejected: it buys nothing on the UI path (bounds arrive aware) and nothing on the script path (an aware bound already says it), while widening a signature [ADR-0010](0010-load-transcribes-format-interprets.md) had just narrowed. + +## Consequences + +- The app behaves as users expect and the library as scripts expect, with no conflict between them — one rule seen from two ends, not a compromise between two rules. +- `filter_data_by_timestamps` takes `naive_bound_tz`, not `display_timezone`; `resolve_display_timezone` gained a `fallback` so the load path's invalid-name branch cannot land on the display default. Load-path tests monkeypatch the new name, which distinguishes the two concepts in tests for free. +- **Accepted cost:** a hand-written naive bound is read as `Europe/Paris`, not its author's setting. The app never writes such a file; the fix is to write an offset. +- Adjacent to [ADR-0005](0005-user-options-are-fallbacks.md) but distinct — that one ranks user options below database options for *display*; this one keeps them off the load path entirely, a question of determinism. A setting changing how rows *render* follows 0005; one changing *which rows return* follows this. +- **Revisit if** a script-facing case genuinely needs per-user interpretation. The answer is then a parameter on the public `extract_*` functions — an explicit boundary — not a load-path lookup. diff --git a/src/clinical_scope/constants.py b/src/clinical_scope/constants.py index 1bb6edf..040dae4 100644 --- a/src/clinical_scope/constants.py +++ b/src/clinical_scope/constants.py @@ -16,7 +16,11 @@ ) LIBRARY_TZ = "UTC" +# Timezone plots, annotations and inspect() reports are rendered in, absent a user setting. DISPLAY_TIMEZONE = "Europe/Paris" +# Timezone a tz-naive datetime_start/datetime_end patient option is interpreted in on the +# load path. Equal to DISPLAY_TIMEZONE by convention, separate by construction: see ADR-0011. +NAIVE_BOUND_TZ = "Europe/Paris" DATETIME_INDEX_NAME = "datetime_index" QUALIFIED_NAME_SEPARATOR = "::" diff --git a/src/clinical_scope/datasource/base.py b/src/clinical_scope/datasource/base.py index e49178e..32c5f03 100644 --- a/src/clinical_scope/datasource/base.py +++ b/src/clinical_scope/datasource/base.py @@ -165,19 +165,18 @@ def _pushdown_bounds( ) shift_td = pd.Timedelta(seconds=time_shift) buffer = pd.Timedelta(seconds=cst.DATETIME_PUSHDOWN_BUFFER_SECONDS) - # A tz-naive bound is interpreted in the default *display* timezone, not LIBRARY_TZ: - # naive bounds are what the form used to write, and it typed them in display time. - # Deliberately the constant rather than the user option -- this is the load path, and - # extract_* output must not depend on ~/.clinical_scope/user_options.json. - display_timezone = cst.DISPLAY_TIMEZONE + # An aware bound passes through untouched -- the UI qualifies its bounds at Submit, so + # this constant is reached only by bounds that never met a user (scripts, hand-edited + # files). Interpreting those in the user option would make extract_* output depend on + # ~/.clinical_scope/user_options.json. See ADR-0011. def _to_aware(raw_value: str | None) -> pd.Timestamp | None: if not raw_value: return None timestamp = pd.Timestamp(raw_value) if timestamp.tzinfo is not None: return timestamp - return timestamp.tz_localize(display_timezone) + return timestamp.tz_localize(cst.NAIVE_BOUND_TZ) start_aware = _to_aware(datetime_start) end_aware = _to_aware(datetime_end) @@ -426,14 +425,13 @@ def _filter_by_datetime( datetime_end = patient_options.get(cst.PatientOptions.DatetimeEnd.NAME) datetime_start = pd.Timestamp(datetime_start) if datetime_start else None datetime_end = pd.Timestamp(datetime_end) if datetime_end else None - # See _pushdown_bounds: a tz-naive bound is interpreted in the default display timezone. - display_timezone = cst.DISPLAY_TIMEZONE + # See _pushdown_bounds: only a tz-naive bound is interpreted here. return filter_data_by_timestamps( df, time_start=datetime_start, time_end=datetime_end, filter_date=filter_date, - display_timezone=display_timezone, + naive_bound_tz=cst.NAIVE_BOUND_TZ, ) @classmethod diff --git a/src/clinical_scope/datasource/formatting/timezone.py b/src/clinical_scope/datasource/formatting/timezone.py index 6fcb59f..a251e29 100644 --- a/src/clinical_scope/datasource/formatting/timezone.py +++ b/src/clinical_scope/datasource/formatting/timezone.py @@ -18,25 +18,30 @@ # ================================================================================================== -def resolve_display_timezone(display_timezone: str | None) -> str: +def resolve_display_timezone(display_timezone: str | None, fallback: str | None = None) -> str: """ Return a usable IANA timezone name, falling back to ``cst.DISPLAY_TIMEZONE``. An absent value is the normal case and stays silent; a present-but-invalid name (hand-edited file, programmatic call) is logged rather than left to raise deep inside pandas/zoneinfo. + + *fallback* lets a caller that is not resolving a *display* timezone name its own default + -- the load path passes ``cst.NAIVE_BOUND_TZ`` (ADR-0011) so an invalid name there cannot + silently land on the user-facing display default. """ + fallback = fallback or cst.DISPLAY_TIMEZONE if not display_timezone: - return cst.DISPLAY_TIMEZONE + return fallback try: ZoneInfo(display_timezone) except (ZoneInfoNotFoundError, KeyError, ValueError): logger.warning( - "display_timezone %r is not a valid IANA name; using %s", + "timezone %r is not a valid IANA name; using %s", display_timezone, - cst.DISPLAY_TIMEZONE, + fallback, ) - return cst.DISPLAY_TIMEZONE + return fallback return display_timezone @@ -91,16 +96,21 @@ def filter_data_by_timestamps( time_start: pd.Timestamp | None, time_end: pd.Timestamp | None, filter_date: bool = True, - display_timezone: str | None = None, + naive_bound_tz: str | None = None, ) -> pd.DataFrame: - """Filter data between time_start and time_end timestamps using a hardcoded library timezone.""" + """ + Filter data between time_start and time_end, comparing in ``cst.LIBRARY_TZ``. + + An aware bound is converted; a naive one is first localized in *naive_bound_tz*, which + is a load-path default and not the user's display timezone (ADR-0011). + """ if not pd.api.types.is_datetime64_any_dtype(data.index): logger.warning("Data index is not datetime. Skipping filtering.") return data # Shallow copy since below only rebinds the index or row-filters, never mutates columns. filtered = data.copy(deep=False) - resolved_timezone = resolve_display_timezone(display_timezone) + resolved_timezone = resolve_display_timezone(naive_bound_tz, fallback=cst.NAIVE_BOUND_TZ) if filtered.index.tz is None: msg = "Dataframe 'data' index should be timezone-aware" diff --git a/tests/datasource/test_column_pruning.py b/tests/datasource/test_column_pruning.py index 8c5b9eb..dd15295 100644 --- a/tests/datasource/test_column_pruning.py +++ b/tests/datasource/test_column_pruning.py @@ -304,9 +304,9 @@ def test_absent_field_display_reads_all_columns(self, servo_u_cls, tmp_path): assert list(out.columns) == ["HR", "SpO2", "RR"] def test_composes_row_and_column_pruning(self, servo_u_cls, tmp_path, monkeypatch): - # The materialized cache index is tz-aware UTC (_make_cache); pin the display default - # to UTC so the naive window lands inside it. - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + # The materialized cache index is tz-aware UTC (_make_cache); pin the naive-bound + # default to UTC so the naive window lands inside it. + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") path = _make_cache(tmp_path) full = pd.read_parquet(path) patient_options = { @@ -486,7 +486,7 @@ def test_flag_on_a_first_load_reports_every_column(self, tmp_path): @pytest.mark.parametrize("configured_columns_only", [False, True]) def test_rows_are_never_pruned(self, cached_patient, monkeypatch, configured_columns_only): # The window must cut only *after* the read, or "% retained" would always be 100%. - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") result = _make_source().inspect( _patient_options( cached_patient, diff --git a/tests/datasource/test_datetime_pushdown.py b/tests/datasource/test_datetime_pushdown.py index 37b47f1..93ec18a 100644 --- a/tests/datasource/test_datetime_pushdown.py +++ b/tests/datasource/test_datetime_pushdown.py @@ -164,10 +164,10 @@ class TestPushdownBounds: def test_no_window_returns_none(self, other_cls): assert other_cls._pushdown_bounds({}, {}, index_tz=None) is None - def test_naive_target_converts_from_display_tz_and_pads(self, other_cls, monkeypatch): - # 'other' source tz is UTC; pin the display default to UTC too, isolating the + def test_naive_target_converts_from_naive_bound_tz_and_pads(self, other_cls, monkeypatch): + # 'other' source tz is UTC; pin the naive-bound default to UTC too, isolating the # ± buffer as the only transformation left to verify. - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "datetime_start": "2004-09-15 08:20:00", "datetime_end": "2004-09-15 08:25:00", @@ -178,7 +178,7 @@ def test_naive_target_converts_from_display_tz_and_pads(self, other_cls, monkeyp assert end == pd.Timestamp("2004-09-15 08:25:00") + buffer def test_time_shift_is_inverted(self, other_cls, monkeypatch): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "datetime_start": "2004-09-15 08:20:00", "datetime_end": "2004-09-15 08:25:00", @@ -190,7 +190,7 @@ def test_time_shift_is_inverted(self, other_cls, monkeypatch): assert end == pd.Timestamp("2004-09-15 08:25:00") - pd.Timedelta(seconds=30.0) + buffer def test_negative_time_shift_is_inverted(self, other_cls, monkeypatch): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "datetime_start": "2004-09-15 08:20:00", "datetime_end": "2004-09-15 08:25:00", @@ -212,11 +212,11 @@ def test_naive_target_falls_back_to_database_options_timezone_override( ): """ The exact configuration behind issue #57's bug: no materialized index tz, and a - per-source timezone override that differs from both the library display default and + per-source timezone override that differs from both the naive-bound default and the datasource default — must actually shift the bounds, not just take the fallback branch as a no-op (both prior unit tests above used UTC==UTC, hiding this). """ - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "datetime_start": "2004-09-15 08:20:00", "datetime_end": "2004-09-15 08:25:00", @@ -237,7 +237,7 @@ def test_naive_target_falls_back_to_database_options_timezone_override( assert end.tzinfo is None def test_aware_target_uses_index_tz_directly(self, other_cls, monkeypatch): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "datetime_start": "2004-09-15 08:20:00", "datetime_end": "2004-09-15 08:25:00", @@ -275,7 +275,7 @@ class TestQuickLoadCachePushdownEquality: def test_naive_cache_pushdown_matches_disabled( self, servo_u_cls, patient_full_path, monkeypatch, datetime_start, datetime_end ): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "data_folder": str(patient_full_path), "datetime_start": datetime_start, @@ -295,7 +295,7 @@ def test_naive_cache_pushdown_matches_disabled( def test_aware_cache_pushdown_matches_disabled_with_time_shift( self, mindray_respi_numerics_cls, patient_full_path, monkeypatch ): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "data_folder": str(patient_full_path), "datetime_start": "2004-09-15 08:12:40", @@ -317,27 +317,27 @@ def test_aware_cache_pushdown_matches_disabled_with_time_shift( class TestNaiveAwareBoundEquivalence: """ - A tz-aware bound must select exactly what the equivalent naive + display_timezone - pair would. + A tz-aware bound must select exactly what the equivalent naive + cst.NAIVE_BOUND_TZ + pair would -- and, being already qualified, must not depend on that constant at all. """ @pytest.mark.parametrize( - "datetime_start,datetime_end,display_timezone", + "datetime_start,datetime_end,naive_bound_tz", [ ("2004-09-15 08:12:40", "2004-09-15 08:12:50", "UTC"), ("2004-09-15 10:12:40", "2004-09-15 10:12:50", "Europe/Paris"), ], ) - def test_aware_bound_matches_naive_plus_display_timezone( + def test_aware_bound_matches_naive_plus_naive_bound_tz( self, servo_u_cls, patient_full_path, monkeypatch, datetime_start, datetime_end, - display_timezone, + naive_bound_tz, ): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", display_timezone) + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", naive_bound_tz) naive_options = { "data_folder": str(patient_full_path), "datetime_start": datetime_start, @@ -346,12 +346,12 @@ def test_aware_bound_matches_naive_plus_display_timezone( } df_naive = servo_u_cls.extract(naive_options, {}) - # Deliberately a different library default: an aware bound must not need this to agree. - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "America/New_York") + # Deliberately a different default: an aware bound must not need this to agree. + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "America/New_York") aware_options = { "data_folder": str(patient_full_path), - "datetime_start": to_aware_display_ts(datetime_start, display_timezone), - "datetime_end": to_aware_display_ts(datetime_end, display_timezone), + "datetime_start": to_aware_display_ts(datetime_start, naive_bound_tz), + "datetime_end": to_aware_display_ts(datetime_end, naive_bound_tz), "quick_load": True, } df_aware = servo_u_cls.extract(aware_options, {}) @@ -397,7 +397,7 @@ class TestOtherParquetPushdownEquality: def test_pushdown_matches_disabled( self, other_cls, patient_difficult_path, monkeypatch, datetime_start, datetime_end ): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") patient_options = { "data_folder": str(patient_difficult_path), "datetime_start": datetime_start, @@ -443,7 +443,7 @@ class TestOtherDatetimeColumnDetectionTimezoneOverride: def test_naive_utc_named_column_with_timezone_override_matches_disabled( self, other_cls, tmp_path, monkeypatch ): - monkeypatch.setattr(cst, "DISPLAY_TIMEZONE", "UTC") + monkeypatch.setattr(cst, "NAIVE_BOUND_TZ", "UTC") other_dir = tmp_path / "other" other_dir.mkdir() file_path = other_dir / "device_export.parquet" From c50e1e1482268df89f3ca47b93035609cdd1a8c7 Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Wed, 26 Aug 2026 10:04:51 +0200 Subject: [PATCH 3/9] Add annotations set and group classes --- ...012-annotation-dicts-are-an-open-schema.md | 36 ++ docs/user_guide/tutorial.md | 2 + .../dash_api/annotations/model.py | 216 +++++++++- .../callbacks/annotation_callbacks.py | 200 +++------ tests/unit/test_annotation_model.py | 383 ++++++++++++++++++ 5 files changed, 693 insertions(+), 144 deletions(-) create mode 100644 docs/adr/0012-annotation-dicts-are-an-open-schema.md create mode 100644 tests/unit/test_annotation_model.py diff --git a/docs/adr/0012-annotation-dicts-are-an-open-schema.md b/docs/adr/0012-annotation-dicts-are-an-open-schema.md new file mode 100644 index 0000000..8027b18 --- /dev/null +++ b/docs/adr/0012-annotation-dicts-are-an-open-schema.md @@ -0,0 +1,36 @@ +# 12. Annotation dicts are an open schema + +Date: 2026-08-25 + +## Status + +Accepted + +## Context + +`annotations.json` is public library surface, not just a file the app writes to itself: `load_annotations` and `load_database_annotations` are exported from `clinical_scope/__init__.py`, and the format is documented for users who hand-write a file or generate one from another tool. + +`Annotation.to_dict` wrote a fixed thirteen-key literal, so any key the class did not recognise was erased. It was tempting to read that as a mutation-time concern — the callbacks manipulate raw dicts, so a delete or a label toggle looked like the place a stray key might survive or die. It is not. `auto_load_annotations` hydrates the file through `from_dict` and re-serialises through `to_dict` straight into `annotation-store`, so an externally authored file lost its keys **at load**, before the user clicked anything, and the stripped version was what the next save wrote back. The raw-dict mutations preserved keys that could no longer be present. + +That leaves exactly one choke point, and it is the one every path already goes through. + +## Decision + +**An annotation dict is an open schema: keys the app does not own are carried, not dropped.** + +`Annotation` holds an `extra` dict. `from_dict` collects every key outside the owned set into it; `to_dict` splats `extra` back **before** the owned keys, so a stale or hostile duplicate (`"id"`, `"type"`) can never shadow a real field. A round-trip through the app — load, edit, save — is lossless for anything the app does not understand. + +The owned set is derived from `dataclasses.fields(Annotation)` rather than restated as a literal, so promoting a convention to a real field automatically stops it landing in `extra`. + +## Scope + +This is a **record-level** promise. Sibling keys in the JSON envelope — anything alongside `"annotations"`, including the `"version"` the `io.py` docstring anticipates — are still dropped, because `save_annotations` writes a fresh envelope rather than reading and merging the existing one. + +That was considered and deliberately left out. Read-before-write introduces a merge-or-clobber question when the file has changed on disk underneath a running app, which is a larger decision than the one this ADR makes, and no consumer needs it yet. **Revisit if** a tool appears that stores state in the envelope. + +## Consequences + +- Users can carry their own per-annotation fields — a reviewer, a confidence, an external record id — through the app without the app understanding them. +- Promotion is free: when a convention becomes a real field, `from_dict` starts claiming it and `extra` quietly stops holding it. No migration. +- **Accepted cost:** `Annotation` cannot later be swapped for a strict-schema model without breaking this, and the guarantee needs a test that a foreign key survives load → mutate → save rather than merely load → save. +- Sits beside a second rule that fell out of the same work and stays at docstring level: **`from_dict` transcribes, `create` interprets.** Creation-time defaulting — a point is never global, and its label starts hidden — lives in `Annotation.create`, never in `__post_init__`, because deserialisation must reproduce a stored annotation verbatim: a point whose label the user explicitly un-hid has to load back un-hidden. The same shape as [ADR-0010](0010-load-transcribes-format-interprets.md), one layer up. diff --git a/docs/user_guide/tutorial.md b/docs/user_guide/tutorial.md index 6e112c8..1a48029 100644 --- a/docs/user_guide/tutorial.md +++ b/docs/user_guide/tutorial.md @@ -499,6 +499,8 @@ subsequent annotations will belong to that group until you switch or create anot `annotations.json` sits in the patient data folder, next to the datasource sub-folders. Annotations are reloaded automatically when you re-process the same patient. +If you write the file by hand or generate it from another tool, any extra fields you add to an annotation are kept as they are. + ## Python API Annotations can be loaded programmatically for analysis: diff --git a/src/clinical_scope/dash_api/annotations/model.py b/src/clinical_scope/dash_api/annotations/model.py index 188154c..5515fa9 100644 --- a/src/clinical_scope/dash_api/annotations/model.py +++ b/src/clinical_scope/dash_api/annotations/model.py @@ -2,22 +2,33 @@ Annotation data model. An Annotation represents a user-created mark on a plot: a time event (vertical -line), a time window (shaded rectangle), or a point (arrow + label). +line), a time window (shaded rectangle), or a point (arrow + label). It is +serialisable to a plain dict so it can be stored in a dcc.Store and written to JSON. -Each Annotation is serialisable to a plain dict so it can be stored in a -dcc.Store and written to JSON. +An AnnotationSet is an immutable collection of them, and a Group is a set of annotations +sharing a ``group_id``. Groups are never persisted (see :mod:`.io`): a group exists only as +the annotations that carry its id, and its name, colour, type and scope are read off the +first of them. That derivation is defined here, once, rather than at each call site. + +Nothing here imports Dash: a callback hydrates the store's list of dicts into an +AnnotationSet, asks a question or produces a modified copy, and serialises back out. +The set itself is never stored — ``dcc.Store`` holds JSON only. """ from __future__ import annotations import re import uuid -from dataclasses import dataclass, field +from dataclasses import dataclass, field, fields, replace from datetime import UTC, datetime from enum import StrEnum +from typing import TYPE_CHECKING import clinical_scope.constants as cst +if TYPE_CHECKING: + from collections.abc import Iterator + def _now_iso() -> str: return datetime.now(tz=UTC).isoformat() @@ -112,6 +123,10 @@ class Annotation: for loop-plot points where per-point timing is available. created_at ISO datetime string of creation time. + extra + Keys present in the source dict that this class does not own, kept verbatim so a + hand-written or externally generated ``annotations.json`` survives a round-trip + through the app (ADR-0012). """ @@ -128,10 +143,46 @@ class Annotation: patient: str | None = None id: str = field(default_factory=lambda: str(uuid.uuid4())) created_at: str = field(default_factory=_now_iso) + extra: dict = field(default_factory=dict) + + @classmethod + def create( + cls, + *, + annotation_type: AnnotationType, + plot_name: str, + data: dict, + is_global: bool = False, + subplot_name: str | None = None, + **kwargs, + ) -> Annotation: + """ + Build a *new* annotation, applying the creation-time defaults for its type. + + Deserialisation must not come through here: :meth:`from_dict` reproduces a stored + annotation verbatim, so a point whose label the user explicitly un-hid loads back un-hidden. + """ + if annotation_type is AnnotationType.POINT: + # A point is anchored to one (x, y) inside a single subplot, so it can never be + # global, and its label starts hidden so the dot marker alone shows. + is_global = False + kwargs.setdefault("label_hidden", True) + return cls( + type=annotation_type, + plot_name=plot_name, + data=data, + subplot_name=None if is_global else subplot_name, + **kwargs, + ) def to_dict(self) -> dict: - """Serialise to a JSON-safe dict.""" + """ + Serialise to a JSON-safe dict. + + Unowned keys go first so a known field can never be shadowed by a stale one in `extra`. + """ return { + **self.extra, "id": self.id, "type": self.type.value, "label": self.label, @@ -164,4 +215,159 @@ def from_dict(cls, annotation_dict: dict) -> Annotation: trace_metadata=annotation_dict.get("trace_metadata"), label_hidden=annotation_dict.get("label_hidden", False), created_at=annotation_dict.get("created_at", _now_iso()), + extra={ + key: value + for key, value in annotation_dict.items() + if key not in _OWNED_ANNOTATION_KEYS + }, + ) + + +# Derived from the dataclass rather than restated, so a promoted field stops landing in `extra` +# on its own. `extra` itself is not a serialised key. +_OWNED_ANNOTATION_KEYS: frozenset[str] = frozenset( + annotation_field.name for annotation_field in fields(Annotation) +) - {"extra"} + + +@dataclass(frozen=True) +class Group: + """ + A set of annotations sharing a ``group_id``, plus the metadata derived from its first member. + + Always holds at least one annotation: a group with no members cannot be derived, and so + cannot be represented. + """ + + id: str + name: str + color: str + type: AnnotationType + is_global: bool + annotations: list[Annotation] + + @property + def is_hidden(self) -> bool: + """Whether every member's label is hidden — the state the group's eye icon shows.""" + return all(annotation.label_hidden for annotation in self.annotations) + + def __len__(self) -> int: + return len(self.annotations) + + +class AnnotationSet: + """An ordered, immutable collection of annotations. Every mutator returns a new set.""" + + def __init__(self, annotations: list[Annotation] | tuple[Annotation, ...] = ()) -> None: + self._annotations: tuple[Annotation, ...] = tuple(annotations) + + # ---------------------------------------------------------------------------------------- + # Boundaries + # ---------------------------------------------------------------------------------------- + + @classmethod + def from_dicts(cls, annotation_dicts: list[dict] | None) -> AnnotationSet: + """Hydrate from a ``dcc.Store`` payload, tolerating the ``None`` of an untouched store.""" + return cls([Annotation.from_dict(item) for item in (annotation_dicts or [])]) + + def to_dicts(self) -> list[dict]: + """Serialise back to a JSON-safe list for a ``dcc.Store`` payload.""" + return [annotation.to_dict() for annotation in self._annotations] + + @property + def annotations(self) -> list[Annotation]: + """The annotations, in order.""" + return list(self._annotations) + + def __iter__(self) -> Iterator[Annotation]: + return iter(self._annotations) + + def __len__(self) -> int: + return len(self._annotations) + + # ---------------------------------------------------------------------------------------- + # Derivation + # ---------------------------------------------------------------------------------------- + + def groups(self) -> list[Group]: + """Derive the groups, in first-seen order, each carrying its members in creation order.""" + members: dict[str, list[Annotation]] = {} + metadata: dict[str, Annotation] = {} + for annotation in self._annotations: + if not annotation.group_id: + continue + if annotation.group_id not in members: + members[annotation.group_id] = [] + metadata[annotation.group_id] = annotation + members[annotation.group_id].append(annotation) + + return [ + Group( + id=group_id, + name=first.group_name or "", + color=first.color, + type=first.type, + is_global=first.subplot_name is None, + annotations=members[group_id], + ) + for group_id, first in metadata.items() + ] + + def group(self, group_id: str) -> Group | None: + """Return the group with this id, or ``None`` if no annotation carries it.""" + return next((group for group in self.groups() if group.id == group_id), None) + + def ungrouped(self) -> list[Annotation]: + """Annotations belonging to no group, in order.""" + return [annotation for annotation in self._annotations if not annotation.group_id] + + # ---------------------------------------------------------------------------------------- + # Mutators — each returns a new set, leaving this one untouched + # ---------------------------------------------------------------------------------------- + + def with_added(self, annotation: Annotation) -> AnnotationSet: + """Append an annotation to the end of the set.""" + return AnnotationSet([*self._annotations, annotation]) + + def without(self, annotation_id: str) -> AnnotationSet: + """Drop the annotation with this id.""" + return AnnotationSet( + [annotation for annotation in self._annotations if annotation.id != annotation_id] + ) + + def without_group(self, group_id: str) -> AnnotationSet: + """Drop every annotation belonging to this group.""" + return AnnotationSet( + [annotation for annotation in self._annotations if annotation.group_id != group_id] + ) + + def with_label_toggled(self, annotation_id: str) -> AnnotationSet: + """Flip ``label_hidden`` on the annotation with this id.""" + return AnnotationSet( + [ + replace(annotation, label_hidden=not annotation.label_hidden) + if annotation.id == annotation_id + else annotation + for annotation in self._annotations + ] + ) + + def with_group_labels_toggled(self, group_id: str) -> AnnotationSet: + """ + Flip a whole group's labels to the state its eye icon is not showing. + + Pairing the target with :attr:`Group.is_hidden` here is what keeps the icon and the + button it sits on from drifting apart. + """ + group = self.group(group_id) + if group is None: + return self + target_hidden = not group.is_hidden + return AnnotationSet( + [ + replace(annotation, label_hidden=target_hidden) + if annotation.group_id == group_id + else annotation + for annotation in self._annotations + ] ) diff --git a/src/clinical_scope/dash_api/callbacks/annotation_callbacks.py b/src/clinical_scope/dash_api/callbacks/annotation_callbacks.py index 05779ce..2de77dc 100644 --- a/src/clinical_scope/dash_api/callbacks/annotation_callbacks.py +++ b/src/clinical_scope/dash_api/callbacks/annotation_callbacks.py @@ -23,7 +23,9 @@ ANNOTATION_COLORS, TIME_BASED_ANNOTATION_TYPES, Annotation, + AnnotationSet, AnnotationType, + Group, normalize_hex_color, ) from clinical_scope.dash_api.annotations.renderer import ( @@ -472,7 +474,7 @@ def handle_graph_click( group_name = mode.get("group_name", "") group_color = mode.get("group_color", ANNOTATION_COLORS[0]) group_is_global = mode.get("group_is_global", False) - current_annotations = list(annotations_raw or []) + annotation_set = AnnotationSet.from_dicts(annotations_raw) if annotation_type == AnnotationType.TIME_WINDOW.value: is_first, stored_x0, new_mode = _check_pending_x0(mode, x_str, plot_name) @@ -487,12 +489,13 @@ def handle_graph_click( ) data: dict[str, Any] = {"x0": stored_x0, "x1": x_str, "xaxis": xaxis_ref} - annotation = Annotation( - type=AnnotationType(annotation_type), + annotation = Annotation.create( + annotation_type=AnnotationType(annotation_type), plot_name=plot_name, label=group_name, color=group_color, - subplot_name=None if group_is_global else subplot_name, + is_global=group_is_global, + subplot_name=subplot_name, group_id=group_id, group_name=group_name, data=data, @@ -504,7 +507,7 @@ def handle_graph_click( ANNOTATION_MODAL_STYLE_HIDDEN, no_update_patches, "", - [*current_annotations, annotation.to_dict()], + annotation_set.with_added(annotation).to_dicts(), ) if annotation_type == AnnotationType.POINT.value: @@ -519,22 +522,20 @@ def handle_graph_click( if raw_t: with contextlib.suppress(Exception): data["t"] = pd.Timestamp(str(raw_t)).tz_localize(display_tz).isoformat() - ann_subplot = subplot_name else: data = {"x": x_str, "xaxis": xaxis_ref} - ann_subplot = None if group_is_global else subplot_name - annotation = Annotation( - type=AnnotationType(annotation_type), + annotation = Annotation.create( + annotation_type=AnnotationType(annotation_type), plot_name=plot_name, label=group_name, color=group_color, - subplot_name=ann_subplot, + is_global=group_is_global, + subplot_name=subplot_name, group_id=group_id, group_name=group_name, data=data, trace_metadata=trace_metadata or None, - label_hidden=annotation_type == AnnotationType.POINT.value, ) return ( mode, @@ -542,7 +543,7 @@ def handle_graph_click( ANNOTATION_MODAL_STYLE_HIDDEN, no_update_patches, "", - [*current_annotations, annotation.to_dict()], + annotation_set.with_added(annotation).to_dicts(), ) # --- Normal mode --- @@ -566,7 +567,6 @@ def handle_graph_click( "x0": stored_x0, "x1": x_str, "xaxis": xaxis_ref, - "auto_subplot_row": auto_subplot_row, "subplot_name": subplot_name, "suggested_color": suggested_color, "display_timezone": display_tz, @@ -587,7 +587,6 @@ def handle_graph_click( "plot_name": plot_name, "x": x_str, "xaxis": xaxis_ref, - "auto_subplot_row": auto_subplot_row, "subplot_name": subplot_name, "suggested_color": suggested_color, "display_timezone": display_tz, @@ -639,11 +638,11 @@ def update_modal_ui(modal_data: dict) -> tuple[str, str, list, str]: prevent_initial_call=True, ) def toggle_global_checkbox_visibility(modal_data: dict) -> dict: - """Hide the global checkbox for POINT annotations (they're always subplot-specific).""" + """Hide the global checkbox for annotations that cannot be global (points).""" if not modal_data: raise PreventUpdate - annotation_type = modal_data.get("type", "") - if annotation_type == AnnotationType.POINT.value: + # StrEnum members hash as their value, so the raw payload string tests directly. + if modal_data.get("type", "") not in TIME_BASED_ANNOTATION_TYPES: return {"marginBottom": "20px", "display": "none"} return {"marginBottom": "20px"} @@ -730,7 +729,6 @@ def create_annotation( annotation_type = AnnotationType(modal_data["type"]) is_global = "global" in (global_checkbox or []) - subplot_name = None if is_global else modal_data.get("subplot_name") color = normalize_hex_color(color) if annotation_type == AnnotationType.TIME_EVENT: @@ -753,18 +751,18 @@ def create_annotation( else: raise PreventUpdate - annotation = Annotation( - type=annotation_type, + annotation = Annotation.create( + annotation_type=annotation_type, plot_name=modal_data["plot_name"], label=label or "", color=color, - subplot_name=subplot_name, + is_global=is_global, + subplot_name=modal_data.get("subplot_name"), data=data, trace_metadata=modal_data.get("trace_metadata"), - label_hidden=annotation_type == AnnotationType.POINT, ) - new_annotations = [*(annotations_raw or []), annotation.to_dict()] + new_annotations = AnnotationSet.from_dicts(annotations_raw).with_added(annotation).to_dicts() new_mode = {**(mode or default_mode()), "pending_x0": None, "pending_plot_name": None} return new_annotations, new_mode, ANNOTATION_MODAL_STYLE_HIDDEN @@ -817,8 +815,8 @@ def render_annotations( display_tz = display_timezone or cst.DISPLAY_TIMEZONE annotations = [ - normalize_annotation_for_display(Annotation.from_dict(annotation_dict), display_tz) - for annotation_dict in (annotations_raw or []) + normalize_annotation_for_display(annotation, display_tz) + for annotation in AnnotationSet.from_dicts(annotations_raw) ] mode = mode or default_mode() pending_x0 = mode.get("pending_x0") @@ -873,26 +871,21 @@ def render_annotations( # --------------------------------------------------------------------------- -def _group_header_row( - group: dict, - annotation_count: int, - is_expanded: bool, - is_hidden: bool = False, -) -> html.Div: +def _group_header_row(group: Group, is_expanded: bool) -> html.Div: """Build a collapsible group header row with per-group action buttons.""" toggle_icon = "▼" if is_expanded else "▶" - labels_label = "Labels: off" if is_hidden else "Labels: on" - labels_style = {**_SMALL_BTN, "backgroundColor": "#6c757d" if is_hidden else "#adb5bd"} + labels_label = "Labels: off" if group.is_hidden else "Labels: on" + labels_style = {**_SMALL_BTN, "backgroundColor": "#6c757d" if group.is_hidden else "#adb5bd"} - type_icon = _TYPE_ICONS.get(group["type"], "?") - type_label = _TYPE_LABELS.get(group["type"], group["type"]) + type_icon = _TYPE_ICONS.get(group.type, "?") + type_label = _TYPE_LABELS.get(group.type, group.type) # Scope badge only for time-based annotations (global/subplot distinction is # meaningless for points which are always subplot-specific). scope_badge = None - if AnnotationType(group["type"]) in TIME_BASED_ANNOTATION_TYPES: - scope_text = "Global" if group["is_global"] else "Subplot" - scope_color = "#5a9fd4" if group["is_global"] else "#e67e00" + if group.type in TIME_BASED_ANNOTATION_TYPES: + scope_text = "Global" if group.is_global else "Subplot" + scope_color = "#5a9fd4" if group.is_global else "#e67e00" scope_badge = html.Span( scope_text, style={ @@ -909,7 +902,7 @@ def _group_header_row( [ html.Button( toggle_icon, - id={"type": "group-toggle-btn", "id": group["id"]}, + id={"type": "group-toggle-btn", "id": group.id}, n_clicks=0, style={ "background": "none", @@ -924,7 +917,7 @@ def _group_header_row( html.Span( type_icon, style={ - "color": group["color"], + "color": group.color, "fontSize": "16px", "fontWeight": "bold", "minWidth": "16px", @@ -933,7 +926,7 @@ def _group_header_row( }, ), html.Span( - group["name"], + group.name, style={"fontWeight": "bold", "fontSize": "13px", "flex": 1, "color": "#333"}, ), html.Span( @@ -942,24 +935,24 @@ def _group_header_row( ), *([scope_badge] if scope_badge else []), html.Span( - f"({annotation_count})", + f"({len(group)})", style={"color": "#888", "fontSize": "12px", "flexShrink": 0}, ), html.Button( "▶ Continue", - id={"type": "group-continue-btn", "id": group["id"]}, + id={"type": "group-continue-btn", "id": group.id}, n_clicks=0, style=_SMALL_BTN, ), html.Button( labels_label, - id={"type": "group-hide-btn", "id": group["id"]}, + id={"type": "group-hide-btn", "id": group.id}, n_clicks=0, style=labels_style, ), html.Button( "Delete all", - id={"type": "group-delete-btn", "id": group["id"]}, + id={"type": "group-delete-btn", "id": group.id}, n_clicks=0, style={**_SMALL_BTN, "backgroundColor": "#dc3545"}, ), @@ -989,63 +982,28 @@ def update_annotation_list( display_timezone: str | None, ) -> tuple[list | html.Div, str]: """Rebuild the annotation list, always sorted by group with collapsible group sections.""" - annotations = [ - Annotation.from_dict(annotation_dict) for annotation_dict in (annotations_raw or []) - ] - if not annotations: + annotation_set = AnnotationSet.from_dicts(annotations_raw) + if not len(annotation_set): return [], "" expanded_set = set(expanded_groups or []) - - # Derive group metadata from the annotations themselves. - # is_global is encoded as subplot_name is None on the first annotation in the group. - groups_by_id: dict[str, dict] = {} - for annotation in annotations: - if annotation.group_id and annotation.group_id not in groups_by_id: - groups_by_id[annotation.group_id] = { - "id": annotation.group_id, - "name": annotation.group_name or "", - "color": annotation.color, - "type": annotation.type.value, - "is_global": annotation.subplot_name is None, - } - - # Bucket annotations: grouped (preserve creation order within group) + ungrouped - grouped: dict[str, list[Annotation]] = {} - ungrouped: list[Annotation] = [] - group_order: list[str] = [] # first-seen order of group IDs - - for annotation in annotations: - if annotation.group_id and annotation.group_id in groups_by_id: - if annotation.group_id not in grouped: - grouped[annotation.group_id] = [] - group_order.append(annotation.group_id) - grouped[annotation.group_id].append(annotation) - else: - ungrouped.append(annotation) + groups = annotation_set.groups() + ungrouped = annotation_set.ungrouped() rows: list = [] - for group_id in group_order: - group = groups_by_id[group_id] - group_annotations = grouped[group_id] - is_expanded = group_id in expanded_set - is_hidden = bool(group_annotations) and all( - annotation.label_hidden for annotation in group_annotations - ) - - rows.append(_group_header_row(group, len(group_annotations), is_expanded, is_hidden)) + for group in groups: + is_expanded = group.id in expanded_set + rows.append(_group_header_row(group, is_expanded)) if is_expanded: rows.extend( - _annotation_list_row( - annotation, group_name=group["name"], display_tz=display_timezone - ) - for annotation in group_annotations + _annotation_list_row(annotation, group_name=group.name, display_tz=display_timezone) + for annotation in group.annotations ) if ungrouped: - if group_order: + if groups: rows.append( html.Div( "Other annotations", @@ -1064,7 +1022,8 @@ def update_annotation_list( for annotation in ungrouped ) - count_text = f"{len(annotations)} annotation{'s' if len(annotations) != 1 else ''}" + count = len(annotation_set) + count_text = f"{count} annotation{'s' if count != 1 else ''}" panel = html.Div( [ html.Div( @@ -1136,19 +1095,7 @@ def toggle_group_labels(_n: list, annotations_raw: list) -> list: if triggered_id is None: raise PreventUpdate group_id = triggered_id["id"] - annotations = [ - Annotation.from_dict(annotation_dict) for annotation_dict in (annotations_raw or []) - ] - group_annotations = [ - annotation for annotation in annotations if annotation.group_id == group_id - ] - target_hidden = any(not annotation.label_hidden for annotation in group_annotations) - return [ - {**annotation_dict, "label_hidden": target_hidden} - if annotation_dict.get("group_id") == group_id - else annotation_dict - for annotation_dict in (annotations_raw or []) - ] + return AnnotationSet.from_dicts(annotations_raw).with_group_labels_toggled(group_id).to_dicts() # --------------------------------------------------------------------------- @@ -1181,11 +1128,7 @@ def delete_group( if triggered_id is None: raise PreventUpdate group_id = triggered_id["id"] - new_annotations = [ - annotation_dict - for annotation_dict in (annotations_raw or []) - if annotation_dict.get("group_id") != group_id - ] + new_annotations = AnnotationSet.from_dicts(annotations_raw).without_group(group_id).to_dicts() new_expanded = [ expanded_group_id for expanded_group_id in (expanded_groups or []) @@ -1234,9 +1177,7 @@ def save_annotations_cb(_n: int, annotations_raw: list, folder: str) -> tuple[st if not folder: return "No patient folder loaded.", BUTTON_ANNOTATION_SAVE try: - annotations = [ - Annotation.from_dict(annotation_dict) for annotation_dict in (annotations_raw or []) - ] + annotations = AnnotationSet.from_dicts(annotations_raw).annotations save_annotations(annotations, folder) return f"Saved ({len(annotations)})", { **BUTTON_ANNOTATION_SAVE, @@ -1293,11 +1234,7 @@ def delete_annotation(n_clicks_list: list, annotations_raw: list) -> list: if triggered_id is None or not any(n_clicks_list): raise PreventUpdate annotation_id = triggered_id["id"] - return [ - annotation_dict - for annotation_dict in (annotations_raw or []) - if annotation_dict["id"] != annotation_id - ] + return AnnotationSet.from_dicts(annotations_raw).without(annotation_id).to_dicts() # --------------------------------------------------------------------------- @@ -1389,28 +1326,18 @@ def activate_group( if not any(_continue_list): raise PreventUpdate group_id = triggered_id["id"] - reference_annotation = next( - ( - annotation_dict - for annotation_dict in (annotations_raw or []) - if annotation_dict.get("group_id") == group_id - ), - None, - ) - if not reference_annotation: + group = AnnotationSet.from_dicts(annotations_raw).group(group_id) + if group is None: raise PreventUpdate - group_name = reference_annotation.get("group_name", "") - group_color = reference_annotation.get("color", ANNOTATION_COLORS[0]) - group_type = AnnotationType(reference_annotation["type"]) - group_is_global = reference_annotation.get("subplot_name") is None + group_name = group.name new_mode = { **(mode or default_mode()), "active": True, - "type": group_type.value, + "type": group.type.value, "group_id": group_id, "group_name": group_name, - "group_color": group_color, - "group_is_global": group_is_global, + "group_color": group.color, + "group_is_global": group.is_global, "pending_x0": None, "pending_plot_name": None, } @@ -1455,9 +1382,4 @@ def toggle_annotation_label(_n: list, annotations_raw: list) -> list: if triggered_id is None: raise PreventUpdate annotation_id = triggered_id["id"] - return [ - {**annotation_dict, "label_hidden": not annotation_dict.get("label_hidden", False)} - if annotation_dict["id"] == annotation_id - else annotation_dict - for annotation_dict in (annotations_raw or []) - ] + return AnnotationSet.from_dicts(annotations_raw).with_label_toggled(annotation_id).to_dicts() diff --git a/tests/unit/test_annotation_model.py b/tests/unit/test_annotation_model.py new file mode 100644 index 0000000..222b6cf --- /dev/null +++ b/tests/unit/test_annotation_model.py @@ -0,0 +1,383 @@ +""" +Unit tests for Annotation, Group and AnnotationSet in annotations/model.py. + +These exercise the logic that used to live inline in the annotation callbacks, where it +could not be reached without a Dash context. +""" + +from __future__ import annotations + +import pytest + +from clinical_scope.dash_api.annotations.model import ( + Annotation, + AnnotationSet, + AnnotationType, + Group, +) + +TWO = 2 +THREE = 3 + + +# ================================================================================================== +# Helpers +# ================================================================================================== + + +def make_annotation( + annotation_id: str, + *, + group_id: str | None = None, + group_name: str | None = None, + color: str = "#e74c3c", + annotation_type: AnnotationType = AnnotationType.TIME_EVENT, + subplot_name: str | None = "Pressure", + label_hidden: bool = False, +) -> Annotation: + """Build an annotation with an explicit id so ordering assertions stay readable.""" + return Annotation( + id=annotation_id, + type=annotation_type, + plot_name="time_series", + data={"x": "2024-01-01T00:00:00+00:00"}, + group_id=group_id, + group_name=group_name, + color=color, + subplot_name=subplot_name, + label_hidden=label_hidden, + ) + + +# ================================================================================================== +# Group derivation +# ================================================================================================== + + +class TestGroupDerivation: + """Groups are rebuilt from the annotations alone; no group metadata is ever persisted.""" + + def test_groups_are_returned_in_first_seen_order(self): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g2", group_name="Second"), + make_annotation("b", group_id="g1", group_name="First"), + make_annotation("c", group_id="g2", group_name="Second"), + ] + ) + assert [group.id for group in annotation_set.groups()] == ["g2", "g1"] + + def test_group_metadata_comes_from_the_first_member(self): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1", group_name="Weaning", color="#3498db"), + make_annotation("b", group_id="g1", group_name="IGNORED", color="#000000"), + ] + ) + group = annotation_set.groups()[0] + assert group.name == "Weaning" + assert group.color == "#3498db" + + def test_members_keep_creation_order(self): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1"), + make_annotation("b"), + make_annotation("c", group_id="g1"), + ] + ) + assert [a.id for a in annotation_set.groups()[0].annotations] == ["a", "c"] + + def test_is_global_is_encoded_as_a_missing_subplot_name(self): + grouped = AnnotationSet([make_annotation("a", group_id="g1", subplot_name=None)]) + assert grouped.groups()[0].is_global is True + scoped = AnnotationSet([make_annotation("a", group_id="g1", subplot_name="Flow")]) + assert scoped.groups()[0].is_global is False + + def test_ungrouped_holds_only_annotations_without_a_group(self): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1"), + make_annotation("b"), + make_annotation("c"), + ] + ) + assert [a.id for a in annotation_set.ungrouped()] == ["b", "c"] + + def test_group_lookup_returns_none_for_an_unknown_id(self): + annotation_set = AnnotationSet([make_annotation("a", group_id="g1")]) + assert annotation_set.group("g1") is not None + assert annotation_set.group("nope") is None + + def test_a_group_can_never_be_empty(self): + """Every derived group carries at least one member, so `is_hidden` is always defined.""" + annotation_set = AnnotationSet( + [make_annotation("a", group_id="g1"), make_annotation("b", group_id="g2")] + ) + assert all(len(group) >= 1 for group in annotation_set.groups()) + + +# ================================================================================================== +# Label visibility +# ================================================================================================== + + +class TestLabelVisibility: + """The group eye icon and the button that flips it must read the same rule.""" + + def test_group_is_hidden_only_when_every_member_is_hidden(self): + partly = AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=True), + make_annotation("b", group_id="g1", label_hidden=False), + ] + ) + assert partly.groups()[0].is_hidden is False + + fully = AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=True), + make_annotation("b", group_id="g1", label_hidden=True), + ] + ) + assert fully.groups()[0].is_hidden is True + + @pytest.mark.parametrize("start_hidden", [True, False]) + def test_toggling_a_group_flips_it_to_the_state_the_icon_is_not_showing(self, start_hidden): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=start_hidden), + make_annotation("b", group_id="g1", label_hidden=start_hidden), + ] + ) + before = annotation_set.groups()[0].is_hidden + after = annotation_set.with_group_labels_toggled("g1").groups()[0].is_hidden + assert after is not before + + def test_a_mixed_group_hides_every_member(self): + """Any visible label means the group reads as shown, so one click hides all of it.""" + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=True), + make_annotation("b", group_id="g1", label_hidden=False), + ] + ) + toggled = annotation_set.with_group_labels_toggled("g1") + assert all(annotation.label_hidden for annotation in toggled) + + def test_toggling_a_group_leaves_other_groups_alone(self): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=False), + make_annotation("b", group_id="g2", label_hidden=False), + ] + ) + toggled = annotation_set.with_group_labels_toggled("g1") + assert toggled.group("g2").is_hidden is False + + def test_toggling_an_unknown_group_is_a_no_op(self): + annotation_set = AnnotationSet([make_annotation("a", group_id="g1")]) + assert annotation_set.with_group_labels_toggled("nope").to_dicts() == ( + annotation_set.to_dicts() + ) + + def test_toggling_one_annotation_flips_only_that_one(self): + annotation_set = AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=False), + make_annotation("b", group_id="g1", label_hidden=False), + ] + ) + toggled = annotation_set.with_label_toggled("a") + assert [annotation.label_hidden for annotation in toggled] == [True, False] + + +# ================================================================================================== +# Mutators return new sets +# ================================================================================================== + + +class TestImmutability: + """Every mutator returns a new set; callbacks never mutate the store payload in place.""" + + @pytest.fixture + def annotation_set(self) -> AnnotationSet: + return AnnotationSet( + [ + make_annotation("a", group_id="g1", label_hidden=False), + make_annotation("b", group_id="g1", label_hidden=False), + make_annotation("c"), + ] + ) + + @pytest.mark.parametrize( + ("method", "argument"), + [ + ("without", "a"), + ("without_group", "g1"), + ("with_label_toggled", "a"), + ("with_group_labels_toggled", "g1"), + ], + ) + def test_source_set_is_unchanged(self, annotation_set, method, argument): + before = annotation_set.to_dicts() + getattr(annotation_set, method)(argument) + assert annotation_set.to_dicts() == before + + def test_without_drops_one_annotation(self, annotation_set): + assert [a.id for a in annotation_set.without("a")] == ["b", "c"] + + def test_without_group_drops_every_member(self, annotation_set): + assert [a.id for a in annotation_set.without_group("g1")] == ["c"] + + def test_with_added_appends_at_the_end(self, annotation_set): + grown = annotation_set.with_added(make_annotation("d")) + assert [a.id for a in grown] == ["a", "b", "c", "d"] + assert len(annotation_set) == THREE + + +# ================================================================================================== +# Serialisation boundary +# ================================================================================================== + + +class TestOpenSchema: + """Keys the app does not own survive a round-trip (ADR-0012).""" + + def test_unknown_keys_survive_a_round_trip(self): + raw = { + "id": "a", + "type": "time_event", + "plot_name": "time_series", + "data": {"x": "2024-01-01T00:00:00+00:00"}, + "reviewer": "dr-who", + "confidence": 0.8, + } + round_tripped = Annotation.from_dict(raw).to_dict() + assert round_tripped["reviewer"] == "dr-who" + assert round_tripped["confidence"] == pytest.approx(0.8) + + def test_unknown_keys_survive_a_mutation(self): + raw = [ + { + "id": "a", + "type": "time_event", + "plot_name": "time_series", + "data": {}, + "group_id": "g1", + "reviewer": "dr-who", + }, + {"id": "b", "type": "time_event", "plot_name": "time_series", "data": {}}, + ] + survivor = AnnotationSet.from_dicts(raw).without("b").to_dicts()[0] + assert survivor["reviewer"] == "dr-who" + + def test_a_known_field_can_never_be_shadowed(self): + """`extra` is splatted first, so a stale duplicate key cannot overwrite a real field.""" + annotation = Annotation.from_dict( + {"id": "a", "type": "time_event", "plot_name": "time_series", "data": {}} + ) + annotation.extra["id"] = "hijacked" + assert annotation.to_dict()["id"] == "a" + + def test_an_annotation_with_no_extra_keys_serialises_the_owned_set(self): + raw = {"id": "a", "type": "time_event", "plot_name": "time_series", "data": {}} + assert Annotation.from_dict(raw).extra == {} + + def test_from_dicts_tolerates_an_untouched_store(self): + assert len(AnnotationSet.from_dicts(None)) == 0 + + +# ================================================================================================== +# Creation-time defaulting +# ================================================================================================== + + +class TestAnnotationCreate: + """`create` interprets; `from_dict` transcribes.""" + + def test_a_point_is_never_global(self): + annotation = Annotation.create( + annotation_type=AnnotationType.POINT, + plot_name="time_series", + data={}, + is_global=True, + subplot_name="Pressure", + ) + assert annotation.subplot_name == "Pressure" + + def test_a_point_starts_with_its_label_hidden(self): + annotation = Annotation.create( + annotation_type=AnnotationType.POINT, plot_name="time_series", data={} + ) + assert annotation.label_hidden is True + + def test_an_explicit_label_hidden_wins_over_the_point_default(self): + annotation = Annotation.create( + annotation_type=AnnotationType.POINT, + plot_name="time_series", + data={}, + label_hidden=False, + ) + assert annotation.label_hidden is False + + def test_a_time_event_keeps_its_label_shown(self): + annotation = Annotation.create( + annotation_type=AnnotationType.TIME_EVENT, plot_name="time_series", data={} + ) + assert annotation.label_hidden is False + + def test_a_global_time_event_drops_its_subplot_name(self): + annotation = Annotation.create( + annotation_type=AnnotationType.TIME_EVENT, + plot_name="time_series", + data={}, + is_global=True, + subplot_name="Pressure", + ) + assert annotation.subplot_name is None + + def test_a_scoped_time_event_keeps_its_subplot_name(self): + annotation = Annotation.create( + annotation_type=AnnotationType.TIME_EVENT, + plot_name="time_series", + data={}, + is_global=False, + subplot_name="Pressure", + ) + assert annotation.subplot_name == "Pressure" + + def test_deserialisation_does_not_apply_creation_defaults(self): + """A point whose label the user un-hid must load back un-hidden.""" + raw = { + "id": "a", + "type": "point", + "plot_name": "time_series", + "data": {}, + "label_hidden": False, + "subplot_name": None, + } + annotation = Annotation.from_dict(raw) + assert annotation.label_hidden is False + assert annotation.subplot_name is None + + +# ================================================================================================== +# Group is a value object +# ================================================================================================== + + +def test_group_length_is_its_member_count(): + annotation_set = AnnotationSet( + [make_annotation("a", group_id="g1"), make_annotation("b", group_id="g1")] + ) + assert len(annotation_set.groups()[0]) == TWO + + +def test_group_carries_the_annotation_type_as_an_enum(): + annotation_set = AnnotationSet( + [make_annotation("a", group_id="g1", annotation_type=AnnotationType.TIME_WINDOW)] + ) + group = annotation_set.groups()[0] + assert isinstance(group, Group) + assert group.type is AnnotationType.TIME_WINDOW From e15f00cfa2dad41310b3528fec1b5003c3a77fb0 Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Wed, 26 Aug 2026 12:38:53 +0200 Subject: [PATCH 4/9] Split io/file_utils along its real seams file_utils.py held seven unrelated concerns behind one name and had the second-highest fan-in in the package. Its call graph is a stack, not a partition: parquet pruning calls into datetime detection, sharing four helpers. Split accordingly, one file per ADR: io/time_axis.py ADR-0004's rule plus both its adapters io/parquet_pruning.py ADR-0007: plan the prunings, then execute one io/discovery.py find_files and the folder predicates io/column_patterns.py one definition of what a `*` matches io/export.py save_df + print_out_figure Schema-only detection stays with full-frame detection: the two must pick the same column, and that invariant is only cheap to hold in one file. Provenance is now structural. `read_parquet_pruned(index_is_time_axis=)` became two front doors -- read_parquet_pruned for a file of unknown origin, read_cache_pruned for one we wrote -- so ADR-0010's guarantee is carried by the name a caller reaches for instead of by a comment saying "no other caller may claim that". _build_datetime_row_filters now converts a bound into the column's tz before dropping the label, rather than trusting _pushdown_bounds to have converted already. Same behaviour, one less contract held at a distance; the branch had no coverage, and now has a test that fails without it. Also removed: load_csv_with_datetime_index (no callers), load_parquet_with_datetime_index (both callers used the bare path, now inlined), and timezone.py's dead _first_last_timestamp twin. Hard cut, no compatibility shim. Test assertions are unchanged except where a deleted wrapper had to become the code it contained. Co-Authored-By: Claude Opus 5 --- ...004-validate-datetime-column-candidates.md | 4 + ...0010-load-transcribes-format-interprets.md | 4 + src/clinical_scope/constants.py | 2 +- src/clinical_scope/datasource/base.py | 18 +- .../datasource/formatting/timezone.py | 10 - src/clinical_scope/datasource/registry.py | 2 +- .../sources/edf/find_load_format.py | 2 +- .../sources/eit/find_load_format.py | 2 +- .../fluxmed_parameters/find_load_format.py | 4 +- .../fluxmed_signals/find_load_format.py | 7 +- .../find_load_format.py | 2 +- .../mindray_respi_waves/find_load_format.py | 2 +- .../sources/mindray_scope/find_load_format.py | 2 +- .../sources/other/find_load_format.py | 9 +- .../sources/servo_u/find_load_format.py | 2 +- src/clinical_scope/io/column_patterns.py | 88 ++ src/clinical_scope/io/discovery.py | 164 ++++ src/clinical_scope/io/export.py | 60 ++ src/clinical_scope/io/file_utils.py | 784 ------------------ src/clinical_scope/io/parquet_pruning.py | 227 +++++ src/clinical_scope/io/time_axis.py | 341 ++++++++ src/clinical_scope/signal_container.py | 25 +- tests/datasource/test_column_pruning.py | 13 +- tests/datasource/test_datetime_pushdown.py | 145 ++-- tests/unit/test_data_processor.py | 4 +- .../unit/test_deduplicate_then_sort_index.py | 2 +- tests/unit/test_find_files.py | 2 +- tests/unit/test_junk_files.py | 2 +- tests/unit/test_signal_container.py | 2 +- ...find_datetime_col.py => test_time_axis.py} | 56 +- 30 files changed, 1049 insertions(+), 938 deletions(-) create mode 100644 src/clinical_scope/io/column_patterns.py create mode 100644 src/clinical_scope/io/discovery.py create mode 100644 src/clinical_scope/io/export.py delete mode 100644 src/clinical_scope/io/file_utils.py create mode 100644 src/clinical_scope/io/parquet_pruning.py create mode 100644 src/clinical_scope/io/time_axis.py rename tests/unit/{test_find_datetime_col.py => test_time_axis.py} (86%) diff --git a/docs/adr/0004-validate-datetime-column-candidates.md b/docs/adr/0004-validate-datetime-column-candidates.md index d43bfc4..a0a4646 100644 --- a/docs/adr/0004-validate-datetime-column-candidates.md +++ b/docs/adr/0004-validate-datetime-column-candidates.md @@ -46,3 +46,7 @@ Migrated to the shared detector: `load_csv_with_datetime_index`, the new `load_p - **Harder / accepted trade-offs:** raising the parse threshold from `other`'s previous 50% to 90% means some previously-tolerated messy files (mostly-garbage timestamp column) now fail outright instead of loading with a partially-broken time axis — intentional, since that was never actually useful data. Per-datasource candidate-name priority is gone; if a real device's column repeatedly loses to a wrong candidate under the universal list, revisit. - **Explicitly deferred:** combining separate date + time columns into one datetime (no current datasource needs it). Epoch detection beyond nanoseconds (s/ms/µs) — revisit if a real datasource surfaces raw second/millisecond epoch timestamps. - **Not fully resolved, by design:** a file with several genuinely-plausible time columns (like the anesthesia record above) may land on any of the ones that survive validation + the uniqueness/`utc` tiebreak, not necessarily the single "best" one a human would pick. Per [0001](0001-diagnose-dont-resolve-patient-folders.md)'s precedent, deep disambiguation of a badly-overloaded file is left to the user (e.g. pre-pruning columns before use), not solved inside the detector. + +## Update — 2026-08-26 + +The detector now has a file of its own, `io/time_axis.py`, holding this rule and nothing else. It exposes two adapters over the same tiers — `detect_time_axis_in_frame` for a loaded frame, `detect_time_axis_in_parquet` for a file read only by schema and bounded sample. They must pick the same column, so they stay in one module; `_is_numeric_pa_type`'s agreement with `pd.api.types.is_numeric_dtype` is the tripwire for that. diff --git a/docs/adr/0010-load-transcribes-format-interprets.md b/docs/adr/0010-load-transcribes-format-interprets.md index 05840fc..0f3ef46 100644 --- a/docs/adr/0010-load-transcribes-format-interprets.md +++ b/docs/adr/0010-load-transcribes-format-interprets.md @@ -76,3 +76,7 @@ The rule is now carried by the signature. `_load(file_path)` takes the file and The `configured_field_display` guard the Consequences flagged for removal is gone — the *guard*, not the parameter. The fresh-load branch that restored `field_display` for non-caching sources could never fire under this rule and has been deleted; the parameter survives on the quick-load branch, where it still prunes the cache read that serves `inspect(configured_columns_only=True)`. One small behaviour note: the base declines to write a cache for an empty frame, and does not create the output folder for one either. Four sources' early returns already skipped the save for that reason; the rule is now uniform and stated once. + +## Update — 2026-08-26 + +The cache's provenance is now expressed by which function a caller reaches for. `io/file_utils.py` split along its concerns — `time_axis`, `parquet_pruning`, `discovery`, `column_patterns`, `export` — and the `index_is_time_axis` flag that carried this ADR's guarantee became a second front door: `read_cache_pruned` for a file we wrote, `read_parquet_pruned` for one we did not. "No other caller may claim that" was a comment at the call site; it is now the absence of any way to say it. The path cited in the Consequences above, `io/file_utils.py:568-571`, is today `_pruning_plan` in `io/parquet_pruning.py`. diff --git a/src/clinical_scope/constants.py b/src/clinical_scope/constants.py index 040dae4..7dbac1f 100644 --- a/src/clinical_scope/constants.py +++ b/src/clinical_scope/constants.py @@ -95,7 +95,7 @@ class DatetimeColumnDetection: class ParquetPushdownKind: - """How a detected parquet datetime column can carry a row predicate (see io/file_utils).""" + """How a detected parquet datetime column can carry a row predicate (see io/parquet_pruning).""" TIMESTAMP = "timestamp" # real timestamp column — bounds filter it directly EPOCH_NS = "epoch_ns" # numeric nanoseconds since epoch — bounds convert to int first diff --git a/src/clinical_scope/datasource/base.py b/src/clinical_scope/datasource/base.py index 32c5f03..54c9c79 100644 --- a/src/clinical_scope/datasource/base.py +++ b/src/clinical_scope/datasource/base.py @@ -28,13 +28,10 @@ _column_infos, ) from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import ( - find_files, - folder_name_matches_keywords, - make_column_selector, - read_parquet_pruned, - save_df, -) +from clinical_scope.io.column_patterns import make_column_selector +from clinical_scope.io.discovery import find_files, folder_name_matches_keywords +from clinical_scope.io.export import save_df +from clinical_scope.io.parquet_pruning import read_cache_pruned from clinical_scope.io.paths import get_datasource_cache_path from clinical_scope.signal_container import DisplayFallbacks, Signal @@ -255,13 +252,10 @@ def _quick_load( """ database_options_specific = database_options_specific or {} - return read_parquet_pruned( + return read_cache_pruned( path_dataframe, compute_bounds=cls._make_bounds_computer(patient_options, database_options_specific), select_columns=make_column_selector(database_options_specific), - # A cache is a file we wrote, so its index is the time axis by construction whatever - # its dtype (EIT's is float64 fractional days). No other caller may claim that. - index_is_time_axis=True, ) @classmethod @@ -596,7 +590,7 @@ def extract( Args: patient_options: Patient-specific options (same as :meth:`main`). save_path: If given, save the formatted DataFrame to this path using - :func:`io.file_utils.save_df` (supports ``.csv`` and ``.parquet``). + :func:`io.export.save_df` (supports ``.csv`` and ``.parquet``). Returns: Formatted ``pd.DataFrame``, or ``None`` if the file was not found or diff --git a/src/clinical_scope/datasource/formatting/timezone.py b/src/clinical_scope/datasource/formatting/timezone.py index a251e29..4bd9627 100644 --- a/src/clinical_scope/datasource/formatting/timezone.py +++ b/src/clinical_scope/datasource/formatting/timezone.py @@ -371,13 +371,3 @@ def _date_range(df: pd.DataFrame) -> tuple[str, str] | None: return (fmt_ts(df.index.min()), fmt_ts(df.index.max())) except Exception: # noqa: BLE001 return None - - -def _first_last_timestamp(df: pd.DataFrame, column: str) -> tuple[str | None, str | None]: - """Return (first, last) compact timestamp strings for valid (non-NaN) values in a column.""" - if column not in df.columns: - return None, None - valid_index = df.index[df[column].notna()] - if valid_index.empty: - return None, None - return fmt_ts(valid_index.min()), fmt_ts(valid_index.max()) diff --git a/src/clinical_scope/datasource/registry.py b/src/clinical_scope/datasource/registry.py index 596fc8a..12e6086 100644 --- a/src/clinical_scope/datasource/registry.py +++ b/src/clinical_scope/datasource/registry.py @@ -26,7 +26,7 @@ ) from clinical_scope.datasource.sources.other import find_load_format as _other from clinical_scope.datasource.sources.servo_u import find_load_format as _servo_u -from clinical_scope.io.file_utils import ( +from clinical_scope.io.discovery import ( folder_has_real_content, folder_name_matches_keywords, is_junk_file, diff --git a/src/clinical_scope/datasource/sources/edf/find_load_format.py b/src/clinical_scope/datasource/sources/edf/find_load_format.py index 5bc6062..ae6b332 100644 --- a/src/clinical_scope/datasource/sources/edf/find_load_format.py +++ b/src/clinical_scope/datasource/sources/edf/find_load_format.py @@ -9,7 +9,7 @@ import clinical_scope.datasource.sources.edf.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index logger = logging.getLogger(__name__) diff --git a/src/clinical_scope/datasource/sources/eit/find_load_format.py b/src/clinical_scope/datasource/sources/eit/find_load_format.py index 9a90014..e381a68 100644 --- a/src/clinical_scope/datasource/sources/eit/find_load_format.py +++ b/src/clinical_scope/datasource/sources/eit/find_load_format.py @@ -9,7 +9,7 @@ import clinical_scope.datasource.sources.eit.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index logger = logging.getLogger(__name__) diff --git a/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py b/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py index 4bf4a6a..7cace98 100644 --- a/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py +++ b/src/clinical_scope/datasource/sources/fluxmed_parameters/find_load_format.py @@ -8,7 +8,7 @@ import clinical_scope.datasource.sources.fluxmed_parameters.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import load_parquet_with_datetime_index +from clinical_scope.io.time_axis import set_datetime_index logger = logging.getLogger(__name__) @@ -30,7 +30,7 @@ class FluxmedParametersDataSource(DataSourceBase): @time_it def _load(cls, file_path: Path) -> pd.DataFrame: if file_path.suffix.lower() == ".parquet": - df = load_parquet_with_datetime_index(file_path) + df = set_datetime_index(pd.read_parquet(file_path)) elif file_path.suffix.lower() in [".txt", ".csv"]: filename = file_path.name match = re.search(r"(\d+_\d+_\d+-\d+_\d+_\d+)", filename) diff --git a/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py b/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py index 0ceb966..f09a2cf 100644 --- a/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py +++ b/src/clinical_scope/datasource/sources/fluxmed_signals/find_load_format.py @@ -8,10 +8,7 @@ import clinical_scope.datasource.sources.fluxmed_signals.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import ( - deduplicate_then_sort_index, - load_parquet_with_datetime_index, -) +from clinical_scope.io.time_axis import deduplicate_then_sort_index, set_datetime_index logger = logging.getLogger(__name__) @@ -33,7 +30,7 @@ class FluxmedSignalsDataSource(DataSourceBase): @time_it def _load(cls, file_path: Path) -> pd.DataFrame: if file_path.suffix.lower() == ".parquet": - df = load_parquet_with_datetime_index(file_path) + df = set_datetime_index(pd.read_parquet(file_path)) elif file_path.suffix.lower() in [".txt", ".csv"]: filename = file_path.name match = re.search(r"(\d+_\d+_\d+-\d+_\d+_\d+)", filename) diff --git a/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py b/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py index 845a74d..b0092e5 100644 --- a/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py +++ b/src/clinical_scope/datasource/sources/mindray_respi_numerics/find_load_format.py @@ -7,7 +7,7 @@ import clinical_scope.datasource.sources.mindray_respi_numerics.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index logger = logging.getLogger(__name__) diff --git a/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py b/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py index 38becff..4d70c4b 100644 --- a/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py +++ b/src/clinical_scope/datasource/sources/mindray_respi_waves/find_load_format.py @@ -9,7 +9,7 @@ import clinical_scope.datasource.sources.mindray_respi_waves.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index logger = logging.getLogger(__name__) diff --git a/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py b/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py index 2ee7157..6133983 100644 --- a/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py +++ b/src/clinical_scope/datasource/sources/mindray_scope/find_load_format.py @@ -10,7 +10,7 @@ import clinical_scope.datasource.sources.mindray_scope.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index logger = logging.getLogger(__name__) diff --git a/src/clinical_scope/datasource/sources/other/find_load_format.py b/src/clinical_scope/datasource/sources/other/find_load_format.py index 3801c83..6e33de1 100644 --- a/src/clinical_scope/datasource/sources/other/find_load_format.py +++ b/src/clinical_scope/datasource/sources/other/find_load_format.py @@ -9,13 +9,10 @@ import clinical_scope.datasource.sources.other.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.inspection import DataSourceInspection -from clinical_scope.io.file_utils import ( - deduplicate_then_sort_index, - make_column_selector, - read_parquet_pruned, - set_datetime_index, -) +from clinical_scope.io.column_patterns import make_column_selector +from clinical_scope.io.parquet_pruning import read_parquet_pruned from clinical_scope.io.paths import get_output_folder +from clinical_scope.io.time_axis import deduplicate_then_sort_index, set_datetime_index from clinical_scope.signal_container import ( DisplayFallbacks, Signal, diff --git a/src/clinical_scope/datasource/sources/servo_u/find_load_format.py b/src/clinical_scope/datasource/sources/servo_u/find_load_format.py index abfa0ff..bfe1406 100644 --- a/src/clinical_scope/datasource/sources/servo_u/find_load_format.py +++ b/src/clinical_scope/datasource/sources/servo_u/find_load_format.py @@ -8,7 +8,7 @@ import clinical_scope.datasource.sources.servo_u.options as options_naming from clinical_scope.datasource.base import DataSourceBase from clinical_scope.datasource.timing import time_it -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index logger = logging.getLogger(__name__) diff --git a/src/clinical_scope/io/column_patterns.py b/src/clinical_scope/io/column_patterns.py new file mode 100644 index 0000000..ab78992 --- /dev/null +++ b/src/clinical_scope/io/column_patterns.py @@ -0,0 +1,88 @@ +""" +Resolving configured signal patterns against a file's actual column names. + +A pattern is either a literal column name or a trailing-``*`` wildcard. Both consumers +share one definition of what ``*`` matches, so a pruned read can never select a different +set than the full read would. +""" + +import logging +from collections.abc import Callable + +import pandas as pd + +import clinical_scope.constants as cst + +logger = logging.getLogger(__name__) + + +def _wildcard_matches(pattern: str, columns: pd.Index | list[str]) -> list[str] | None: + """ + Prefix-match *columns* for a trailing-``*`` wildcard; ``None`` if *pattern* is a literal. + + Single definition of what a ``*`` matches, shared by :func:`get_column_name_from_pattern` + and :func:`_pruned_columns` so their wildcard handling can't drift. ``None`` (literal) is + distinct from ``[]`` (wildcard with zero matches) — each caller handles literals its own way. + """ + if not (pattern and pattern.endswith(cst.DatabaseOptions.WILDCARD_SUFFIX)): + return None + prefix = pattern.rstrip(cst.DatabaseOptions.WILDCARD_SUFFIX) + return [column_name for column_name in columns if column_name.startswith(prefix)] + + +def get_column_name_from_pattern(columns: pd.Index | list[str], pattern: str) -> str | None: + """Find a column name matching a pattern (supports wildcard suffix '*').""" + matching_columns = _wildcard_matches(pattern, columns) + if matching_columns is None: + return pattern # literal: assume the caller-supplied name is the column + + if len(matching_columns) == 1: + return matching_columns[0] + if len(matching_columns) == 0: + logger.warning("No column found in dataframe from the pattern %s", pattern) + else: + logger.warning( + "More than one column found in dataframe with the pattern %s. -> Ignored", pattern + ) + return None + + +def _pruned_columns(field_display: list[str] | None, file_columns: list[str]) -> list[str] | None: + """ + Resolve which parquet columns to read for a set of configured signal patterns. + + Pure column-name logic (no data read). Shares :func:`_wildcard_matches` with + :func:`get_column_name_from_pattern`, so the result is by construction a superset of the + columns that matcher finally selects — every 0/1/2+ match count, and thus every warning, + is identical to a full read. + + - wildcard ``pre*`` → all file columns starting with ``pre`` (1) + - literal → included iff present (an absent name in ``columns=`` would raise) (2) + - *field_display* absent (``None``) → ``None`` ⇒ read all columns (3) + """ + if field_display is None: # (3) + return None + selected: list[str] = [] + seen: set[str] = set() + for pattern in field_display: + matches = _wildcard_matches(pattern, file_columns) + if matches is None: # (2) + matches = [pattern] if pattern in file_columns else [] + for name in matches: # (1) + if name not in seen: + seen.add(name) + selected.append(name) + return selected + + +def make_column_selector( + database_options_specific: dict | None, +) -> Callable[[list[str]], list[str] | None]: + """ + Build a *select_columns* callable for :func:`read_parquet_pruned` from a datasource's options. + + Centralizes the ``field_display`` lookup shared by every parquet call site. The returned + closure defers pattern resolution until the file's columns are known. + """ + field_display = (database_options_specific or {}).get(cst.DatabaseOptions.FIELD_DISPLAY) + return lambda file_columns: _pruned_columns(field_display, file_columns) diff --git a/src/clinical_scope/io/discovery.py b/src/clinical_scope/io/discovery.py new file mode 100644 index 0000000..b03cad6 --- /dev/null +++ b/src/clinical_scope/io/discovery.py @@ -0,0 +1,164 @@ +""" +Locating a patient folder's datasource folders and the data files inside them. + +Folder-level predicates identify which datasource a subfolder holds; ``find_files`` then +picks the file(s) to load inside one. +""" + +import logging +import re +from pathlib import Path + +import clinical_scope.constants as cst + +logger = logging.getLogger(__name__) + + +def folder_name_matches_keywords(folder_name: str, keywords: list[str]) -> bool: + """Check if *folder_name* contains every keyword (case-insensitive).""" + name_lower = folder_name.lower() + return all(keyword.lower() in name_lower for keyword in keywords) + + +_JUNK_FILENAME_RE = re.compile("|".join(cst.JUNK_FILENAME_PATTERNS)) + + +def is_junk_file(path: Path) -> bool: + """Return True if *path* is VCS/OS cruft or documentation (``.gitkeep``, ``readme.txt``).""" + return bool(_JUNK_FILENAME_RE.match(path.name)) + + +def folder_has_real_content(folder_path: Path) -> bool: + """Return True if *folder_path* contains at least one non-junk file (not recursive).""" + return any(entry.is_file() and not is_junk_file(entry) for entry in folder_path.iterdir()) + + +def deduplicate_by_stem(files: list[Path], extensions: list[str]) -> list[Path]: + """ + Keep one file per stem, preferring the extension earliest in *extensions*. + + A device folder routinely holds both a source export and a parquet written from it; + loading both would duplicate every signal under colliding names. + """ + suffix_rank = {extension.lower(): index for index, extension in enumerate(extensions)} + max_rank = len(extensions) + + def rank(file: Path) -> int: + return suffix_rank.get(file.suffix.lower(), max_rank) + + kept_by_stem: dict[str, Path] = {} + for file in files: + stem = file.stem.lower() + incumbent = kept_by_stem.get(stem) + if incumbent is None: + kept_by_stem[stem] = file + continue + winner, shadowed = (file, incumbent) if rank(file) < rank(incumbent) else (incumbent, file) + kept_by_stem[stem] = winner + logger.info( + "Ignoring '%s': '%s' already covers stem '%s'.", shadowed.name, winner.name, stem + ) + return list(kept_by_stem.values()) + + +def find_files( + folder_path: Path, + extensions: list[str], + datasource_name: str, + *, + multi: bool = False, + keywords: list[str] | None = None, +) -> list[Path] | Path | None: + """ + Find data files in *folder_path*. + + When *multi* is ``True``, return **all** files matching *extensions*, deduplicated by + stem and sorted alphabetically, or ``None`` if none found. + + When *multi* is ``False``, return a **single** file (tiered disambiguation): + + 1. Collect files matching *extensions* (or all files if none given). + 2. If one match, return it. + 3. Deduplicate by stem: when multiple extensions exist for the same stem, + keep the most preferred one (earliest in *extensions*). + 4. If one stem remains, return it. + 5. If *keywords* is given, try each keyword in order to narrow the set; + return immediately if exactly one match remains. + 6. If *extensions* is given, narrow the set by the first prefered extension that is available + in the files. Return directly if only one remains. + 6. Warn and return ``None`` if still ambiguous. + """ + if multi: + ext_set = {extension.lower() for extension in extensions} + files = [ + file + for file in folder_path.iterdir() + if file.is_file() and file.suffix.lower() in ext_set + ] + if not files: + logger.debug("Could not find any %s files in folder '%s'", datasource_name, folder_path) + return None + files = sorted(deduplicate_by_stem(files, extensions)) + logger.debug("Found %s: %s in folder %s", datasource_name, files, folder_path) + return files + + # --- single-file mode --- + if extensions: + suffix_set = {extension.lower() for extension in extensions} + matches = [ + file + for file in folder_path.iterdir() + if file.is_file() and file.suffix.lower() in suffix_set + ] + else: + # No extension filter: all non-junk files are candidates. + matches = [ + file for file in folder_path.iterdir() if file.is_file() and not is_junk_file(file) + ] + + if not matches: + logger.warning("No file for '%s' found in folder '%s'.", datasource_name, folder_path) + return None + + if len(matches) == 1: + logger.info("Selected file for '%s': %s", datasource_name, matches[0]) + return matches[0] + + if extensions: + matches = deduplicate_by_stem(matches, extensions) + + if len(matches) == 1: + logger.info("Selected file for '%s': %s", datasource_name, matches[0]) + return matches[0] + + # Keyword filtering on stem (ordered by preference) + if keywords: + for keyword in keywords: + keyword_lower = keyword.lower() + keyword_matches = [file for file in matches if keyword_lower in file.stem.lower()] + if len(keyword_matches) == 1: + logger.info( + "Selected file by keyword for '%s': %s", datasource_name, keyword_matches[0] + ) + return keyword_matches[0] + if keyword_matches: + matches = keyword_matches + + if extensions: + suffix_rank = {extension.lower(): index for index, extension in enumerate(extensions)} + matches.sort(key=lambda file: suffix_rank.get(file.suffix.lower(), len(extensions))) + if suffix_rank.get(matches[0].suffix.lower(), len(extensions)) < suffix_rank.get( + matches[1].suffix.lower(), len(extensions) + ): + logger.info( + "Selected file for '%s' by extension preference: %s", datasource_name, matches[0] + ) + return matches[0] + + logger.warning( + "Multiple '%s' files found in '%s', could not resolve a unique match: %s", + datasource_name, + folder_path, + [file.name for file in matches], + ) + return None diff --git a/src/clinical_scope/io/export.py b/src/clinical_scope/io/export.py new file mode 100644 index 0000000..1c4c5c6 --- /dev/null +++ b/src/clinical_scope/io/export.py @@ -0,0 +1,60 @@ +""" +Writing the terminal output of a pipeline run to disk. + +Unlike the parquet cache or the app's own state files, an export is output the user asked +for, so a failure here is raised rather than logged and swallowed. +""" + +import logging +from pathlib import Path + +import pandas as pd + +import clinical_scope.constants as cst + +logger = logging.getLogger(__name__) + + +def save_df(df: pd.DataFrame, path: str | Path) -> None: + """ + Save *df* to *path* as CSV (``.csv``) or parquet (any other recognised extension). + + Args: + path: Destination path. Extension must be ``.csv`` or ``.parquet``. + + Raises: + ValueError: If *path* has an unsupported extension. + + """ + path = Path(path) + if path.suffix == ".csv": + path.parent.mkdir(parents=True, exist_ok=True) + df.to_csv(path) + elif path.suffix == ".parquet": + path.parent.mkdir(parents=True, exist_ok=True) + df.to_parquet(path) + else: + msg = f"Unsupported file format '{path.suffix}'. Use '.csv' or '.parquet'." + raise ValueError(msg) + logger.info("Saved %d rows to %s", len(df), path) + + +def print_out_figure(path_output: Path, fig_list: list, self_contained: bool = False) -> None: + """ + Export Plotly figures to a single HTML file. + + With *self_contained*, plotly.js is embedded once (in the first figure; the rest reuse it) + so the file renders on a machine with no network — at ~3.5 MB. Otherwise it is fetched + from a CDN, which keeps the file small but shows a blank page offline. + """ + path_output.parent.mkdir(parents=True, exist_ok=True) + with Path.open(path_output, "w") as file_out: + for figure_index, fig in enumerate(fig_list): + if self_contained: + # Embedding the ~3.5 MB bundle once per file, not once per figure. + include_plotlyjs = ( + cst.HtmlExport.INLINE if figure_index == 0 else cst.HtmlExport.OMIT + ) + else: + include_plotlyjs = cst.HtmlExport.CDN + file_out.write(fig.to_html(full_html=False, include_plotlyjs=include_plotlyjs)) diff --git a/src/clinical_scope/io/file_utils.py b/src/clinical_scope/io/file_utils.py deleted file mode 100644 index 2f40b73..0000000 --- a/src/clinical_scope/io/file_utils.py +++ /dev/null @@ -1,784 +0,0 @@ -"""File I/O utilities for reading, writing, and discovering data files.""" - -import logging -import re -import warnings -from collections.abc import Callable -from pathlib import Path - -import pandas as pd -import pyarrow as pa -import pyarrow.parquet as pq - -import clinical_scope.constants as cst - -logger = logging.getLogger(__name__) - - -# ================================================================================================== -def save_df(df: pd.DataFrame, path: str | Path) -> None: - """ - Save *df* to *path* as CSV (``.csv``) or parquet (any other recognised extension). - - Args: - path: Destination path. Extension must be ``.csv`` or ``.parquet``. - - Raises: - ValueError: If *path* has an unsupported extension. - - """ - path = Path(path) - if path.suffix == ".csv": - path.parent.mkdir(parents=True, exist_ok=True) - df.to_csv(path) - elif path.suffix == ".parquet": - path.parent.mkdir(parents=True, exist_ok=True) - df.to_parquet(path) - else: - msg = f"Unsupported file format '{path.suffix}'. Use '.csv' or '.parquet'." - raise ValueError(msg) - logger.info("Saved %d rows to %s", len(df), path) - - -# ================================================================================================== -def folder_name_matches_keywords(folder_name: str, keywords: list[str]) -> bool: - """Check if *folder_name* contains every keyword (case-insensitive).""" - name_lower = folder_name.lower() - return all(keyword.lower() in name_lower for keyword in keywords) - - -# ================================================================================================== -_JUNK_FILENAME_RE = re.compile("|".join(cst.JUNK_FILENAME_PATTERNS)) - - -def is_junk_file(path: Path) -> bool: - """Return True if *path* is VCS/OS cruft or documentation (``.gitkeep``, ``readme.txt``).""" - return bool(_JUNK_FILENAME_RE.match(path.name)) - - -def folder_has_real_content(folder_path: Path) -> bool: - """Return True if *folder_path* contains at least one non-junk file (not recursive).""" - return any(entry.is_file() and not is_junk_file(entry) for entry in folder_path.iterdir()) - - -# ================================================================================================== -def deduplicate_by_stem(files: list[Path], extensions: list[str]) -> list[Path]: - """ - Keep one file per stem, preferring the extension earliest in *extensions*. - - A device folder routinely holds both a source export and a parquet written from it; - loading both would duplicate every signal under colliding names. - """ - suffix_rank = {extension.lower(): index for index, extension in enumerate(extensions)} - max_rank = len(extensions) - - def rank(file: Path) -> int: - return suffix_rank.get(file.suffix.lower(), max_rank) - - kept_by_stem: dict[str, Path] = {} - for file in files: - stem = file.stem.lower() - incumbent = kept_by_stem.get(stem) - if incumbent is None: - kept_by_stem[stem] = file - continue - winner, shadowed = (file, incumbent) if rank(file) < rank(incumbent) else (incumbent, file) - kept_by_stem[stem] = winner - logger.info( - "Ignoring '%s': '%s' already covers stem '%s'.", shadowed.name, winner.name, stem - ) - return list(kept_by_stem.values()) - - -def find_files( - folder_path: Path, - extensions: list[str], - datasource_name: str, - *, - multi: bool = False, - keywords: list[str] | None = None, -) -> list[Path] | Path | None: - """ - Find data files in *folder_path*. - - When *multi* is ``True``, return **all** files matching *extensions*, deduplicated by - stem and sorted alphabetically, or ``None`` if none found. - - When *multi* is ``False``, return a **single** file (tiered disambiguation): - - 1. Collect files matching *extensions* (or all files if none given). - 2. If one match, return it. - 3. Deduplicate by stem: when multiple extensions exist for the same stem, - keep the most preferred one (earliest in *extensions*). - 4. If one stem remains, return it. - 5. If *keywords* is given, try each keyword in order to narrow the set; - return immediately if exactly one match remains. - 6. If *extensions* is given, narrow the set by the first prefered extension that is available - in the files. Return directly if only one remains. - 6. Warn and return ``None`` if still ambiguous. - """ - if multi: - ext_set = {extension.lower() for extension in extensions} - files = [ - file - for file in folder_path.iterdir() - if file.is_file() and file.suffix.lower() in ext_set - ] - if not files: - logger.debug("Could not find any %s files in folder '%s'", datasource_name, folder_path) - return None - files = sorted(deduplicate_by_stem(files, extensions)) - logger.debug("Found %s: %s in folder %s", datasource_name, files, folder_path) - return files - - # --- single-file mode --- - if extensions: - suffix_set = {extension.lower() for extension in extensions} - matches = [ - file - for file in folder_path.iterdir() - if file.is_file() and file.suffix.lower() in suffix_set - ] - else: - # No extension filter: all non-junk files are candidates. - matches = [ - file for file in folder_path.iterdir() if file.is_file() and not is_junk_file(file) - ] - - if not matches: - logger.warning("No file for '%s' found in folder '%s'.", datasource_name, folder_path) - return None - - if len(matches) == 1: - logger.info("Selected file for '%s': %s", datasource_name, matches[0]) - return matches[0] - - if extensions: - matches = deduplicate_by_stem(matches, extensions) - - if len(matches) == 1: - logger.info("Selected file for '%s': %s", datasource_name, matches[0]) - return matches[0] - - # Keyword filtering on stem (ordered by preference) - if keywords: - for keyword in keywords: - keyword_lower = keyword.lower() - keyword_matches = [file for file in matches if keyword_lower in file.stem.lower()] - if len(keyword_matches) == 1: - logger.info( - "Selected file by keyword for '%s': %s", datasource_name, keyword_matches[0] - ) - return keyword_matches[0] - if keyword_matches: - matches = keyword_matches - - if extensions: - suffix_rank = {extension.lower(): index for index, extension in enumerate(extensions)} - matches.sort(key=lambda file: suffix_rank.get(file.suffix.lower(), len(extensions))) - if suffix_rank.get(matches[0].suffix.lower(), len(extensions)) < suffix_rank.get( - matches[1].suffix.lower(), len(extensions) - ): - logger.info( - "Selected file for '%s' by extension preference: %s", datasource_name, matches[0] - ) - return matches[0] - - logger.warning( - "Multiple '%s' files found in '%s', could not resolve a unique match: %s", - datasource_name, - folder_path, - [file.name for file in matches], - ) - return None - - -# ================================================================================================== -# Datetime-column detection (ADR 0004): tiered name search gated by content validation. -# Name lists and validation thresholds live in constants.py (DATETIME_* constants). - -_DATETIME_SUBSTRING_TIER_RES = [ - re.compile(pattern) for pattern in cst.DatetimeColumnDetection.SUBSTRING_TIERS -] - - -def _validate_parsed_datetimes(parsed: pd.Series) -> bool: - """Gate a parsed datetime Series: ≥90% valid in-range values, ≥90% non-decreasing.""" - if len(parsed) == 0: - return False - valid = parsed.dropna() - in_range = valid[ - (valid.dt.year >= cst.DatetimeColumnDetection.MIN_YEAR) - & (valid.dt.year <= cst.DatetimeColumnDetection.MAX_YEAR) - ] - if len(in_range) < cst.DatetimeColumnDetection.MIN_VALID_FRACTION * len(parsed): - return False - if len(in_range) > 1: - sorted_fraction = (in_range.diff().iloc[1:] >= pd.Timedelta(0)).mean() - if sorted_fraction < cst.DatetimeColumnDetection.MIN_SORTED_FRACTION: - return False - return True - - -def _try_parse_datetime_column(series: pd.Series) -> pd.Series | None: - """Parse a non-numeric Series as datetimes; return the parsed Series or None if gated out.""" - if pd.api.types.is_datetime64_any_dtype(series): - parsed = series - else: - try: - # Probing arbitrary columns triggers pandas' "could not infer format" - # warning on every garbage candidate — noise, not signal, here. - with warnings.catch_warnings(): - warnings.simplefilter("ignore", UserWarning) - parsed = pd.to_datetime(series, errors="coerce") - except (ValueError, TypeError, OverflowError): - return None - return parsed if _validate_parsed_datetimes(parsed) else None - - -def _pick_best_candidate(passing: list[tuple[str, pd.Series]]) -> tuple[str, pd.Series]: - """ - Tiebreak same-tier candidates that all passed validation. - - Prefer the highest uniqueness (penalizes batchy DB-artifact columns), then - utc-named columns (unambiguous vs DST-prone naive-local), then column order. - A utc-named winner that's still tz-naive after parsing gets localized to UTC. - """ - best_uniqueness = max(parsed.nunique() for _, parsed in passing) - top = [ - (column_name, parsed) - for column_name, parsed in passing - if parsed.nunique() == best_uniqueness - ] - utc_named = [ - (column_name, parsed) for column_name, parsed in top if "utc" in str(column_name).lower() - ] - column_name, parsed = (utc_named or top)[0] - if "utc" in str(column_name).lower() and parsed.dt.tz is None: - parsed = parsed.dt.tz_localize(cst.LIBRARY_TZ) - return column_name, parsed - - -def _name_tiers(columns: list[str]) -> list[list[str]]: - """ - Build the datetime-column name-priority tiers (exact names, then substring buckets). - - Shared by full-frame detection (:func:`_find_datetime_col_parsed`) and - schema-only detection (:func:`_detect_datetime_column_from_parquet`), so both walk - the same priority order without duplicating it. Each name/pattern is its own tier - so list order is a real priority: a lower-priority name never competes via - uniqueness against a higher-priority one that's also present and valid. - """ - lower_names = {column_name: str(column_name).lower().strip() for column_name in columns} - tiers = [ - [column_name for column_name in columns if lower_names[column_name] == name] - for name in cst.DatetimeColumnDetection.EXACT_NAMES - ] - tiers += [ - [column_name for column_name in columns if pattern.search(lower_names[column_name])] - for pattern in _DATETIME_SUBSTRING_TIER_RES - ] - # Widen tier: every column, ignoring name (numeric ones still deferred to epoch tier). - tiers.append(list(columns)) - return tiers - - -def _find_datetime_col_parsed(df: pd.DataFrame) -> tuple[str, pd.Series]: - """ - Detect the datetime column, returning ``(column_name, parsed_series)``. - - Walks the name tiers (exact, then substring buckets), validating content at every - tier; numeric columns are deferred to the epoch tier. Raises ValueError when no - column passes validation (fail loudly — never guess a time axis). - """ - for tier in _name_tiers(list(df.columns)): - passing = [ - (column_name, parsed) - for column_name in tier - if not pd.api.types.is_numeric_dtype(df[column_name]) - and (parsed := _try_parse_datetime_column(df[column_name])) is not None - ] - if passing: - return _pick_best_candidate(passing) - - # Numeric-epoch tier, tried last: nanosecond epochs only (~1.6e18 is unambiguous - # against real measurement data), gated by the same validation. - epoch_passing = [] - for column_name in df.columns: - if not pd.api.types.is_numeric_dtype(df[column_name]): - continue - try: - parsed = pd.to_datetime(df[column_name], unit="ns", errors="coerce") - except (ValueError, TypeError, OverflowError): - continue - if _validate_parsed_datetimes(parsed): - epoch_passing.append((column_name, parsed)) - if epoch_passing: - return _pick_best_candidate(epoch_passing) - - msg = ( - "No datetime column detected: no column passed content validation " - f"(≥90% parseable in [{cst.DatetimeColumnDetection.MIN_YEAR}, " - f"{cst.DatetimeColumnDetection.MAX_YEAR}], ≥90% non-decreasing). " - f"Columns: {list(df.columns)}" - ) - raise ValueError(msg) - - -# ================================================================================================== - - -def resolve_stored_index_field(path: Path) -> pa.Field | None: - """ - Return the parquet file's materialized index column as a pyarrow field, if any. - - The field of the single index column *path* stores (e.g. written by our own - ``to_parquet``), else ``None``: a plain ``RangeIndex`` is recorded as a descriptor dict - rather than a physical column name, and a MultiIndex resolves to several columns. - - The field's *type* is deliberately not judged here: "which column is the index" and "is - that index a range-comparable time axis" are separate questions, answered at the call site. - """ - schema = pq.ParquetFile(path).schema_arrow - pandas_metadata = schema.pandas_metadata - if not pandas_metadata: - return None - index_columns = pandas_metadata.get("index_columns") or [] - if len(index_columns) != 1 or not isinstance(index_columns[0], str): - return None - return schema.field(index_columns[0]) - - -def _is_numeric_pa_type(field_type: pa.DataType) -> bool: - """ - Schema-only "numeric, defer to the epoch tier" predicate for a pyarrow field type. - - Must agree with :func:`_find_datetime_col_parsed`'s ``pd.api.types.is_numeric_dtype`` - check on every dtype that can appear in a clinical parquet export — the two datetime - detectors (schema-only vs. full-frame) rely on picking the same candidate column. - See :class:`tests.datasource.test_datetime_pushdown.TestNumericTypeClassificationAgreement`. - """ - return pa.types.is_integer(field_type) or pa.types.is_floating(field_type) - - -def _sample_parquet_columns(parquet_file: pq.ParquetFile, columns: list[str]) -> pd.DataFrame: - """ - Read a bounded, spread sample of *columns* for datetime detection. - - Reads whole row groups (parquet's random-access unit) evenly spread across the file, - head-slicing each to ``SAMPLE_ROWS_PER_BLOCK`` — contiguous slices preserve duplicate-value - runs, so the uniqueness and sorted checks in :func:`_pick_best_candidate` / - :func:`_validate_parsed_datetimes` stay meaningful. The whole file is read only when it - fits the decode budget or has a single row group. When it has several, at least - ``SAMPLE_MIN_GROUPS`` are sampled even if the budget alone would pick fewer (huge row - groups), so detection always sees two independent places. - """ - parquet_metadata = parquet_file.metadata - row_group_count = parquet_metadata.num_row_groups - detection_constants = cst.DatetimeColumnDetection - max_row_decoded = detection_constants.SAMPLE_MAX_ROW_DECODED - if row_group_count <= 1 or parquet_metadata.num_rows <= max_row_decoded: - return parquet_file.read(columns=columns).to_pandas() - - rows_per_group = parquet_metadata.row_group(0).num_rows - budget_groups = max(1, max_row_decoded // rows_per_group) - sample_group_count = min(detection_constants.SAMPLE_MAX_GROUPS, row_group_count, budget_groups) - sample_group_count = max( - sample_group_count, min(detection_constants.SAMPLE_MIN_GROUPS, row_group_count) - ) # ≥2 places when ≥2 groups exist - indices = sorted( - { - round(sample_index * (row_group_count - 1) / (sample_group_count - 1)) - for sample_index in range(sample_group_count) - } - ) - rows_per_block = detection_constants.SAMPLE_ROWS_PER_BLOCK - tables = [ - parquet_file.read_row_group(group_index, columns=columns).slice(0, rows_per_block) - for group_index in indices - ] - return pa.concat_tables(tables).to_pandas() - - -def _detect_datetime_column_from_parquet( - path: Path, -) -> tuple[str, str, str | None, bool] | None: - """ - Detect the datetime column of a parquet file without a materialized index. - - Mirrors :func:`_find_datetime_col_parsed`'s tiered name search, but reads only - each tier's candidate columns (progressively widening) to validate content, - rather than loading the whole file upfront. - - Returns ``(column_name, kind, tz, physically_naive)`` where *kind* is - ``TIMESTAMP`` (direct range filter, *tz* set for tz-aware columns) or - ``EPOCH_NS`` (nanosecond-epoch numeric column, *tz* is ``None``) — see - :class:`~clinical_scope.constants.ParquetPushdownKind`. Both are safe for - an unambiguous parquet row filter. Any other resolved type (e.g. a string datetime - column, unparsed) is not pushdown-safe and yields ``None``, so the caller falls - back to a full unfiltered read. - - *tz* is the *semantic* timezone (from :func:`_pick_best_candidate`, which - force-localizes a tz-naive utc-named column to UTC — matching what - :func:`set_datetime_index` does downstream) and can therefore diverge from the - column's on-disk type, which stays physically tz-naive. *physically_naive* flags - that case so the caller can strip the tz label back off before filtering — pyarrow - filter values must match the physical on-disk type exactly. - - Candidate columns are validated on a bounded sample (:func:`_sample_parquet_columns`), - not the whole file, so this pick can diverge from the downstream full-frame - :func:`set_datetime_index` — which would filter one column and index another (silent row - loss). To stay safe, detection only ever consults the **highest-priority tier that has any - named candidate**, and commits only if that tier yields **exactly one** sample-validated - column; otherwise it abstains (returns ``None`` → full read, full-frame decides): - - - **zero passing** there — a higher-priority column we couldn't confirm on the sample - (e.g. valid over the whole file but garbage in exactly the sampled row groups) may still - validate on the full frame and outrank any lower-tier pick, so we must not look lower. - - **more than one** — the sample-based uniqueness tiebreak in :func:`_pick_best_candidate` - isn't stable, so the pick could differ from the full frame's. - """ - parquet_file = pq.ParquetFile(path) - schema = parquet_file.schema_arrow - - def _is_numeric(column_name: str) -> bool: - return _is_numeric_pa_type(schema.field(column_name).type) - - columns = list(schema.names) - for tier in _name_tiers(columns): - candidates = [column_name for column_name in tier if not _is_numeric(column_name)] - if not candidates: - continue - sample = _sample_parquet_columns(parquet_file, candidates) - passing = [ - (column_name, parsed) - for column_name in candidates - if (parsed := _try_parse_datetime_column(sample[column_name])) is not None - ] - if len(passing) != 1: - return None - column_name, parsed = _pick_best_candidate(passing) - field_type = schema.field(column_name).type - if pa.types.is_timestamp(field_type): - # Use the resolved parsed tz, not the raw physical field's — _pick_best_candidate - # force-localizes utc-named naive columns to UTC, matching what the real pipeline - # (set_datetime_index) later does, and the row-filter bounds must agree with that. - # The physical on-disk type stays naive though, so flag it for the caller. - tz = parsed.dt.tz - return ( - column_name, - cst.ParquetPushdownKind.TIMESTAMP, - (str(tz) if tz else None), - field_type.tz is None, - ) - return column_name, cst.ParquetPushdownKind.OTHER, None, False - - numeric_columns = [column_name for column_name in columns if _is_numeric(column_name)] - if numeric_columns: - sample = _sample_parquet_columns(parquet_file, numeric_columns) - epoch_passing = [] - for column_name in numeric_columns: - try: - parsed = pd.to_datetime(sample[column_name], unit="ns", errors="coerce") - except (ValueError, TypeError, OverflowError): - continue - if _validate_parsed_datetimes(parsed): - epoch_passing.append((column_name, parsed)) - if len(epoch_passing) > 1: # no named tier hid a candidate here — only the tiebreak can - return None - if epoch_passing: - column_name, _parsed = _pick_best_candidate(epoch_passing) - return column_name, cst.ParquetPushdownKind.EPOCH_NS, None, True - - return None - - -def _build_datetime_row_filters( - column_name: str, - kind: str, - physically_naive: bool, - bounds: tuple[pd.Timestamp | None, pd.Timestamp | None], -) -> list[tuple] | None: - """Turn resolved ``(start, end)`` bounds into pyarrow row filters, or ``None`` if empty.""" - start, end = bounds - if kind == cst.ParquetPushdownKind.EPOCH_NS: - start = None if start is None else start.value - end = None if end is None else end.value - elif kind == cst.ParquetPushdownKind.TIMESTAMP and physically_naive: - # A naive column may carry a semantic tz from name detection (e.g. "*utc*"); strip the - # tz label so the wall-clock bounds match the physical, tz-naive on-disk column. - if start is not None and start.tzinfo is not None: - start = start.tz_localize(None) - if end is not None and end.tzinfo is not None: - end = end.tz_localize(None) - - filters = [ - filter_clause - for filter_clause in [ - (column_name, ">=", start) if start is not None else None, - (column_name, "<=", end) if end is not None else None, - ] - if filter_clause is not None - ] - return filters or None - - -def read_parquet_pruned( - path: Path, - compute_bounds: Callable[[str | None], tuple[pd.Timestamp | None, pd.Timestamp | None] | None] - | None = None, - select_columns: Callable[[list[str]], list[str] | None] | None = None, - *, - index_is_time_axis: bool = False, -) -> pd.DataFrame: - """ - Read a parquet file, pruning out-of-window rows *and* unconfigured columns at read time. - - Two orthogonal prunings, each safe on its own: - - - **Rows** — *compute_bounds* receives the datetime column's tz (``None`` if tz-naive/epoch) - and returns loose ``(start, end)`` bounds in that tz (either side may be ``None``), or - ``None`` for no window. Applied only for a range-comparable datetime column; otherwise - the read is unfiltered (under-pruning only costs extra rows). - - **Columns** — *select_columns* receives the file's column names and returns the subset to - read (a superset of the finally-selected signals), or ``None`` to read all. Independent - of any window — the common case is a wide cache with no window set. - - Index-safe: a materialized index is auto-restored by pandas even when omitted from - ``columns=``; a non-materialized datetime column is unioned back so ``set_datetime_index`` - still finds it; if that column can't be resolved, column pruning is skipped (never drop the - time axis). - - *index_is_time_axis* lets a caller that knows the file's provenance declare that its stored - index is the time axis whatever its dtype, so a non-temporal one (EIT's float64 fractional - days) prunes columns instead of falling back to reading all of them. A declared axis is not - thereby known to be range-comparable, so it never carries a row filter — and, since the axis - is already accounted for, detection is skipped entirely, giving up any pushdown it might - have found on a data column. - """ - file_columns = pq.ParquetFile(path).schema_arrow.names - requested_columns = None if select_columns is None else select_columns(list(file_columns)) - - index_field = resolve_stored_index_field(path) - # A timestamp index is also range-comparable, so it can carry the row filter; a declared - # one is only known to be the axis — enough to prune columns, never enough to filter rows. - temporal_index = index_field is not None and pa.types.is_timestamp(index_field.type) - axis_survives_pruning = temporal_index or (index_is_time_axis and index_field is not None) - want_pushdown = compute_bounds is not None - - # Resolve the datetime column only when needed (row filter, or protecting an axis that - # isn't the index); detection samples data, so skip it otherwise. - column_name = kind = tz = None - physically_naive = False - if temporal_index: - column_name = index_field.name - tz = str(index_field.type.tz) if index_field.type.tz else None - kind = cst.ParquetPushdownKind.TIMESTAMP - physically_naive = tz is None - elif not axis_survives_pruning and (want_pushdown or requested_columns is not None): - detected = _detect_datetime_column_from_parquet(path) - if detected is not None: - column_name, kind, tz, physically_naive = detected - - columns_to_read = requested_columns - if columns_to_read is not None and not axis_survives_pruning: - if column_name is None: - columns_to_read = None # unknown datetime axis → don't risk dropping it, read all - elif column_name not in columns_to_read: - columns_to_read = [column_name, *columns_to_read] # keep the time axis in the read - - filters = None - if ( - want_pushdown - and column_name is not None - and kind is not None - and kind != cst.ParquetPushdownKind.OTHER - ): - bounds = compute_bounds(tz if kind == cst.ParquetPushdownKind.TIMESTAMP else None) - if bounds is not None: - filters = _build_datetime_row_filters(column_name, kind, physically_naive, bounds) - - if columns_to_read is not None: - logger.debug( - "Parquet column pruning on '%s': reading %d/%d columns.", - path, - len(columns_to_read), - len(file_columns), - ) - - if filters is None: - return pd.read_parquet(path, columns=columns_to_read) - - total_rows = pq.ParquetFile(path).metadata.num_rows - df = pd.read_parquet(path, filters=filters, columns=columns_to_read) - pruned_pct = 100 * (1 - len(df) / total_rows) if total_rows else 0.0 - logger.info( - "Parquet pushdown on '%s': read %d/%d rows (%.0f%% pruned).", - path, - len(df), - total_rows, - pruned_pct, - ) - return df - - -def set_datetime_index(df: pd.DataFrame) -> pd.DataFrame: - """ - Return *df* indexed by its detected datetime column. - - Short-circuits when the index is already a DatetimeIndex; otherwise detects, - parses, and sets the best-validated datetime column (raises if none passes). - """ - if isinstance(df.index, pd.DatetimeIndex): - return df - column_name, parsed = _find_datetime_col_parsed(df) - df = df.copy() - df[column_name] = parsed - return df.set_index(column_name) - - -def deduplicate_then_sort_index(df: pd.DataFrame) -> pd.DataFrame: - """ - Drop duplicate index entries (keep first) *then* sort by index. - - Deduplicating first keeps the first row in file order on a timestamp - collision, which a non-stable ``sort_index`` would decide arbitrarily. - Skips either step when already satisfied (device exports are usually - already sorted and unique). - """ - if not df.index.is_unique: - df = df[~df.index.duplicated(keep="first")] - if not df.index.is_monotonic_increasing: - df = df.sort_index() - return df - - -# ================================================================================================== -def load_csv_with_datetime_index( - file_path: str | Path, datetime_column: str | None = None, **kwargs -) -> pd.DataFrame: - """ - Load a CSV file and set a datetime column as the index. - - When *datetime_column* is ``None``, auto-detects the datetime column with - ``set_datetime_index`` (raises if no column passes validation). - """ - if datetime_column is not None: - return pd.read_csv(file_path, index_col=datetime_column, parse_dates=True, **kwargs) - - return set_datetime_index(pd.read_csv(file_path, **kwargs)) - - -# ================================================================================================== -def load_parquet_with_datetime_index( - file_path: str | Path, - datetime_column: str | None = None, - compute_bounds: Callable[[str | None], tuple[pd.Timestamp | None, pd.Timestamp | None] | None] - | None = None, - select_columns: Callable[[list[str]], list[str] | None] | None = None, - **kwargs, -) -> pd.DataFrame: - """ - Load a parquet file and ensure it is indexed by datetime. - - Files already carrying a DatetimeIndex are returned as-is; otherwise the datetime column - is detected with ``set_datetime_index`` (raises if none passes validation). - - *datetime_column* names the datetime column explicitly, bypassing detection. Column pruning - (*select_columns*) still applies then — *datetime_column* is always kept in the read — but - row pushdown (*compute_bounds*) does not, since an explicit column may still need parsing - before it is range-comparable. Without *datetime_column*, both prunings go through - :func:`read_parquet_pruned`. - """ - if datetime_column is not None: - columns = None - if select_columns is not None: - file_columns = pq.ParquetFile(file_path).schema_arrow.names - columns = select_columns(list(file_columns)) - if columns is not None and datetime_column not in columns: - columns = [datetime_column, *columns] # keep the time axis in the read - df = pd.read_parquet(file_path, columns=columns, **kwargs) - df[datetime_column] = pd.to_datetime(df[datetime_column]) - return df.set_index(datetime_column) - - if compute_bounds is not None or select_columns is not None: - df = read_parquet_pruned( - Path(file_path), compute_bounds=compute_bounds, select_columns=select_columns - ) - else: - df = pd.read_parquet(file_path, **kwargs) - return set_datetime_index(df) - - -# ================================================================================================== -def _wildcard_matches(pattern: str, columns: pd.Index | list[str]) -> list[str] | None: - """ - Prefix-match *columns* for a trailing-``*`` wildcard; ``None`` if *pattern* is a literal. - - Single definition of what a ``*`` matches, shared by :func:`get_column_name_from_pattern` - and :func:`_pruned_columns` so their wildcard handling can't drift. ``None`` (literal) is - distinct from ``[]`` (wildcard with zero matches) — each caller handles literals its own way. - """ - if not (pattern and pattern.endswith(cst.DatabaseOptions.WILDCARD_SUFFIX)): - return None - prefix = pattern.rstrip(cst.DatabaseOptions.WILDCARD_SUFFIX) - return [column_name for column_name in columns if column_name.startswith(prefix)] - - -def get_column_name_from_pattern(columns: pd.Index | list[str], pattern: str) -> str | None: - """Find a column name matching a pattern (supports wildcard suffix '*').""" - matching_columns = _wildcard_matches(pattern, columns) - if matching_columns is None: - return pattern # literal: assume the caller-supplied name is the column - - if len(matching_columns) == 1: - return matching_columns[0] - if len(matching_columns) == 0: - logger.warning("No column found in dataframe from the pattern %s", pattern) - else: - logger.warning( - "More than one column found in dataframe with the pattern %s. -> Ignored", pattern - ) - return None - - -# ================================================================================================== -def _pruned_columns(field_display: list[str] | None, file_columns: list[str]) -> list[str] | None: - """ - Resolve which parquet columns to read for a set of configured signal patterns. - - Pure column-name logic (no data read). Shares :func:`_wildcard_matches` with - :func:`get_column_name_from_pattern`, so the result is by construction a superset of the - columns that matcher finally selects — every 0/1/2+ match count, and thus every warning, - is identical to a full read. - - - wildcard ``pre*`` → all file columns starting with ``pre`` (1) - - literal → included iff present (an absent name in ``columns=`` would raise) (2) - - *field_display* absent (``None``) → ``None`` ⇒ read all columns (3) - """ - if field_display is None: # (3) - return None - selected: list[str] = [] - seen: set[str] = set() - for pattern in field_display: - matches = _wildcard_matches(pattern, file_columns) - if matches is None: # (2) - matches = [pattern] if pattern in file_columns else [] - for name in matches: # (1) - if name not in seen: - seen.add(name) - selected.append(name) - return selected - - -def make_column_selector( - database_options_specific: dict | None, -) -> Callable[[list[str]], list[str] | None]: - """ - Build a *select_columns* callable for :func:`read_parquet_pruned` from a datasource's options. - - Centralizes the ``field_display`` lookup shared by every parquet call site. The returned - closure defers pattern resolution until the file's columns are known. - """ - field_display = (database_options_specific or {}).get(cst.DatabaseOptions.FIELD_DISPLAY) - return lambda file_columns: _pruned_columns(field_display, file_columns) diff --git a/src/clinical_scope/io/parquet_pruning.py b/src/clinical_scope/io/parquet_pruning.py new file mode 100644 index 0000000..4e866b0 --- /dev/null +++ b/src/clinical_scope/io/parquet_pruning.py @@ -0,0 +1,227 @@ +""" +Read-time row and column pruning for parquet reads (ADR 0007). + +Pruning is an optimization: every decision here may safely decline, and under-pruning only +costs a wider read. Which file is being read decides how much can be pruned, so the two +provenances get their own front door rather than a flag — +:func:`read_parquet_pruned` for a user's file, :func:`read_cache_pruned` for one we wrote. +""" + +import logging +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq + +import clinical_scope.constants as cst +from clinical_scope.io.time_axis import detect_time_axis_in_parquet + +logger = logging.getLogger(__name__) + +ComputeBounds = Callable[[str | None], tuple[pd.Timestamp | None, pd.Timestamp | None] | None] +SelectColumns = Callable[[list[str]], list[str] | None] + + +@dataclass(frozen=True) +class _PruningPlan: + """What a pruned read will ask parquet for, decided without reading any bulk data.""" + + columns: list[str] | None # None → read every column + row_filters: list[tuple] | None # None → no row predicate + file_column_count: int # for the pruning log line + + +def _resolve_stored_index_field(path: Path) -> pa.Field | None: + """ + Return the parquet file's materialized index column as a pyarrow field, if any. + + The field of the single index column *path* stores (e.g. written by our own + ``to_parquet``), else ``None``: a plain ``RangeIndex`` is recorded as a descriptor dict + rather than a physical column name, and a MultiIndex resolves to several columns. + + The field's *type* is deliberately not judged here: "which column is the index" and "is + that index a range-comparable time axis" are separate questions, answered at the call site. + """ + schema = pq.ParquetFile(path).schema_arrow + pandas_metadata = schema.pandas_metadata + if not pandas_metadata: + return None + index_columns = pandas_metadata.get("index_columns") or [] + if len(index_columns) != 1 or not isinstance(index_columns[0], str): + return None + return schema.field(index_columns[0]) + + +def _to_stored_naive(bound: pd.Timestamp | None, tz: str) -> pd.Timestamp | None: + """Express an aware *bound* in *tz*, then drop the label to match a tz-naive column.""" + if bound is None or bound.tzinfo is None: + return bound + return bound.tz_convert(tz).tz_localize(None) + + +def _build_datetime_row_filters( + column_name: str, + kind: str, + tz: str | None, + tz_from_name: bool, + bounds: tuple[pd.Timestamp | None, pd.Timestamp | None], +) -> list[tuple] | None: + """Turn resolved ``(start, end)`` bounds into pyarrow row filters, or ``None`` if empty.""" + start, end = bounds + if kind == cst.ParquetPushdownKind.EPOCH_NS: + start = None if start is None else start.value + end = None if end is None else end.value + elif kind == cst.ParquetPushdownKind.TIMESTAMP and tz_from_name and tz is not None: + # The tz was asserted from the column name, never stored, so the on-disk values are + # bare wall clock in it. Converting before dropping the label is what keeps a bound + # expressed in any other zone landing on the right instant. + start = _to_stored_naive(start, tz) + end = _to_stored_naive(end, tz) + + filters = [ + filter_clause + for filter_clause in [ + (column_name, ">=", start) if start is not None else None, + (column_name, "<=", end) if end is not None else None, + ] + if filter_clause is not None + ] + return filters or None + + +def _pruning_plan( + path: Path, + compute_bounds: ComputeBounds | None, + select_columns: SelectColumns | None, + *, + index_is_time_axis: bool, +) -> _PruningPlan: + """ + Decide both prunings for *path*, reading only its schema and a detection sample. + + Two orthogonal prunings, each safe on its own: + + - **Rows** — *compute_bounds* receives the datetime column's tz (``None`` if tz-naive/epoch) + and returns loose ``(start, end)`` bounds in that tz (either side may be ``None``), or + ``None`` for no window. Planned only for a range-comparable datetime column. + - **Columns** — *select_columns* receives the file's column names and returns the subset to + read (a superset of the finally-selected signals), or ``None`` to read all. Independent + of any window — the common case is a wide cache with no window set. + + Index-safe: a materialized index is auto-restored by pandas even when omitted from + ``columns=``; a non-materialized datetime column is unioned back so ``set_datetime_index`` + still finds it; if that column can't be resolved, column pruning is skipped (never drop the + time axis). + + *index_is_time_axis* declares that the stored index is the time axis whatever its dtype, so + a non-temporal one (EIT's float64 fractional days) prunes columns instead of falling back to + reading all of them. A declared axis is not thereby range-comparable, so it never carries a + row filter — and, since the axis is already accounted for, detection is skipped entirely, + giving up any pushdown it might have found on a data column. + """ + file_columns = pq.ParquetFile(path).schema_arrow.names + requested_columns = None if select_columns is None else select_columns(list(file_columns)) + + index_field = _resolve_stored_index_field(path) + # A timestamp index is also range-comparable, so it can carry the row filter; a declared + # one is only known to be the axis — enough to prune columns, never enough to filter rows. + temporal_index = index_field is not None and pa.types.is_timestamp(index_field.type) + axis_survives_pruning = temporal_index or (index_is_time_axis and index_field is not None) + want_pushdown = compute_bounds is not None + + # Resolve the datetime column only when needed (row filter, or protecting an axis that + # isn't the index); detection samples data, so skip it otherwise. + column_name = kind = tz = None + tz_from_name = False # a stored index never asserts one; only name detection can + if temporal_index: + column_name = index_field.name + tz = str(index_field.type.tz) if index_field.type.tz else None + kind = cst.ParquetPushdownKind.TIMESTAMP + elif not axis_survives_pruning and (want_pushdown or requested_columns is not None): + detected = detect_time_axis_in_parquet(path) + if detected is not None: + column_name = detected.column_name + kind = detected.kind + tz = detected.tz + tz_from_name = detected.tz_from_name + + columns_to_read = requested_columns + if columns_to_read is not None and not axis_survives_pruning: + if column_name is None: + columns_to_read = None # unknown datetime axis → don't risk dropping it, read all + elif column_name not in columns_to_read: + columns_to_read = [column_name, *columns_to_read] # keep the time axis in the read + + row_filters = None + if ( + want_pushdown + and column_name is not None + and kind is not None + and kind != cst.ParquetPushdownKind.OTHER + ): + bounds = compute_bounds(tz if kind == cst.ParquetPushdownKind.TIMESTAMP else None) + if bounds is not None: + row_filters = _build_datetime_row_filters(column_name, kind, tz, tz_from_name, bounds) + + return _PruningPlan( + columns=columns_to_read, row_filters=row_filters, file_column_count=len(file_columns) + ) + + +def _read_with_plan(path: Path, plan: _PruningPlan) -> pd.DataFrame: + """Execute *plan*, reporting what each pruning actually saved.""" + if plan.columns is not None: + logger.debug( + "Parquet column pruning on '%s': reading %d/%d columns.", + path, + len(plan.columns), + plan.file_column_count, + ) + + if plan.row_filters is None: + return pd.read_parquet(path, columns=plan.columns) + + total_rows = pq.ParquetFile(path).metadata.num_rows + df = pd.read_parquet(path, filters=plan.row_filters, columns=plan.columns) + pruned_pct = 100 * (1 - len(df) / total_rows) if total_rows else 0.0 + logger.info( + "Parquet pushdown on '%s': read %d/%d rows (%.0f%% pruned).", + path, + len(df), + total_rows, + pruned_pct, + ) + return df + + +def read_parquet_pruned( + path: Path, + compute_bounds: ComputeBounds | None = None, + select_columns: SelectColumns | None = None, +) -> pd.DataFrame: + """ + Read a parquet file of unknown provenance, pruning only what detection can establish. + + A stored index of a non-timestamp type could be anything, so column pruning is declined + rather than risk dropping the time axis. See :func:`_pruning_plan` for both prunings. + """ + plan = _pruning_plan(path, compute_bounds, select_columns, index_is_time_axis=False) + return _read_with_plan(path, plan) + + +def read_cache_pruned( + path: Path, + compute_bounds: ComputeBounds | None = None, + select_columns: SelectColumns | None = None, +) -> pd.DataFrame: + """ + Read a parquet cache *we* wrote, whose index is the time axis by construction (ADR 0010). + + That provenance is the one thing detection cannot infer, and it lets a non-timestamp index + (EIT's float64 fractional days) still prune columns. + """ + plan = _pruning_plan(path, compute_bounds, select_columns, index_is_time_axis=True) + return _read_with_plan(path, plan) diff --git a/src/clinical_scope/io/time_axis.py b/src/clinical_scope/io/time_axis.py new file mode 100644 index 0000000..7520975 --- /dev/null +++ b/src/clinical_scope/io/time_axis.py @@ -0,0 +1,341 @@ +""" +Establishing a dataframe's time axis: which column carries it, and normalising it. + +Implements ADR 0004's tiered name search gated by content validation. The rule has two +adapters over the same tiers — :func:`detect_time_axis_in_frame` for a loaded frame and +:func:`detect_time_axis_in_parquet` for a file read only by schema and sample. They must +pick the same column, so they live together. + +Name lists and validation thresholds live in constants.py (``DatetimeColumnDetection``). +""" + +import re +import warnings +from dataclasses import dataclass +from pathlib import Path + +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq + +import clinical_scope.constants as cst + + +@dataclass(frozen=True) +class DetectedTimeAxis: + """ + A parquet file's time axis, resolved from its schema and a bounded sample. + + *tz* is semantic: for a utc-named but tz-naive column it is asserted from the name + rather than read from the type. *tz_from_name* marks that one case, so a caller can + strip the label back off before comparing against the stored values. + """ + + column_name: str + kind: str + tz: str | None + tz_from_name: bool + + +_DATETIME_SUBSTRING_TIER_RES = [ + re.compile(pattern) for pattern in cst.DatetimeColumnDetection.SUBSTRING_TIERS +] + + +def _validate_parsed_datetimes(parsed: pd.Series) -> bool: + """Gate a parsed datetime Series: ≥90% valid in-range values, ≥90% non-decreasing.""" + if len(parsed) == 0: + return False + valid = parsed.dropna() + in_range = valid[ + (valid.dt.year >= cst.DatetimeColumnDetection.MIN_YEAR) + & (valid.dt.year <= cst.DatetimeColumnDetection.MAX_YEAR) + ] + if len(in_range) < cst.DatetimeColumnDetection.MIN_VALID_FRACTION * len(parsed): + return False + if len(in_range) > 1: + sorted_fraction = (in_range.diff().iloc[1:] >= pd.Timedelta(0)).mean() + if sorted_fraction < cst.DatetimeColumnDetection.MIN_SORTED_FRACTION: + return False + return True + + +def _try_parse_datetime_column(series: pd.Series) -> pd.Series | None: + """Parse a non-numeric Series as datetimes; return the parsed Series or None if gated out.""" + if pd.api.types.is_datetime64_any_dtype(series): + parsed = series + else: + try: + # Probing arbitrary columns triggers pandas' "could not infer format" + # warning on every garbage candidate — noise, not signal, here. + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UserWarning) + parsed = pd.to_datetime(series, errors="coerce") + except (ValueError, TypeError, OverflowError): + return None + return parsed if _validate_parsed_datetimes(parsed) else None + + +def _pick_best_candidate(passing: list[tuple[str, pd.Series]]) -> tuple[str, pd.Series]: + """ + Tiebreak same-tier candidates that all passed validation. + + Prefer the highest uniqueness (penalizes batchy DB-artifact columns), then + utc-named columns (unambiguous vs DST-prone naive-local), then column order. + A utc-named winner that's still tz-naive after parsing gets localized to UTC. + """ + best_uniqueness = max(parsed.nunique() for _, parsed in passing) + top = [ + (column_name, parsed) + for column_name, parsed in passing + if parsed.nunique() == best_uniqueness + ] + utc_named = [ + (column_name, parsed) for column_name, parsed in top if "utc" in str(column_name).lower() + ] + column_name, parsed = (utc_named or top)[0] + if "utc" in str(column_name).lower() and parsed.dt.tz is None: + parsed = parsed.dt.tz_localize(cst.LIBRARY_TZ) + return column_name, parsed + + +def _name_tiers(columns: list[str]) -> list[list[str]]: + """ + Build the datetime-column name-priority tiers (exact names, then substring buckets). + + Shared by full-frame detection (:func:`detect_time_axis_in_frame`) and + schema-only detection (:func:`detect_time_axis_in_parquet`), so both walk + the same priority order without duplicating it. Each name/pattern is its own tier + so list order is a real priority: a lower-priority name never competes via + uniqueness against a higher-priority one that's also present and valid. + """ + lower_names = {column_name: str(column_name).lower().strip() for column_name in columns} + tiers = [ + [column_name for column_name in columns if lower_names[column_name] == name] + for name in cst.DatetimeColumnDetection.EXACT_NAMES + ] + tiers += [ + [column_name for column_name in columns if pattern.search(lower_names[column_name])] + for pattern in _DATETIME_SUBSTRING_TIER_RES + ] + # Widen tier: every column, ignoring name (numeric ones still deferred to epoch tier). + tiers.append(list(columns)) + return tiers + + +def detect_time_axis_in_frame(df: pd.DataFrame) -> tuple[str, pd.Series]: + """ + Detect the datetime column, returning ``(column_name, parsed_series)``. + + Walks the name tiers (exact, then substring buckets), validating content at every + tier; numeric columns are deferred to the epoch tier. Raises ValueError when no + column passes validation (fail loudly — never guess a time axis). + """ + for tier in _name_tiers(list(df.columns)): + passing = [ + (column_name, parsed) + for column_name in tier + if not pd.api.types.is_numeric_dtype(df[column_name]) + and (parsed := _try_parse_datetime_column(df[column_name])) is not None + ] + if passing: + return _pick_best_candidate(passing) + + # Numeric-epoch tier, tried last: nanosecond epochs only (~1.6e18 is unambiguous + # against real measurement data), gated by the same validation. + epoch_passing = [] + for column_name in df.columns: + if not pd.api.types.is_numeric_dtype(df[column_name]): + continue + try: + parsed = pd.to_datetime(df[column_name], unit="ns", errors="coerce") + except (ValueError, TypeError, OverflowError): + continue + if _validate_parsed_datetimes(parsed): + epoch_passing.append((column_name, parsed)) + if epoch_passing: + return _pick_best_candidate(epoch_passing) + + msg = ( + "No datetime column detected: no column passed content validation " + f"(≥90% parseable in [{cst.DatetimeColumnDetection.MIN_YEAR}, " + f"{cst.DatetimeColumnDetection.MAX_YEAR}], ≥90% non-decreasing). " + f"Columns: {list(df.columns)}" + ) + raise ValueError(msg) + + +def _is_numeric_pa_type(field_type: pa.DataType) -> bool: + """ + Schema-only "numeric, defer to the epoch tier" predicate for a pyarrow field type. + + Must agree with :func:`detect_time_axis_in_frame`'s ``pd.api.types.is_numeric_dtype`` + check on every dtype that can appear in a clinical parquet export — the two datetime + detectors (schema-only vs. full-frame) rely on picking the same candidate column. + See :class:`tests.datasource.test_datetime_pushdown.TestNumericTypeClassificationAgreement`. + """ + return pa.types.is_integer(field_type) or pa.types.is_floating(field_type) + + +def _sample_parquet_columns(parquet_file: pq.ParquetFile, columns: list[str]) -> pd.DataFrame: + """ + Read a bounded, spread sample of *columns* for datetime detection. + + Reads whole row groups (parquet's random-access unit) evenly spread across the file, + head-slicing each to ``SAMPLE_ROWS_PER_BLOCK`` — contiguous slices preserve duplicate-value + runs, so the uniqueness and sorted checks in :func:`_pick_best_candidate` / + :func:`_validate_parsed_datetimes` stay meaningful. The whole file is read only when it + fits the decode budget or has a single row group. When it has several, at least + ``SAMPLE_MIN_GROUPS`` are sampled even if the budget alone would pick fewer (huge row + groups), so detection always sees two independent places. + """ + parquet_metadata = parquet_file.metadata + row_group_count = parquet_metadata.num_row_groups + detection_constants = cst.DatetimeColumnDetection + max_row_decoded = detection_constants.SAMPLE_MAX_ROW_DECODED + if row_group_count <= 1 or parquet_metadata.num_rows <= max_row_decoded: + return parquet_file.read(columns=columns).to_pandas() + + rows_per_group = parquet_metadata.row_group(0).num_rows + budget_groups = max(1, max_row_decoded // rows_per_group) + sample_group_count = min(detection_constants.SAMPLE_MAX_GROUPS, row_group_count, budget_groups) + sample_group_count = max( + sample_group_count, min(detection_constants.SAMPLE_MIN_GROUPS, row_group_count) + ) # ≥2 places when ≥2 groups exist + indices = sorted( + { + round(sample_index * (row_group_count - 1) / (sample_group_count - 1)) + for sample_index in range(sample_group_count) + } + ) + rows_per_block = detection_constants.SAMPLE_ROWS_PER_BLOCK + tables = [ + parquet_file.read_row_group(group_index, columns=columns).slice(0, rows_per_block) + for group_index in indices + ] + return pa.concat_tables(tables).to_pandas() + + +def detect_time_axis_in_parquet(path: Path) -> DetectedTimeAxis | None: + """ + Detect the datetime column of a parquet file without a materialized index. + + Mirrors :func:`detect_time_axis_in_frame`'s tiered name search, but reads only + each tier's candidate columns (progressively widening) to validate content, + rather than loading the whole file upfront. + + Returns a :class:`DetectedTimeAxis` whose *kind* is + ``TIMESTAMP`` (direct range filter, *tz* set for tz-aware columns) or + ``EPOCH_NS`` (nanosecond-epoch numeric column, *tz* is ``None``) — see + :class:`~clinical_scope.constants.ParquetPushdownKind`. Both are safe for + an unambiguous parquet row filter. Any other resolved type (e.g. a string datetime + column, unparsed) is not pushdown-safe and yields ``None``, so the caller falls + back to a full unfiltered read. + + Its *tz* is the *semantic* timezone (from :func:`_pick_best_candidate`, which + force-localizes a tz-naive utc-named column to UTC — matching what + :func:`set_datetime_index` does downstream) and can therefore diverge from the + column's on-disk type, which stays tz-naive. Its *tz_from_name* flags that one case, so + the caller can strip the label back off before filtering — pyarrow filter values must + match the stored type exactly. + + Candidate columns are validated on a bounded sample (:func:`_sample_parquet_columns`), + not the whole file, so this pick can diverge from the downstream full-frame + :func:`set_datetime_index` — which would filter one column and index another (silent row + loss). To stay safe, detection only ever consults the **highest-priority tier that has any + named candidate**, and commits only if that tier yields **exactly one** sample-validated + column; otherwise it abstains (returns ``None`` → full read, full-frame decides): + + - **zero passing** there — a higher-priority column we couldn't confirm on the sample + (e.g. valid over the whole file but garbage in exactly the sampled row groups) may still + validate on the full frame and outrank any lower-tier pick, so we must not look lower. + - **more than one** — the sample-based uniqueness tiebreak in :func:`_pick_best_candidate` + isn't stable, so the pick could differ from the full frame's. + """ + parquet_file = pq.ParquetFile(path) + schema = parquet_file.schema_arrow + + def _is_numeric(column_name: str) -> bool: + return _is_numeric_pa_type(schema.field(column_name).type) + + columns = list(schema.names) + for tier in _name_tiers(columns): + candidates = [column_name for column_name in tier if not _is_numeric(column_name)] + if not candidates: + continue + sample = _sample_parquet_columns(parquet_file, candidates) + passing = [ + (column_name, parsed) + for column_name in candidates + if (parsed := _try_parse_datetime_column(sample[column_name])) is not None + ] + if len(passing) != 1: + return None + column_name, parsed = _pick_best_candidate(passing) + field_type = schema.field(column_name).type + if pa.types.is_timestamp(field_type): + # Use the resolved parsed tz, not the stored field's — _pick_best_candidate + # force-localizes utc-named naive columns to UTC, matching set_datetime_index + # downstream, so the row-filter bounds must agree with it rather than with disk. + tz = parsed.dt.tz + return DetectedTimeAxis( + column_name, + cst.ParquetPushdownKind.TIMESTAMP, + str(tz) if tz else None, + tz_from_name=tz is not None and field_type.tz is None, + ) + return DetectedTimeAxis( + column_name, cst.ParquetPushdownKind.OTHER, None, tz_from_name=False + ) + + numeric_columns = [column_name for column_name in columns if _is_numeric(column_name)] + if numeric_columns: + sample = _sample_parquet_columns(parquet_file, numeric_columns) + epoch_passing = [] + for column_name in numeric_columns: + try: + parsed = pd.to_datetime(sample[column_name], unit="ns", errors="coerce") + except (ValueError, TypeError, OverflowError): + continue + if _validate_parsed_datetimes(parsed): + epoch_passing.append((column_name, parsed)) + if len(epoch_passing) > 1: # no named tier hid a candidate here — only the tiebreak can + return None + if epoch_passing: + column_name, _parsed = _pick_best_candidate(epoch_passing) + return DetectedTimeAxis( + column_name, cst.ParquetPushdownKind.EPOCH_NS, None, tz_from_name=False + ) + + return None + + +def set_datetime_index(df: pd.DataFrame) -> pd.DataFrame: + """ + Return *df* indexed by its detected datetime column. + + Short-circuits when the index is already a DatetimeIndex; otherwise detects, + parses, and sets the best-validated datetime column (raises if none passes). + """ + if isinstance(df.index, pd.DatetimeIndex): + return df + column_name, parsed = detect_time_axis_in_frame(df) + df = df.copy() + df[column_name] = parsed + return df.set_index(column_name) + + +def deduplicate_then_sort_index(df: pd.DataFrame) -> pd.DataFrame: + """ + Drop duplicate index entries (keep first) *then* sort by index. + + Deduplicating first keeps the first row in file order on a timestamp + collision, which a non-stable ``sort_index`` would decide arbitrarily. + Skips either step when already satisfied (device exports are usually + already sorted and unique). + """ + if not df.index.is_unique: + df = df[~df.index.duplicated(keep="first")] + if not df.index.is_monotonic_increasing: + df = df.sort_index() + return df diff --git a/src/clinical_scope/signal_container.py b/src/clinical_scope/signal_container.py index d512c2c..7f353f7 100644 --- a/src/clinical_scope/signal_container.py +++ b/src/clinical_scope/signal_container.py @@ -19,7 +19,8 @@ resolve_display_timezone, to_float_seconds, ) -from clinical_scope.io.file_utils import get_column_name_from_pattern +from clinical_scope.io.column_patterns import get_column_name_from_pattern +from clinical_scope.io.export import print_out_figure from clinical_scope.io.paths import get_visualization_path logger = logging.getLogger(__name__) @@ -1250,25 +1251,3 @@ def wrap_label(text: str, max_line_length: int = 12, break_chars: str = r"[ \-_] lines.append(current_line.strip()) return "
".join(lines) - - -# ================================================================================================== -def print_out_figure(path_output: Path, fig_list: list, self_contained: bool = False) -> None: - """ - Export Plotly figures to a single HTML file. - - With *self_contained*, plotly.js is embedded once (in the first figure; the rest reuse it) - so the file renders on a machine with no network — at ~3.5 MB. Otherwise it is fetched - from a CDN, which keeps the file small but shows a blank page offline. - """ - path_output.parent.mkdir(parents=True, exist_ok=True) - with Path.open(path_output, "w") as file_out: - for figure_index, fig in enumerate(fig_list): - if self_contained: - # Embedding the ~3.5 MB bundle once per file, not once per figure. - include_plotlyjs = ( - cst.HtmlExport.INLINE if figure_index == 0 else cst.HtmlExport.OMIT - ) - else: - include_plotlyjs = cst.HtmlExport.CDN - file_out.write(fig.to_html(full_html=False, include_plotlyjs=include_plotlyjs)) diff --git a/tests/datasource/test_column_pruning.py b/tests/datasource/test_column_pruning.py index dd15295..4bd2c90 100644 --- a/tests/datasource/test_column_pruning.py +++ b/tests/datasource/test_column_pruning.py @@ -27,11 +27,8 @@ import clinical_scope.constants as cst from clinical_scope.datasource.base import DataSourceBase -from clinical_scope.io.file_utils import ( - _pruned_columns, - get_column_name_from_pattern, - read_parquet_pruned, -) +from clinical_scope.io.column_patterns import _pruned_columns, get_column_name_from_pattern +from clinical_scope.io.parquet_pruning import read_cache_pruned, read_parquet_pruned OTHER_DIR = ( Path(__file__).resolve().parent.parent @@ -235,10 +232,9 @@ class TestReadParquetPrunedNonTemporalIndex: def test_pruned_read_equals_full_subset(self, tmp_path, field_display): path = _make_non_temporal_index_parquet(tmp_path) full = pd.read_parquet(path) - actual = read_parquet_pruned( + actual = read_cache_pruned( path, select_columns=lambda names: _pruned_columns(field_display, names), - index_is_time_axis=True, ) assert list(actual.columns) == _expected_cols(full, field_display) # pandas restores the index without it ever being named in columns=, so unlike the @@ -259,10 +255,9 @@ def test_declaration_is_inert_without_a_stored_index(self, tmp_path): """A RangeIndex file has no index column to vouch for, so detection still runs.""" path = _make_nonmaterialized_parquet(tmp_path) full = pd.read_parquet(path) - actual = read_parquet_pruned( + actual = read_cache_pruned( path, select_columns=lambda names: _pruned_columns(["pre_a"], names), - index_is_time_axis=True, ) assert set(actual.columns) == {"timestamp", "pre_a"} pd.testing.assert_frame_equal(actual, full[list(actual.columns)]) diff --git a/tests/datasource/test_datetime_pushdown.py b/tests/datasource/test_datetime_pushdown.py index 93ec18a..eb5a023 100644 --- a/tests/datasource/test_datetime_pushdown.py +++ b/tests/datasource/test_datetime_pushdown.py @@ -17,12 +17,11 @@ import clinical_scope.constants as cst from clinical_scope.datasource.formatting.timezone import to_aware_display_ts -from clinical_scope.io.file_utils import ( - _detect_datetime_column_from_parquet, - _find_datetime_col_parsed, - _is_numeric_pa_type, - load_parquet_with_datetime_index, - read_parquet_pruned, +from clinical_scope.io.parquet_pruning import read_cache_pruned, read_parquet_pruned +from clinical_scope.io.time_axis import ( + detect_time_axis_in_frame, + detect_time_axis_in_parquet, + set_datetime_index, ) OTHER_DIR = ( @@ -131,9 +130,7 @@ def test_window_between_two_rows_is_empty_but_matches(self, path, tz): pd.testing.assert_frame_equal(actual, expected) def test_no_bounds_returns_full_unfiltered_read(self): - actual = read_parquet_pruned( - TZ_AWARE_STORED_INDEX_PARQUET, compute_bounds=lambda _tz: None - ) + actual = read_parquet_pruned(TZ_AWARE_STORED_INDEX_PARQUET, compute_bounds=lambda _tz: None) expected = pd.read_parquet(TZ_AWARE_STORED_INDEX_PARQUET) pd.testing.assert_frame_equal(actual, expected) @@ -475,9 +472,7 @@ def test_naive_utc_named_column_with_timezone_override_matches_disabled( finally: other_cls.ALLOW_DATETIME_PUSHDOWN = original target_disabled = { - s.raw_name: s - for s in signals_disabled - if s.raw_name.startswith("device_export::") + s.raw_name: s for s in signals_disabled if s.raw_name.startswith("device_export::") } # Guards the original bug directly: pushdown used to silently return zero signals here. @@ -511,9 +506,9 @@ def test_spread_sample_still_detects_across_many_row_groups(self, tmp_path, monk df.to_parquet(path, row_group_size=100) # 20 row groups assert pq.ParquetFile(path).num_row_groups > 1 # spread path is actually exercised - detected = _detect_datetime_column_from_parquet(path) + detected = detect_time_axis_in_parquet(path) assert detected is not None - col, kind, _tz, _naive = detected + col, kind = detected.column_name, detected.kind assert (col, kind) == ("timestamp", "timestamp") def test_two_datetime_columns_in_one_tier_abstains_and_reads_all(self, tmp_path): @@ -533,7 +528,7 @@ def test_two_datetime_columns_in_one_tier_abstains_and_reads_all(self, tmp_path) ) df.to_parquet(path) # default RangeIndex → detection path, not stored-index - assert _detect_datetime_column_from_parquet(path) is None + assert detect_time_axis_in_parquet(path) is None start = pd.Timestamp("2020-01-01 00:00:10") end = pd.Timestamp("2020-01-01 00:00:20") @@ -568,17 +563,19 @@ def test_higher_tier_column_hidden_by_sample_does_not_desync(self, tmp_path, mon # The file genuinely has a valid higher-priority 'datetime' column (garbage only in the # sampled groups), so the authoritative full-frame detector indexes on it... - assert _find_datetime_col_parsed(pd.read_parquet(path))[0] == "datetime" + assert detect_time_axis_in_frame(pd.read_parquet(path))[0] == "datetime" # ...while sampled pushdown can't confirm it and must abstain — never pick 'timestamp' # and prune on it (the desync). Abstaining falls back to a full read. - assert _detect_datetime_column_from_parquet(path) is None + assert detect_time_axis_in_parquet(path) is None # Window over the middle (valid) region, expressed on the authoritative 'datetime' axis. start = pd.Timestamp("2020-01-01 10:00:00") end = pd.Timestamp("2020-01-01 13:00:00") - enabled = load_parquet_with_datetime_index(path, compute_bounds=lambda _tz: (start, end)) - disabled = load_parquet_with_datetime_index(path) # no pushdown → full read + enabled = set_datetime_index( + read_parquet_pruned(path, compute_bounds=lambda _tz: (start, end)) + ) + disabled = set_datetime_index(pd.read_parquet(path)) # no pushdown → full read # The authoritative datetime-window cut (_filter_by_datetime) runs on both downstream. enabled = enabled[(enabled.index >= start) & (enabled.index <= end)] @@ -587,14 +584,72 @@ def test_higher_tier_column_hidden_by_sample_does_not_desync(self, tmp_path, mon pd.testing.assert_frame_equal(enabled, disabled) +class TestNameAssertedTimezoneBounds: + """ + The one input that reaches the tz-label strip: a utc-named column stored tz-NAIVE. + + Detection asserts UTC from the name (`_pick_best_candidate`), so the semantic tz and the + on-disk type disagree. A bound expressed in any zone must still land on the right instant: + pyarrow compares against the bare stored values, and an unconverted label would shift the + window by the offset. + """ + + @staticmethod + def _naive_utc_parquet(tmp_path): + path = tmp_path / "naive_utc.parquet" + pd.DataFrame( + { + "time_utc": pd.to_datetime([f"2020-07-01 {hour}:00:00" for hour in range(11, 16)]), + "value": range(5), + } + ).to_parquet(path, index=False) + return path + + def test_detection_asserts_utc_over_a_naive_column(self, tmp_path): + detected = detect_time_axis_in_parquet(self._naive_utc_parquet(tmp_path)) + assert (detected.tz, detected.tz_from_name) == ("UTC", True) + + @pytest.mark.parametrize( + "start_text,end_text,bound_tz,expected_hours", + [ + ("12:00", "15:00", "UTC", [12, 13, 14, 15]), + # Two different zones naming the same instants must select the same rows. + ("12:00", "15:00", "Europe/Paris", [11, 12, 13]), # +02:00 -> 10:00-13:00Z + ("07:00", "09:00", "America/New_York", [11, 12, 13]), # -04:00 -> 11:00-13:00Z + ], + ) + def test_window_lands_on_the_same_instant_whatever_zone_expressed_it( + self, tmp_path, start_text, end_text, bound_tz, expected_hours + ): + path = self._naive_utc_parquet(tmp_path) + start = pd.Timestamp(f"2020-07-01 {start_text}", tz=bound_tz) + end = pd.Timestamp(f"2020-07-01 {end_text}", tz=bound_tz) + + pushed = read_parquet_pruned(path, compute_bounds=lambda tz: (start, end)) + assert sorted(set_datetime_index(pushed).index.hour.tolist()) == expected_hours + + def test_pushdown_never_drops_a_row_the_authoritative_cut_keeps(self, tmp_path): + """Pushdown may under-prune; it may never lose a row the downstream filter would keep.""" + path = self._naive_utc_parquet(tmp_path) + start = pd.Timestamp("2020-07-01 12:00", tz="Europe/Paris") + end = pd.Timestamp("2020-07-01 15:00", tz="Europe/Paris") + + pushed = set_datetime_index( + read_parquet_pruned(path, compute_bounds=lambda tz: (start, end)) + ) + full = set_datetime_index(pd.read_parquet(path)) + cut = full[(full.index >= start) & (full.index <= end)] + + assert cut.index.isin(pushed.index).all() + assert len(pushed) < len(full) # it really did prune + + class TestInspectIgnoresPushdownWindow: """inspect() always passes apply_datetime_pushdown=False (base.py) — it needs whole-file raw stats, so a narrow datetime_start/end window set by patient_options must not shrink the reported raw_date_range. By design, not incidental.""" - def test_narrow_window_does_not_shrink_raw_date_range( - self, servo_u_cls, patient_full_path - ): + def test_narrow_window_does_not_shrink_raw_date_range(self, servo_u_cls, patient_full_path): base_options = { "data_folder": str(patient_full_path), "quick_load": False, @@ -612,47 +667,6 @@ def test_narrow_window_does_not_shrink_raw_date_range( assert narrow.raw_date_range == full.raw_date_range -class TestNumericTypeClassificationAgreement: - """ - Tripwire for a code-review finding on issue #57: schema-only detection - (`_is_numeric_pa_type`, pyarrow-type-based) and full-frame detection - (`_find_datetime_col_parsed`, `pd.api.types.is_numeric_dtype`-based) each decide - independently whether a column is "numeric" and should be deferred to the epoch tier. - If they ever disagree on a dtype that also passes datetime-content validation, the - pushdown fast-path and the full unfiltered read could pick *different* datetime - columns — silently wrong filter results, not an error. - - This pins today's known-good agreement so a future pandas/pyarrow upgrade that shifts - either classification is caught here first, rather than downstream as a silent mismatch. - """ - - # Every dtype that can plausibly appear as a real clinical parquet column. - NUMERIC_TYPES = [pa.int32(), pa.int64(), pa.float32(), pa.float64()] - NON_NUMERIC_TYPES = [pa.string(), pa.timestamp("ns"), pa.timestamp("ns", tz="UTC")] - - @pytest.mark.parametrize("pa_type", NUMERIC_TYPES) - def test_numeric_types_agree(self, pa_type): - assert _is_numeric_pa_type(pa_type) is True - assert pd.api.types.is_numeric_dtype(pa_type.to_pandas_dtype()) is True - - @pytest.mark.parametrize("pa_type", NON_NUMERIC_TYPES) - def test_non_numeric_types_agree(self, pa_type): - assert _is_numeric_pa_type(pa_type) is False - assert pd.api.types.is_numeric_dtype(pa_type.to_pandas_dtype()) is False - - def test_known_bool_divergence_is_unchanged(self): - """ - The one known gap (code-review finding, deliberately not fixed): schema-only - treats bool as non-numeric, pandas treats it as numeric. Harmless today because - both paths still reject a bool column as a datetime candidate (schema-only fails - string-parse; full-frame fails the epoch-ns year-range check) — but if this - assertion ever starts failing, the two paths' agreement has shifted and - `_is_numeric_pa_type` should be revisited. - """ - assert _is_numeric_pa_type(pa.bool_()) is False - assert pd.api.types.is_numeric_dtype(pa.bool_().to_pandas_dtype()) is True - - class TestEitPushdownOptOut: """EIT filters by time-of-day — a min/max pushdown predicate can't express that.""" @@ -679,7 +693,7 @@ def compute_bounds(tz): requested_tz.append(tz) return pd.Timestamp("2020-01-01"), pd.Timestamp("2020-01-02") - actual = read_parquet_pruned(path, compute_bounds=compute_bounds, index_is_time_axis=True) + actual = read_cache_pruned(path, compute_bounds=compute_bounds) assert requested_tz == [] pd.testing.assert_frame_equal(actual, pd.read_parquet(path)) @@ -687,11 +701,10 @@ def compute_bounds(tz): def test_row_filter_would_have_emptied_the_frame(self, tmp_path): """Every row survives, so the window could not have been quietly applied.""" path = self._float_index_parquet(tmp_path) - actual = read_parquet_pruned( + actual = read_cache_pruned( path, compute_bounds=lambda _tz: (pd.Timestamp("2020-01-01"), pd.Timestamp("2020-01-02")), select_columns=lambda names: ["Global"], - index_is_time_axis=True, ) assert list(actual.columns) == ["Global"] assert len(actual) == len(pd.read_parquet(path)) diff --git a/tests/unit/test_data_processor.py b/tests/unit/test_data_processor.py index 779d8c5..47f3ece 100644 --- a/tests/unit/test_data_processor.py +++ b/tests/unit/test_data_processor.py @@ -6,7 +6,7 @@ - wrapper.extract_datasource() (wrapper.py) - wrapper.extract_patient() (wrapper.py) - wrapper.batch_extract() (wrapper.py) -- file_utils.save_df() (file_utils.py) +- export.save_df() (io/export.py) - datasource_list.detect_datasource_from_folder() (datasource_list.py) """ @@ -18,7 +18,7 @@ from clinical_scope import wrapper from clinical_scope.datasource.registry import detect_datasource_from_folder -from clinical_scope.io.file_utils import save_df +from clinical_scope.io.export import save_df # ================================================================================================== # Helpers diff --git a/tests/unit/test_deduplicate_then_sort_index.py b/tests/unit/test_deduplicate_then_sort_index.py index 6d43811..1841c31 100644 --- a/tests/unit/test_deduplicate_then_sort_index.py +++ b/tests/unit/test_deduplicate_then_sort_index.py @@ -8,7 +8,7 @@ import numpy as np import pandas as pd -from clinical_scope.io.file_utils import deduplicate_then_sort_index +from clinical_scope.io.time_axis import deduplicate_then_sort_index def test_already_sorted_and_unique_is_returned_without_copying(): diff --git a/tests/unit/test_find_files.py b/tests/unit/test_find_files.py index 1f684a9..1cf1004 100644 --- a/tests/unit/test_find_files.py +++ b/tests/unit/test_find_files.py @@ -12,7 +12,7 @@ from pathlib import Path -from clinical_scope.io.file_utils import find_files +from clinical_scope.io.discovery import find_files # --------------------------------------------------------------------------- # Helpers diff --git a/tests/unit/test_junk_files.py b/tests/unit/test_junk_files.py index 978f9a9..70ae9fc 100644 --- a/tests/unit/test_junk_files.py +++ b/tests/unit/test_junk_files.py @@ -8,7 +8,7 @@ from pathlib import Path -from clinical_scope.io.file_utils import folder_has_real_content, is_junk_file +from clinical_scope.io.discovery import folder_has_real_content, is_junk_file def create(tmp_path: Path, *names: str) -> list[Path]: diff --git a/tests/unit/test_signal_container.py b/tests/unit/test_signal_container.py index 760e4c9..df846a6 100644 --- a/tests/unit/test_signal_container.py +++ b/tests/unit/test_signal_container.py @@ -17,8 +17,8 @@ compute_average_priority, get_unique_or_raise, merge_y_ranges, - print_out_figure, ) +from clinical_scope.io.export import print_out_figure from clinical_scope.spectral import SpectralRefusalError # --------------------------------------------------------------------------- diff --git a/tests/unit/test_find_datetime_col.py b/tests/unit/test_time_axis.py similarity index 86% rename from tests/unit/test_find_datetime_col.py rename to tests/unit/test_time_axis.py index afc7fd1..48088c0 100644 --- a/tests/unit/test_find_datetime_col.py +++ b/tests/unit/test_time_axis.py @@ -10,11 +10,12 @@ """ import pandas as pd +import pyarrow as pa import pytest -from clinical_scope.io.file_utils import ( - load_csv_with_datetime_index, - load_parquet_with_datetime_index, +from clinical_scope.io.time_axis import ( + _is_numeric_pa_type, + detect_time_axis_in_frame, set_datetime_index, ) @@ -196,7 +197,7 @@ def test_load_parquet_short_circuits_pre_indexed_file(self, tmp_path): ) path = tmp_path / "data.parquet" df.to_parquet(path) - result = load_parquet_with_datetime_index(path) + result = set_datetime_index(pd.read_parquet(path)) assert isinstance(result.index, pd.DatetimeIndex) assert list(result.columns) == ["value"] @@ -204,7 +205,7 @@ def test_load_parquet_detects_datetime_column(self, tmp_path): df = pd.DataFrame({"timestamp": dt_strings(10), "value": range(10)}) path = tmp_path / "data.parquet" df.to_parquet(path) - result = load_parquet_with_datetime_index(path) + result = set_datetime_index(pd.read_parquet(path)) assert isinstance(result.index, pd.DatetimeIndex) assert result.index.name == "timestamp" @@ -212,7 +213,7 @@ def test_load_csv_detects_datetime_column(self, tmp_path): df = pd.DataFrame({"acquisition_time": dt_strings(10), "value": range(10)}) path = tmp_path / "data.csv" df.to_csv(path, index=False) - result = load_csv_with_datetime_index(path) + result = set_datetime_index(pd.read_csv(path)) assert isinstance(result.index, pd.DatetimeIndex) assert result.index.name == "acquisition_time" @@ -221,7 +222,7 @@ def test_load_csv_raises_on_undetectable_file(self, tmp_path): path = tmp_path / "data.csv" df.to_csv(path, index=False) with pytest.raises(ValueError, match="No datetime column detected"): - load_csv_with_datetime_index(path) + set_datetime_index(pd.read_csv(path)) # =========================================================================== @@ -446,3 +447,44 @@ def test_non_utc_named_winner_stays_naive(self): df = pd.DataFrame({"timestamp": dt_strings(10), "value": range(10)}) result = set_datetime_index(df) assert result.index.tz is None + + +class TestNumericTypeClassificationAgreement: + """ + Tripwire for a code-review finding on issue #57: schema-only detection + (`_is_numeric_pa_type`, pyarrow-type-based) and full-frame detection + (`detect_time_axis_in_frame`, `pd.api.types.is_numeric_dtype`-based) each decide + independently whether a column is "numeric" and should be deferred to the epoch tier. + If they ever disagree on a dtype that also passes datetime-content validation, the + pushdown fast-path and the full unfiltered read could pick *different* datetime + columns — silently wrong filter results, not an error. + + This pins today's known-good agreement so a future pandas/pyarrow upgrade that shifts + either classification is caught here first, rather than downstream as a silent mismatch. + """ + + # Every dtype that can plausibly appear as a real clinical parquet column. + NUMERIC_TYPES = [pa.int32(), pa.int64(), pa.float32(), pa.float64()] + NON_NUMERIC_TYPES = [pa.string(), pa.timestamp("ns"), pa.timestamp("ns", tz="UTC")] + + @pytest.mark.parametrize("pa_type", NUMERIC_TYPES) + def test_numeric_types_agree(self, pa_type): + assert _is_numeric_pa_type(pa_type) is True + assert pd.api.types.is_numeric_dtype(pa_type.to_pandas_dtype()) is True + + @pytest.mark.parametrize("pa_type", NON_NUMERIC_TYPES) + def test_non_numeric_types_agree(self, pa_type): + assert _is_numeric_pa_type(pa_type) is False + assert pd.api.types.is_numeric_dtype(pa_type.to_pandas_dtype()) is False + + def test_known_bool_divergence_is_unchanged(self): + """ + The one known gap (code-review finding, deliberately not fixed): schema-only + treats bool as non-numeric, pandas treats it as numeric. Harmless today because + both paths still reject a bool column as a datetime candidate (schema-only fails + string-parse; full-frame fails the epoch-ns year-range check) — but if this + assertion ever starts failing, the two paths' agreement has shifted and + `_is_numeric_pa_type` should be revisited. + """ + assert _is_numeric_pa_type(pa.bool_()) is False + assert pd.api.types.is_numeric_dtype(pa.bool_().to_pandas_dtype()) is True From 7acc156faeaf08684a5e814a764e6a0229193a96 Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Wed, 26 Aug 2026 12:51:14 +0200 Subject: [PATCH 5/9] Fix latent KeyError when sampling a stored non-temporal index Detection samples candidate columns by name, but _sample_parquet_columns let pyarrow restore pandas metadata -- so a materialized index column came back as the frame's index and `sample[column_name]` raised KeyError. Any plain parquet with a stored non-temporal index (a float or int index, not one of our own caches) crashed instead of declining to prune. Predates the io/file_utils split; found by the tests added here. Reading with ignore_metadata=True keeps every requested column addressable by name. Our own caches are unaffected: they declare provenance through read_cache_pruned and never reach detection. Tests cover the pruning decisions through the two readers rather than through the plan they produce: the plan's shape is implementation, and "a row predicate was built" is an optimization under ADR-0007, not behaviour. Expectations come from a full read plus a plain pandas slice, never from replaying the library's own resolution. Co-Authored-By: Claude Opus 5 --- src/clinical_scope/io/time_axis.py | 6 +- tests/datasource/test_datetime_pushdown.py | 89 ++++++++++++++++++++++ 2 files changed, 93 insertions(+), 2 deletions(-) diff --git a/src/clinical_scope/io/time_axis.py b/src/clinical_scope/io/time_axis.py index 7520975..23ef391 100644 --- a/src/clinical_scope/io/time_axis.py +++ b/src/clinical_scope/io/time_axis.py @@ -189,12 +189,14 @@ def _sample_parquet_columns(parquet_file: pq.ParquetFile, columns: list[str]) -> ``SAMPLE_MIN_GROUPS`` are sampled even if the budget alone would pick fewer (huge row groups), so detection always sees two independent places. """ + # ignore_metadata keeps a materialized index column addressable by name: pandas metadata + # would restore it as the frame's index, and a candidate can be exactly that column. parquet_metadata = parquet_file.metadata row_group_count = parquet_metadata.num_row_groups detection_constants = cst.DatetimeColumnDetection max_row_decoded = detection_constants.SAMPLE_MAX_ROW_DECODED if row_group_count <= 1 or parquet_metadata.num_rows <= max_row_decoded: - return parquet_file.read(columns=columns).to_pandas() + return parquet_file.read(columns=columns).to_pandas(ignore_metadata=True) rows_per_group = parquet_metadata.row_group(0).num_rows budget_groups = max(1, max_row_decoded // rows_per_group) @@ -213,7 +215,7 @@ def _sample_parquet_columns(parquet_file: pq.ParquetFile, columns: list[str]) -> parquet_file.read_row_group(group_index, columns=columns).slice(0, rows_per_block) for group_index in indices ] - return pa.concat_tables(tables).to_pandas() + return pa.concat_tables(tables).to_pandas(ignore_metadata=True) def detect_time_axis_in_parquet(path: Path) -> DetectedTimeAxis | None: diff --git a/tests/datasource/test_datetime_pushdown.py b/tests/datasource/test_datetime_pushdown.py index eb5a023..87323be 100644 --- a/tests/datasource/test_datetime_pushdown.py +++ b/tests/datasource/test_datetime_pushdown.py @@ -150,6 +150,95 @@ def compute_bounds(tz): assert seen_tz == [None] +# --------------------------------------------------------------------------- +# Pruning decisions, observed through the two readers +# --------------------------------------------------------------------------- + + +class TestPruningDecisionsThroughTheReader: + """ + What each pruning decision does to the frame a caller receives. + + Every expectation is an independent full read plus a plain pandas slice -- never the + library's own resolution replayed -- so a wrong decision shows up as wrong data rather + than as agreement with itself. + """ + + @staticmethod + def _detection_parquet(tmp_path: Path) -> Path: + """RangeIndex file: the time axis is a column, so detection has to find it.""" + path = tmp_path / "detected.parquet" + pd.DataFrame( + { + "timestamp": pd.date_range("2020-01-01", periods=50, freq="1min"), + "pre_a": range(50), + "other": range(50), + } + ).to_parquet(path, index=False) + return path + + @staticmethod + def _float_index_parquet(tmp_path: Path) -> Path: + """EIT-shaped: a stored index that is the time axis but is not range-comparable.""" + path = tmp_path / "float_index.parquet" + index = pd.Index([minute / 1440 for minute in range(60)], name="Time") + pd.DataFrame({"Global": range(60), "Local 1": range(60, 120)}, index=index).to_parquet(path) + return path + + def test_time_axis_survives_a_selection_that_omits_it(self, tmp_path): + """Dropping the axis would strand the frame with no index to set downstream.""" + path = self._detection_parquet(tmp_path) + actual = read_parquet_pruned(path, select_columns=lambda _names: ["pre_a"]) + assert "timestamp" in actual.columns + pd.testing.assert_frame_equal(actual, pd.read_parquet(path)[list(actual.columns)]) + + def test_unresolvable_axis_reads_every_column(self, tmp_path): + """With no detectable axis there is nothing to protect, so pruning is declined.""" + path = tmp_path / "no_axis.parquet" + pd.DataFrame({"pre_a": range(10), "other": range(10)}).to_parquet(path, index=False) + actual = read_parquet_pruned(path, select_columns=lambda _names: ["pre_a"]) + pd.testing.assert_frame_equal(actual, pd.read_parquet(path)) + + def test_column_pruning_does_not_need_a_window(self, tmp_path): + """The two prunings are orthogonal -- the common case is a wide file and no window.""" + path = self._detection_parquet(tmp_path) + full = pd.read_parquet(path) + actual = read_parquet_pruned( + path, compute_bounds=lambda _tz: None, select_columns=lambda _names: ["pre_a"] + ) + assert "other" not in actual.columns + assert len(actual) == len(full) + + def test_epoch_column_window_matches_a_plain_slice(self, tmp_path): + """A numeric epoch axis needs integer bounds on disk; wrong types read wrong rows.""" + path = tmp_path / "epoch.parquet" + stamps = pd.date_range("2020-01-01", periods=50, freq="1min") + pd.DataFrame( + {"epoch": stamps.as_unit("ns").astype("int64"), "value": range(50)} + ).to_parquet(path, index=False) + start, end = stamps[10], stamps[20] + + actual = read_parquet_pruned(path, compute_bounds=lambda _tz: (start, end)) + + full = pd.read_parquet(path) + as_time = pd.to_datetime(full["epoch"], unit="ns") + expected = full[(as_time >= start) & (as_time <= end)] + pd.testing.assert_frame_equal( + actual.reset_index(drop=True), expected.reset_index(drop=True) + ) + assert 0 < len(actual) < len(full) + + def test_stored_non_temporal_index_is_read_not_rejected(self, tmp_path): + """ + Regression: detection samples a materialized index column by name. Restoring pandas + metadata would turn it back into the frame's index mid-detection and raise KeyError, + so a plain parquet with a float index used to crash instead of declining to prune. + """ + path = self._float_index_parquet(tmp_path) + actual = read_parquet_pruned(path, select_columns=lambda _names: ["Global"]) + pd.testing.assert_frame_equal(actual, pd.read_parquet(path)) + + # --------------------------------------------------------------------------- # _pushdown_bounds: conservative-loose bounds computation (base.py) # --------------------------------------------------------------------------- From 707905520f33645af79da8a2f9b1b63bcb5c0348 Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Wed, 26 Aug 2026 15:30:27 +0200 Subject: [PATCH 6/9] Lift PlotGroup assembly out of wrapper.main (ADR-0013) Grouping and derived-plot construction were pure domain logic reachable only by running the whole pipeline against a folder on disk. They now live in plot_assembly.py, called once after the datasource loop, and are exercised with in-memory Signals and literal config dicts. Two rules replace the two-path shape (ADR-0013): Config scope is desugared once. A per-datasource section is a namespace, not a different kind of grouping, so its references are resolved against that datasource's own signals and re-emitted as qualified global ones before anything else runs. Downstream there is one resolver, one suppression rule and one spelling of a reference. Both config spellings stay valid: no parser change, no config migration. Group membership joins on signal identity. A raw name is unique only within a datasource, so the string-keyed accumulator and the post-hoc prune each dropped a plot the first time two sources shared a name -- ordinary for HR, SpO2 and ABP, and silent when it happened. Derived signals are new objects that were never in the input list, so they are structurally immune to suppression and need no naming disguise. Three deliberate behaviour changes, all in CHANGELOG: same-named signals in different datasources stop suppressing each other; a group that resolves to one signal keeps the group's name; and local loop / spectrogram / grouped_fields resolve display names. wrapper.py 931 -> 562 lines; main gains a docstring and loses the outer try/except that wrapped the moved steps. Co-Authored-By: Claude Opus 5 --- .../skills/generate-database-options/SKILL.md | 2 +- CHANGELOG.md | 6 + CLAUDE.md | 5 +- docs/adr/0009-other-stem-is-a-config-scope.md | 4 +- ...eferences-are-qualified-before-assembly.md | 48 ++ src/clinical_scope/plot_assembly.py | 526 ++++++++++++++++++ src/clinical_scope/signal_container.py | 4 +- src/clinical_scope/wrapper.py | 453 ++------------- tests/datasource/test_other.py | 2 +- tests/unit/test_plot_assembly.py | 201 +++++++ .../unit/test_signal_reference_resolution.py | 4 +- 11 files changed, 835 insertions(+), 420 deletions(-) create mode 100644 docs/adr/0013-signal-references-are-qualified-before-assembly.md create mode 100644 src/clinical_scope/plot_assembly.py create mode 100644 tests/unit/test_plot_assembly.py diff --git a/.claude/skills/generate-database-options/SKILL.md b/.claude/skills/generate-database-options/SKILL.md index c88b73c..b57f8ef 100644 --- a/.claude/skills/generate-database-options/SKILL.md +++ b/.claude/skills/generate-database-options/SKILL.md @@ -129,7 +129,7 @@ Use this table to guide refinements and answer user questions. ### Known limitations (tell the user if relevant) - ℹ️ **`global.loop`** (cross-datasource phase loops): **supported** — see - `wrapper.py::_resolve_signal_references`. Two signals per loop, each resolved via the + `plot_assembly.py::_resolve_signal_references`. Two signals per loop, each resolved via the 3-mode chain (qualified `datasource::raw_name` → display name → raw name fallback). See `docs/user_guide/tutorial.md` → *Global Loops vs. Per-Source Loops*. - ℹ️ **Qualified signal references** (`"datasource::signal_name"`): supported in both diff --git a/CHANGELOG.md b/CHANGELOG.md index c79d66a..ac33a31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ All notable changes to this project will be documented in this file. ## [Unreleased] ### Fixed +- **A group in one data source no longer hides a same-named signal in another.** Grouping matched signals by name across the whole run, but a name like `HR`, `SpO2` or `ABP` is only unique *within* one source. So grouping `HR` under your monitor could silently drop the ventilator's `HR` from the page — no error, just a missing plot, and which one disappeared depended on the order the sources happened to load in. Groups now take the signals they actually name, and nothing else. + +- **A group that finds only one of its signals now keeps the group's name.** A group configured over four pressures but resolving to one used to be titled after that surviving signal — so the same configuration gave a differently-named panel depending on how much data a recording happened to contain. It is now always titled after the group. + +- **`loop`, `spectrogram` and `psd` written inside a data source's own section accept the same signal references as `global` ones.** They matched raw column names only; a display name (the `label` you configured) silently matched nothing. They now resolve display names too. A loop given other than exactly two signals is reported as a skipped plot instead of an unexplained error in the log. + - **`trace_options` now applies to every datasource, not only `other::` files.** The block was accepted and validated in any `database_options` section, and the Excel sentinel row wrote it for any datasource, but only `other` ever read it — anywhere else it validated cleanly and did nothing. It now works everywhere it was already accepted. **What changes for you:** a configuration that already sets `trace_options` (or the Excel `trace_mode` / `line_width` / `opacity` / `marker_symbol` columns) on a device datasource starts taking effect, where before it was ignored. Where a datasource ships its own trace style, your block now wins key by key over it; keys you leave unset keep the shipped value. Nothing changes for a configuration that only styled `other::` files. diff --git a/CLAUDE.md b/CLAUDE.md index 681d4c6..7ca5da6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,6 +16,7 @@ CLI scripts (extract / inspect / visualize) and the Python API are documented in ``` src/clinical_scope/ wrapper.py main pipeline — visualize / extract / inspect + plot_assembly.py Signals + database_options → PlotGroups (grouping, derived plots) signal_container.py Signal / PlotGroup / PlotModel data models constants.py global constants + option schema classes datasource/ @@ -41,7 +42,9 @@ src/clinical_scope/ - **Extract** (`wrapper.extract_patient` / `batch_extract` / `extract_datasource`, also `from clinical_scope import extract_datasource, extract_patient, batch_extract`) — stop at `format`, return DataFrame(s). `save_path`/`save_folder` write explicit output, independent of the per-patient `clinical_scope_output/` parquet cache (always written; reused when `quick_load` is set). - **Inspect** (`wrapper.inspect`) — stop at `format`, return `list[DataSourceInspection]` (columns, point counts, time ranges). `OtherDataSource.inspect()` returns **one entry per file** (`other::`); the wrapper handles single-or-list returns. -**Signal references** in `grouped_fields` and `global.loop` resolve via a 3-mode lookup in `_resolve_signal_references`: qualified `datasource::raw_name` → display name → raw-name fallback. +**Signal references** in `grouped_fields`, `loop`, `spectrogram` and `psd` resolve via a 3-mode lookup in `plot_assembly._resolve_signal_references`: qualified `datasource::raw_name` → display name → raw-name fallback. + +**Config scope is desugared once, and grouping joins on signal identity** ([ADR-0013](docs/adr/0013-signal-references-are-qualified-before-assembly.md)). `assemble_plot_groups` is called once, after the datasource loop, and its first step rewrites every per-datasource reference as a qualified global one — downstream, local scope does not exist. Both config spellings stay valid; the desugaring is a property of the code, not of the file format. A signal is left out of the default one-plot-per-signal pass only when a group took *that object*, never when something merely shares its `raw_name` (unique only within a datasource). `wrapper.main`/`inspect` call an optional `progress_callback(current, total, name)` between datasources, which drives the UI progress bar. diff --git a/docs/adr/0009-other-stem-is-a-config-scope.md b/docs/adr/0009-other-stem-is-a-config-scope.md index 327b3cc..c4ff772 100644 --- a/docs/adr/0009-other-stem-is-a-config-scope.md +++ b/docs/adr/0009-other-stem-is-a-config-scope.md @@ -26,11 +26,11 @@ So `other::waves` names a configuration scope — one file — carrying its own The asymmetry with other datasources is deliberate and is the point of recording this: **`other/` is the only source whose files are unrelated by construction.** Everywhere else, a folder is one recording. A contributor who reaches for `edf::` by analogy is generalising from the exception. -Signal references resolve in three modes, in order — qualified `datasource::raw_name`, then display name, then bare raw name (`_resolve_signal_references`, `wrapper.py`). +Signal references resolve in three modes, in order — qualified `datasource::raw_name`, then display name, then bare raw name (`_resolve_signal_references`, `plot_assembly.py`). ## Consequences - **Easier:** an `other/` folder can hold files from unrelated machines, each configured independently, without any of them earning a module. Cross-source `grouped_fields` and `loop` entries can address a specific file's column unambiguously. -- **Harder / accepted trade-offs:** a file in `other/` named after a registered datasource makes a reference genuinely readable two ways — `other/servo_u.parquet` gives a scope whose qualified names collide with the real `servo_u` source. Both readings are legitimate, so this cannot be resolved by rule alone: the precedence order above decides, and `_warn_if_also_a_raw_name` (`wrapper.py:46-60`) logs the collision naming the losing signal and the spelling that reaches it. Silent shadowing was the alternative and was rejected. +- **Harder / accepted trade-offs:** a file in `other/` named after a registered datasource makes a reference genuinely readable two ways — `other/servo_u.parquet` gives a scope whose qualified names collide with the real `servo_u` source. Both readings are legitimate, so this cannot be resolved by rule alone: the precedence order above decides, and `_warn_if_also_a_raw_name` (`plot_assembly.py`) logs the collision naming the losing signal and the spelling that reaches it. Silent shadowing was the alternative and was rejected. - **Also:** signals inside `other/` are named `::` rather than bare column names, so configurations written against the old single-block form need their references rewritten. This is part of the [ADR-0008](0008-datasource-modules-need-format-specific-parsing.md) migration. - **Revisit if:** a second datasource appears whose folder genuinely holds unrelated recordings rather than chunks of one. At that point the scope mechanism generalises — but it should generalise to *that* source explicitly, not to all of them by default. diff --git a/docs/adr/0013-signal-references-are-qualified-before-assembly.md b/docs/adr/0013-signal-references-are-qualified-before-assembly.md new file mode 100644 index 0000000..0b719ed --- /dev/null +++ b/docs/adr/0013-signal-references-are-qualified-before-assembly.md @@ -0,0 +1,48 @@ +# 13. Signal references are qualified before assembly + +Date: 2026-08-26 + +## Status + +Accepted — implemented in `plot_assembly.py` ([#89](https://github.com/larib-data/clinical-scope/issues/89)). + +## Context + +`database_options` lets a plot be configured in two places. A group whose signals come from one datasource is written in that datasource's own section (`icca.grouped_fields`); a group spanning several is written in `global.grouped_fields`. The same split exists for `loop`, and does not exist at all for `spectrogram` and `psd`, which are per-datasource only. + +Nobody authors that distinction deliberately. The XLSX derives it — a group lands in `global` iff its signals span more than one datasource — so a spreadsheet author writes a group name in a column and never sees the scope. The two formats do not even agree on what is expressible: the `loops` sheet always writes a per-datasource loop, so a global loop is reachable only from hand-written JSON. + +Despite reading as a naming convention, the split carried real semantics. The per-datasource path matched a reference against that datasource's signals by bare `raw_name` equality; the global path resolved through the three-mode chain (qualified name → display name → raw name) against every loaded signal. So the same group, written the two ways, resolved by different rules. And because the third mode of that chain returns **every** raw-name match, a bare reference evaluated globally pulls in a signal of the same name from every datasource — the per-datasource section was silently acting as a namespace qualifier. + +Two suppression bugs grew in the gap. Deciding which signals get a default one-signal-per-plot group was done twice, by two mechanisms, both keyed on `raw_name` strings: + +- an accumulator that was never reset between datasources, so a raw name appearing in an earlier datasource's `grouped_fields` suppressed an identically-named signal in a later one, with load order fixed by `DataSource.AVAILABLE`; +- a post-hoc filter that removed single-signal groups whose `raw_name` appeared in a global group — again by string, so one datasource's grouped `HR` removed *every* datasource's single `HR`. + +`raw_name` is unique only *within* a datasource. Any collection keyed on it across datasources is a namespace collision waiting for the right patient folder, and shared vitals names (`HR`, `SpO2`, `ABP`) make that folder ordinary rather than exotic. The failure mode is a silently missing plot, not an error. + +The post-hoc filter also reached across a module boundary to dictate naming: `Signal.psd_from_signal` qualified its `raw_name` as `psd_name::label` explicitly so the filter would not swallow the PSD. A derived plot had to disguise its name to survive a rule in another module. + +## Decision + +**Config scope is desugared at exactly one point, and downstream of it every signal reference is qualified.** + +`assemble_plot_groups` flattens every per-datasource section into qualified global references (`icca.grouped_fields.Vitals: [HR]` becomes `Vitals: [icca::HR]`) as its first step, and builds an internal value rather than writing back. Each reference is *resolved before it is qualified* — through the three-mode chain, against that datasource's own signals — so the section keeps scoping the candidates rather than merely prefixing the string, and `[Heart Rate]` desugars to `[icca::HR]` as readily as `[HR]` does. A reference that resolves to nothing is qualified all the same, so it cannot fall through and match a namesake elsewhere. Nothing after that pass knows local scope exists. There is one resolution path, one suppression rule, and one spelling of a reference — extending [ADR-0009](0009-other-stem-is-a-config-scope.md), which established `::` as the qualified-name separator, into a full internal normal form. + +Both config spellings stay valid. The desugaring is a property of the code, not of the file format: no parser changes, no config migration, and the per-datasource section keeps its authoring advantages — locality with the rest of a source's config, and no repetition of the datasource name per signal. + +The pass must run at the head of assembly rather than at config-parse time, because the `other` datasource injects its derived `grouped_fields` into its section during load. By the time assembly runs — once, after every datasource has loaded — that writeback is already present. + +**Group membership joins on signal identity, not on `raw_name`.** + +Memberships are resolved first; the `Signal` objects that land in any group that produces a plot are collected; default single-signal groups are then built only for input signals absent from that set. The join is on identity rather than count: a group that resolved to one signal still takes it, or that signal would be plotted twice — once under the group's name and once under its own. This replaces both string-keyed mechanisms at once. Derived signals — loop, spectrogram, PSD — are newly constructed objects that were never in the input list, so they cannot be identity-matched and are structurally immune to suppression, with no naming disguise required. + +## Consequences + +- A datasource's `grouped_fields` can no longer suppress a same-named signal belonging to a different datasource. Two independent bugs disappear together, because both were symptoms of the same string join. +- `Signal.psd_from_signal` keeps its `psd_name::label` raw name — it still distinguishes two PSD traces built from one source signal with different parameters — but for that reason alone. The cross-module coupling is gone and the comment citing it no longer applies. +- Per-datasource `grouped_fields`, `loop` and `spectrogram` gain the three-mode resolver, scoped to their own datasource. This is a strict widening: mode three *is* the bare `raw_name` match they did before, so nothing that resolved stops resolving, and a display name now resolves where it silently matched nothing. +- A group resolving to exactly one signal keeps the **group's** name. Both paths previously gave the *signal's* name — per-datasource by rebuilding it through `PlotGroup.from_single_signal`, global by leaving the default plot standing — and the group name is the one that degrades continuously: the same config gives a `Pressure` panel whether the recording has four signals or one, instead of a title that depends on how much data happened to load. +- Grouping and derived-plot construction become reachable with in-memory `Signal` objects and a literal config dict, with no patient folder on disk. That is the point: the cross-datasource collisions above cannot be expressed by the single demo patient the integration suite runs against, so they were untestable where the logic previously lived. +- **Accepted cost:** the desugaring is invisible in the config file. A reader of `database_options.json` sees two spellings and must know they mean the same thing. That is the price of not breaking existing files, and it is recorded rather than hidden — [#88](https://github.com/larib-data/clinical-scope/issues/88) tracks collapsing the surface itself along an ingestion/presentation seam, which this ADR makes safe to do later as a parser-and-docs change with no risk to assembly logic. +- **Deliberately not decided:** whether assembly's log-and-continue failures should become a reported result. Nothing consumes such a list today, and the resilience is intentional — one bad `database_options` entry must not blank a clinician's screen. diff --git a/src/clinical_scope/plot_assembly.py b/src/clinical_scope/plot_assembly.py new file mode 100644 index 0000000..b423d34 --- /dev/null +++ b/src/clinical_scope/plot_assembly.py @@ -0,0 +1,526 @@ +""" +Turn loaded Signals plus a ``database_options`` dict into the PlotGroups a figure is drawn from. + +This is the last purely-domain step of the visualize pipeline: it runs once, after every +datasource has loaded, and needs nothing on disk. Two rules govern it (ADR-0013): + +* **Config scope is desugared once.** A per-datasource section is a namespace, not a + different kind of grouping, so its references are rewritten as qualified global ones + before anything else happens. Downstream, local scope does not exist -- one resolver, + one suppression rule, one spelling of a reference. +* **Group membership joins on signal identity, not on ``raw_name``.** A raw name is unique + only *within* a datasource, so any join on it across datasources silently drops a plot + the first time two sources share a name (``HR``, ``SpO2``, ``ABP``). + +Every failure here is logged and skipped: one bad ``database_options`` entry must not blank +a clinician's screen. +""" + +import logging +from collections.abc import Callable +from dataclasses import dataclass +from functools import partial +from typing import Any + +from clinical_scope import constants as cst +from clinical_scope.signal_container import PlotGroup, Signal +from clinical_scope.spectral import SpectralRefusalError + +# ================================================================================================== +logger = logging.getLogger(__name__) + + +# ================================================================================================== +# Reference resolution +# ================================================================================================== +def _warn_if_also_a_raw_name( + ref: str, chosen: Signal, all_signals: list[Signal], separator: str +) -> None: + """ + Log when *ref* reads as a qualified name *and* as some signal's bare raw_name. + + Only an 'other' file named after a registered datasource can cause this, so it is rare -- + but silent, since both readings are legitimate. The log names the loser and the spelling + that reaches it. + """ + shadowed = [signal for signal in all_signals if signal.raw_name == ref and signal is not chosen] + if not shadowed: + return + logger.warning( + "⚠️ Ambiguous signal reference '%s': read as datasource '%s', but it is also the raw " + "name of a signal in datasource '%s'. Using the former -- write '%s' for the latter.", + ref, + chosen.metadata.datasource_name, + shadowed[0].metadata.datasource_name, + f"{shadowed[0].metadata.datasource_name}{separator}{ref}", + ) + + +def _resolve_signal_references(field_list: list[str], all_signals: list[Signal]) -> list[Signal]: + """ + Resolve signal references using a three-mode fallback chain. + + 1. Qualified name ``"datasource::raw_name"`` -- explicit, unambiguous. + 2. Display name -- matches ``signal.name``. Warns if ambiguous. + 3. Raw name -- current behaviour, backward compatible. + + A ref containing the separator tries mode 1 first but still falls through when it finds + nothing: an 'other' file's raw_name is itself ``::``, so ``waves::art`` is a + mode-3 hit while ``other::waves::art`` is the mode-1 one, and both must resolve. + + Because of that double meaning a ref can match under both readings at once -- a file + ``other/servo_u.parquet`` makes ``servo_u::Paw`` name both the servo_u datasource's column + and that file's. Mode 1 wins (an explicit datasource beats a coincidence of file naming) + and the collision is logged, since the fully qualified form reaches the other one. + """ + matched: list[Signal] = [] + + separator = cst.QUALIFIED_NAME_SEPARATOR + for ref in field_list: + # Mode 1: qualified "datasource::raw_name" + if separator in ref: + matched_signal = next( + ( + signal + for signal in all_signals + if f"{signal.metadata.datasource_name}{separator}{signal.raw_name}" == ref + ), + None, + ) + if matched_signal: + _warn_if_also_a_raw_name(ref, matched_signal, all_signals, separator) + matched.append(matched_signal) + continue + + # Mode 2: display name + by_name = [signal for signal in all_signals if signal.name == ref] + if len(by_name) == 1: + matched.append(by_name[0]) + elif len(by_name) > 1: + logger.warning( + "Ambiguous display name '%s' matched %d signals -- " + "use 'datasource::raw_name' to disambiguate.", + ref, + len(by_name), + ) + else: + # Mode 3: raw name fallback (no display name matched) + by_raw = [signal for signal in all_signals if signal.raw_name == ref] + if by_raw: + matched.extend(by_raw) + elif separator in ref: + logger.warning("Qualified reference '%s' did not match any signal.", ref) + + return matched + + +# ================================================================================================== +# Derived-plot builders +# ================================================================================================== +class _SourceSignalNotFoundError(Exception): + """Raised by a plot-group builder when its source signal isn't among the loaded signals.""" + + +class _DerivedPlotArityError(Exception): + """Raised by a plot-group builder given the wrong number of signal references.""" + + +def _resolve_one(reference: str, all_signals: list[Signal]) -> Signal: + matched = _resolve_signal_references([reference], all_signals) if reference else [] + if not matched: + raise _SourceSignalNotFoundError(reference) + return matched[0] + + +def _build_loop_signal( + all_signals: list[Signal], loop_name: str, loop_field_list: list[str] +) -> Signal: + if len(loop_field_list) != 2: # noqa: PLR2004 + msg = f"needs exactly 2 signal references, got {len(loop_field_list)}" + raise _DerivedPlotArityError(msg) + signal_x, signal_y = (_resolve_one(reference, all_signals) for reference in loop_field_list) + return Signal.loop_from_signals(signal_x, signal_y, name=loop_name) + + +def _build_spectrogram_signal( + all_signals: list[Signal], spectrogram_name: str, spectrogram_config: dict +) -> Signal: + config_cls = cst.DatabaseOptions.SpectrogramConfig + source_signal = _resolve_one(spectrogram_config.get(config_cls.SIGNAL), all_signals) + try: + return Signal.spectrogram_from_signal( + source_signal, + name=spectrogram_name, + freq_range=tuple(spectrogram_config[config_cls.FREQ_RANGE]), + db_range=spectrogram_config.get(config_cls.DB_RANGE), + window_s=spectrogram_config.get(config_cls.WINDOW_S), + overlap=spectrogram_config.get(config_cls.OVERLAP), + ) + except SpectralRefusalError as exc: + msg = f"signal '{source_signal.name}' -- {exc}" + raise SpectralRefusalError(msg) from exc + + +def _build_psd_signals(all_signals: list[Signal], psd_name: str, psd_config: dict) -> list[Signal]: + """Build one PSD trace per configured entry; they share a subplot, so a list comes back.""" + config_cls = cst.DatabaseOptions.PsdConfig + entry_cls = config_cls.Entry + # A plain string is shorthand for an Entry naming just a signal, no per-trace overrides. + entries = [ + entry if isinstance(entry, dict) else {entry_cls.SIGNAL: entry} + for entry in psd_config.get(config_cls.SIGNALS) or [] + ] + + freq_range = tuple(psd_config[config_cls.FREQ_RANGE]) + db_range = psd_config.get(config_cls.DB_RANGE) + psd_signals = [] + not_found = 0 + for entry in entries: + reference = entry[entry_cls.SIGNAL] + # Resolved one entry at a time (rather than batched) so a per-entry window_s/overlap + # override stays attached to the right match. + source_signals = _resolve_signal_references([reference], all_signals) + if not source_signals: + not_found += 1 + continue + for source_signal in source_signals: + try: + psd_signals.append( + Signal.psd_from_signal( + source_signal, + psd_name=psd_name, + freq_range=freq_range, + db_range=db_range, + window_s=entry.get(entry_cls.WINDOW_S), + overlap=entry.get(entry_cls.OVERLAP), + label=entry.get(entry_cls.LABEL), + color=entry.get(entry_cls.COLOR), + line_dash=entry.get(entry_cls.LINE_DASH), + ) + ) + except SpectralRefusalError as exc: + # Refuse the whole entry: a comparison missing one of its channels invites the + # wrong reading more than an absent plot does. + msg = f"signal '{source_signal.name}' -- {exc}" + raise SpectralRefusalError(msg) from exc + + if not psd_signals: + raise _SourceSignalNotFoundError( + ", ".join(str(entry[entry_cls.SIGNAL]) for entry in entries) + ) + if not_found: + logger.warning( + "⚠️ PSD '%s': %d of %d signal(s) not found; plotting the rest.", + psd_name, + not_found, + len(entries), + ) + return psd_signals + + +# ================================================================================================== +# Desugaring per-datasource sections into qualified global ones +# ================================================================================================== +def _qualify(reference: Any, datasource_name: str, datasource_signals: list[Signal]) -> str: + """ + Rewrite a per-datasource reference as the qualified ``datasource::raw_name`` (ADR-0013). + + Resolved against that datasource's own signals, so the section keeps acting as the + namespace it always implicitly was -- and an unresolvable reference is qualified all the + same, so it cannot fall through and match a namesake belonging to another source. + """ + matched = _resolve_signal_references([reference], datasource_signals) + target = matched[0].raw_name if matched else reference + return f"{datasource_name}{cst.QUALIFIED_NAME_SEPARATOR}{target}" + + +def _qualify_loop(config: Any, datasource_name: str, datasource_signals: list[Signal]) -> Any: + if not isinstance(config, (list, tuple)): + return config + return [_qualify(reference, datasource_name, datasource_signals) for reference in config] + + +def _qualify_spectrogram( + config: Any, datasource_name: str, datasource_signals: list[Signal] +) -> Any: + key = cst.DatabaseOptions.SpectrogramConfig.SIGNAL + if not isinstance(config, dict) or key not in config: + return config + return {**config, key: _qualify(config[key], datasource_name, datasource_signals)} + + +def _qualify_psd(config: Any, datasource_name: str, datasource_signals: list[Signal]) -> Any: + key = cst.DatabaseOptions.PsdConfig.SIGNALS + entry_key = cst.DatabaseOptions.PsdConfig.Entry.SIGNAL + if not isinstance(config, dict) or not isinstance(config.get(key), (list, tuple)): + return config + qualified = [ + {**entry, entry_key: _qualify(entry[entry_key], datasource_name, datasource_signals)} + if isinstance(entry, dict) + else _qualify(entry, datasource_name, datasource_signals) + for entry in config[key] + ] + return {**config, key: qualified} + + +@dataclass(frozen=True) +class _DerivedPlotKind: + """One kind of plot derived from already-loaded signals, and how to read its config.""" + + section_key: str + build: Callable[[list[Signal], str, Any], Signal | list[Signal]] + qualify: Callable[[Any, str, list[Signal]], Any] + refusals: tuple[type[Exception], ...] = () + + +# In the order their sections are read. Adding a derived plot type is a row here plus its +# builder and its qualifier -- assemble_plot_groups itself does not change. +_DERIVED_PLOTS = ( + _DerivedPlotKind( + cst.DatabaseOptions.LOOP, _build_loop_signal, _qualify_loop, (_DerivedPlotArityError,) + ), + _DerivedPlotKind( + cst.DatabaseOptions.SPECTROGRAM, + _build_spectrogram_signal, + _qualify_spectrogram, + (SpectralRefusalError,), + ), + _DerivedPlotKind( + cst.DatabaseOptions.PSD, _build_psd_signals, _qualify_psd, (SpectralRefusalError,) + ), +) + + +@dataclass(frozen=True) +class _GroupSpec: + """One configured group of signals, its references already qualified.""" + + name: str + references: list[str] + origin: str + + +@dataclass(frozen=True) +class _DerivedSpec: + """One configured derived plot, its references already qualified.""" + + kind: _DerivedPlotKind + name: str + config: Any + origin: str + + +def _flatten_config( + database_options_global: dict, signals: list[Signal] +) -> tuple[list[_GroupSpec], list[_DerivedSpec]]: + """ + Desugar every per-datasource section into qualified global references. + + Runs at the head of assembly rather than at parse time because ``other`` injects its + derived sections into its own section *during load*, after normalization has run. + Returns internal values; *database_options_global* is never written back to. + """ + group_specs: list[_GroupSpec] = [] + derived_specs: list[_DerivedSpec] = [] + + for section_name, section in database_options_global.items(): + if not isinstance(section, dict): + continue + is_global = section_name == cst.DatabaseOptions.GLOBAL + section_signals = [ + signal for signal in signals if signal.metadata.datasource_name == section_name + ] + configured_groups = section.get(cst.DatabaseOptions.GROUPED_FIELDS, {}) + try: + for group_name, references in configured_groups.items(): + qualified = ( + list(references) + if is_global + else [_qualify(ref, section_name, section_signals) for ref in references] + ) + group_specs.append(_GroupSpec(group_name, qualified, section_name)) + + for kind in _DERIVED_PLOTS: + for item_name, item_config in section.get(kind.section_key, {}).items(): + config = ( + item_config + if is_global + else kind.qualify(item_config, section_name, section_signals) + ) + derived_specs.append(_DerivedSpec(kind, item_name, config, section_name)) + except Exception: + logger.exception("⚠️ Unreadable database_options section '%s'; skipping.", section_name) + + return group_specs, derived_specs + + +# ================================================================================================== +# Assembly +# ================================================================================================== +def _origin_order(signals: list[Signal], database_options_global: dict) -> list[str | None]: + """ + The order plots are emitted in: datasources in load order, then ``global`` last. + + Taken from *signals* rather than from the registry, so assembly stays free of the + loading machinery; a configured datasource that loaded nothing still gets its turn, + so its unresolved entries are reported where a reader expects them. + """ + order: list[str | None] = [] + for signal in signals: + if signal.metadata.datasource_name not in order: + order.append(signal.metadata.datasource_name) + for section_name in database_options_global: + if section_name != cst.DatabaseOptions.GLOBAL and section_name not in order: + order.append(section_name) + order.append(cst.DatabaseOptions.GLOBAL) + return order + + +def _resolve_members(spec: _GroupSpec, signals: list[Signal]) -> list[Signal]: + members = _resolve_signal_references(spec.references, signals) + missing = len(spec.references) - len(members) + if missing > 0: + logger.warning( + "⚠️ Group '%s' (%s): %d of %d signal(s) not found.", + spec.name, + spec.origin, + missing, + len(spec.references), + ) + return members + + +def _add_derived_plot_group( + kind: str, + item_name: str, + datasource_name: str, + build_signal: Callable[[], Signal | list[Signal]], + plot_group_list: list[PlotGroup], + refusal_exceptions: tuple[type[Exception], ...] = (), +) -> None: + """ + Build one derived plot (loop, spectrogram, psd, ...) and add it as its own PlotGroup. + + *build_signal* returns one Signal, or a list of Signals to overlay on a single subplot + (a psd entry naming several signals). A list is titled by *item_name*, since the entry + names the plot rather than any one trace in it. + + Every failure is logged and skipped rather than raised, so one bad entry in + ``database_options`` doesn't abort the rest of a datasource's plots. *build_signal* + should raise ``_SourceSignalNotFoundError`` for a missing source signal and, optionally, + one of *refusal_exceptions* for a deliberate, named refusal -- both are logged as + warnings; anything else is logged with a full traceback. + """ + try: + signal = build_signal() + except _SourceSignalNotFoundError as exc: + logger.warning( + "⚠️ Could not construct %s '%s' in datasource '%s'. Missing signal '%s'.", + kind, + item_name, + datasource_name, + exc, + ) + return + except refusal_exceptions as exc: + logger.warning( + "⚠️ %s '%s' in datasource '%s' refused: %s", + kind.capitalize(), + item_name, + datasource_name, + exc, + ) + return + except Exception: + logger.exception( + "⚠️ Error constructing %s '%s' in datasource '%s'.", kind, item_name, datasource_name + ) + return + + try: + if isinstance(signal, list): + plot_group_list.append( + PlotGroup(name=item_name, signals=signal, allow_secondary_y=False) + ) + else: + plot_group_list.append(PlotGroup.from_single_signal(signal)) + except Exception: + logger.exception( + "⚠️ Failed to create PlotGroup from %s signal '%s' in datasource '%s'.", + kind, + item_name, + datasource_name, + ) + + +def assemble_plot_groups(signals: list[Signal], database_options_global: dict) -> list[PlotGroup]: + """ + Build the plot groups a visualization is drawn from, out of loaded signals and config. + + Every signal gets a plot of its own unless it belongs to a configured group; groups and + derived plots (loops, spectrograms, PSDs) follow their datasource's own plots, and the + ``global`` section's come last. *database_options_global* is only read. + + Args: + signals: Every signal loaded by the run, in datasource load order. + database_options_global: The full database options dict, per-datasource sections + and ``global`` alike. + + Returns: + The plot groups, in page order. + + """ + group_specs, derived_specs = _flatten_config(database_options_global, signals) + + # Memberships first, so a default one-signal-per-plot group can be skipped for exactly the + # signal objects a configured group took -- never for anything that merely shares a name. + members_by_spec = [(spec, _resolve_members(spec, signals)) for spec in group_specs] + grouped_ids = {id(signal) for _, members in members_by_spec for signal in members} + + plot_group_list: list[PlotGroup] = [] + for origin in _origin_order(signals, database_options_global): + for signal in signals: + if signal.metadata.datasource_name != origin or id(signal) in grouped_ids: + continue + try: + plot_group_list.append(PlotGroup.from_single_signal(signal)) + except Exception: + logger.exception( + "⚠️ Failed to create PlotGroup from single signal '%s' in datasource '%s'.", + signal.raw_name, + origin, + ) + + for spec, members in members_by_spec: + if spec.origin != origin or not members: + continue + try: + # A group that resolved to one signal keeps the *group's* name: the same + # config then titles the panel identically however much data happened to load. + plot_group_list.append( + PlotGroup( + name=spec.name, + signals=members, + allow_secondary_y=len(members) > 1, + ) + ) + except Exception: + logger.exception( + "⚠️ Failed to create grouped PlotGroup '%s' in datasource '%s'.", + spec.name, + origin, + ) + + for spec in derived_specs: + if spec.origin != origin: + continue + _add_derived_plot_group( + kind=spec.kind.section_key, + item_name=spec.name, + datasource_name=spec.origin, + build_signal=partial(spec.kind.build, signals, spec.name, spec.config), + plot_group_list=plot_group_list, + refusal_exceptions=spec.kind.refusals, + ) + + return plot_group_list diff --git a/src/clinical_scope/signal_container.py b/src/clinical_scope/signal_container.py index 7f353f7..1c8ceb5 100644 --- a/src/clinical_scope/signal_container.py +++ b/src/clinical_scope/signal_container.py @@ -774,8 +774,8 @@ def psd_from_signal( line_dash=line_dash or signal.trace_options.line_dash, ) return cls( - # Qualified rather than the bare source raw_name: wrapper.main prunes single-signal - # PlotGroups whose raw_name is in a global group, which would swallow the PSD too. + # Qualified by the PSD's own name: two entries built from one source signal with + # different window_s would otherwise share a raw_name as well as a display name. raw_name=f"{psd_name}{cst.QUALIFIED_NAME_SEPARATOR}{label or signal.raw_name}", name=label or signal.name, data=data, diff --git a/src/clinical_scope/wrapper.py b/src/clinical_scope/wrapper.py index 42ef583..848d45e 100644 --- a/src/clinical_scope/wrapper.py +++ b/src/clinical_scope/wrapper.py @@ -1,6 +1,5 @@ import logging from collections.abc import Callable -from functools import partial from pathlib import Path import pandas as pd @@ -15,13 +14,11 @@ from clinical_scope.datasource import registry as datasource_list from clinical_scope.datasource.inspection import DataSourceInspection from clinical_scope.io.paths import get_annotations_path +from clinical_scope.plot_assembly import assemble_plot_groups from clinical_scope.signal_container import ( DisplayFallbacks, - PlotGroup, PlotModel, - Signal, ) -from clinical_scope.spectral import SpectralRefusalError # ================================================================================================== logger = logging.getLogger(__name__) @@ -43,87 +40,6 @@ def _resolve_database_options(database_options_global: dict | None) -> dict: # ================================================================================================== -def _warn_if_also_a_raw_name( - ref: str, chosen: Signal, all_signals: list[Signal], separator: str -) -> None: - """ - Log when *ref* reads as a qualified name *and* as some signal's bare raw_name. - - Only an 'other' file named after a registered datasource can cause this, so it is rare -- - but silent, since both readings are legitimate. The log names the loser and the spelling - that reaches it. - """ - shadowed = [signal for signal in all_signals if signal.raw_name == ref and signal is not chosen] - if not shadowed: - return - logger.warning( - "⚠️ Ambiguous signal reference '%s': read as datasource '%s', but it is also the raw " - "name of a signal in datasource '%s'. Using the former -- write '%s' for the latter.", - ref, - chosen.metadata.datasource_name, - shadowed[0].metadata.datasource_name, - f"{shadowed[0].metadata.datasource_name}{separator}{ref}", - ) - - -def _resolve_signal_references(field_list: list[str], all_signals: list[Signal]) -> list[Signal]: - """ - Resolve signal references using a three-mode fallback chain. - - 1. Qualified name ``"datasource::raw_name"`` -- explicit, unambiguous. - 2. Display name -- matches ``signal.name``. Warns if ambiguous. - 3. Raw name -- current behaviour, backward compatible. - - A ref containing the separator tries mode 1 first but still falls through when it finds - nothing: an 'other' file's raw_name is itself ``::``, so ``waves::art`` is a - mode-3 hit while ``other::waves::art`` is the mode-1 one, and both must resolve. - - Because of that double meaning a ref can match under both readings at once -- a file - ``other/servo_u.parquet`` makes ``servo_u::Paw`` name both the servo_u datasource's column - and that file's. Mode 1 wins (an explicit datasource beats a coincidence of file naming) - and the collision is logged, since the fully qualified form reaches the other one. - """ - matched: list[Signal] = [] - - separator = cst.QUALIFIED_NAME_SEPARATOR - for ref in field_list: - # Mode 1: qualified "datasource::raw_name" - if separator in ref: - matched_signal = next( - ( - signal - for signal in all_signals - if f"{signal.metadata.datasource_name}{separator}{signal.raw_name}" == ref - ), - None, - ) - if matched_signal: - _warn_if_also_a_raw_name(ref, matched_signal, all_signals, separator) - matched.append(matched_signal) - continue - - # Mode 2: display name - by_name = [signal for signal in all_signals if signal.name == ref] - if len(by_name) == 1: - matched.append(by_name[0]) - elif len(by_name) > 1: - logger.warning( - "Ambiguous display name '%s' matched %d signals -- " - "use 'datasource::raw_name' to disambiguate.", - ref, - len(by_name), - ) - else: - # Mode 3: raw name fallback (no display name matched) - by_raw = [signal for signal in all_signals if signal.raw_name == ref] - if by_raw: - matched.extend(by_raw) - elif separator in ref: - logger.warning("Qualified reference '%s' did not match any signal.", ref) - - return matched - - def _format_datasource_summary(found: dict[str, str], requested: list[str]) -> str: """ Render a one-line found/not-found tally for a single-patient run. @@ -146,187 +62,37 @@ def _format_datasource_summary(found: dict[str, str], requested: list[str]) -> s return " | ".join(parts) if parts else "No datasource requested." -class _SourceSignalNotFoundError(Exception): - """Raised by a plot-group builder when its source signal isn't in ``list_signal``.""" - - -def _add_derived_plot_group( - kind: str, - item_name: str, - datasource_name: str, - build_signal: Callable[[], Signal | list[Signal]], - plot_group_list: list[PlotGroup], - refusal_exceptions: tuple[type[Exception], ...] = (), -) -> None: - """ - Build one derived plot (loop, spectrogram, psd, ...) and add it as its own PlotGroup. - - *build_signal* returns one Signal, or a list of Signals to overlay on a single subplot - (a psd entry naming several signals). A list is titled by *item_name*, since the entry - names the plot rather than any one trace in it. - - Every failure is logged and skipped rather than raised, so one bad entry in - ``database_options`` doesn't abort the rest of a datasource's plots. *build_signal* - should raise ``_SourceSignalNotFoundError`` for a missing source signal and, optionally, - one of *refusal_exceptions* for a deliberate, named refusal -- both are logged as - warnings; anything else is logged with a full traceback. - """ - try: - signal = build_signal() - except _SourceSignalNotFoundError as exc: - logger.warning( - "⚠️ Could not construct %s '%s' in datasource '%s'. Missing signal '%s'.", - kind, - item_name, - datasource_name, - exc, - ) - return - except refusal_exceptions as exc: - logger.warning( - "⚠️ %s '%s' in datasource '%s' refused: %s", - kind.capitalize(), - item_name, - datasource_name, - exc, - ) - return - except Exception: - logger.exception( - "⚠️ Error constructing %s '%s' in datasource '%s'.", kind, item_name, datasource_name - ) - return - - try: - if isinstance(signal, list): - plot_group_list.append( - PlotGroup(name=item_name, signals=signal, allow_secondary_y=False) - ) - else: - plot_group_list.append(PlotGroup.from_single_signal(signal)) - except Exception: - logger.exception( - "⚠️ Failed to create PlotGroup from %s signal '%s' in datasource '%s'.", - kind, - item_name, - datasource_name, - ) - - -def _build_loop_signal( - list_signal: list[Signal], loop_name: str, loop_field_list: list[str] -) -> Signal: - signal_x = next((s for s in list_signal if s.raw_name == loop_field_list[0]), None) - signal_y = next((s for s in list_signal if s.raw_name == loop_field_list[1]), None) - if signal_x is None or signal_y is None: - missing = loop_field_list[0] if signal_x is None else loop_field_list[1] - raise _SourceSignalNotFoundError(missing) - return Signal.loop_from_signals(signal_x, signal_y, name=loop_name) - - -def _build_spectrogram_signal( - list_signal: list[Signal], spectrogram_name: str, spectrogram_config: dict -) -> Signal: - config_cls = cst.DatabaseOptions.SpectrogramConfig - source_raw_name = spectrogram_config.get(config_cls.SIGNAL) - source_signal = next((s for s in list_signal if s.raw_name == source_raw_name), None) - if source_signal is None: - raise _SourceSignalNotFoundError(source_raw_name) - try: - return Signal.spectrogram_from_signal( - source_signal, - name=spectrogram_name, - freq_range=tuple(spectrogram_config[config_cls.FREQ_RANGE]), - db_range=spectrogram_config.get(config_cls.DB_RANGE), - window_s=spectrogram_config.get(config_cls.WINDOW_S), - overlap=spectrogram_config.get(config_cls.OVERLAP), - ) - except SpectralRefusalError as exc: - msg = f"signal '{source_signal.name}' -- {exc}" - raise SpectralRefusalError(msg) from exc - - -def _build_psd_signals(list_signal: list[Signal], psd_name: str, psd_config: dict) -> list[Signal]: - """Build one PSD trace per configured entry; they share a subplot, so a list comes back.""" - config_cls = cst.DatabaseOptions.PsdConfig - entry_cls = config_cls.Entry - # A plain string is shorthand for an Entry naming just a signal, no per-trace overrides. - entries = [ - entry if isinstance(entry, dict) else {entry_cls.SIGNAL: entry} - for entry in psd_config.get(config_cls.SIGNALS) or [] - ] - - freq_range = tuple(psd_config[config_cls.FREQ_RANGE]) - db_range = psd_config.get(config_cls.DB_RANGE) - psd_signals = [] - not_found = 0 - for entry in entries: - reference = entry[entry_cls.SIGNAL] - # Same three-mode chain as grouped_fields: qualified, display name, or raw name. - # Resolved one entry at a time (rather than batched) so a per-entry window_s/overlap - # override stays attached to the right match. - source_signals = _resolve_signal_references([reference], list_signal) - if not source_signals: - not_found += 1 - continue - for source_signal in source_signals: - try: - psd_signals.append( - Signal.psd_from_signal( - source_signal, - psd_name=psd_name, - freq_range=freq_range, - db_range=db_range, - window_s=entry.get(entry_cls.WINDOW_S), - overlap=entry.get(entry_cls.OVERLAP), - label=entry.get(entry_cls.LABEL), - color=entry.get(entry_cls.COLOR), - line_dash=entry.get(entry_cls.LINE_DASH), - ) - ) - except SpectralRefusalError as exc: - # Refuse the whole entry: a comparison missing one of its channels invites the - # wrong reading more than an absent plot does. - msg = f"signal '{source_signal.name}' -- {exc}" - raise SpectralRefusalError(msg) from exc - - if not psd_signals: - raise _SourceSignalNotFoundError( - ", ".join(str(entry[entry_cls.SIGNAL]) for entry in entries) - ) - if not_found: - logger.warning( - "⚠️ PSD '%s': %d of %d signal(s) not found; plotting the rest.", - psd_name, - not_found, - len(entries), - ) - return psd_signals - - -# Plots derived from already-loaded signals, in the order their sections are read. Each row is -# (database_options section key, builder, exceptions treated as a deliberate refusal). Adding a -# derived plot type is a row here plus its builder -- main() itself does not change. -_DERIVED_PLOTS = ( - (cst.DatabaseOptions.LOOP, _build_loop_signal, ()), - (cst.DatabaseOptions.SPECTROGRAM, _build_spectrogram_signal, (SpectralRefusalError,)), - (cst.DatabaseOptions.PSD, _build_psd_signals, (SpectralRefusalError,)), -) - - def main( patient_options: dict, database_options_global: dict | None = None, progress_callback: Callable[[int, int, str], None] | None = None, user_options: dict | None = None, ) -> list[PlotModel]: + """ + Run the visualize pipeline: load every configured datasource, then build its figures. + + Loading is isolated per datasource — a source that fails is logged and skipped, and the + rest of the patient still renders. ``progress_callback(current, total, name)`` is called + once per datasource, before it loads. + + Args: + patient_options: Per-run settings (``data_folder``, the datetime window, ``quick_load``, + per-source options). + database_options_global: Full database options dict. Defaults to all available + datasources with their default options. + progress_callback: Optional UI progress hook. + user_options: Per-person display fallbacks; they never override *database_options_global* + (ADR-0005). + + Returns: + One :class:`PlotModel` per plot type present, in page order. Empty if nothing rendered. + + """ database_options_global = _resolve_database_options(database_options_global) # User options are passed in explicitly by the UI layer; the core never reads the on-disk # file. Their display tenants travel as one carrier, built once here (ADR-0005). display_fallbacks = DisplayFallbacks.from_user_options(user_options) all_signal_list = [] - already_used_in_group = [] - plot_group_list = [] found_datasources: dict[str, str] = {} requested_sources = [ @@ -357,164 +123,29 @@ def main( database_options = database_options_global[name] try: - # (1) Create signals - try: - list_signal = data_source.MAIN_MODULE( - patient_options, database_options, display_fallbacks - ) - all_signal_list.extend(list_signal) - logger.info("✅ [%s] %d signal(s) loaded.", name, len(list_signal)) - if list_signal: - if name == datasource_list.DataSource.Other.NAME: - # "other" is a generic multi-file catch-all: which files actually - # matched is the useful signal, not a bare "found". - file_stems = { - sig.raw_name.split(cst.QUALIFIED_NAME_SEPARATOR, 1)[0] - for sig in list_signal - } - found_datasources[name] = ", ".join(sorted(file_stems)) - else: - found_datasources[name] = "" - except Exception: - logger.exception("❌ Failed to create signals for datasource '%s'. Skipping.", name) - continue - - # Read grouped fields after MAIN_MODULE (datasource may populate dynamically) - local_group = database_options.get(cst.DatabaseOptions.GROUPED_FIELDS, {}) - for grouped_field_list in local_group.values(): - already_used_in_group.extend(grouped_field_list) - - # (2) Add default groups (one signal = one group) - for signal in list_signal: - if signal.raw_name not in already_used_in_group: - try: - plot_group = PlotGroup.from_single_signal(signal) - plot_group_list.append(plot_group) - except Exception: - logger.exception( - "⚠️ Failed to create PlotGroup from single signal '%s' in datasource " - "'%s'.", - signal.raw_name, - name, - ) - - # (3) Add explicit user-defined groups - for group_name, grouped_field_list in local_group.items(): - try: - signals = [ - signal for signal in list_signal if signal.raw_name in grouped_field_list - ] - loaded_names = {signal.raw_name for signal in signals} - missing = [ - field_name - for field_name in grouped_field_list - if field_name not in loaded_names - ] - if missing: - logger.warning( - "⚠️ Group '%s' in datasource '%s': signals not found: %s", - group_name, - name, - missing, - ) - if len(signals) >= 2: # noqa: PLR2004 - plot_group_list.append(PlotGroup(name=group_name, signals=signals)) - elif len(signals) == 1: - plot_group_list.append(PlotGroup.from_single_signal(signals[0])) - except Exception: - logger.exception( - "⚠️ Failed to create grouped PlotGroup '%s' in datasource '%s'.", - group_name, - name, - ) - - # (4) Add derived plots: loops, spectrograms, PSDs - for section_key, builder, refusal_exceptions in _DERIVED_PLOTS: - for item_name, item_config in database_options.get(section_key, {}).items(): - _add_derived_plot_group( - kind=section_key, - item_name=item_name, - datasource_name=name, - build_signal=partial(builder, list_signal, item_name, item_config), - plot_group_list=plot_group_list, - refusal_exceptions=refusal_exceptions, - ) - - except Exception: - logger.exception("❌ Error while treating datasource '%s'.", name) - - # Global grouping (must be done at the end) - grouped_fields_global = database_options_global.get(cst.DatabaseOptions.GLOBAL, {}).get( - cst.DatabaseOptions.GROUPED_FIELDS, {} - ) - - global_grouped_raw_names: set[str] = set() - for group_name, grouped_field_list in grouped_fields_global.items(): - try: - signals = _resolve_signal_references(grouped_field_list, all_signal_list) - n_missing = len(grouped_field_list) - len(signals) - if n_missing > 0: - logger.warning( - "⚠️ Global group '%s': %d of %d signal(s) not found.", - group_name, - n_missing, - len(grouped_field_list), - ) - if len(signals) >= 2: # noqa: PLR2004 - plot_group_list.append(PlotGroup(name=group_name, signals=signals)) - global_grouped_raw_names.update(s.raw_name for s in signals) - elif len(signals) == 1: - logger.info( - "Global group '%s' degraded to 1 signal; shown individually.", group_name - ) - except Exception: - logger.exception("⚠️ Failed to create global PlotGroup '%s'.", group_name) - - # Remove individual PlotGroups for signals that are now in a global group - if global_grouped_raw_names: - plot_group_list = [ - plot_group - for plot_group in plot_group_list - if len(plot_group.signals) > 1 - or plot_group.signals[0].raw_name not in global_grouped_raw_names - ] - - # Global loops (cross-datasource) - global_loop_group = database_options_global.get(cst.DatabaseOptions.GLOBAL, {}).get( - cst.DatabaseOptions.LOOP, {} - ) - for loop_name, loop_field_list in global_loop_group.items(): - try: - if len(loop_field_list) != 2: # noqa: PLR2004 - logger.warning( - "⚠️ Global loop '%s' needs exactly 2 signal refs, got %d.", - loop_name, - len(loop_field_list), - ) - continue - signals = _resolve_signal_references(loop_field_list[:2], all_signal_list) - if len(signals) != 2: # noqa: PLR2004 - logger.warning( - "⚠️ Could not resolve both signals for global loop '%s' (resolved %d/2).", - loop_name, - len(signals), - ) - continue - signal_x, signal_y = signals - try: - loop_signal = Signal.loop_from_signals(signal_x, signal_y, name=loop_name) - plot_group_list.append(PlotGroup.from_single_signal(loop_signal)) - except Exception: - logger.exception("⚠️ Error constructing global loop '%s'.", loop_name) - continue - logger.info( - "✅ Global loop '%s' created (%s x %s).", - loop_name, - signal_x.raw_name, - signal_y.raw_name, + list_signal = data_source.MAIN_MODULE( + patient_options, database_options, display_fallbacks ) except Exception: - logger.exception("❌ Unexpected error while processing global loop '%s'.", loop_name) + logger.exception("❌ Failed to create signals for datasource '%s'. Skipping.", name) + continue + + all_signal_list.extend(list_signal) + logger.info("✅ [%s] %d signal(s) loaded.", name, len(list_signal)) + if list_signal: + if name == datasource_list.DataSource.Other.NAME: + # "other" is a generic multi-file catch-all: which files actually matched is + # the useful signal, not a bare "found". + file_stems = { + sig.raw_name.split(cst.QUALIFIED_NAME_SEPARATOR, 1)[0] for sig in list_signal + } + found_datasources[name] = ", ".join(sorted(file_stems)) + else: + found_datasources[name] = "" + + # Grouping runs once, on every signal at once: a group may span datasources, and no + # datasource's configuration may reach into another's signals (ADR-0013). + plot_group_list = assemble_plot_groups(all_signal_list, database_options_global) try: plot_model_list = PlotModel.assign_plot_model( diff --git a/tests/datasource/test_other.py b/tests/datasource/test_other.py index cda4dd4..0cc9dbd 100644 --- a/tests/datasource/test_other.py +++ b/tests/datasource/test_other.py @@ -353,7 +353,7 @@ def test_per_file_psd_injected_with_prefix(self, patient_difficult_path): def test_psd_overlays_signals_from_two_different_files(self, tmp_path): """One PSD subplot may compare channels living in separate files under other/.""" - from clinical_scope.wrapper import _build_psd_signals + from clinical_scope.plot_assembly import _build_psd_signals folder = tmp_path / "other" folder.mkdir(parents=True) diff --git a/tests/unit/test_plot_assembly.py b/tests/unit/test_plot_assembly.py new file mode 100644 index 0000000..8ec6ad8 --- /dev/null +++ b/tests/unit/test_plot_assembly.py @@ -0,0 +1,201 @@ +""" +Unit tests for plot_assembly.assemble_plot_groups. + +Assembly is the last purely-domain step of the visualize pipeline, so these run on in-memory +Signals and literal config dicts -- no patient folder. That is what makes the cross-datasource +name collisions below expressible at all: the single demo patient the integration suite runs +against cannot produce two datasources that share a raw name. +""" + +import copy + +import numpy as np +import pandas as pd +import pytest + +from clinical_scope import constants as cst +from clinical_scope.plot_assembly import assemble_plot_groups +from clinical_scope.signal_container import ( + Data, + Metadata, + PlotOptions, + Signal, + TraceOptions, +) + + +def _signal(raw_name: str, name: str | None = None, datasource: str = "icca") -> Signal: + """A minimal time-series Signal: enough data for a loop to be built from two of them.""" + points = 64 + return Signal( + raw_name=raw_name, + name=name or raw_name, + data=Data( + x=pd.date_range("2024-01-01", periods=points, freq="s").to_numpy(), + y=np.linspace(0.0, 1.0, points), + ), + trace_options=TraceOptions(plot_options=PlotOptions(plot_type=cst.PlotType.TIME_SERIES)), + metadata=Metadata(datasource_name=datasource), + ) + + +def _names(plot_groups) -> list[str]: + return [plot_group.name for plot_group in plot_groups] + + +class TestDefaultGroups: + def test_an_unconfigured_signal_gets_its_own_plot(self): + groups = assemble_plot_groups([_signal("HR", "Heart Rate")], {}) + assert _names(groups) == ["Heart Rate"] + + def test_defaults_come_before_the_configured_groups_of_the_same_datasource(self): + signals = [_signal("HR"), _signal("Paw"), _signal("Vol")] + options = {"icca": {"grouped_fields": {"Ventilation": ["Paw", "Vol"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["HR", "Ventilation"] + + +class TestLocalSectionsAreFlattened: + """A per-datasource section is a namespace, and desugars into qualified references.""" + + def test_a_local_group_of_two_signals_becomes_one_plot(self): + signals = [_signal("HR"), _signal("SpO2")] + options = {"icca": {"grouped_fields": {"Vitals": ["HR", "SpO2"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Vitals"] + + def test_a_local_reference_may_be_a_display_name(self): + """The local path resolves through the same three-mode chain as the global one.""" + signals = [_signal("HR", "Heart Rate"), _signal("SpO2", "Oxygen Saturation")] + options = {"icca": {"grouped_fields": {"Vitals": ["Heart Rate", "Oxygen Saturation"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Vitals"] + + def test_an_other_file_reference_keeps_resolving(self): + """``other`` injects ``::`` references into its own section during load.""" + signals = [ + _signal("waves::art", "Arterial Pressure", "other"), + _signal("numerics::FC", "Heart Rate", "other"), + ] + options = {"other": {"grouped_fields": {"Vitals": ["waves::art", "numerics::FC"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Vitals"] + + def test_an_unresolvable_local_reference_cannot_reach_another_datasource(self): + """Qualified even when it matches nothing, so it can never fall through to a namesake.""" + signals = [_signal("HR", "HR", "icca"), _signal("Paw", "Paw", "servo_u")] + options = {"icca": {"grouped_fields": {"Ventilation": ["Paw", "HR"]}}} + groups = assemble_plot_groups(signals, options) + grouped = [signal.metadata.datasource_name for signal in groups[0].signals] + assert grouped == ["icca"] + + +class TestGroupsThatResolveToOneSignal: + def test_a_local_group_of_one_keeps_the_group_name(self): + signals = [_signal("HR", "Heart Rate")] + options = {"icca": {"grouped_fields": {"Vitals": ["HR", "SpO2"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Vitals"] + + def test_a_global_group_of_one_keeps_the_group_name(self): + signals = [_signal("HR", "Heart Rate")] + options = {"global": {"grouped_fields": {"Vitals": ["icca::HR"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Vitals"] + + def test_the_signal_is_not_also_plotted_on_its_own(self): + signals = [_signal("HR", "Heart Rate")] + options = {"global": {"grouped_fields": {"Vitals": ["icca::HR"]}}} + assert len(assemble_plot_groups(signals, options)) == 1 + + def test_a_group_that_resolves_to_nothing_leaves_its_signals_alone(self): + signals = [_signal("HR", "Heart Rate")] + options = {"global": {"grouped_fields": {"Vitals": ["icca::SpO2"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Heart Rate"] + + +class TestCrossDatasourceNameCollisions: + """``raw_name`` is unique only within a datasource, so grouping must join on identity.""" + + def test_a_local_group_does_not_suppress_a_namesake_in_another_datasource(self): + signals = [ + _signal("HR", "HR", "icca"), + _signal("SpO2", "SpO2", "icca"), + _signal("HR", "HR", "mindray_scope"), + ] + options = {"icca": {"grouped_fields": {"Vitals": ["HR", "SpO2"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["Vitals", "HR"] + + def test_a_global_group_does_not_suppress_a_namesake_it_did_not_include(self): + signals = [ + _signal("HR", "HR", "icca"), + _signal("ABP", "ABP", "icca"), + _signal("HR", "HR", "mindray_scope"), + ] + options = {"global": {"grouped_fields": {"Pressure": ["icca::HR", "icca::ABP"]}}} + assert _names(assemble_plot_groups(signals, options)) == ["HR", "Pressure"] + + +class TestDerivedPlots: + @pytest.fixture + def ventilator(self) -> list[Signal]: + """Two signals of one datasource — enough to draw a loop from.""" + return [_signal("Paw", "Airway Pressure", "servo_u"), _signal("Vol", "Volume", "servo_u")] + + def test_a_local_loop_is_built_beside_its_source_signals(self, ventilator): + """A loop is an extra plot; the signals it is drawn from keep their own.""" + options = {"servo_u": {"loop": {"PV loop": ["Paw", "Vol"]}}} + groups = assemble_plot_groups(ventilator, options) + assert _names(groups) == ["Airway Pressure", "Volume", "PV loop"] + assert groups[-1].plot_options.plot_type == "loop" + + def test_a_local_loop_reference_may_be_a_display_name(self, ventilator): + options = {"servo_u": {"loop": {"PV loop": ["Airway Pressure", "Volume"]}}} + assert "PV loop" in _names(assemble_plot_groups(ventilator, options)) + + def test_a_global_loop_is_built_the_same_way(self): + signals = [_signal("Paw", "Airway Pressure", "servo_u"), _signal("Vol", "Vol", "mindray")] + options = {"global": {"loop": {"PV loop": ["servo_u::Paw", "mindray::Vol"]}}} + assert "PV loop" in _names(assemble_plot_groups(signals, options)) + + def test_a_loop_with_the_wrong_number_of_references_is_refused_not_raised( + self, ventilator, caplog + ): + options = {"servo_u": {"loop": {"PV loop": ["Paw"]}}} + assert _names(assemble_plot_groups(ventilator, options)) == ["Airway Pressure", "Volume"] + assert "refused" in caplog.text + + def test_a_derived_plot_survives_a_group_that_shares_its_name(self): + """A derived signal is a new object, so no string can suppress it.""" + signals = [_signal("HR", "HR", "icca"), _signal("ABP", "ABP", "icca")] + options = { + "icca": {"loop": {"HR": ["HR", "ABP"]}}, + "global": {"grouped_fields": {"Pressure": ["icca::HR", "icca::ABP"]}}, + } + assert _names(assemble_plot_groups(signals, options)) == ["HR", "Pressure"] + + +class TestAssemblyIsPure: + def test_the_config_dict_is_not_written_back_to(self): + signals = [_signal("HR"), _signal("SpO2")] + options = { + "icca": { + "grouped_fields": {"Vitals": ["HR", "SpO2"]}, + "loop": {"PV loop": ["HR", "SpO2"]}, + "psd": {"HR PSD": {"signals": ["HR"], "freq_range": [0.5, 30.0]}}, + }, + "global": {"grouped_fields": {"Pressure": ["icca::HR"]}}, + } + before = copy.deepcopy(options) + assemble_plot_groups(signals, options) + assert options == before + + +class TestMalformedConfigIsSurvived: + @pytest.mark.parametrize( + "options", + [ + {"icca": "not a section"}, + {"icca": {"grouped_fields": "not a mapping"}}, + {"icca": {"loop": {"PV loop": "not a list"}}}, + {"icca": {"spectrogram": {"S": {}}}}, + ], + ) + def test_the_rest_of_the_signals_still_get_their_plots(self, options): + """One bad database_options entry must not blank the screen.""" + groups = assemble_plot_groups([_signal("HR", "Heart Rate")], options) + assert "Heart Rate" in _names(groups) diff --git a/tests/unit/test_signal_reference_resolution.py b/tests/unit/test_signal_reference_resolution.py index 30444f1..b70798b 100644 --- a/tests/unit/test_signal_reference_resolution.py +++ b/tests/unit/test_signal_reference_resolution.py @@ -1,5 +1,5 @@ """ -Unit tests for wrapper._resolve_signal_references. +Unit tests for plot_assembly._resolve_signal_references. The three-mode chain carries the whole weight of `grouped_fields`, `global.loop` and `psd` signal references, and `::` means two different things depending on the datasource: for most @@ -9,8 +9,8 @@ import pytest +from clinical_scope.plot_assembly import _resolve_signal_references from clinical_scope.signal_container import Metadata, Signal -from clinical_scope.wrapper import _resolve_signal_references def _signal(raw_name: str, name: str, datasource_name: str) -> Signal: From 86099f68ae98f754cb4ea852670a087c6560482d Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Wed, 26 Aug 2026 15:37:09 +0200 Subject: [PATCH 7/9] Make the global-grouping integration test actually assert TestMainGlobalGrouping guarded its only assertion behind `global_fields & all_signal_names`, an intersection that is provably always empty: an 'other' signal's raw_name is `::`, so the set holds `numerics::PNId`, never the bare `PNId` the guard compared against. The assertion had never run. It is replaced by three that pin what global grouping is for -- a group spanning three datasources, a three-segment `other::::` reference resolving, and a grouped signal not also being plotted on its own. The `pytest.skip("No time_series models produced")` guards go with it, here and in its siblings. demo_patient/ is a fixed committed fixture, so "maybe there is no data" is not a real condition -- it only meant the tests would fall silent if the fixture ever regressed. Co-Authored-By: Claude Opus 5 --- tests/integration/test_main_visualization.py | 74 ++++++++++++-------- 1 file changed, 45 insertions(+), 29 deletions(-) diff --git a/tests/integration/test_main_visualization.py b/tests/integration/test_main_visualization.py index 835a167..6d445f0 100644 --- a/tests/integration/test_main_visualization.py +++ b/tests/integration/test_main_visualization.py @@ -55,27 +55,52 @@ def test_returns_plot_models(self, patient_options_full, default_database_option class TestMainGlobalGrouping: - def test_global_grouped_fields(self, patient_options_full, example_database_options): - """The example config has global.grouped_fields.Pressure — verify grouping works.""" + """ + ``global.grouped_fields`` is the only place a group may span datasources. + + The demo patient has no raw name shared by two sources, so the collisions identity-based + grouping exists to prevent are not expressible here — see tests/unit/test_plot_assembly.py. + """ + + @pytest.fixture(scope="class") + def time_series_groups(self, patient_options_full, example_database_options): models = main(patient_options_full, example_database_options) - ts_models = [m for m in models if m.plot_type == "time_series"] - if not ts_models: - pytest.skip("No time_series models produced") - ts_model = ts_models[0] - pressure_groups = [g for g in ts_model.groups if "Pressure" in g.name] - # With synthetic data, the ART/PNId/etc. signals may or may not exist. - # If they do, the Pressure group should be there. - all_signal_names = {s.raw_name for g in ts_model.groups for s in g.signals} - global_fields = {"ART", "PNId", "PNIm", "PNIs"} - if global_fields & all_signal_names: - assert len(pressure_groups) > 0, "Expected a 'Pressure' group" + return next(model for model in models if model.plot_type == "time_series").groups + + @staticmethod + def _named(groups, name): + return next(group for group in groups if group.name == name) + + def test_a_global_group_gathers_signals_from_several_datasources(self, time_series_groups): + group = self._named(time_series_groups, "Paw [cross-source qualified refs]") + assert [signal.metadata.datasource_name for signal in group.signals] == [ + "fluxmed_signals", + "mindray_scope", + "mindray_respi_waves", + ] + + def test_a_qualified_other_reference_resolves(self, time_series_groups): + """``other::numerics::PNId`` is three segments, the last two the signal's own raw name.""" + group = self._named(time_series_groups, "NI Pressure") + assert [signal.raw_name for signal in group.signals] == [ + "numerics::PNId", + "numerics::PNIm", + "numerics::PNIs", + ] + + def test_a_globally_grouped_signal_is_not_also_plotted_alone(self, time_series_groups): + grouped = { + signal.raw_name for signal in self._named(time_series_groups, "NI Pressure").signals + } + alone = { + group.signals[0].raw_name for group in time_series_groups if len(group.signals) == 1 + } + assert grouped & alone == set() class TestToHtml: def test_to_html_writes_file(self, patient_options_full, example_database_options, tmp_path): models = main(patient_options_full, example_database_options) - if not models: - pytest.skip("No models produced") opts = dict(patient_options_full) opts["data_folder"] = str(tmp_path) # Create the output directory (to_html expects it or helper creates it) @@ -103,10 +128,7 @@ def user_options(self): @pytest.fixture(scope="class") def time_series_model(self, patient_options_full, example_database_options, user_options): models = main(patient_options_full, example_database_options, user_options=user_options) - ts_models = [model for model in models if model.plot_type == "time_series"] - if not ts_models: - pytest.skip("No time_series models produced") - return ts_models[0] + return next(model for model in models if model.plot_type == "time_series") def test_subplot_height_applied(self, time_series_model): assert time_series_model.computed_height == 175 * len(time_series_model.groups) @@ -133,16 +155,13 @@ def test_configured_hover_template_survives(self, time_series_model): heart_rate = [ trace for trace in time_series_model.figure.data if trace.name == "Heart Rate" ] - if not heart_rate: - pytest.skip("Heart Rate signal absent from the demo data") + assert heart_rate assert "%{y:.0f}" in heart_rate[0].hovertemplate def test_no_user_options_keeps_defaults(self, patient_options_full, example_database_options): models = main(patient_options_full, example_database_options) - ts_models = [model for model in models if model.plot_type == "time_series"] - if not ts_models: - pytest.skip("No time_series models produced") - assert ts_models[0].computed_height == 300 * len(ts_models[0].groups) + ts_model = next(model for model in models if model.plot_type == "time_series") + assert ts_model.computed_height == 300 * len(ts_model.groups) class TestMainGlobalLoops: @@ -173,8 +192,6 @@ def test_global_loop_model_has_correct_name( """The loop PlotModel group must be named after the loop key.""" models = main(patient_options_full, db_opts_with_global_loop) loop_models = [m for m in models if m.plot_type == "loop"] - if not loop_models: - pytest.skip("No loop PlotModel produced — signal data may be absent") group_names = [g.name for m in loop_models for g in m.groups] assert "pv_loop" in group_names @@ -182,8 +199,7 @@ def test_global_loop_model_has_figure(self, patient_options_full, db_opts_with_g """The loop PlotModel must carry a rendered Plotly figure.""" models = main(patient_options_full, db_opts_with_global_loop) loop_models = [m for m in models if m.plot_type == "loop"] - if not loop_models: - pytest.skip("No loop PlotModel produced — signal data may be absent") + assert loop_models for m in loop_models: assert isinstance(m.figure, go.Figure) assert len(m.figure.data) > 0 From 31d578312106f8475e14ef7b068ea7c1fd9e5873 Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Wed, 26 Aug 2026 16:29:43 +0200 Subject: [PATCH 8/9] Validate user options at the boundary (ADR-0014) The user_options schema rules -- MIN/MAX, CHOICES, valid IANA name -- were walked twice, in the settings modal on the way in and in DisplayFallbacks.from_user_options on the way out, and on neither occasion at load. A hand-edited ~/.clinical_scope/user_options.json therefore reached the store raw: two copies of one rule set, and the path that most needed them had none. They now live once, in a new core module user_options.py, called by every boundary that accepts a value. The module is pure -- no Path.home(), no file reads -- which is what makes wrapper.main's "the core never reads the on-disk file" structural rather than a matter of nobody calling the function. Disk I/O stays in helper_api. validate(raw) returns (clean, list[Correction]) rather than logging. Detection is shared; the reaction belongs to the boundary that knows the user's situation -- the modal discards silently because its widget re-renders showing the corrected value, the loader logs because nobody is watching a widget. That collapsed persist_user_options' corrected_a_widget flag to bool(corrections): it existed only to re-derive what per-field coercion had thrown away. from_user_options becomes a projection that converts but does not check. resolve_display_timezone stays, as the one tenant whose bad value raises inside pandas/zoneinfo instead of merely rendering oddly, covering a dict a library caller hand-built. Unknown keys warn and drop, three lines in the loader. Deliberately opposite to ADR-0012's Annotation.extra passthrough: the discriminator is provenance. Annotations are human-authored and get shared, so an unknown key is someone's data; user options are per-person state this app writes, so an unknown key is a value stranded under a name the schema no longer has. Two behaviour changes, both in CHANGELOG: a bad value in a hand-edited settings file is now corrected and reported at load, and the modal refuses an inverted spectrogram dB pair on save rather than leaving the render layer to fix it -- the cross-field rule moved into validate with the rest. validate also always returns every schema field, so the store can no longer be partial. Validation tests move to tests/unit/test_user_options.py and assert on returned Correction values, not caplog prose, which no longer breaks on a reworded message. src is +48/-144 across three files, +150 in the new one. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 4 + CLAUDE.md | 3 +- ...r-options-are-validated-at-the-boundary.md | 30 +++ .../callbacks/user_options_callbacks.py | 63 +---- src/clinical_scope/dash_api/helper_api.py | 38 +-- src/clinical_scope/signal_container.py | 91 ++----- src/clinical_scope/user_options.py | 150 +++++++++++ tests/dash/test_callbacks_user_options.py | 28 +- tests/dash/test_ui_components.py | 6 +- tests/unit/test_display_fallbacks.py | 112 +------- tests/unit/test_user_options.py | 239 ++++++++++++++++++ 11 files changed, 510 insertions(+), 254 deletions(-) create mode 100644 docs/adr/0014-user-options-are-validated-at-the-boundary.md create mode 100644 src/clinical_scope/user_options.py create mode 100644 tests/unit/test_user_options.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ac33a31..ef2d222 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,10 @@ All notable changes to this project will be documented in this file. **What changes for you:** a code pasted without its leading `#` is accepted, a malformed one is flagged as you leave the field, and a colour that is not a valid six-digit hex now falls back to the default instead of being written into the annotation file as-is. +- **A hand-edited `~/.clinical_scope/user_options.json` is now checked when it loads.** Settings were only validated as you typed them into the Settings modal, so a file edited by hand — or one holding a value from an older version — could carry a subplot height of `99999`, a palette that no longer exists, or a misspelled timezone, and reach the app unchecked. Each such value now falls back to its default and says which one it was in the log, and a setting stored under a name the app no longer knows is reported rather than dropped in silence. + + **What changes for you:** the Settings modal also refuses a spectrogram colour range whose minimum is not below its maximum — both bounds snap back to their defaults as you save, instead of the pair being stored and quietly corrected at plot time. + --- ## [1.1.0] — 2026-08-24 diff --git a/CLAUDE.md b/CLAUDE.md index 7ca5da6..58fbe3f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,7 @@ src/clinical_scope/ plot_assembly.py Signals + database_options → PlotGroups (grouping, derived plots) signal_container.py Signal / PlotGroup / PlotModel data models constants.py global constants + option schema classes + user_options.py UserOptions schema as data: traversal, defaults, validate() datasource/ base.py DataSourceBase — find/load/format/extract/inspect template registry.py registered sources (DataSource.AVAILABLE; keep Other last) @@ -65,7 +66,7 @@ Registered in `datasource/registry.py` (`DataSource.AVAILABLE`); the canonical l Field-by-field reference is in the [tutorial](docs/user_guide/tutorial.md). The three tiers: - **`database_options`** (`.json` or `.xlsx`) — per-source signal config: `field_display`, `signals` (labels/units/colors), `grouped_fields`, `loop`; plus `global.grouped_fields`. Uploading one in the UI caches it to `~/.clinical_scope/last_database_options.json` (signal metadata only, no PHI). - **`patient_options`** (`.json`) — per-run settings: `data_folder`, `datetime_start`/`datetime_end`, `quick_load`, and per-source options (`time_shift`, `day`, …). -- **`user_options`** (`~/.clinical_scope/user_options.json`) — the third tier: per-person app behaviour + display fallbacks, edited only in the Settings modal. **Never overrides `database_options`** ([ADR-0005](docs/adr/0005-user-options-are-fallbacks.md)). A new display setting = a `UserOptions` schema class (with `SECTION`) + a field on `DisplayFallbacks` (`signal_container.py`) + one read site; the carrier is threaded from `wrapper.main` down to both `Signal` and `PlotModel` construction, so no signature grows. +- **`user_options`** (`~/.clinical_scope/user_options.json`) — the third tier: per-person app behaviour + display fallbacks, edited only in the Settings modal. **Never overrides `database_options`** ([ADR-0005](docs/adr/0005-user-options-are-fallbacks.md)). A new display setting = a `UserOptions` schema class (with `SECTION`) + a field on `DisplayFallbacks` (`signal_container.py`) + one read site; the carrier is threaded from `wrapper.main` down to both `Signal` and `PlotModel` construction, so no signature grows. Values are held to the schema by `user_options.validate()` at every boundary that accepts one, and only `dash_api` may read the file ([ADR-0014](docs/adr/0014-user-options-are-validated-at-the-boundary.md)). Reference configs: `example/demo_database/database_options.{xlsx,json}` — the canonical example, in both formats, runnable against `demo_patient/` and covering **every** datasource it ships. **The `.json` is generated from the `.xlsx`**; edit the spreadsheet and regenerate. `tests/unit/test_example_assets.py` enforces both the coverage and the parity, and prints the regeneration one-liner. `example/option_files/patient_options_example.json` covers the other tier, for library users who never launch the app and so never get an app-written one. diff --git a/docs/adr/0014-user-options-are-validated-at-the-boundary.md b/docs/adr/0014-user-options-are-validated-at-the-boundary.md new file mode 100644 index 0000000..0094666 --- /dev/null +++ b/docs/adr/0014-user-options-are-validated-at-the-boundary.md @@ -0,0 +1,30 @@ +# 14. User options are validated at the boundary, never read ambiently + +Date: 2026-08-26 + +## Status + +Accepted + +## Context + +`~/.clinical_scope/user_options.json` is hand-editable and machine-written. Its schema rules (MIN/MAX, CHOICES, valid IANA name) were walked in two places — the settings modal on the way in, the `DisplayFallbacks` projection on the way out — and in neither case on load, so a hand-edited file reached the store raw. Two copies of one rule set, and the path that most needed them had none. + +Who *may* read the file is a separate question with the same answer shape. [ADR-0011](0011-datetime-bounds-are-qualified-at-the-boundary.md) already forbids the load path resolving a user option, for determinism; nothing structurally stopped the core from reading the file directly. + +## Decision + +**One implementation of the schema rules, run at every boundary that accepts a value; and only boundary code may read the file.** + +- `user_options.validate(raw) -> (clean, list[Correction])` is the single implementation. It is **pure** — no `Path.home()`, no file reads — which is what keeps the core structurally unable to pick up an ambient settings file. +- Corrections are returned as data, not logged, because the right reaction depends on the boundary: the settings modal discards silently (its widget re-renders showing the corrected value), `helper_api.load_user_options` logs (nobody is watching a widget). +- `DisplayFallbacks.from_user_options` converts, it does not check. The one exception is `display_timezone`, whose bad value raises inside pandas/zoneinfo rather than merely rendering oddly. +- An unknown key is **warned about and dropped**. Deliberately opposite to [ADR-0012](0012-annotation-dicts-are-an-open-schema.md)'s `extra` passthrough: the discriminator is provenance. Annotations are human-authored and get shared, so unknown keys are someone's data; user options are per-person state this app writes, so an unknown key is a value stranded under a name the schema no longer has. +- Scripts do not read the file. A setting a script needs is re-offered as an explicit flag — `inspect_patient_data.py --configured-columns-only` is the precedent. + +## Consequences + +- Adding a setting still costs one schema class; validation follows from its `API_TYPE`. +- The modal now refuses an inverted `spectrogram_db_min`/`max` pair on save rather than leaving the render layer to fix it, since the cross-field rule moved into `validate` with the rest. +- Corrections are testable as values instead of log prose, which no longer breaks on a reworded message. +- Distinct from [ADR-0005](0005-user-options-are-fallbacks.md): that one ranks user options *below* database options. This one is about which values are trusted, and where they may be read. diff --git a/src/clinical_scope/dash_api/callbacks/user_options_callbacks.py b/src/clinical_scope/dash_api/callbacks/user_options_callbacks.py index 14539de..7a5b221 100644 --- a/src/clinical_scope/dash_api/callbacks/user_options_callbacks.py +++ b/src/clinical_scope/dash_api/callbacks/user_options_callbacks.py @@ -14,13 +14,13 @@ from dash import ALL, Input, Output, State, callback, ctx, no_update import clinical_scope.constants as cst +from clinical_scope import user_options from clinical_scope.dash_api import helper_api as ui_helper from clinical_scope.dash_api import ui_components from clinical_scope.dash_api.styles import ( ANNOTATION_MODAL_STYLE_HIDDEN, ANNOTATION_MODAL_STYLE_SHOWN, ) -from clinical_scope.datasource.formatting.timezone import resolve_display_timezone logger = logging.getLogger(__name__) @@ -28,51 +28,11 @@ _INSPECT_PRUNING = cst.UserOptions.InspectConfiguredColumnsOnly.NAME -def _field_by_name(name: str) -> Any | None: - """Return the UserOptions nested schema class whose NAME matches, or None.""" - return next( - (field for field in ui_helper.iter_user_option_fields() if name == field.NAME), None - ) - - def _option_key(widget_id: dict[str, str]) -> str: """Extract the bare option name from a ``prefix.name`` pattern-matching widget id.""" return widget_id["name"].split(".")[-1] -def _api_type(name: str) -> str | None: - """API_TYPE of the named UserOptions field, or None if unknown.""" - return getattr(_field_by_name(name), "API_TYPE", None) - - -def _coerce(name: str, value: Any) -> Any: - """ - Keep a stored user option inside what its schema allows. - - Numeric fields are clamped to MIN/MAX, choices and timezones fall back to DEFAULT, so a - cleared input or a stale value from an older file can never reach the render layer. - """ - schema = _field_by_name(name) - if schema is None: - return value - - if schema.API_TYPE in (cst.ApiType.INT, cst.ApiType.FLOAT): - cast = int if schema.API_TYPE == cst.ApiType.INT else float - try: - return max(schema.MIN, min(schema.MAX, cast(value))) - except (TypeError, ValueError): - return schema.DEFAULT - - if schema.API_TYPE == cst.ApiType.CHOICE: - allowed = {choice_value for choice_value, _ in schema.CHOICES} - return value if value in allowed else schema.DEFAULT - - if schema.API_TYPE == cst.ApiType.TIMEZONE: - return resolve_display_timezone(value) - - return value - - # ================================================================================================== @callback( Output("settings-modal", "style"), @@ -101,22 +61,18 @@ def persist_user_options( store: dict[str, Any] | None, ) -> dict[str, Any] | Any: """Persist a settings-modal change to the store and to disk.""" - updated = dict(store or {}) - corrected_a_widget = False - # Decode each modal widget value to its Python form (BOOL checklist [True]/[] → bool), - # then hold it to what its schema allows. + # then hold the whole set to what the schema allows. + edited = dict(store or {}) for value, widget_id in zip(widget_values, widget_ids, strict=False): key = _option_key(widget_id) - decoded = ui_components.from_widget_value(_api_type(key), value) - fixed = _coerce(key, decoded) - updated[key] = fixed - corrected_a_widget = corrected_a_widget or fixed != decoded + edited[key] = ui_components.from_widget_value(user_options.api_type(key), value) + updated, corrections = user_options.validate(edited) - # A coercion can land back on the value already in the store (invalid entry falls back + # A correction can land back on the value already in the store (invalid entry falls back # to an already-stored default) — updated == store alone would then wrongly skip the # resync, leaving the widget showing raw invalid text. - if updated == store and not corrected_a_widget: + if updated == store and not corrections: return no_update ui_helper.save_user_options(updated) @@ -138,12 +94,13 @@ def reflect_user_options( ) -> tuple[list[Any], str, str]: """Mirror the store onto the modal widgets and the Process-/Inspect-side indicators.""" store = store or {} + defaults = user_options.defaults() values: list[Any] = [] for widget_id in widget_ids: key = _option_key(widget_id) # A key absent from the store (older options file) shows its schema default, not a blank. - stored = store.get(key, getattr(_field_by_name(key), "DEFAULT", None)) - values.append(ui_components.to_widget_value(_api_type(key), stored)) + stored = store.get(key, defaults.get(key)) + values.append(ui_components.to_widget_value(user_options.api_type(key), stored)) save_html_indicator = ui_components.save_html_indicator_text(bool(store.get(_SAVE_HTML))) pruning_indicator = ui_components.inspect_pruning_indicator_text( bool(store.get(_INSPECT_PRUNING)) diff --git a/src/clinical_scope/dash_api/helper_api.py b/src/clinical_scope/dash_api/helper_api.py index 9fc55df..86fa5e7 100644 --- a/src/clinical_scope/dash_api/helper_api.py +++ b/src/clinical_scope/dash_api/helper_api.py @@ -5,6 +5,7 @@ from typing import Any import clinical_scope.constants as cst +from clinical_scope import user_options # ================================================================================================== logger = logging.getLogger(__name__) @@ -40,20 +41,6 @@ def get_user_options_path() -> Path: return Path.home() / cst.CLINICAL_SCOPE_DIR_NAME / cst.USER_OPTIONS_FILE_NAME -def iter_user_option_fields() -> list[Any]: - """Return the UserOptions nested schema classes (those exposing a NAME).""" - return [ - getattr(cst.UserOptions, attr) - for attr in dir(cst.UserOptions) - if hasattr(getattr(cst.UserOptions, attr), "NAME") - ] - - -def user_options_defaults() -> dict[str, Any]: - """Build the default user_options dict from the UserOptions schema classes.""" - return {field.NAME: field.DEFAULT for field in iter_user_option_fields()} - - def save_user_options(data: dict[str, Any]) -> None: """Persist the user_options dict to its cache path (best-effort).""" try: @@ -63,16 +50,31 @@ def save_user_options(data: dict[str, Any]) -> None: def load_user_options() -> dict[str, Any]: - """Load user options, filling any missing/unknown keys from schema defaults.""" - options = user_options_defaults() + """ + Load the on-disk user options, held to the schema. + + Nothing else reads this file: it is hand-editable, so a stored value gets the same + checks the settings modal applies, and both a rejected value and a name the schema no + longer knows are reported rather than dropped in silence. + """ path = get_user_options_path() + stored: dict[str, Any] = {} if path.exists(): try: with path.open() as file: - stored = json.load(file) - options.update({key: value for key, value in stored.items() if key in options}) + loaded = json.load(file) + stored = loaded if isinstance(loaded, dict) else {} except Exception: logger.exception("Failed to load user options:") + + known = user_options.defaults() + unknown = sorted(key for key in stored if key not in known) + if unknown: + logger.warning("Ignoring unknown user option(s): %s", ", ".join(unknown)) + + options, corrections = user_options.validate(stored) + for correction in corrections: + logger.warning("%s", correction.message) return options diff --git a/src/clinical_scope/signal_container.py b/src/clinical_scope/signal_container.py index 1c8ceb5..696eb29 100644 --- a/src/clinical_scope/signal_container.py +++ b/src/clinical_scope/signal_container.py @@ -1,7 +1,6 @@ import logging import re import time -from collections.abc import Callable from dataclasses import dataclass, field, fields from pathlib import Path from typing import Any @@ -99,86 +98,32 @@ class DisplayFallbacks: @classmethod def from_user_options(cls, user_options: dict[str, Any] | None) -> "DisplayFallbacks": """ - Read the display tenants of *user_options*; missing or unusable values keep defaults. + Project the display tenants of *user_options* onto the carrier. - An absent key is the normal case (a settings file predating the option), so it stays - silent. A value that is present but discarded is logged — the settings modal already - validates, so it only happens to a hand-edited ``user_options.json``. + A projection, not a check: range and choice rules live in + :func:`clinical_scope.user_options.validate`, which the UI runs before a value can be + stored. The timezone is the exception — it is the one tenant whose bad value raises + inside pandas/zoneinfo rather than merely rendering oddly, so it is resolved here too, + covering a dict hand-built by a library caller. """ options = user_options or {} schema = cst.UserOptions - def bounded_number(field_schema: Any, cast: Callable[[Any], Any] = int) -> int | float: - if field_schema.NAME not in options: - return field_schema.DEFAULT - try: - value = cast(options[field_schema.NAME]) - except (TypeError, ValueError): - logger.warning( - "user_options['%s'] = %r is not a number; using %s", - field_schema.NAME, - options[field_schema.NAME], - field_schema.DEFAULT, - ) - return field_schema.DEFAULT - clamped = max(field_schema.MIN, min(field_schema.MAX, value)) - if clamped != value: - logger.warning( - "user_options['%s'] = %s is outside [%s, %s]; using %s", - field_schema.NAME, - value, - field_schema.MIN, - field_schema.MAX, - clamped, - ) - return clamped - - def ordered_bounds(min_schema: Any, max_schema: Any) -> tuple[float, float]: - # Each bound is in range on its own yet the pair can still be inverted, which - # reaches Plotly as zmin > zmax and renders an unreadable scale. - low = bounded_number(min_schema, cast=float) - high = bounded_number(max_schema, cast=float) - if low >= high: - logger.warning( - "user_options['%s'] = %s is not below '%s' = %s; using [%s, %s]", - min_schema.NAME, - low, - max_schema.NAME, - high, - min_schema.DEFAULT, - max_schema.DEFAULT, - ) - return (min_schema.DEFAULT, max_schema.DEFAULT) - return (low, high) - - def one_of(field_schema: Any) -> Any: - if field_schema.NAME not in options: - return field_schema.DEFAULT - value = options[field_schema.NAME] - allowed = [choice_value for choice_value, _ in field_schema.CHOICES] - if value not in allowed: - logger.warning( - "user_options['%s'] = %r is not one of %s; using %r", - field_schema.NAME, - value, - allowed, - field_schema.DEFAULT, - ) - return field_schema.DEFAULT - return value + def stored(field_schema: Any) -> Any: + return options.get(field_schema.NAME, field_schema.DEFAULT) return cls( - subplot_height=bounded_number(schema.DefaultSubplotHeight), - loop_subplot_height=bounded_number(schema.LoopSubplotHeight), - loops_per_row=one_of(schema.LoopsPerRow), - legend_entry_width=bounded_number(schema.LegendEntryWidth), - y_significant_digits=one_of(schema.YSignificantDigits), - colorway=one_of(schema.FallbackColorway), - template=one_of(schema.Template), - hovermode=one_of(schema.HoverModeOption), - hover_time_format=one_of(schema.HoverTimeFormatOption), + subplot_height=stored(schema.DefaultSubplotHeight), + loop_subplot_height=stored(schema.LoopSubplotHeight), + loops_per_row=stored(schema.LoopsPerRow), + legend_entry_width=stored(schema.LegendEntryWidth), + y_significant_digits=stored(schema.YSignificantDigits), + colorway=stored(schema.FallbackColorway), + template=stored(schema.Template), + hovermode=stored(schema.HoverModeOption), + hover_time_format=stored(schema.HoverTimeFormatOption), display_timezone=resolve_display_timezone(options.get(schema.DisplayTimezone.NAME)), - spectrogram_db_range=ordered_bounds(schema.SpectrogramDbMin, schema.SpectrogramDbMax), + spectrogram_db_range=(stored(schema.SpectrogramDbMin), stored(schema.SpectrogramDbMax)), ) @property diff --git a/src/clinical_scope/user_options.py b/src/clinical_scope/user_options.py new file mode 100644 index 0000000..a735658 --- /dev/null +++ b/src/clinical_scope/user_options.py @@ -0,0 +1,150 @@ +""" +The user_options schema as data: traversal, defaults, and validation. + +Pure by design — nothing here reads ``~/.clinical_scope/user_options.json``. Disk I/O lives in +``dash_api.helper_api``, so the core is structurally unable to pick up an ambient settings file +and an ``extract_*`` run cannot depend on who is at the keyboard (ADR-0014). +""" + +from dataclasses import dataclass +from typing import Any + +import clinical_scope.constants as cst +from clinical_scope.datasource.formatting.timezone import resolve_display_timezone + + +@dataclass(frozen=True) +class Correction: + """ + One stored value the schema rejected, and what replaced it. + + Returned rather than logged so each boundary can react on its own: the settings modal + discards silently (its widget re-renders showing *used*), the loader logs, because a + hand-edited file has nobody watching a widget. + """ + + name: str + given: Any + used: Any + reason: str + + @property + def message(self) -> str: + """Log-ready sentence naming the option, what it held, and what was used instead.""" + return f"user_options[{self.name!r}] = {self.given!r} {self.reason}; using {self.used!r}" + + +def iter_fields() -> list[Any]: + """Return the UserOptions nested schema classes (those exposing a NAME).""" + return [ + getattr(cst.UserOptions, attr) + for attr in dir(cst.UserOptions) + if hasattr(getattr(cst.UserOptions, attr), "NAME") + ] + + +def defaults() -> dict[str, Any]: + """Build the default user_options dict from the UserOptions schema classes.""" + return {field.NAME: field.DEFAULT for field in iter_fields()} + + +def api_type(name: str) -> str | None: + """API_TYPE of the named field, or None if the schema has no field by that name.""" + return getattr(_field_by_name(name), "API_TYPE", None) + + +def validate(raw: dict[str, Any] | None) -> tuple[dict[str, Any], list[Correction]]: + """ + Hold every user option to its schema, returning the clean dict and what was corrected. + + The result always carries every schema field: a missing key takes its default silently, + since a settings file predating an option is the normal case. Keys the schema does not + know are absent from the result — this walks the schema, not *raw*. + """ + given = raw or {} + clean: dict[str, Any] = {} + corrections: list[Correction] = [] + + for field in iter_fields(): + if field.NAME not in given: + clean[field.NAME] = field.DEFAULT + continue + value, correction = _validated_field(field, given[field.NAME]) + clean[field.NAME] = value + if correction is not None: + corrections.append(correction) + + pair_correction = _order_spectrogram_bounds(clean) + if pair_correction is not None: + corrections.append(pair_correction) + + return clean, corrections + + +# ================================================================================================== +def _field_by_name(name: str) -> Any | None: + """Return the UserOptions nested schema class whose NAME matches, or None.""" + return next((field for field in iter_fields() if name == field.NAME), None) + + +def _validated_field(field: Any, value: Any) -> tuple[Any, Correction | None]: + """Dispatch one present value to the check its API_TYPE calls for.""" + if field.API_TYPE in (cst.ApiType.INT, cst.ApiType.FLOAT): + return _bounded_number(field, value) + if field.API_TYPE == cst.ApiType.CHOICE: + return _one_of(field, value) + if field.API_TYPE == cst.ApiType.TIMEZONE: + return _valid_timezone(field, value) + return value, None + + +def _bounded_number(field: Any, value: Any) -> tuple[Any, Correction | None]: + cast = int if field.API_TYPE == cst.ApiType.INT else float + try: + number = cast(value) + except (TypeError, ValueError): + return field.DEFAULT, Correction(field.NAME, value, field.DEFAULT, "is not a number") + clamped = max(field.MIN, min(field.MAX, number)) + if clamped != number: + reason = f"is outside [{field.MIN}, {field.MAX}]" + return clamped, Correction(field.NAME, number, clamped, reason) + return clamped, None + + +def _one_of(field: Any, value: Any) -> tuple[Any, Correction | None]: + allowed = [choice_value for choice_value, _ in field.CHOICES] + if value in allowed: + return value, None + reason = f"is not one of {allowed}" + return field.DEFAULT, Correction(field.NAME, value, field.DEFAULT, reason) + + +def _valid_timezone(field: Any, value: Any) -> tuple[Any, Correction | None]: + resolved = resolve_display_timezone(value, fallback=field.DEFAULT) + if resolved == value: + return value, None + reason = "is not a usable IANA timezone name" + return resolved, Correction(field.NAME, value, resolved, reason) + + +def _order_spectrogram_bounds(clean: dict[str, Any]) -> Correction | None: + """ + Reset the spectrogram colour range in place if its bounds are not strictly increasing. + + Each bound can sit inside MIN/MAX on its own while the pair is still inverted, which + reaches Plotly as zmin > zmax and renders an unreadable scale. + """ + low_field = cst.UserOptions.SpectrogramDbMin + high_field = cst.UserOptions.SpectrogramDbMax + low, high = clean[low_field.NAME], clean[high_field.NAME] + if low < high: + return None + + clean[low_field.NAME] = low_field.DEFAULT + clean[high_field.NAME] = high_field.DEFAULT + return Correction( + name=low_field.NAME, + given=(low, high), + used=(low_field.DEFAULT, high_field.DEFAULT), + reason=f"is not below {high_field.NAME!r}", + ) diff --git a/tests/dash/test_callbacks_user_options.py b/tests/dash/test_callbacks_user_options.py index e368080..190a831 100644 --- a/tests/dash/test_callbacks_user_options.py +++ b/tests/dash/test_callbacks_user_options.py @@ -12,6 +12,7 @@ reflect_user_options, ) from clinical_scope.signal_container import DisplayFallbacks +from clinical_scope.user_options import defaults # --------------------------------------------------------------------------- # Helpers @@ -48,7 +49,20 @@ def test_int_clamped_to_bounds(self, user_options_home): def test_float_clamped_to_bounds(self, user_options_home): name = cst.UserOptions.SpectrogramDbMax.NAME assert self._saved(name, 99999.0) == 100.0 - assert self._saved(name, -99999.0) == -100.0 + assert self._saved(cst.UserOptions.SpectrogramDbMin.NAME, -99999.0) == -100.0 + + def test_inverted_db_pair_is_refused_on_save(self, user_options_home): + """Both bounds are in range on their own; the modal must not store zmin > zmax.""" + saved = persist_user_options( + [80.0, 10.0], + [ + _widget_id(cst.UserOptions.SpectrogramDbMin.NAME), + _widget_id(cst.UserOptions.SpectrogramDbMax.NAME), + ], + {}, + ) + assert saved[cst.UserOptions.SpectrogramDbMin.NAME] == 0.0 + assert saved[cst.UserOptions.SpectrogramDbMax.NAME] == 40.0 def test_cleared_int_input_falls_back_to_default(self, user_options_home): assert self._saved(cst.UserOptions.LegendEntryWidth.NAME, None) == 220 @@ -77,10 +91,10 @@ def test_invalid_value_still_forces_a_widget_resync_when_it_falls_back_to_the_st so ``updated == store`` alone must not read as "nothing changed" and skip the resync. """ name = cst.UserOptions.DisplayTimezone.NAME - store = {name: cst.DISPLAY_TIMEZONE} + store = defaults() result = persist_user_options(["NotATimezone"], [_widget_id(name)], store) + assert result == store assert result is not no_update - assert result[name] == cst.DISPLAY_TIMEZONE def test_checklist_is_stored_as_a_bool(self, user_options_home): assert self._saved(cst.UserOptions.SaveHtmlOnProcess.NAME, [True]) is True @@ -103,11 +117,13 @@ def test_writes_every_widget_to_the_store(self, user_options_home): store = persist_user_options(values, [_widget_id(name) for name in names], {}) - assert store == { + assert {name: store[name] for name in names} == { cst.UserOptions.SaveHtmlOnProcess.NAME: True, cst.UserOptions.FallbackColorway.NAME: cst.Colorway.TOL_MUTED, cst.UserOptions.LoopsPerRow.NAME: 3, } + # Untouched settings are filled from the schema, so the store is never partial. + assert set(store) == set(defaults()) def test_persists_to_disk(self, user_options_home): name = cst.UserOptions.Template.NAME @@ -153,10 +169,6 @@ def test_pruning_indicator_reflects_the_store(self): class TestUserOptionsRoundTrip: - def test_defaults_cover_every_schema_field(self): - defaults = ui_helper.user_options_defaults() - assert set(defaults) == {field.NAME for field in ui_helper.iter_user_option_fields()} - def test_saved_options_reload_and_reach_the_carrier(self, user_options_home): persist_user_options( [cst.Colorway.TOL_MUTED, 3], diff --git a/tests/dash/test_ui_components.py b/tests/dash/test_ui_components.py index cfc3aea..09be495 100644 --- a/tests/dash/test_ui_components.py +++ b/tests/dash/test_ui_components.py @@ -4,7 +4,6 @@ import clinical_scope.constants as cst from clinical_scope.dash_api import ui_components -from clinical_scope.dash_api.helper_api import iter_user_option_fields from clinical_scope.dash_api.ui_components import ( build_ui_and_schema_registry, dash_widget_factory, @@ -12,6 +11,7 @@ to_widget_value, ) from clinical_scope.dash_api.validation import validate_value +from clinical_scope.user_options import iter_fields # --------------------------------------------------------------------------- # Helpers @@ -124,7 +124,7 @@ def test_every_user_option_still_gets_a_widget(self): cst.UserOptions, "user_options", id_type="user-option" ) built = {component_id for component_id, _ in _widgets(container)} - assert built == {f"user_options.{field.NAME}" for field in iter_user_option_fields()} + assert built == {f"user_options.{field.NAME}" for field in iter_fields()} def test_display_timezone_callback_id_matches_a_real_widget(self): """ @@ -141,7 +141,7 @@ def test_display_timezone_callback_id_matches_a_real_widget(self): def test_every_declared_section_is_ranked(self): """A section missing from SECTION_ORDER would silently sort to the top.""" - declared = {getattr(schema, "SECTION", None) for schema in iter_user_option_fields()} + declared = {getattr(schema, "SECTION", None) for schema in iter_fields()} assert declared <= set(cst.UserOptions.SECTION_ORDER) diff --git a/tests/unit/test_display_fallbacks.py b/tests/unit/test_display_fallbacks.py index affa418..28d729f 100644 --- a/tests/unit/test_display_fallbacks.py +++ b/tests/unit/test_display_fallbacks.py @@ -45,6 +45,8 @@ def test_is_frozen(self): class TestFromUserOptions: + """A projection, not a check — validation is tested in test_user_options.py.""" + def test_reads_every_display_tenant(self): fallbacks = DisplayFallbacks.from_user_options( { @@ -58,6 +60,8 @@ def test_reads_every_display_tenant(self): cst.UserOptions.HoverModeOption.NAME: cst.HoverMode.CLOSEST, cst.UserOptions.HoverTimeFormatOption.NAME: cst.HoverTimeFormat.DATE_TIME, cst.UserOptions.DisplayTimezone.NAME: "America/New_York", + cst.UserOptions.SpectrogramDbMin.NAME: -20.0, + cst.UserOptions.SpectrogramDbMax.NAME: 60.0, } ) assert fallbacks == DisplayFallbacks( @@ -71,10 +75,19 @@ def test_reads_every_display_tenant(self): hovermode=cst.HoverMode.CLOSEST, hover_time_format=cst.HoverTimeFormat.DATE_TIME, display_timezone="America/New_York", + spectrogram_db_range=(-20.0, 60.0), ) + def test_a_missing_tenant_keeps_its_default(self): + """Options predating a setting are the normal case, not something to warn about.""" + fallbacks = DisplayFallbacks.from_user_options({cst.UserOptions.LoopsPerRow.NAME: 3}) + assert fallbacks == DisplayFallbacks(loops_per_row=3) + def test_invalid_display_timezone_falls_back_to_default(self): - """display_timezone delegates to resolve_display_timezone, same as everywhere else.""" + """ + The one tenant still resolved here: a bad IANA name raises inside pandas/zoneinfo, + so a dict hand-built by a library caller must not carry it into the render layer. + """ fallbacks = DisplayFallbacks.from_user_options( {cst.UserOptions.DisplayTimezone.NAME: "NotATimezone"} ) @@ -87,103 +100,6 @@ def test_app_behaviour_tenants_are_ignored(self): ) assert fallbacks == DisplayFallbacks() - def test_height_clamped_to_schema_bounds(self): - too_big = DisplayFallbacks.from_user_options( - {cst.UserOptions.DefaultSubplotHeight.NAME: 99999} - ) - too_small = DisplayFallbacks.from_user_options( - {cst.UserOptions.DefaultSubplotHeight.NAME: 1} - ) - assert too_big.subplot_height == 2000 - assert too_small.subplot_height == 100 - - def test_valid_db_range_is_kept(self): - fallbacks = DisplayFallbacks.from_user_options( - { - cst.UserOptions.SpectrogramDbMin.NAME: -20.0, - cst.UserOptions.SpectrogramDbMax.NAME: 60.0, - } - ) - assert fallbacks.spectrogram_db_range == (-20.0, 60.0) - - def test_inverted_db_range_falls_back_to_defaults(self): - """Each bound is in range on its own, yet the pair is inverted.""" - fallbacks = DisplayFallbacks.from_user_options( - { - cst.UserOptions.SpectrogramDbMin.NAME: 100.0, - cst.UserOptions.SpectrogramDbMax.NAME: 40.0, - } - ) - assert fallbacks.spectrogram_db_range == (0.0, 40.0) - - def test_equal_db_bounds_fall_back_to_defaults(self): - fallbacks = DisplayFallbacks.from_user_options( - { - cst.UserOptions.SpectrogramDbMin.NAME: 20.0, - cst.UserOptions.SpectrogramDbMax.NAME: 20.0, - } - ) - assert fallbacks.spectrogram_db_range == (0.0, 40.0) - - def test_out_of_range_db_bounds_are_clamped_then_ordered(self): - """Clamping each bound on its own still leaves the pair inverted.""" - fallbacks = DisplayFallbacks.from_user_options( - { - cst.UserOptions.SpectrogramDbMin.NAME: 99999.0, - cst.UserOptions.SpectrogramDbMax.NAME: -99999.0, - } - ) - assert fallbacks.spectrogram_db_range == (0.0, 40.0) - - def test_unparseable_int_falls_back_to_default(self): - fallbacks = DisplayFallbacks.from_user_options( - {cst.UserOptions.DefaultSubplotHeight.NAME: "tall"} - ) - assert fallbacks.subplot_height == 300 - - def test_none_int_falls_back_to_default(self): - """A cleared number input arrives as None.""" - fallbacks = DisplayFallbacks.from_user_options( - {cst.UserOptions.LegendEntryWidth.NAME: None} - ) - assert fallbacks.legend_entry_width == 220 - - def test_discarded_value_is_logged(self, caplog): - """The modal validates, so a bad value here means a hand-edited file — say so in the log.""" - with caplog.at_level("WARNING"): - DisplayFallbacks.from_user_options( - { - cst.UserOptions.DefaultSubplotHeight.NAME: "tall", - cst.UserOptions.FallbackColorway.NAME: "retired_palette", - } - ) - assert len(caplog.records) == 2 - - def test_absent_key_is_not_logged(self, caplog): - """Options predating a setting are the normal case, not something to warn about.""" - with caplog.at_level("WARNING"): - DisplayFallbacks.from_user_options({}) - assert caplog.records == [] - - def test_string_int_is_accepted(self): - fallbacks = DisplayFallbacks.from_user_options( - {cst.UserOptions.DefaultSubplotHeight.NAME: "420"} - ) - assert fallbacks.subplot_height == 420 - - def test_unknown_choice_falls_back_to_default(self): - """A value from an older options file must not reach the render layer.""" - fallbacks = DisplayFallbacks.from_user_options( - { - cst.UserOptions.FallbackColorway.NAME: "retired_palette", - cst.UserOptions.HoverModeOption.NAME: "y unified", - cst.UserOptions.LoopsPerRow.NAME: 12, - } - ) - assert fallbacks.colorway == "okabe_ito" - assert fallbacks.hovermode == "x unified" - assert fallbacks.loops_per_row == 2 - # --------------------------------------------------------------------------- # Derived values diff --git a/tests/unit/test_user_options.py b/tests/unit/test_user_options.py new file mode 100644 index 0000000..70f7e9f --- /dev/null +++ b/tests/unit/test_user_options.py @@ -0,0 +1,239 @@ +"""Unit tests for the user_options schema module — traversal and validation (ADR-0014).""" + +from dataclasses import FrozenInstanceError +from pathlib import Path + +import pytest + +import clinical_scope.constants as cst +from clinical_scope import user_options +from clinical_scope.user_options import Correction, api_type, defaults, iter_fields, validate + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _clean(raw): + """Validated dict only, for cases that do not care about the corrections.""" + return validate(raw)[0] + + +def _corrections(raw): + """Corrections only, for cases that do not care about the cleaned values.""" + return validate(raw)[1] + + +def _by_name(corrections, name): + return next(correction for correction in corrections if correction.name == name) + + +# --------------------------------------------------------------------------- +# Schema traversal +# --------------------------------------------------------------------------- + + +class TestTraversal: + def test_every_field_is_reachable(self): + """A nested class the traversal misses would never get a widget or a default.""" + assert {field.NAME for field in iter_fields()} == { + "save_html_on_process", + "self_contained_html", + "inspect_configured_columns_only", + "display_timezone", + "default_subplot_height", + "loop_subplot_height", + "loops_per_row", + "legend_entry_width", + "colorway", + "plot_template", + "hover_time_format", + "hovermode", + "y_significant_digits", + "spectrogram_db_min", + "spectrogram_db_max", + } + + def test_defaults_are_the_pre_settings_look(self): + """Golden values: what the app did before any of this was settable.""" + values = defaults() + assert values["save_html_on_process"] is False + assert values["default_subplot_height"] == 300 + assert values["loops_per_row"] == 2 + assert values["colorway"] == "okabe_ito" + assert values["spectrogram_db_min"] == 0.0 + assert values["spectrogram_db_max"] == 40.0 + + def test_api_type_of_a_known_field(self): + assert api_type("default_subplot_height") == cst.ApiType.INT + assert api_type("colorway") == cst.ApiType.CHOICE + + def test_api_type_of_an_unknown_field_is_none(self): + """The widget helpers pass whatever id they were given; an unknown one must not raise.""" + assert api_type("retired_setting") is None + + +# --------------------------------------------------------------------------- +# validate — the quiet path +# --------------------------------------------------------------------------- + + +class TestValidateAcceptsGoodInput: + def test_empty_input_gives_defaults_silently(self): + """A settings file predating an option is the normal case, not something to report.""" + assert validate({}) == (defaults(), []) + + def test_none_input_gives_defaults_silently(self): + assert validate(None) == (defaults(), []) + + def test_result_always_carries_every_field(self): + assert set(_clean({"colorway": cst.Colorway.TOL_MUTED})) == set(defaults()) + + def test_valid_values_are_kept(self): + raw = { + "default_subplot_height": 450, + "loops_per_row": 3, + "colorway": cst.Colorway.TOL_MUTED, + "display_timezone": "America/New_York", + "spectrogram_db_min": -20.0, + "spectrogram_db_max": 60.0, + } + clean, corrections = validate(raw) + assert corrections == [] + assert {key: clean[key] for key in raw} == raw + + def test_a_string_number_is_accepted(self): + """A hand-edited file can quote its numbers; that is not a mistake worth reporting.""" + assert validate({"default_subplot_height": "420"}) == ( + {**defaults(), "default_subplot_height": 420}, + [], + ) + + def test_booleans_pass_through(self): + assert _clean({"save_html_on_process": True})["save_html_on_process"] is True + + def test_unknown_keys_are_absent_from_the_result(self): + """validate walks the schema, so a retired name cannot survive it.""" + assert "retired_setting" not in _clean({"retired_setting": 1}) + + +# --------------------------------------------------------------------------- +# validate — numbers +# --------------------------------------------------------------------------- + + +class TestValidateNumbers: + def test_int_above_max_is_clamped(self): + clean, corrections = validate({"default_subplot_height": 99999}) + assert clean["default_subplot_height"] == 2000 + assert _by_name(corrections, "default_subplot_height").used == 2000 + + def test_int_below_min_is_clamped(self): + assert _clean({"default_subplot_height": 1})["default_subplot_height"] == 100 + + def test_float_bounds_are_clamped(self): + assert _clean({"spectrogram_db_min": -99999.0})["spectrogram_db_min"] == -100.0 + + def test_unparseable_number_falls_back_to_default(self): + clean, corrections = validate({"default_subplot_height": "tall"}) + assert clean["default_subplot_height"] == 300 + assert _by_name(corrections, "default_subplot_height").given == "tall" + + def test_cleared_number_input_falls_back_to_default(self): + """A cleared Dash number input arrives as None.""" + assert _clean({"legend_entry_width": None})["legend_entry_width"] == 220 + + def test_a_value_already_in_range_is_not_reported(self): + assert _corrections({"default_subplot_height": 450}) == [] + + +# --------------------------------------------------------------------------- +# validate — choices and timezone +# --------------------------------------------------------------------------- + + +class TestValidateChoices: + def test_unknown_choice_falls_back_to_default(self): + clean, corrections = validate({"colorway": "retired_palette"}) + assert clean["colorway"] == "okabe_ito" + assert _by_name(corrections, "colorway").given == "retired_palette" + + def test_choice_outside_the_set_falls_back_to_default(self): + assert _clean({"loops_per_row": 12})["loops_per_row"] == 2 + + def test_invalid_timezone_falls_back_to_default(self): + clean, corrections = validate({"display_timezone": "NotATimezone"}) + assert clean["display_timezone"] == cst.DISPLAY_TIMEZONE + assert _by_name(corrections, "display_timezone").used == cst.DISPLAY_TIMEZONE + + def test_valid_timezone_is_kept_silently(self): + assert validate({"display_timezone": "Asia/Tokyo"}) == ( + {**defaults(), "display_timezone": "Asia/Tokyo"}, + [], + ) + + def test_cleared_timezone_falls_back_to_default(self): + assert _clean({"display_timezone": ""})["display_timezone"] == cst.DISPLAY_TIMEZONE + + +# --------------------------------------------------------------------------- +# validate — the cross-field spectrogram rule +# --------------------------------------------------------------------------- + + +class TestSpectrogramRange: + def test_inverted_pair_resets_both_bounds(self): + """Each bound is inside its own MIN/MAX, yet the pair reaches Plotly as zmin > zmax.""" + clean, corrections = validate({"spectrogram_db_min": 100.0, "spectrogram_db_max": 40.0}) + assert (clean["spectrogram_db_min"], clean["spectrogram_db_max"]) == (0.0, 40.0) + assert _by_name(corrections, "spectrogram_db_min").used == (0.0, 40.0) + + def test_equal_bounds_reset_both(self): + clean = _clean({"spectrogram_db_min": 20.0, "spectrogram_db_max": 20.0}) + assert (clean["spectrogram_db_min"], clean["spectrogram_db_max"]) == (0.0, 40.0) + + def test_out_of_range_bounds_are_clamped_then_ordered(self): + """Clamping each bound on its own still leaves the pair inverted — two corrections.""" + clean, corrections = validate( + {"spectrogram_db_min": 99999.0, "spectrogram_db_max": -99999.0} + ) + assert (clean["spectrogram_db_min"], clean["spectrogram_db_max"]) == (0.0, 40.0) + assert len(corrections) == 3 + + def test_one_bound_alone_is_ordered_against_the_default_of_the_other(self): + clean = _clean({"spectrogram_db_max": -10.0}) + assert (clean["spectrogram_db_min"], clean["spectrogram_db_max"]) == (0.0, 40.0) + + +# --------------------------------------------------------------------------- +# Correction +# --------------------------------------------------------------------------- + + +class TestCorrection: + def test_message_names_the_option_and_both_values(self): + """Only the loader renders this; the modal reacts to the object, not the prose.""" + message = Correction("colorway", "retired", "okabe_ito", "is not one of []").message + assert "colorway" in message + assert "retired" in message + assert "okabe_ito" in message + + def test_is_frozen(self): + """Corrections travel to a caller that only reads them; none may edit one in place.""" + with pytest.raises(FrozenInstanceError): + _corrections({"colorway": "retired_palette"})[0].used = "anything" + + +# --------------------------------------------------------------------------- +# Purity — the reason this module is not in dash_api +# --------------------------------------------------------------------------- + + +def test_module_never_touches_the_home_directory(): + """ + The core must stay unable to read ``~/.clinical_scope/user_options.json`` (ADR-0014): + an ``extract_*`` run may not depend on who is at the keyboard. Disk I/O is helper_api's. + """ + source = Path(user_options.__file__).read_text() + assert "Path.home" not in source + assert "open(" not in source From 91daf8a27f6d45cd3c0a4ba1deb068412fbe500c Mon Sep 17 00:00:00 2001 From: Alexis Janin Date: Thu, 27 Aug 2026 13:25:45 +0200 Subject: [PATCH 9/9] Assert independent literals for display_timezone fallback, drop issue ref from docstring test_user_options.py compared corrected display_timezone against cst.DISPLAY_TIMEZONE, so the assertion could never fail independently of the constant it exercises. Also drops an issue-number reference from a test_time_axis.py docstring, keeping the rationale in prose. Co-Authored-By: Claude Sonnet 5 --- tests/unit/test_time_axis.py | 2 +- tests/unit/test_user_options.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/unit/test_time_axis.py b/tests/unit/test_time_axis.py index 48088c0..6c12850 100644 --- a/tests/unit/test_time_axis.py +++ b/tests/unit/test_time_axis.py @@ -451,7 +451,7 @@ def test_non_utc_named_winner_stays_naive(self): class TestNumericTypeClassificationAgreement: """ - Tripwire for a code-review finding on issue #57: schema-only detection + Tripwire for schema-only detection (`_is_numeric_pa_type`, pyarrow-type-based) and full-frame detection (`detect_time_axis_in_frame`, `pd.api.types.is_numeric_dtype`-based) each decide independently whether a column is "numeric" and should be deferred to the epoch tier. diff --git a/tests/unit/test_user_options.py b/tests/unit/test_user_options.py index 70f7e9f..4c31942 100644 --- a/tests/unit/test_user_options.py +++ b/tests/unit/test_user_options.py @@ -163,8 +163,8 @@ def test_choice_outside_the_set_falls_back_to_default(self): def test_invalid_timezone_falls_back_to_default(self): clean, corrections = validate({"display_timezone": "NotATimezone"}) - assert clean["display_timezone"] == cst.DISPLAY_TIMEZONE - assert _by_name(corrections, "display_timezone").used == cst.DISPLAY_TIMEZONE + assert clean["display_timezone"] == "Europe/Paris" + assert _by_name(corrections, "display_timezone").used == "Europe/Paris" def test_valid_timezone_is_kept_silently(self): assert validate({"display_timezone": "Asia/Tokyo"}) == ( @@ -173,7 +173,7 @@ def test_valid_timezone_is_kept_silently(self): ) def test_cleared_timezone_falls_back_to_default(self): - assert _clean({"display_timezone": ""})["display_timezone"] == cst.DISPLAY_TIMEZONE + assert _clean({"display_timezone": ""})["display_timezone"] == "Europe/Paris" # ---------------------------------------------------------------------------