Regression baseline machinery: portable native baselines + reconstruction-input capture

changelog/761.fix.1.md

@@ -0,0 +1 @@
+PMP climatology reference datasets whose institution identifier contains a hyphen (for example the `GPCP-3-3` precipitation climatology from `NASA-GISS`) are now resolved by the registry instead of being silently skipped, so diagnostics that depend on them no longer fail with "No datasets found matching facets".

changelog/761.fix.md

@@ -0,0 +1 @@
+Committed regression baselines are now reproducible from their stored native blobs on any machine: native outputs are rewritten to portable placeholders (`<OUTPUT_DIR>` / `<TEST_DATA_DIR>` / `<SOFTWARE_ROOT_DIR>`) before snapshotting, so re-minting on a different host yields identical digests instead of churning the baseline store. Diagnostics can also declare `reconstruction_inputs` — extra output globs such as ESMValTool `diagnostic_provenance.yml` or the PMP driver `*_cmec.json` — that are persisted into the baseline, so `ref test-cases replay` rebuilds the bundle by re-running `build_execution_result` against the captured inputs.

changelog/761.trivial.md

@@ -0,0 +1 @@
+Committed regression bundle JSON (`diagnostic.json`, `output.json`, `series.json`) is now written with a trailing newline, matching `manifest.json` and standard end-of-file conventions. The example provider's committed baselines were regenerated into this canonical form.

packages/climate-ref-core/src/climate_ref_core/diagnostics.py

@@ -566,6 +566,21 @@ class Diagnostic(AbstractDiagnostic):
     files: Sequence[FileDefinition] = tuple()
     test_data_spec: TestDataSpecification | None = None
 
+    reconstruction_inputs: tuple[str, ...] = ()
+    """
+    Extra output globs to persist into the results/regression set, beyond the files the
+    CMEC output bundle references.
+
+    :meth:`build_execution_result` for some providers re-derives the bundle by re-scanning raw
+    execution artefacts (e.g. ESMValTool ``diagnostic_provenance.yml`` or the PMP driver's ``*_cmec.json``)
+    that the curated output set would otherwise exclude.
+    Declaring those globs here makes them part of the persisted baseline,
+    so a ``replay`` can rebuild the bundle from the stored native set alone.
+
+    Patterns are relative to the execution output directory (see :meth:`pathlib.Path.glob`).
+    The copied files are sanitised for portability like every other text artefact.
+    """
+
     def __init__(self) -> None:
         super().__init__()
         self._provider: DiagnosticProvider | None = None

packages/climate-ref-core/src/climate_ref_core/esgf/registry.py

@@ -113,11 +113,12 @@ def _parse_pmp_climatology_key(key: str) -> dict[str, Any]:
     _, _variable_id_dir, _grid_label, _version, filename = parts
 
     # Parse filename: {var}_mon_{source_id}_{inst_id}_{grid}_{time}_AC_{ver}_{res}.nc
-    # Handle source_ids with hyphens (e.g., "ERA-5", "GPCP-Monthly-3-2")
+    # source_id and institution_id may both contain hyphens (e.g. "GPCP-3-3", "NASA-GISS");
+    # the literal "_" separators keep tokenisation unambiguous.
     filename_pattern = re.compile(
         r"^(?P<variable_id>[a-z]+)_mon_"
         r"(?P<source_id>[A-Za-z0-9-]+)_"
-        r"(?P<institution_id>[A-Za-z0-9]+)_"
+        r"(?P<institution_id>[A-Za-z0-9-]+)_"
         r"(?P<grid_label>[a-z]+)_"
         r"(?P<time_range>\d+-\d+)_AC_"
         r"(?P<version>v\d+)_"

packages/climate-ref-core/src/climate_ref_core/output_files.py

@@ -15,7 +15,7 @@
 Only files in the results directory are accessed by the API/public.
 
 For some tests we must sanitise paths to files as well as the contents of text files
-(:func:`to_placeholders` / :func:`from_placeholders`).
+(:class:`PlaceholderMap`).
 This ensures that the regression data is machine independent.
 """
 
@@ -25,6 +25,8 @@
 from pathlib import Path
 from typing import TYPE_CHECKING
 
+from attrs import frozen
+
 from climate_ref_core.diagnostics import ensure_relative_path
 from climate_ref_core.logging import EXECUTION_LOG_FILENAME
 from climate_ref_core.pycmec.output import CMECOutput
@@ -38,6 +40,13 @@
 PLACEHOLDER_TEST_DATA_DIR = "<TEST_DATA_DIR>"
 """Placeholder substituted for the absolute provider test-data directory."""
 
+PLACEHOLDER_SOFTWARE_ROOT_DIR = "<SOFTWARE_ROOT_DIR>"
+"""Placeholder substituted for the absolute shared-software root directory.
+
+Provider command lines stamped into CMEC provenance reference the installed
+software environment (e.g. a conda prefix) under this root.
+"""
+
 SANITISED_FILE_GLOBS: tuple[str, ...] = (
     "*.json",
     "*.txt",
@@ -102,68 +111,91 @@ def rewrite_tree(
                 file.write_text(rewritten, encoding="utf-8")
 
 
-def to_placeholders(
-    directory: Path,
-    *,
-    output_dir: Path,
-    test_data_dir: Path,
-    globs: tuple[str, ...] = SANITISED_FILE_GLOBS,
-) -> None:
+@frozen
+class PlaceholderMap:
     """
-    Rewrite absolute paths in committed artefacts to portable placeholders ("to").
+    Map between absolute runtime directories and portable ``<TOKEN>`` placeholders.
 
-    Replaces the absolute ``output_dir`` with ``<OUTPUT_DIR>`` and the absolute
-    ``test_data_dir`` with ``<TEST_DATA_DIR>`` in every text artefact under ``directory``.
-    Binary files are never touched.
-
-    Parameters
-    ----------
-    directory
-        The tree of committed artefacts to sanitise in place.
-    output_dir
-        The absolute execution output directory.
-    test_data_dir
-        The absolute provider test-data directory.
-    globs
-        File globs whose contents are rewritten.
-    """
-    rewrite_tree(
-        directory,
-        {str(output_dir): PLACEHOLDER_OUTPUT_DIR, str(test_data_dir): PLACEHOLDER_TEST_DATA_DIR},
-        globs,
-    )
+    A committed regression bundle is made machine-independent
+    by replacing host-specific absolute paths with stable tokens.
+    The *same* map drives every direction:
 
+    - :meth:`sanitise` rewrites absolute paths to tokens (capture / mint).
+    - :meth:`hydrate` rewrites tokens back to absolute paths (replay / rebuild).
+    - :meth:`as_replacements` yields the ``{absolute: token}`` mapping
+      the bundle and series comparators apply to a freshly regenerated artefact before diffing.
 
-def from_placeholders(
-    directory: Path,
-    *,
-    output_dir: Path,
-    test_data_dir: Path,
-    globs: tuple[str, ...] = SANITISED_FILE_GLOBS,
-) -> None:
+    The two configuration-stable tokens (``<TEST_DATA_DIR>`` / ``<SOFTWARE_ROOT_DIR>``)
+    are fixed for a whole run,
+    while the per-execution ``<OUTPUT_DIR>`` is late-bound with :meth:`with_output`.
     """
-    Rewrite portable placeholders back to absolute paths ("from").
 
-    Inverse of :func:`to_placeholders`: replaces ``<OUTPUT_DIR>`` with the absolute ``output_dir``
-    and ``<TEST_DATA_DIR>`` with the absolute ``test_data_dir`` in every text artefact under ``directory``.
-    Binary files are never touched.
+    pairs: tuple[tuple[str, Path], ...]
+    """Ordered ``(token, absolute_directory)`` pairs.
 
-    Parameters
-    ----------
-    directory
-        The tree of artefacts to hydrate in place.
-    output_dir
-        The absolute execution output directory to substitute in.
-    test_data_dir
-        The absolute provider test-data directory to substitute in.
-    globs
-        File globs whose contents are rewritten.
+    Application order is not significant:
+    all rewriting goes through :func:`rewrite_tree`,
+    which re-sorts longest-match-first so an overlapping shorter path cannot shadow a longer one.
     """
-    rewrite_tree(
-        directory,
-        {PLACEHOLDER_OUTPUT_DIR: str(output_dir), PLACEHOLDER_TEST_DATA_DIR: str(test_data_dir)},
-        globs,
-    )
+
+    @classmethod
+    def for_baseline(cls, *, test_data_dir: Path, software_root_dir: Path | None = None) -> PlaceholderMap:
+        """
+        Build the configuration-stable placeholder set for a committed baseline.
+
+        Holds every token except the per-execution output directory,
+        which is added with :meth:`with_output`.
+        ``software_root_dir`` is optional:
+        when ``None`` no ``<SOFTWARE_ROOT_DIR>`` substitution is applied --
+        a verification context that does not know the shared-software root relies on this,
+        and the omission is explicit and declared once.
+        """
+        pairs: list[tuple[str, Path]] = [(PLACEHOLDER_TEST_DATA_DIR, test_data_dir)]
+        if software_root_dir is not None:
+            pairs.append((PLACEHOLDER_SOFTWARE_ROOT_DIR, software_root_dir))
+        return cls(pairs=tuple(pairs))
+
+    def with_output(self, output_dir: Path) -> PlaceholderMap:
+        """Return a new map that also binds ``<OUTPUT_DIR>`` to the per-execution ``output_dir``.
+
+        Rebinding replaces any existing ``<OUTPUT_DIR>`` entry,
+        so a second call hydrates to the latest directory rather than the first.
+        """
+        rest = tuple((token, path) for token, path in self.pairs if token != PLACEHOLDER_OUTPUT_DIR)
+        return PlaceholderMap(pairs=((PLACEHOLDER_OUTPUT_DIR, output_dir), *rest))
+
+    @property
+    def is_output_bound(self) -> bool:
+        """Whether ``<OUTPUT_DIR>`` has been bound via :meth:`with_output`.
+
+        The committed-bundle writer requires this:
+        an unbound map would leave execution-specific output paths in the committed bundle.
+        """
+        return any(token == PLACEHOLDER_OUTPUT_DIR for token, _ in self.pairs)
+
+    def as_replacements(self) -> dict[str, str]:
+        """Return the ``{absolute_directory: token}`` mapping (real path -> placeholder).
+
+        This is what the bundle and series comparators apply to a regenerated artefact
+        before diffing it against the committed (already-placeholdered) baseline.
+        """
+        return {str(abs_dir): token for token, abs_dir in self.pairs}
+
+    def sanitise(self, directory: Path, globs: tuple[str, ...] = SANITISED_FILE_GLOBS) -> None:
+        """Rewrite absolute paths to ``<TOKEN>`` placeholders in every text artefact under ``directory``.
+
+        Binary files are never touched.
+        In-place.
+        """
+        rewrite_tree(directory, self.as_replacements(), globs)
+
+    def hydrate(self, directory: Path, globs: tuple[str, ...] = SANITISED_FILE_GLOBS) -> None:
+        """Inverse of :meth:`sanitise`: rewrite ``<TOKEN>`` placeholders back to absolute paths.
+
+        Binary files are never touched.
+        In-place.
+        """
+        rewrite_tree(directory, {token: str(abs_dir) for token, abs_dir in self.pairs}, globs)
 
 
 def copy_output_file(
@@ -228,13 +260,49 @@ def _copy_output_bundle_files(
     return copied
 
 
-def copy_execution_outputs(
+def _copy_extra_globs(
+    scratch_directory: Path,
+    results_directory: Path,
+    fragment: Path | str,
+    extra_globs: tuple[str, ...],
+    exclude: set[Path],
+) -> list[Path]:
+    """
+    Copy every file matching ``extra_globs`` under the execution fragment into results.
+
+    These are *extra* raw artefacts a diagnostic declares
+    (:attr:`~climate_ref_core.diagnostics.Diagnostic.reconstruction_inputs`)
+    beyond the files its CMEC output bundle references,
+    so that :meth:`~climate_ref_core.diagnostics.Diagnostic.build_execution_result`
+    can re-derive its bundle on replay from the persisted set alone.
+    Each glob is resolved against ``scratch_directory / fragment``
+    (so ``**`` and ``/`` in the pattern behave as for :meth:`pathlib.Path.glob`);
+    directories are skipped.
+    """
+    input_directory = scratch_directory / fragment
+
+    copied: list[Path] = []
+    seen: set[Path] = set(exclude)
+    for glob in extra_globs:
+        for match in sorted(input_directory.glob(glob)):
+            if not match.is_file():
+                continue
+            relative = match.relative_to(input_directory)
+            if relative in seen:
+                continue
+            seen.add(relative)
+            copied.append(copy_output_file(scratch_directory, results_directory, fragment, relative))
+    return copied
+
+
+def copy_execution_outputs(  # noqa: PLR0913
     scratch_directory: Path,
     results_directory: Path,
     fragment: Path | str,
     result: ExecutionResult,
     *,
     include_log: bool = False,
+    extra_globs: tuple[str, ...] = (),
 ) -> list[Path]:
     """
     Copy the curated set of persisted outputs from scratch to results.
@@ -246,6 +314,7 @@ def copy_execution_outputs(
     - every file it references (plots/data/html)
     - the series bundle
     - the execution log (if ``include_log=True``)
+    - any files matching ``extra_globs`` (a diagnostic's declared reconstruction inputs)
 
     Parameters
     ----------
@@ -259,6 +328,10 @@ def copy_execution_outputs(
         The successful execution result (must carry a metric bundle filename).
     include_log
         If True, copy the execution log.
+    extra_globs
+        Additional output globs (relative to the execution directory) to persist beyond the
+        bundle-referenced files, e.g. a diagnostic's
+        :attr:`~climate_ref_core.diagnostics.Diagnostic.reconstruction_inputs`.
 
     Returns
     -------
@@ -292,4 +365,11 @@ def copy_execution_outputs(
             copy_output_file(scratch_directory, results_directory, fragment, result.series_filename)
         )
 
+    if extra_globs:
+        copied.extend(
+            _copy_extra_globs(
+                scratch_directory, results_directory, fragment, extra_globs, exclude=set(copied)
+            )
+        )
+
     return copied

packages/climate-ref-core/src/climate_ref_core/regression/_quantise.py

@@ -1,29 +1,21 @@
 """
 Float quantisation for committed regression bundles.
 
-The rounded committed JSON (``series.json`` / ``diagnostic.json``)
-records full-precision floats whose least-significant digits are platform-dependent
+The regression files include floats whose least-significant digits are platform-dependent
 (CPU, BLAS, library versions).
-Those last digits churn byte-for-byte between CI and local runs even when the result is numerically identical,
+The last digits churn byte-for-byte between CI and local runs even when the result is numerically identical,
 producing noisy, unreviewable diffs in the committed bundle.
 
-``output.json`` is excluded from rounding: it is the CMEC *output* bundle (metadata only, no
-float leaves by construction) and is serialised by Pydantic's ``model_dump_json(indent=2)`` which
-does not sort keys, so re-dumping it with ``json.dumps(sort_keys=True)`` would reorder its keys and
-churn the committed bytes (see :data:`climate_ref_core.regression.capture._UNROUNDED_COMMITTED_FILES`).
-
 Rounding every float to a fixed number of significant figures at write time gives
 stable, reviewable committed bytes.
-We round to seven significant figures: one digit finer than the regression compare
-tolerance (``rtol=1e-6`` in :mod:`climate_ref_core.regression.compare`),
+We round to seven significant figures: one digit finer than the regression compare tolerance
+(``rtol=1e-6`` in :mod:`climate_ref_core.regression.compare`),
 so the rounding error stays an order of magnitude under tolerance and can never flip a boundary gate verdict.
 
 This affects only the committed bundle.
 The native blobs (``.nc`` / ``.png``) and their content-addressed digests are never touched.
 """
 
-from __future__ import annotations
-
 from typing import Any
 
 DEFAULT_SIG_FIGS: int = 7