UQ benchmarking

.gitignore

@@ -21,4 +21,7 @@ venv/
 # Editor / IDE
 .cursor/
 .vscode/
-.idea/
+.idea/
+
+# Local, machine-specific benchmark configs (real paths); not for the repo
+*.local.yaml

CHANGELOG.md

@@ -17,9 +17,52 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   **`physicsnemo-cfd-create-custom-metric`** — each with a `SKILL.md` and
   `evals/` cases to guide adding a new `CFDModel`, `DatasetAdapter`, or
   metric.
+- **DrivAerStar surface dataset adapter** (`datasets/adapters/drivaerstar.py`,
+  in-memory transforms with opt-in disk caching) and **`DatasetConfig.label`**
+  for decoupling report display names from adapter lookup / caching.
+- **Probabilistic / uncertainty-quantification (UQ) benchmarking:**
+  additive UQ path across the evaluation stack.
+  - **`CFDModel` capability model:** `SUPPORTS_UQ` / `UQ_METHOD`
+    (`"none"` | `"analytic"` | `"sampling"`) class flags plus an optional
+    `decode_distribution` hook; deterministic wrappers are unchanged
+    (`SUPPORTS_UQ=False` -> UQ metrics report `NaN`).
+  - **`FieldDistribution`** payload (`datasets/schema.py`) carrying
+    `mean` + `std` / `epistemic_std` / `aleatoric_std` (and optional
+    `samples` / `quantiles`), with `build_predictive_distribution` and
+    `as_distribution` helpers; all in physical units.
+  - **Engine sampling loop** (`benchmarks/uq_inference.py`): `run.uq`
+    controls (`enabled`, `num_samples`, `retain_samples`, `device_metrics`),
+    streaming Welford aggregation of `N` stochastic passes, and an optional
+    `predict_ensemble(model_input, n)` fast-path for multi-member models.
+  - **Pooled + sample UQ metrics** (`metrics/builtin/uq.py`): `nlpd`,
+    `nlpd_epistemic`, `calibration_zrms`, `coverage_95`, `sharpness_std`,
+    `sharpness_epistemic_std`, `uncertainty_error_spearman`
+    (+ `_epistemic`), `sparsification_ause` (+ `_epistemic`), and `drag_uq`
+    (configurable `coeff` / `drag_direction`), implemented via
+    reducer/sample-metric protocols so statistics pool correctly across
+    cases and ranks.
+  - **`sparsification_plot`** report visual, and std / epistemic-std
+    companion arrays exported to inference / comparison meshes
+    (`output.std_mesh_field_names`, `output.epistemic_std_mesh_field_names`).
+  - **Example UQ model wrappers** (GeoTransolver family, one per
+    archetype): `geotransolver_gp_surface` (analytic GP field head),
+    `geotransolver_mc_dropout_surface` (Concrete-Dropout sampling), and
+    `geotransolver_ensemble_surface` (explicit multi-member ensemble),
+    alongside the deterministic `geotransolver_drivaerstar_surface`. Shared
+    inference plumbing in
+    `models/common_wrapper_utils/geotransolver_runtime.py`.
+  - **Example config:** `workflows/benchmarking/conf/config_uq_surface.yaml`
+    scores deterministic + analytic GP + MC-Dropout + ensemble on the same
+    cases.
 
 ### Changed
 
+- **`physicsnemo-cfd-create-model-wrapper` skill:** documents the UQ
+  wrapper contract (capability flags, `FieldDistribution`, analytic vs.
+  sampling archetypes, the engine sampling loop, and UQ metrics/config),
+  with `ExampleAnalyticUQWrapper` / `ExampleSamplingUQWrapper` reference
+  implementations.
+
 ### Deprecated
 
 ### Removed

physicsnemo/cfd/evaluation/benchmarks/distributed_utils.py

@@ -85,6 +85,12 @@ def merge_benchmark_result_shards(shards: list[dict[str, Any]]) -> dict[str, Any
                 f"Distributed merge mismatch: expected {model!r}×{dataset!r}, "
                 f"got {s['model']!r}×{s['dataset']!r}"
             )
+    from physicsnemo.cfd.evaluation.benchmarks.uq_inference import (
+        finalize_reducer_metrics,
+        finalize_sample_metrics,
+        is_uq_partial_key,
+    )
+
     per_case: list[dict[str, Any]] = []
     for s in active:
         per_case.extend(s.get("per_case") or [])
@@ -95,8 +101,22 @@ def merge_benchmark_result_shards(shards: list[dict[str, Any]]) -> dict[str, Any
             all_metric_values.setdefault(k, []).append(float(v))
     metrics_summary: dict[str, float] = {}
     for mname, values in all_metric_values.items():
+        # Reducer / sample statistics are finalized (pooled / collected), not mean-over-cases.
+        if is_uq_partial_key(mname):
+            continue
         valid = [v for v in values if v == v]
         metrics_summary[mname] = sum(valid) / len(valid) if valid else float("nan")
+    inference_domain = active[0].get("inference_domain")
+    metrics_summary.update(finalize_reducer_metrics(per_case, inference_domain))
+    metrics_summary.update(finalize_sample_metrics(per_case, inference_domain))
+    # Preserve NaN placeholders for configured-but-unavailable UQ metrics: each shard already
+    # finalized them (via the configured metric names in _run_single_model_dataset), so carry any
+    # summary key missing from the merged set forward as-is. setdefault never clobbers a value
+    # recomputed above from real partials.
+    for s in active:
+        for k, v in (s.get("metrics") or {}).items():
+            if isinstance(v, (int, float)):
+                metrics_summary.setdefault(k, float(v))
     case_ids_raw = [str(r["case_id"]) for r in per_case if r.get("case_id") is not None]
     case_ids_no_dup = sorted(set(case_ids_raw), key=natural_sort_key)
     return {
@@ -105,6 +125,7 @@ def merge_benchmark_result_shards(shards: list[dict[str, Any]]) -> dict[str, Any
         "cases": case_ids_no_dup,
         "metrics": metrics_summary,
         "per_case": per_case,
+        "inference_domain": inference_domain,
     }