feat: multi-and-per-flow rates schema (v2.2.0)

CHANGELOG.md

@@ -5,6 +5,89 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] - 2026-05-10
+
+### Added
+
+- **`enabled: false` annotation on simulations**: Scenarios marked
+  `enabled: false` in `simulate.yaml` are now dropped before strict
+  schema validation, letting externally-managed scenarios (e.g.
+  Monte-Carlo runs whose outputs are produced outside skypro) coexist
+  with live scenarios in the same yaml. Removes the need for callers
+  (e.g. the rebuilder in skypro-service) to write a temp staged yaml
+  to scrub them out.
+
+- **Per-flow `imbalanceDataSourceOverride` on `RatesFiles`**: Each flow
+  in a rates block can now override the block-level `imbalanceDataSource`
+  via a new dict shape:
+
+  ```yaml
+  rates:
+    final:
+      imbalanceDataSource: *imbSrc_default     # block-level default
+      files:
+        gridToBatt:                            # legacy list shape — inherits default
+          - dno_import.json
+          - supply_import.json
+        solarToGrid:                           # new dict shape with override
+          rates:
+            - dno_export.json
+            - supply_statkraft_export.json
+          imbalanceDataSourceOverride: *imbSrc_plain
+  ```
+
+  Closes the structural Axle-premium leak on two-MPAN BSCP550 sites
+  where the BESS and site MPANs settle against different imbalance
+  signals. Backward-compatible — legacy list shape on flows continues
+  to parse.
+
+- **Multi-`finals` per simulation**: Declare N settlement variants in
+  one simulation. Mutually exclusive with the legacy `final: <Rates>`
+  field (same precedent as `peak`/`peaks` in priceCurveAlgo):
+
+  ```yaml
+  rates:
+    live:  *ratesLive_basecase
+    finals:
+      fullfcl:      *ratesFinal_fullfcl
+      trio_imbflex: *ratesFinal_trio_imbflex
+  ```
+
+  Fanned out at parse time into one expanded `SimulationCase` per
+  variant (sim name `<orig>.<variant>`). CSV paths get an automatic
+  variant suffix when they don't use `$_SIM_NAME`. The optimiser runs
+  N times — accepted trade-off for YAML ergonomics; the
+  single-dispatch / multi-settlement-column variant is deferred (see
+  `docs/proposals/multi-final-rates.md`).
+
+### Changed
+
+- `RatesFiles` flow fields are now `FlowFilesType` (a `FlowFiles`
+  dataclass) rather than `List[PathType]`. Schema parsing accepts
+  either list or dict shape transparently. **No migration needed for
+  existing YAML configs.** Internal callers that previously accessed
+  e.g. `rates_files.grid_to_batt[0]` need to use
+  `rates_files.grid_to_batt.rates[0]`. Only consumer affected was
+  `parse_vol_rates_files_for_all_energy_flows`, which has been
+  updated.
+
+- `parse_vol_rates_files_for_all_energy_flows` accepts an optional
+  `flow_imbalance_pricings: Dict[str, pd.Series]` keyword argument
+  for per-flow override pricings. Cache key now
+  `(file_list_str, id(pricing))` so flows with the same files but
+  different overrides don't share rate instances.
+
+### Compatibility
+
+- Legacy single-`final` configs unchanged. Verified by integration
+  tests (`integrationTestPriceCurve`, `integrationTestPriceCurveMultiPeak`,
+  `integrationTestPerfectHindsightLP`) — bit-identical LP output
+  within tolerance `0.01`.
+- Legacy list-shape rate files continue to parse and behave
+  identically.
+- `ratesDB` source rejects per-flow overrides with a clear error
+  (override only supported with the YAML `files` source).
+
 ## [2.0.5] - 2026-05-07
 
 ### Fixed

CLAUDE.md

@@ -123,6 +123,51 @@ TWINE_USERNAME=__token__ TWINE_PASSWORD=$(cat ~/.simt/testpypi.token) python -m
 - **Market** - actual cashflows with suppliers
 - **Internal** - opportunity cost for optimization
 
+### `simulate.yaml` schema reference (post-v2.2.0)
+
+Three small ergonomic features added in v2.2.0. Authoritative
+reference: `CHANGELOG.md` v2.2.0 entry; design history:
+`docs/proposals/`.
+
+**`enabled: false`** on a simulation drops it pre-schema — useful for
+externally-managed scenarios that may use unsupported strategy keys:
+```yaml
+simulations:
+  hmce.202512.imb.mc-reopt:
+    enabled: false   # outputs preserved as-is, schema doesn't see this scenario
+    strategy: { monteCarloAlgo: ... }
+```
+
+**Per-flow `imbalanceDataSourceOverride`** for two-MPAN BSCP550 sites
+(or anywhere flows in a single rates block need different imbalance
+signals). Flow can be either a list (legacy) or a dict carrying an
+override:
+```yaml
+files:
+  gridToBatt: [ a.json, b.json ]                # legacy — inherits block-level
+  solarToGrid:                                  # dict shape with override
+    rates: [ a.json, b.json ]
+    imbalanceDataSourceOverride: *imbSrc_plain
+```
+Apply override **symmetrically** to live AND final blocks — MPAN
+imbalance treatment is structural, not algo-only.
+
+**Multi-`finals`** declares N settlement variants per simulation,
+fanned out at parse time. Mutually exclusive with `final:` (same
+precedent as `peak`/`peaks`). Sim names become `<orig>.<variant>`;
+CSV paths get a variant suffix when not using `$_SIM_NAME`:
+```yaml
+rates:
+  live:  *ratesLive_basecase
+  finals:
+    fullfcl:      *ratesFinal_fullfcl
+    trio_imbflex: *ratesFinal_trio_imbflex
+```
+Each variant runs the optimiser independently — accepted trade-off
+for YAML ergonomics. The single-dispatch / multi-column variant
+(loop `_process_final_rates`, OSAM-mutation-safe via deepcopy) is
+deferred — see `docs/proposals/multi-final-rates.md`.
+
 ### OSAM (P395)
 On-site Allocation Methodology for calculating final demand levies. Runs in parallel with Skypro's own methodology; discrepancies reported as Notices.
 
@@ -185,6 +230,7 @@ pandas, plotly, pulp, pendulum, sqlalchemy, psycopg2-binary, marshmallow, pyyaml
 
 | Date | PR | Branch | Summary |
 |------|-----|--------|---------|
+| 2026-05-09 | TBD | feature/multi-and-per-flow-rates | Three schema additions for multi-MPAN BSCP550 + multi-settlement workflows: (1) `enabled: false` annotation on simulations now accepted by core (drops the rebuilder's temp-yaml workaround); (2) per-flow `imbalanceDataSourceOverride` inside `RatesFiles` flows so two-MPAN sites can settle BESS-MPAN and site-MPAN against different imbalance signals (closes the HMCE Apr-26 BSCP550 ~£500–700/mo Axle leak); (3) `rates.finals: {<name>: Rates, ...}` declares N settlement variants per simulation, fanned out at parse time into one sim per variant with auto-suffixed CSV paths. Backward-compatible — legacy list/single-final shapes unchanged. +14 unit tests (v2.2.0) |
 | 2026-05-08 | TBD | feature/support-axle-flex | New `Peak.dynamic.minEndOfPeakSoe` parameter reserves SoE for post-peak niv-chase. Used in time-to-empty calc as `dischargeable_soe = soe − min_end_of_peak_soe`, creating slack so the dynamic HOLD-on-LONG branch can actually fire instead of always falling through to forced full discharge. Default 0 (legacy behaviour). +5 unit tests (v2.1.1) |
 | 2026-05-08 | TBD | feature/support-axle-flex | Multi-peak support in priceCurveAlgo: new `peaks: [...]` list form alongside legacy `peak: ...` (mutually exclusive). Enables dispatch into multiple price-elevated windows per day. Backward-compat verified — existing single-peak fixture summary unchanged within tolerance (v2.1.0) |
 | 2026-04-30 | TBD | feature/profile-filter-and-nameplate | Opt-in profile anomaly filter (`maxEnergyPerIntervalKwh`) + display-only `nameplateKwp` metadata field on Profile (v2.0.4) |

docs/proposals/multi-final-rates.md

@@ -0,0 +1,155 @@
+# Skypro core change — support multiple `ratesFinal` per simulation
+
+**Audience:** Skypro engineer (Python core)
+**Author:** Damon Rand
+**Status:** ✅ IMPLEMENTED in v2.2.0 (commit `2188e35` on `feature/multi-and-per-flow-rates`)
+**— with a different design.** The shipped implementation is a
+**parse-time fan-out**: `rates.finals: {<name>: Rates, ...}` expands
+into one independent `SimulationCase` per variant, each with its own
+output CSV. The optimiser runs N times. The single-dispatch /
+multi-settlement-column variant described below (loop
+`_process_final_rates`, suffix output columns with
+`.final[<variant>]`) is **deferred** — user priority was YAML
+ergonomics over compute saving. See `CHANGELOG.md` v2.2.0 for the
+final YAML reference. This brief is preserved as the design record
+for if/when the optimised variant lands.
+
+## Background — why this matters
+
+Today skypro's simulation is configured with a single pair of rates:
+
+```yaml
+rates:
+  live:  { ... }     # what the optimiser sees, drives dispatch
+  final: { ... }     # what gets settled, drives reported margin
+```
+
+In several real workflows we want to evaluate the **same dispatch
+decision against multiple alternative settlement rate structures**.
+Re-running the optimiser is unnecessary in those cases — the dispatch
+is locked once `ratesLive` is fixed; only the cost columns change.
+
+### Concrete example — Axle / flex on/off
+
+A common pattern: optimise against base imbalance prices (`ratesLive =
+imbalance`), then settle the same dispatch under two final structures
+in parallel:
+
+1. `final.no_flex = imbalance` — what the operator earns without any
+   flex aggregator (counterfactual).
+2. `final.with_flex = imbalance + axle_premium × bess_share` — what
+   the operator actually banks once Axle / a Virtual Trading Party
+   takes their cut.
+
+Today this requires running skypro twice. The optimiser runs both
+times even though it's the same dispatch question. We want one
+optimiser run, two parallel settlement column-sets in the output CSV.
+
+### Other use cases
+
+- **Tariff alternatives**: Trio vs flat vs Octopus Tracker for the same
+  dispatch. Useful for proposals to housing developers / customers.
+- **OSAM on/off** as a sensitivity (subject to the OSAM-NCSP caveat
+  below — it's dispatch-driven, not rate-driven, so OSAM-NCSP is a
+  shared input across variants).
+- **Per-flow settlement structures** that exist today but require
+  separate sims: e.g. site MPAN settles at imbalance, BESS MPAN
+  settles at imbalance+flex (the BSCP550 pattern at HMCE — though
+  see the per-flow-imbalance-source-override.md brief for a more
+  targeted fix to that specific case).
+
+## What to build
+
+Extend the YAML schema so `rates.final` can be **either** a single
+`Rates` object (backward-compatible) **or** a dict of named variants:
+
+```yaml
+rates:
+  live: { ... }
+  final:
+    no_flex:    { ... }
+    with_flex:  { ... }
+```
+
+Output CSV columns get a variant suffix: `mvRate:battToGrid.final[no_flex]`,
+`mvRate:battToGrid.final[with_flex]`, etc. Single-variant case (current
+schema) stays as `mvRate:battToGrid.final` — no breaking change.
+
+## Existing entry points worth knowing about
+
+(Verified via read of skypro core. Treat as starting points; full
+design is the implementer's call.)
+
+- **Schema**: `AllRates` dataclass at
+  `src/skypro/commands/simulator/config/config.py:378`. Make
+  `final` polymorphic.
+- **Re-rating loop**: `_process_final_rates()` at
+  `src/skypro/commands/simulator/main.py:328`. The function
+  is pure (input → output) given a fixed dispatch. Loop it N times
+  once the optimiser is done.
+- **Output column generation**: `generate_output_df()` at
+  `src/skypro/common/microgrid_analysis/output.py:98`.
+  Currently iterates over `(int_final, mkt_final, int_live, mkt_live)`
+  rate dicts. Extend to iterate per variant.
+- **OSAM/P395 NCSP**: `calculate_osam_ncsp()` at
+  `src/skypro/common/rate_utils/osam.py:15`. NCSP is
+  **dispatch-dependent but rate-variant-independent** — compute once
+  after the optimiser, reuse across all variants.
+- **`skypro report`** reuses `generate_output_df()`. Same change
+  benefits report-side rate switching for free.
+
+## Rough scope
+
+**~3–4 hours** as a focused PR. Not a week-long architectural project.
+
+Roughly:
+1. Schema + parser tweak (singular vs dict `final`).
+2. Hoist the OSAM NCSP calc above the re-rating loop.
+3. Loop `_process_final_rates()` per variant; collect per-variant
+   rate dataframes.
+4. Extend column naming in `generate_output_df()` to emit
+   `*.final[<variant>]` when multiple variants are configured.
+5. Backward-compat regression test on the single-variant path.
+
+## Known risks / red flags
+
+1. **Column naming ambiguity.** Don't keep a rollup
+   `mvRate:battToGrid.final` alongside `mvRate:battToGrid.final[v1]`
+   and `[v2]` — downstream consumers (skypro-fresh dashboards,
+   axle_reconcile, ad-hoc analysis scripts) would have to guess
+   whether the rollup is a sum, an average, or stale. Cleaner: drop
+   the rollup, require variant names in brackets when multiple are
+   configured.
+2. **OSAM rate-instance state mutation.** `OSAMFlatVolRate.add_ncsp()`
+   mutates the rate instance in place. If the same OSAM rate object
+   appears in multiple final variants, the second variant's add_ncsp
+   could double-apply. Either deep-copy rate instances per variant
+   in the loop, or refactor NCSP to be a runtime parameter rather
+   than mutated state.
+3. **CSV row-width explosion.** With N variants, you get roughly
+   N×2 extra columns per flow per HH. For 12-month detail dumps
+   with 3 variants, expect ~3× the file size. Consider an opt-in
+   summary-only output mode for variant-heavy runs, or document
+   the cost.
+
+## Out of scope for this change
+
+- Changes to the optimiser or algorithm layer — none required.
+- Changes to YAML rate parsing primitives — already list-aware.
+- OSAM math itself — unchanged.
+- Per-flow `imbalanceDataSource` override — separate brief at
+  `per-flow-imbalance-source-override.md`. Both changes are
+  independent and can ship in either order.
+- skypro-fresh dashboard surfaces for the new variant columns
+  (consumer-side work, separate).
+
+## Validation hint
+
+Run `hmce.202604` Axle-aware scenarios with `final = { with_axle,
+without_axle }`. The `with_axle` column-set should exactly equal
+the current single-`final` output. The `without_axle` column-set
+should equal what the existing Impr0 family scenarios
+(`hmce.202604.*.imb-imb.*.bscp550.imb-imbflex` etc.) produce today
+when run as standalone sims with `ratesLive ≠ ratesFinal`. After
+this change ships, those Impr0 sims become redundant — one
+multi-`ratesFinal` sim replaces the pair.