From 7b7caa6435b86b83a2caf72781b3ecc726c5319b Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sat, 9 May 2026 18:54:49 +0100 Subject: [PATCH 1/7] docs: add proposals for two skypro-core changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both originated from sims work in skyprospector.com/skypro-service (NCESF + HMCE Axle integration). HMCE 202604 is the validation case for both — its rebuild today landed with known biases that these two changes close. - per-flow-imbalance-source-override: lets BSCP550 wire BESS flows to Axle-stacked imbalance and site flows to plain imbalance within a single rates block. Closes a ~£500-700/mo over-attribution bias on HMCE Apr-26 BSCP550 scenarios where Axle currently leaks onto gridToLoad and solarToGrid. - multi-final-rates: lets one simulation run produce N parallel ratesFinal column-sets against the same dispatch. Avoids re-running the optimiser to compare settlement structures (e.g. Axle on vs off, Trio vs flat, OSAM on/off). Eliminates the standalone Impr0 scenario pattern. Both are ~3-4 hour PRs. Independent — can ship in either order. --- docs/proposals/multi-final-rates.md | 145 ++++++++++++++ .../per-flow-imbalance-source-override.md | 177 ++++++++++++++++++ 2 files changed, 322 insertions(+) create mode 100644 docs/proposals/multi-final-rates.md create mode 100644 docs/proposals/per-flow-imbalance-source-override.md diff --git a/docs/proposals/multi-final-rates.md b/docs/proposals/multi-final-rates.md new file mode 100644 index 0000000..47cf3e9 --- /dev/null +++ b/docs/proposals/multi-final-rates.md @@ -0,0 +1,145 @@ +# Skypro core change — support multiple `ratesFinal` per simulation + +**Audience:** Skypro engineer (Python core) +**Author:** Damon Rand +**Status:** Brief — design + implementation TBD + +## Background — why this matters + +Today skypro's simulation is configured with a single pair of rates: + +```yaml +rates: + live: { ... } # what the optimiser sees, drives dispatch + final: { ... } # what gets settled, drives reported margin +``` + +In several real workflows we want to evaluate the **same dispatch +decision against multiple alternative settlement rate structures**. +Re-running the optimiser is unnecessary in those cases — the dispatch +is locked once `ratesLive` is fixed; only the cost columns change. + +### Concrete example — Axle / flex on/off + +A common pattern: optimise against base imbalance prices (`ratesLive = +imbalance`), then settle the same dispatch under two final structures +in parallel: + +1. `final.no_flex = imbalance` — what the operator earns without any + flex aggregator (counterfactual). +2. `final.with_flex = imbalance + axle_premium × bess_share` — what + the operator actually banks once Axle / a Virtual Trading Party + takes their cut. + +Today this requires running skypro twice. The optimiser runs both +times even though it's the same dispatch question. We want one +optimiser run, two parallel settlement column-sets in the output CSV. + +### Other use cases + +- **Tariff alternatives**: Trio vs flat vs Octopus Tracker for the same + dispatch. Useful for proposals to housing developers / customers. +- **OSAM on/off** as a sensitivity (subject to the OSAM-NCSP caveat + below — it's dispatch-driven, not rate-driven, so OSAM-NCSP is a + shared input across variants). +- **Per-flow settlement structures** that exist today but require + separate sims: e.g. site MPAN settles at imbalance, BESS MPAN + settles at imbalance+flex (the BSCP550 pattern at HMCE — though + see the per-flow-imbalance-source-override.md brief for a more + targeted fix to that specific case). + +## What to build + +Extend the YAML schema so `rates.final` can be **either** a single +`Rates` object (backward-compatible) **or** a dict of named variants: + +```yaml +rates: + live: { ... } + final: + no_flex: { ... } + with_flex: { ... } +``` + +Output CSV columns get a variant suffix: `mvRate:battToGrid.final[no_flex]`, +`mvRate:battToGrid.final[with_flex]`, etc. Single-variant case (current +schema) stays as `mvRate:battToGrid.final` — no breaking change. + +## Existing entry points worth knowing about + +(Verified via read of skypro core. Treat as starting points; full +design is the implementer's call.) + +- **Schema**: `AllRates` dataclass at + `src/skypro/commands/simulator/config/config.py:378`. Make + `final` polymorphic. +- **Re-rating loop**: `_process_final_rates()` at + `src/skypro/commands/simulator/main.py:328`. The function + is pure (input → output) given a fixed dispatch. Loop it N times + once the optimiser is done. +- **Output column generation**: `generate_output_df()` at + `src/skypro/common/microgrid_analysis/output.py:98`. + Currently iterates over `(int_final, mkt_final, int_live, mkt_live)` + rate dicts. Extend to iterate per variant. +- **OSAM/P395 NCSP**: `calculate_osam_ncsp()` at + `src/skypro/common/rate_utils/osam.py:15`. NCSP is + **dispatch-dependent but rate-variant-independent** — compute once + after the optimiser, reuse across all variants. +- **`skypro report`** reuses `generate_output_df()`. Same change + benefits report-side rate switching for free. + +## Rough scope + +**~3–4 hours** as a focused PR. Not a week-long architectural project. + +Roughly: +1. Schema + parser tweak (singular vs dict `final`). +2. Hoist the OSAM NCSP calc above the re-rating loop. +3. Loop `_process_final_rates()` per variant; collect per-variant + rate dataframes. +4. Extend column naming in `generate_output_df()` to emit + `*.final[]` when multiple variants are configured. +5. Backward-compat regression test on the single-variant path. + +## Known risks / red flags + +1. **Column naming ambiguity.** Don't keep a rollup + `mvRate:battToGrid.final` alongside `mvRate:battToGrid.final[v1]` + and `[v2]` — downstream consumers (skypro-fresh dashboards, + axle_reconcile, ad-hoc analysis scripts) would have to guess + whether the rollup is a sum, an average, or stale. Cleaner: drop + the rollup, require variant names in brackets when multiple are + configured. +2. **OSAM rate-instance state mutation.** `OSAMFlatVolRate.add_ncsp()` + mutates the rate instance in place. If the same OSAM rate object + appears in multiple final variants, the second variant's add_ncsp + could double-apply. Either deep-copy rate instances per variant + in the loop, or refactor NCSP to be a runtime parameter rather + than mutated state. +3. **CSV row-width explosion.** With N variants, you get roughly + N×2 extra columns per flow per HH. For 12-month detail dumps + with 3 variants, expect ~3× the file size. Consider an opt-in + summary-only output mode for variant-heavy runs, or document + the cost. + +## Out of scope for this change + +- Changes to the optimiser or algorithm layer — none required. +- Changes to YAML rate parsing primitives — already list-aware. +- OSAM math itself — unchanged. +- Per-flow `imbalanceDataSource` override — separate brief at + `per-flow-imbalance-source-override.md`. Both changes are + independent and can ship in either order. +- skypro-fresh dashboard surfaces for the new variant columns + (consumer-side work, separate). + +## Validation hint + +Run `hmce.202604` Axle-aware scenarios with `final = { with_axle, +without_axle }`. The `with_axle` column-set should exactly equal +the current single-`final` output. The `without_axle` column-set +should equal what the existing Impr0 family scenarios +(`hmce.202604.*.imb-imb.*.bscp550.imb-imbflex` etc.) produce today +when run as standalone sims with `ratesLive ≠ ratesFinal`. After +this change ships, those Impr0 sims become redundant — one +multi-`ratesFinal` sim replaces the pair. diff --git a/docs/proposals/per-flow-imbalance-source-override.md b/docs/proposals/per-flow-imbalance-source-override.md new file mode 100644 index 0000000..520d4a2 --- /dev/null +++ b/docs/proposals/per-flow-imbalance-source-override.md @@ -0,0 +1,177 @@ +# Skypro core change — per-flow `imbalanceDataSource` override + +**Audience:** Skypro engineer (Python core) +**Author:** Damon Rand +**Status:** Brief — design + implementation TBD + +## Background — why this matters + +Today skypro's `Rates` config sets `imbalanceDataSource` once at the +rates-block level. Every flow that consumes imbalance (via the +`imbalance` rate type, optionally with a `multiplierRate` like +Statkraft × imbalance) reads from that single source: + +```yaml +rates: + final: + imbalanceDataSource: + files: + gridToBatt: [ ... ] # all see the same imbalance source + gridToLoad: [ ... ] + battToGrid: [ ... ] + solarToGrid: [ ... ] +``` + +This works for single-MPAN sites where the entire boundary settles +against one imbalance signal. It breaks for **two-MPAN BSCP550 sites** +where the BESS sits on its own MPAN and the residential supply is on +a different one — the two MPANs have structurally different +imbalance treatments and shouldn't share a source. + +### Concrete problem — HMCE BSCP550 + Axle leak + +At HMCE Apr-26 the BSCP550 metering split puts the BESS on its own +MPAN. Under that arrangement: + +- **BESS MPAN** participates in Axle's flex programme → its imbalance + signal gets the Axle premium stacked on top (via + `imbalance_price_with_axle/`). +- **Site MPAN** (residential supply) does NOT participate in Axle — + it should see plain imbalance (`imbalance_price/`). + +Because skypro has only one `imbalanceDataSource` per rates block, +the current `simulate.yaml` configures **all flows** in a BSCP550 +rates anchor to read from `imbSrc_elexon_axle` (Axle-stacked). The +result: `gridToLoad` and `solarToGrid` (site-MPAN flows) absorb the +Axle premium they shouldn't see. + +Quantified bias: **~£500–700/mo over-attribution** on HMCE Apr-26 +BSCP550 scenarios (Axle window HHs × site-load and site-export volumes +× BESS-share-scaled Axle premium). The simulator's `margin` column on +those scenarios overstates HMCE's actual revenue by this amount, and +the optimiser sees biased prices on site flows so dispatch decisions +are slightly off too. + +We worked around this in +`projects/mgfl/hazelmead/202604/tuning_history.md` (2026-05-09 entry) +by publishing the biased numbers with an explicit caveat. This change +closes the gap. + +## What to build + +Allow individual flows in a rates block to override the block-level +`imbalanceDataSource`. Either as a per-flow override or as a property +of the `imbalance` rate type. Sketch (exact schema TBD): + +```yaml +rates: + final: + imbalanceDataSource: *imbSrc_elexon_plain # default for site flows + files: + gridToBatt: # BESS MPAN + imbalanceDataSourceOverride: *imbSrc_elexon_axle + rates: + - dno_fees_southwest_import.json + - supply_fees_unify_import.json + battToGrid: # BESS MPAN + imbalanceDataSourceOverride: *imbSrc_elexon_axle + rates: + - dno_fees_southwest_export.json + - supply_fees_statkraft_export.json + gridToLoad: # Site MPAN — default + rates: + - dno_fees_southwest_import.json + - supply_fees_unify_import.json + - final_consumption_levies.yaml + solarToGrid: # Site MPAN — default + rates: + - dno_fees_southwest_export.json + - supply_fees_statkraft_export.json +``` + +Backward-compat: if no per-flow override is set, behaviour matches +today (block-level source applies to all flows). The current YAML +shape (`gridToBatt: [, ]`) should stay valid; the new +override-capable shape is a per-flow opt-in. + +## Existing entry points worth knowing about + +(Treat as starting points; full design is the implementer's call.) + +- **Schema**: `Rates.files` parsing in + `src/skypro/commands/simulator/config/config.py`. The `files` field + currently accepts `Dict[str, List[str]]`; needs to accept a richer + per-flow shape that carries an optional override. +- **Imbalance rate construction**: wherever the `imbalance` rate type + is built from rate files — that's the consumption point that + currently reads the block-level `imbalanceDataSource`. The override + should plug in there. +- **`generate_output_df()`** in `src/skypro/common/microgrid_analysis/output.py` + — should already cope as long as the rate engine produces the right + per-flow rate dataframes. Output column naming unchanged. + +## Rough scope + +**~4–6 hours** as a focused PR. Slightly bigger than the +multi-`ratesFinal` change because it touches per-flow rate +construction, not just an outer loop. + +Roughly: +1. Schema change: new per-flow override field (config.py). +2. Plumb the override through to the `imbalance` rate constructor. +3. Backward-compat: existing list-shape continues to work. +4. Unit test: a rates block with mixed per-flow imbalance sources + produces the expected per-flow rate dataframes. +5. Integration test: re-run a BSCP550 HMCE scenario before/after, + confirm Axle premium only lands on `battToGrid` / `gridToBatt`. + +## Known risks / red flags + +1. **Multiplier rates referencing the wrong source.** Files like + `supply_fees_statkraft_export.json` reference imbalance via + `multiplierRate`. With per-flow override, the multiplier needs to + resolve to the per-flow source, not the block-level one. Verify + the multiplier-resolve path picks up overrides correctly. +2. **OSAM/P395 NCSP coupling.** OSAM's NCSP factor is dispatch-driven + and shared across flows. The per-flow override doesn't change + NCSP — it just changes which imbalance signal each flow's rate + stack consumes. Sanity-check that NCSP application still resolves + per-flow, not per-imbalance-source. +3. **YAML readability degrades** with the richer per-flow shape. + Consider keeping the simple list shape as the default and only + requiring the dict shape when an override is set. Don't force every + site to migrate. + +## Out of scope for this change + +- Multi-`ratesFinal` per simulation (separate brief at + `multi-final-rates.md`). Both changes are independent and can + ship in either order. +- Changes to the optimiser, algorithm layer, or output column naming. +- Restructuring how `imbSrc_*` anchors are defined in YAML (the + override pattern reuses existing imbalance source anchors). +- skypro-fresh dashboard surfaces (the change is invisible to the + consumer side as long as output column names stay stable). + +## Validation hint + +**HMCE 202604 is the validation case.** After this change ships: + +1. Update `projects/mgfl/hazelmead/202604/simulate.yaml` BSCP550 + rate anchors (`ratesFinal_bscp550`, `ratesLive_impr12`, + `ratesLive_impr2`) to override imbalance source per flow: + - `gridToBatt`, `battToGrid` → `imbSrc_*_axle` + - `gridToLoad`, `solarToGrid` → `imbSrc_*_plain` +2. Re-run `./tools/rebuild projects/mgfl/hazelmead/202604/`. +3. Compare margin columns for the HEADLINE scenario + `hmce.202604.1609kWh-210kWp.imb-imbflex.cc-twopeaks.bscp550` + against the 2026-05-09 entry in + `projects/mgfl/hazelmead/tuning_history.md` — should drop by + ~£500–700/mo, removing the over-attribution to gridToLoad + + solarToGrid. +4. Once 202604 validates, roll the same per-flow wiring out to other + HMCE timeframes (202601, 202602, etc.). + +NCESF doesn't need migration — its solarToGrid uses a flat PPA rate +that doesn't reference imbalance, and `gridToLoad` = 0 (no onsite +load), so there's nothing to leak Axle onto. From ea2d492efd27f7d8915c194c2c1108f74563f2e5 Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sat, 9 May 2026 23:33:10 +0100 Subject: [PATCH 2/7] feat: accept enabled:false on simulations to skip strict schema validation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scenarios marked `enabled: false` in simulate.yaml are now dropped before marshmallow validation, mirroring the rebuilder's existing skip logic. This lets externally-managed scenarios (e.g. Monte-Carlo runs whose outputs are produced outside skypro) coexist with live scenarios in the same simulate.yaml — no need for the rebuilder to write a temp staged yaml that scrubs them out. - parse_config.py: pre-load filter walks simulations dict and removes entries with `enabled is False` before Config.Schema().load - report-side parser unchanged: report.yaml has no enabled annotations in any current tenant config Existing 34 unit + integration tests pass unchanged. --- src/skypro/commands/simulator/config/parse_config.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/skypro/commands/simulator/config/parse_config.py b/src/skypro/commands/simulator/config/parse_config.py index 07d368d..e8d5511 100644 --- a/src/skypro/commands/simulator/config/parse_config.py +++ b/src/skypro/commands/simulator/config/parse_config.py @@ -23,6 +23,15 @@ def parse_config(file_path: str, env_vars: dict) -> Config: version = Version(config_dict["configFormatVersion"]) + # Drop scenarios annotated `enabled: false` before strict schema validation. + # Lets externally-managed scenarios (e.g. Monte-Carlo runs whose outputs are + # produced outside skypro) sit alongside live ones in the same simulate.yaml. + sims = config_dict.get("simulations") or {} + for name in list(sims): + cfg = sims[name] + if isinstance(cfg, dict) and cfg.get("enabled") is False: + del sims[name] + # Set up the variables that are substituted into file paths PathField.vars_for_substitution = env_vars if version.major == 4 and "variables" in config_dict: From ca77995eac83b227429107f5322b85fc1800e4e4 Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sat, 9 May 2026 23:40:10 +0100 Subject: [PATCH 3/7] feat: per-flow imbalanceDataSource override on RatesFiles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the structural Axle-premium leak on two-MPAN BSCP550 sites where the BESS and site MPANs settle against different imbalance signals. Previously every flow in a rates block read from the block-level `imbalanceDataSource`, so site-MPAN flows (gridToLoad, solarToGrid) absorbed any premium intended only for the BESS-MPAN flows (gridToBatt, battToGrid). Schema (backward-compatible — legacy list shape still parses): - New `FlowFiles` dataclass: `{rates: [paths], imbalanceDataSourceOverride: ...}` - New `FlowFilesField` marshmallow field accepts either `gridToBatt: [a.json, b.json]` # legacy or `gridToBatt: {rates: [a.json], imbalanceDataSourceOverride: {price: ..., volume: ...}}` - `RatesFiles` flows now typed `FlowFilesType` (always FlowFiles after parsing) Plumbing: - `parse_vol_rates_files_for_all_energy_flows` accepts `flow_imbalance_pricings: Optional[Dict[str, pd.Series]]` keyed by flow name; cache key now `(files_str, id(pricing))` so flows with same files but different overrides don't share rate instances. - `_get_rates_from_config` collects unique override sources, fetches + normalises each (final-style for final-block overrides, live-style for live-block), passes per-flow pricing dict down. Block-level df continues to drive the canonical `imbalance_volume_*` output columns. - Per-flow override + ratesDB combination rejected with a clear error. - Override sources contribute to the flowsMarketData engine-needed gate. Multiplier and OSAM safety: - MultiplierVolRate.set_all_rates_in_set already operates per-flow, so `Statkraft × imbalance` resolves to whatever imbalance ShapedVolRate is in its flow's set — verified, no change needed. - OSAM NCSP is dispatch-driven and computed once after the algo runs — per-flow overrides don't touch it. Report-side parser unchanged: `commands/report/rates.py` still passes a single `imbalance_pricing` and per-flow override dict defaults to `{}`. Tests: 4 new unit tests cover legacy list shape, dict shape with/without override, and sibling-flow defaulting. Existing 34 tests pass unchanged. --- src/skypro/commands/simulator/main.py | 72 +++++++++++++++- src/skypro/common/config/rates_dataclasses.py | 60 +++++++++++--- src/skypro/common/config/rates_parse_yaml.py | 82 +++++++++---------- .../unit/skypro/common/config/__init__.py | 0 .../skypro/common/config/test_flow_files.py | 80 ++++++++++++++++++ 5 files changed, 237 insertions(+), 57 deletions(-) create mode 100644 src/tests/unit/skypro/common/config/__init__.py create mode 100644 src/tests/unit/skypro/common/config/test_flow_files.py diff --git a/src/skypro/commands/simulator/main.py b/src/skypro/commands/simulator/main.py index bd66734..eca6aa2 100644 --- a/src/skypro/commands/simulator/main.py +++ b/src/skypro/commands/simulator/main.py @@ -46,6 +46,24 @@ assert ((timedelta(minutes=30) / STEP_SIZE) == STEPS_PER_SP) # Check that we have an exact number of steps per SP +def _flow_files_by_name(rates_files): + """Return a flow-name → FlowFiles mapping for the seven canonical microgrid flows.""" + return { + "solar_to_batt": rates_files.solar_to_batt, + "grid_to_batt": rates_files.grid_to_batt, + "batt_to_load": rates_files.batt_to_load, + "batt_to_grid": rates_files.batt_to_grid, + "solar_to_grid": rates_files.solar_to_grid, + "solar_to_load": rates_files.solar_to_load, + "grid_to_load": rates_files.grid_to_load, + } + + +def _iter_flow_files(rates_files): + """Iterate the seven FlowFiles instances on a RatesFiles block.""" + return _flow_files_by_name(rates_files).values() + + @dataclass class ParsedRates: """ @@ -522,6 +540,13 @@ def _get_rates_from_config( rates_config.live.imbalance_data_source.price, rates_config.live.imbalance_data_source.volume, ] + # Per-flow imbalance overrides (if any) may also reference flowsMarketData; include them in the gate. + for rates_block in (rates_config.final, rates_config.live): + if rates_block.files is not None: + for flow_files in _iter_flow_files(rates_block.files): + if flow_files.imbalance_data_source_override is not None: + imbalance_sources.append(flow_files.imbalance_data_source_override.price) + imbalance_sources.append(flow_files.imbalance_data_source_override.volume) needs_flows_db = any( s.flows_market_data_source is not None for s in imbalance_sources ) @@ -562,6 +587,47 @@ def read_imbalance_data(source: TimeseriesDataSource, context: str): live_imbalance_df = normalise_live_imbalance_data(time_index, live_price_df, live_volume_df) df = pd.concat([final_imbalance_df, live_imbalance_df], axis=1) + # Per-flow imbalance source overrides — fetch and normalise the price series for any flow that opts + # in via `imbalanceDataSourceOverride`. These are used only by the rate-construction pipeline; the + # block-level df above remains the canonical source for output `imbalance_volume_*` columns. + fetched_override_pricings: Dict[int, pd.Series] = {} + + def _override_pricing(src, kind: str) -> pd.Series: + key = id(src) + if key in fetched_override_pricings: + return fetched_override_pricings[key] + price_df = read_imbalance_data(src.price, context=f"{kind} imbalance price (override)") + volume_df = read_imbalance_data(src.volume, context=f"{kind} imbalance volume (override)") + if kind == "final": + normalised = normalise_final_imbalance_data(time_index, price_df, volume_df) + pricing = normalised["imbalance_price_final"] + else: + normalised = normalise_live_imbalance_data(time_index, price_df, volume_df) + pricing = normalised["imbalance_price_live"] + fetched_override_pricings[key] = pricing + return pricing + + def _collect_flow_pricings(rates_block, kind: str) -> Dict[str, pd.Series]: + if rates_block.files is None: + return {} + pricings: Dict[str, pd.Series] = {} + for flow_name, flow_files in _flow_files_by_name(rates_block.files).items(): + override = flow_files.imbalance_data_source_override + if override is not None: + pricings[flow_name] = _override_pricing(override, kind) + return pricings + + final_flow_pricings = _collect_flow_pricings(rates_config.final, "final") + live_flow_pricings = _collect_flow_pricings(rates_config.live, "live") + + # Reject per-flow overrides when rates come from the rates DB — that path doesn't (yet) honour them. + if rates_config.final.rates_db is not None or rates_config.live.rates_db is not None: + if final_flow_pricings or live_flow_pricings: + raise ValueError( + "imbalanceDataSourceOverride is only supported with the YAML `files` rates source, " + "not with `ratesDB`." + ) + if (rates_config.live.rates_db is None) != (rates_config.final.rates_db is None): # There is nothing inherent about this limitation: the below code could be refactored to support it. raise ValueError("Both live and final rates must use the same source: either the rates DB or YAML configuration") @@ -613,13 +679,15 @@ def read_imbalance_data(source: TimeseriesDataSource, context: str): rates_files=rates_config.final.files, supply_points=final_supply_points, imbalance_pricing=df["imbalance_price_final"], - file_path_resolver_func=file_path_resolver_func + file_path_resolver_func=file_path_resolver_func, + flow_imbalance_pricings=final_flow_pricings, ) parsed_rates.live_mkt_vol = parse_vol_rates_files_for_all_energy_flows( rates_files=rates_config.live.files, supply_points=live_supply_points, imbalance_pricing=df["imbalance_price_live"], - file_path_resolver_func=file_path_resolver_func + file_path_resolver_func=file_path_resolver_func, + flow_imbalance_pricings=live_flow_pricings, ) # There is an 'experimental' configuration block which has beta supports customer and fixed market rates. diff --git a/src/skypro/common/config/rates_dataclasses.py b/src/skypro/common/config/rates_dataclasses.py index bcca635..4de5eec 100644 --- a/src/skypro/common/config/rates_dataclasses.py +++ b/src/skypro/common/config/rates_dataclasses.py @@ -1,6 +1,7 @@ from typing import List, Optional, Dict -from marshmallow_dataclass import dataclass +from marshmallow import fields +from marshmallow_dataclass import dataclass, NewType from skypro.common.config.data_source import ImbalanceDataSource from skypro.common.config.path_field import PathType @@ -41,19 +42,58 @@ class RatesDB: customer: Optional[CustomerRatesDB] # Optionally define rates for customers - these are only really used for reporting purposes as this doesn't affect control algorithms +@dataclass +class FlowFiles: + """ + Per-flow rate files, with an optional override of the rates-block-level + `imbalanceDataSource` for this specific flow. Used to model two-MPAN + arrangements (e.g. BSCP550) where the BESS and site MPANs settle against + different imbalance signals. + """ + rates: List[PathType] + imbalance_data_source_override: Optional[ImbalanceDataSource] = field_with_opts( + key="imbalanceDataSourceOverride", default=None + ) + + +class FlowFilesField(fields.Field): + """ + Marshmallow field that accepts either: + - a list of paths (legacy shape) — equivalent to FlowFiles(rates=[...]) + - a dict with `rates: [...]` and optional `imbalanceDataSourceOverride: ...` + + Both shapes resolve to a FlowFiles instance, so downstream code only ever + sees the structured form. + """ + def _deserialize(self, value, attr, data, **kwargs): + schema = FlowFiles.Schema() + if isinstance(value, list): + return schema.load({"rates": value}) + if isinstance(value, dict): + return schema.load(value) + raise ValueError( + f"Flow rates must be a list of paths or a dict with `rates`, " + f"got {type(value).__name__}" + ) + + +FlowFilesType = NewType("FlowFiles", FlowFiles, FlowFilesField) + + @dataclass class RatesFiles: """ Configures rates to be pulled from YAML files, with a list of files for each microgrid flow. - Each rate definition file may define one or more rates. - """ - solar_to_batt: List[PathType] = field_with_opts(key="solarToBatt") - grid_to_batt: List[PathType] = field_with_opts(key="gridToBatt") - batt_to_grid: List[PathType] = field_with_opts(key="battToGrid") - batt_to_load: List[PathType] = field_with_opts(key="battToLoad") - solar_to_grid: List[PathType] = field_with_opts(key="solarToGrid") - solar_to_load: List[PathType] = field_with_opts(key="solarToLoad") - grid_to_load: List[PathType] = field_with_opts(key="gridToLoad") + Each rate definition file may define one or more rates. Each flow may also carry an optional + `imbalanceDataSourceOverride` to override the rates-block-level imbalance source for that flow. + """ + solar_to_batt: FlowFilesType = field_with_opts(key="solarToBatt") + grid_to_batt: FlowFilesType = field_with_opts(key="gridToBatt") + batt_to_grid: FlowFilesType = field_with_opts(key="battToGrid") + batt_to_load: FlowFilesType = field_with_opts(key="battToLoad") + solar_to_grid: FlowFilesType = field_with_opts(key="solarToGrid") + solar_to_load: FlowFilesType = field_with_opts(key="solarToLoad") + grid_to_load: FlowFilesType = field_with_opts(key="gridToLoad") @dataclass diff --git a/src/skypro/common/config/rates_parse_yaml.py b/src/skypro/common/config/rates_parse_yaml.py index f937f45..466c716 100644 --- a/src/skypro/common/config/rates_parse_yaml.py +++ b/src/skypro/common/config/rates_parse_yaml.py @@ -1,6 +1,6 @@ import yaml import os -from typing import Dict, List, Optional, Callable, cast +from typing import Dict, List, Optional, Callable, Tuple, cast import pandas as pd @@ -44,46 +44,45 @@ def parse_vol_rates_files_for_all_energy_flows( supply_points: Dict[str, SupplyPoint], imbalance_pricing: pd.Series, file_path_resolver_func: Callable, + flow_imbalance_pricings: Optional[Dict[str, pd.Series]] = None, ) -> VolRatesForEnergyFlows: """ Reads the rates files for each flow (JSON or YAML) and returns only the volume-based rates objects for each energy flow. Fixed charges like £/day are not returned. + + `imbalance_pricing` is the rates-block-level default. `flow_imbalance_pricings` optionally supplies a + per-flow override (keyed by flow name like "grid_to_batt"), used when a flow has an + `imbalanceDataSourceOverride` declared. """ - # This is a rudimentary caching mechanism to spot if two flows have identical files and re-use the same rate - # instances in that case. + flow_imbalance_pricings = flow_imbalance_pricings or {} + + # FlowFiles instances per flow — they carry the file list and any per-flow imbalance override metadata. flows = { - "solar_to_batt": { - "files": rates_files.solar_to_batt - }, - "grid_to_batt": { - "files": rates_files.grid_to_batt - }, - "batt_to_load": { - "files": rates_files.batt_to_load - }, - "batt_to_grid": { - "files": rates_files.batt_to_grid - }, - "solar_to_grid": { - "files": rates_files.solar_to_grid - }, - "solar_to_load": { - "files": rates_files.solar_to_load - }, - "grid_to_load": { - "files": rates_files.grid_to_load - }, + "solar_to_batt": rates_files.solar_to_batt, + "grid_to_batt": rates_files.grid_to_batt, + "batt_to_load": rates_files.batt_to_load, + "batt_to_grid": rates_files.batt_to_grid, + "solar_to_grid": rates_files.solar_to_grid, + "solar_to_load": rates_files.solar_to_load, + "grid_to_load": rates_files.grid_to_load, } - cached: Dict[str, List[VolRate]] = {} - for flow_name, flow_info in flows.items(): - files_str = str(flow_info["files"]) - if files_str not in cached: + # Cache reuses identical (files, imbalance-pricing) pairs. The pricing identity is + # part of the key so flows with the same files but different imbalance overrides + # don't share rate instances. + cached: Dict[Tuple[str, int], List[VolRate]] = {} + pull_keys: Dict[str, Tuple[str, int]] = {} + for flow_name, flow_files in flows.items(): + pricing = flow_imbalance_pricings.get(flow_name, imbalance_pricing) + cache_key = (str(flow_files.rates), id(pricing)) + pull_keys[flow_name] = cache_key + + if cache_key not in cached: rates = parse_rate_files( - files=flow_info["files"], + files=flow_files.rates, supply_points=supply_points, - imbalance_pricing=imbalance_pricing, + imbalance_pricing=pricing, file_path_resolver_func=file_path_resolver_func, ) # check that the rates are all volume-based, and not fixed rates @@ -91,23 +90,16 @@ def parse_vol_rates_files_for_all_energy_flows( if not isinstance(rate, VolRate): raise ValueError(f"Flow '{flow_name}' specifies a non-volume based rate: '{rate.name}'") - cached[files_str] = cast(List[VolRate], rates) - - def pull_from_cache(name: str) -> List[VolRate]: - """ - Convenience function to pull the rates associated with the given flow name from the cache. - This function captures the `cached` variable. - """ - return cached[str(flows[name]["files"])] + cached[cache_key] = cast(List[VolRate], rates) all_rates = VolRatesForEnergyFlows( - solar_to_batt=pull_from_cache("solar_to_batt"), - grid_to_batt=pull_from_cache("grid_to_batt"), - batt_to_load=pull_from_cache("batt_to_load"), - solar_to_grid=pull_from_cache("solar_to_grid"), - solar_to_load=pull_from_cache("solar_to_load"), - grid_to_load=pull_from_cache("grid_to_load"), - batt_to_grid=pull_from_cache("batt_to_grid"), + solar_to_batt=cached[pull_keys["solar_to_batt"]], + grid_to_batt=cached[pull_keys["grid_to_batt"]], + batt_to_load=cached[pull_keys["batt_to_load"]], + solar_to_grid=cached[pull_keys["solar_to_grid"]], + solar_to_load=cached[pull_keys["solar_to_load"]], + grid_to_load=cached[pull_keys["grid_to_load"]], + batt_to_grid=cached[pull_keys["batt_to_grid"]], ) # This runs through all the rates in each set and if there is a multiplier rate present then it will be diff --git a/src/tests/unit/skypro/common/config/__init__.py b/src/tests/unit/skypro/common/config/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/tests/unit/skypro/common/config/test_flow_files.py b/src/tests/unit/skypro/common/config/test_flow_files.py new file mode 100644 index 0000000..dce539f --- /dev/null +++ b/src/tests/unit/skypro/common/config/test_flow_files.py @@ -0,0 +1,80 @@ +""" +Tests for the FlowFiles polymorphic schema added in skypro v2.x. + +Each flow on a `RatesFiles` block can be expressed as either: + - a legacy list of paths: `gridToBatt: [a.json, b.json]` + - a dict carrying an override: `gridToBatt: { rates: [...], imbalanceDataSourceOverride: ... }` + +Both shapes deserialise to a `FlowFiles` instance. +""" +import unittest + +from skypro.common.config.path_field import PathField +from skypro.common.config.rates_dataclasses import RatesFiles + + +def _make_rates_files_dict(grid_to_batt): + """Helper: minimal RatesFiles input with all flows present.""" + return { + "solarToBatt": [], + "gridToBatt": grid_to_batt, + "battToGrid": [], + "battToLoad": [], + "solarToGrid": [], + "solarToLoad": [], + "gridToLoad": [], + } + + +class TestFlowFilesShapes(unittest.TestCase): + + def setUp(self): + # PathField uses class-level state for env var substitution; ensure deterministic. + PathField.vars_for_substitution = {} + + def test_legacy_list_shape(self): + rates_files = RatesFiles.Schema().load(_make_rates_files_dict( + grid_to_batt=["/tmp/a.json", "/tmp/b.json"] + )) + self.assertEqual(rates_files.grid_to_batt.rates, ["/tmp/a.json", "/tmp/b.json"]) + self.assertIsNone(rates_files.grid_to_batt.imbalance_data_source_override) + + def test_dict_shape_without_override(self): + rates_files = RatesFiles.Schema().load(_make_rates_files_dict( + grid_to_batt={"rates": ["/tmp/a.json"]} + )) + self.assertEqual(rates_files.grid_to_batt.rates, ["/tmp/a.json"]) + self.assertIsNone(rates_files.grid_to_batt.imbalance_data_source_override) + + def test_dict_shape_with_override(self): + rates_files = RatesFiles.Schema().load(_make_rates_files_dict( + grid_to_batt={ + "rates": ["/tmp/a.json"], + "imbalanceDataSourceOverride": { + "price": {"csvTimeseries": {"dir": "/tmp/price"}}, + "volume": {"csvTimeseries": {"dir": "/tmp/volume"}}, + }, + } + )) + override = rates_files.grid_to_batt.imbalance_data_source_override + self.assertIsNotNone(override) + self.assertEqual(override.price.csv_timeseries_data_source.dir, "/tmp/price") + self.assertEqual(override.volume.csv_timeseries_data_source.dir, "/tmp/volume") + + def test_other_flows_default_to_no_override(self): + rates_files = RatesFiles.Schema().load(_make_rates_files_dict( + grid_to_batt={ + "rates": ["/tmp/a.json"], + "imbalanceDataSourceOverride": { + "price": {"csvTimeseries": {"dir": "/tmp/price"}}, + "volume": {"csvTimeseries": {"dir": "/tmp/volume"}}, + }, + } + )) + # Sibling flows declared as legacy lists must still parse and have no override. + self.assertIsNone(rates_files.grid_to_load.imbalance_data_source_override) + self.assertEqual(rates_files.grid_to_load.rates, []) + + +if __name__ == "__main__": + unittest.main() From 2188e35d6e9015161081b07533d6c452cdcf16f6 Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sat, 9 May 2026 23:43:58 +0100 Subject: [PATCH 4/7] =?UTF-8?q?feat:=20multi-final=20fan-out=20=E2=80=94?= =?UTF-8?q?=20declare=20N=20settlement=20variants=20in=20one=20simulation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lets a single simulation declare multiple `finals: {: Rates, ...}` and get N output CSVs back, each settling the same dispatch under a different final-rates structure. Removes the cut-and-paste duplication for fullfcl-vs-trio-imbflex-style scenario pairs. Schema (mutually exclusive with the legacy `final: Rates` field, mirrors the `peak`/`peaks` precedent in PriceCurveAlgo): rates: live: *ratesLive_basecase finals: fullfcl: *ratesFinal_fullfcl trio_imbflex: *ratesFinal_trio_imbflex Implementation: parse-time fan-out in `parse_config`. Each multi-final sim is replaced with N entries `.`, each carrying a deep-copied SimulationCase with `rates.final` resolved to that variant. The simulator main loop never sees a multi-final scenario — zero changes to `_run_one_simulation`, `_get_rates_from_config`, `_process_final_rates`, or `generate_output_df`. The optimiser runs N times; accepted trade-off for YAML ergonomics over compute saving (deferred — see proposals/multi-final-rates.md for the column-multiplex alternative). CSV path de-clashing: - Paths containing `$_SIM_NAME` are left alone — the substitution loop runs after fan-out and naturally produces unique paths via `.`. - Hardcoded paths get `.` inserted before the extension to avoid two variants overwriting the same file. Composes orthogonally with commit 2 — each variant is a complete `Rates` block and can carry its own per-flow `imbalanceDataSourceOverride`. Tests: 10 new unit tests cover variant suffix logic, fan-out expansion, deep- copy isolation, declaration-order preservation, and the legacy single-final no-op path. All 48 tests (34 existing + 4 FlowFiles + 10 fan-out) pass. --- .../commands/simulator/config/config.py | 11 +- .../commands/simulator/config/parse_config.py | 53 +++++++++ .../commands/simulator/test_parse_config.py | 110 ++++++++++++++++++ 3 files changed, 173 insertions(+), 1 deletion(-) create mode 100644 src/tests/unit/skypro/commands/simulator/test_parse_config.py diff --git a/src/skypro/commands/simulator/config/config.py b/src/skypro/commands/simulator/config/config.py index 213725f..66f4352 100644 --- a/src/skypro/commands/simulator/config/config.py +++ b/src/skypro/commands/simulator/config/config.py @@ -381,11 +381,20 @@ class AllRates: - live: these are the rates which the algorithm thinks are going to happen at the time it is making decisions - final: these are the rates which are used after the algorithm has run, and determine the actual costs/revenues of the simulation run. + Either declare a single `final` (the legacy form) or multiple named variants via `finals` — + when `finals` is set, the simulation is fanned out into one run per variant at parse time, each + run sharing dispatch/`live` rates but settling against its own final rates. Output CSV paths get + a per-variant suffix to avoid collisions. + It is sometimes useful to have two sets of rates, for example, we might want to use Modo imbalance price predictions as 'live' and the actual Elexon prices as 'final'. """ live: Rates - final: Rates + final: Optional[Rates] = field_with_opts(default=None) + finals: Optional[Dict[str, Rates]] = field_with_opts(default=None) + + def __post_init__(self): + enforce_one_option([self.final, self.finals], "'final' (single) or 'finals' (named variants)") @dataclass diff --git a/src/skypro/commands/simulator/config/parse_config.py b/src/skypro/commands/simulator/config/parse_config.py index e8d5511..800d38d 100644 --- a/src/skypro/commands/simulator/config/parse_config.py +++ b/src/skypro/commands/simulator/config/parse_config.py @@ -1,3 +1,5 @@ +import copy +import os from packaging.version import Version import yaml @@ -44,6 +46,13 @@ def parse_config(file_path: str, env_vars: dict) -> Config: config = Config.Schema().load(config_dict) + # Fan out any sim with a `finals: {name: Rates, ...}` block into one expanded sim per variant, + # sharing dispatch and live rates but settling against its own final rates. The expanded + # sim_name is `.`. Each variant's CSV paths are de-clashed: paths containing + # `$_SIM_NAME` get the variant suffix via the substitution loop below; hardcoded paths get + # `.` inserted before the extension. + config.simulations = _expand_multi_final_simulations(config.simulations) + if version.major == 4: # There is also a special variable `$CASE_NAME` which should resolve to the name of the case, which can't # be handled with the above mechanism... manually go through a substitute that here... this isn't a @@ -60,3 +69,47 @@ def parse_config(file_path: str, env_vars: dict) -> Config: sim_config.output.summary.csv = substitute_vars(sim_config.output.summary.csv, case_name_dict) return config + + +def _expand_multi_final_simulations(simulations): + """Replace each multi-`finals` simulation with N single-`final` simulations. + + Sim name becomes `.`; CSV paths get a `.` suffix when the original + path doesn't already use `$_SIM_NAME` (which the later substitution loop will rewrite). Order + is preserved: variants land in the position of their original sim, in declaration order. + """ + expanded = {} + for sim_name, sim_config in simulations.items(): + finals = sim_config.rates.finals + if finals is None: + expanded[sim_name] = sim_config + continue + for variant_name, variant_final in finals.items(): + expanded_name = f"{sim_name}.{variant_name}" + expanded_sim = copy.deepcopy(sim_config) + expanded_sim.rates.final = variant_final + expanded_sim.rates.finals = None + if expanded_sim.output is not None: + if expanded_sim.output.simulation is not None: + expanded_sim.output.simulation.csv = _add_variant_suffix( + expanded_sim.output.simulation.csv, variant_name + ) + if expanded_sim.output.summary is not None: + expanded_sim.output.summary.csv = _add_variant_suffix( + expanded_sim.output.summary.csv, variant_name + ) + expanded[expanded_name] = expanded_sim + return expanded + + +def _add_variant_suffix(csv_path: str, variant_name: str) -> str: + """Insert `.` before the extension when the path doesn't already use `$_SIM_NAME`. + + Paths using `$_SIM_NAME` get the variant suffix naturally because the sim_name is now + `.` and the substitution loop runs after fan-out — leave those untouched. + Hardcoded paths must be de-clashed manually so two variants don't overwrite the same file. + """ + if "$_SIM_NAME" in csv_path: + return csv_path + base, ext = os.path.splitext(csv_path) + return f"{base}.{variant_name}{ext}" diff --git a/src/tests/unit/skypro/commands/simulator/test_parse_config.py b/src/tests/unit/skypro/commands/simulator/test_parse_config.py new file mode 100644 index 0000000..0c3d10f --- /dev/null +++ b/src/tests/unit/skypro/commands/simulator/test_parse_config.py @@ -0,0 +1,110 @@ +""" +Unit tests for the YAML parser helpers added in the multi-and-per-flow-rates work: + + - `_add_variant_suffix`: collision-free CSV path naming for fanned-out variants + - `_expand_multi_final_simulations`: fan-out from `rates.finals` dict into + one SimulationCase per variant + +End-to-end behaviour (`enabled:false` drop, schema validation) is exercised +via the existing integration test in `test_integration_simulator.py`. +""" +import copy +import unittest +from types import SimpleNamespace + +from skypro.commands.simulator.config.parse_config import ( + _add_variant_suffix, + _expand_multi_final_simulations, +) + + +def _make_sim(*, finals=None, final="legacy_final", csv="$_SIM_NAME.csv"): + """Build a SimulationCase-shaped object with just enough structure for the + fan-out helper. We use SimpleNamespace because `_expand_multi_final_simulations` + only accesses `.rates.{final,finals}` and `.output.summary.csv`. + """ + rates = SimpleNamespace(final=final, finals=finals) + summary = SimpleNamespace(csv=csv) + output = SimpleNamespace(summary=summary, simulation=None) + return SimpleNamespace(rates=rates, output=output) + + +class TestAddVariantSuffix(unittest.TestCase): + + def test_inserts_before_extension(self): + self.assertEqual(_add_variant_suffix("/tmp/out.csv", "v1"), "/tmp/out.v1.csv") + + def test_skipped_when_sim_name_token_present(self): + # $_SIM_NAME paths get the variant suffix naturally via the substitution + # loop after fan-out (sim name is now `.`); leaving these + # untouched here avoids double-suffixing. + self.assertEqual(_add_variant_suffix("$_SIM_NAME.csv", "v1"), "$_SIM_NAME.csv") + self.assertEqual( + _add_variant_suffix("outputs/$_SIM_NAME.summary.csv", "v1"), + "outputs/$_SIM_NAME.summary.csv", + ) + + def test_works_without_extension(self): + self.assertEqual(_add_variant_suffix("/tmp/out", "v1"), "/tmp/out.v1") + + +class TestExpandMultiFinalSimulations(unittest.TestCase): + + def test_legacy_single_final_unchanged(self): + sim = _make_sim(final="legacy_final", finals=None) + result = _expand_multi_final_simulations({"legacy": sim}) + self.assertEqual(list(result.keys()), ["legacy"]) + self.assertIs(result["legacy"], sim) + + def test_finals_block_expands_to_n_sims(self): + sim = _make_sim(final=None, finals={"plain": "rates_a", "alt": "rates_b"}) + result = _expand_multi_final_simulations({"hmce": sim}) + self.assertEqual(set(result.keys()), {"hmce.plain", "hmce.alt"}) + self.assertNotIn("hmce", result) + + def test_each_variant_has_resolved_final_and_no_finals(self): + sim = _make_sim(final=None, finals={"a": "rates_a", "b": "rates_b"}) + result = _expand_multi_final_simulations({"hmce": sim}) + self.assertEqual(result["hmce.a"].rates.final, "rates_a") + self.assertIsNone(result["hmce.a"].rates.finals) + self.assertEqual(result["hmce.b"].rates.final, "rates_b") + self.assertIsNone(result["hmce.b"].rates.finals) + + def test_sim_name_token_paths_pass_through(self): + # The substitution loop downstream substitutes $_SIM_NAME with `.` + # so we don't add the suffix here. + sim = _make_sim(final=None, finals={"a": "_", "b": "_"}, csv="$_SIM_NAME.csv") + result = _expand_multi_final_simulations({"sim": sim}) + self.assertEqual(result["sim.a"].output.summary.csv, "$_SIM_NAME.csv") + self.assertEqual(result["sim.b"].output.summary.csv, "$_SIM_NAME.csv") + + def test_hardcoded_csv_path_gets_variant_suffix(self): + sim = _make_sim(final=None, finals={"a": "_", "b": "_"}, csv="/tmp/out.csv") + result = _expand_multi_final_simulations({"clash": sim}) + self.assertEqual(result["clash.a"].output.summary.csv, "/tmp/out.a.csv") + self.assertEqual(result["clash.b"].output.summary.csv, "/tmp/out.b.csv") + + def test_variants_are_deep_copied(self): + # Mutating one variant's output must not bleed into a sibling's. + sim = _make_sim(final=None, finals={"a": "_", "b": "_"}) + result = _expand_multi_final_simulations({"sim": sim}) + result["sim.a"].output.summary.csv = "MUTATED" + self.assertNotEqual(result["sim.b"].output.summary.csv, "MUTATED") + + def test_order_preserved(self): + # Mixed legacy + multi-final preserves declaration order; each variant lands + # in the position of its parent sim, in finals-declaration order. + first = _make_sim(final="x", finals=None) + second = _make_sim(final=None, finals={"alpha": "_", "beta": "_"}) + third = _make_sim(final="y", finals=None) + result = _expand_multi_final_simulations({ + "first": first, "second": second, "third": third, + }) + self.assertEqual( + list(result.keys()), + ["first", "second.alpha", "second.beta", "third"], + ) + + +if __name__ == "__main__": + unittest.main() From c54c7cc1939005b0f70ef384f024803cc4f4ecd2 Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sat, 9 May 2026 23:53:21 +0100 Subject: [PATCH 5/7] chore: bump version to 2.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pyproject.toml: 2.1.1 → 2.2.0 (minor bump for the three additive schema features on feature/multi-and-per-flow-rates) - CLAUDE.md merge log: summary entry for the v2.2.0 changes Three commits in this minor release: - feat: accept enabled:false on simulations (ea2d492) - feat: per-flow imbalanceDataSource override on RatesFiles (ca77995) - feat: multi-final fan-out — declare N settlement variants in one simulation (2188e35) All 48 unit + integration tests pass. Backward-compatible — legacy list-shape flows and single-`final` blocks unchanged. --- CLAUDE.md | 1 + pyproject.toml | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index df2b6b9..3e0d1dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -185,6 +185,7 @@ pandas, plotly, pulp, pendulum, sqlalchemy, psycopg2-binary, marshmallow, pyyaml | Date | PR | Branch | Summary | |------|-----|--------|---------| +| 2026-05-09 | TBD | feature/multi-and-per-flow-rates | Three schema additions for multi-MPAN BSCP550 + multi-settlement workflows: (1) `enabled: false` annotation on simulations now accepted by core (drops the rebuilder's temp-yaml workaround); (2) per-flow `imbalanceDataSourceOverride` inside `RatesFiles` flows so two-MPAN sites can settle BESS-MPAN and site-MPAN against different imbalance signals (closes the HMCE Apr-26 BSCP550 ~£500–700/mo Axle leak); (3) `rates.finals: {: Rates, ...}` declares N settlement variants per simulation, fanned out at parse time into one sim per variant with auto-suffixed CSV paths. Backward-compatible — legacy list/single-final shapes unchanged. +14 unit tests (v2.2.0) | | 2026-05-08 | TBD | feature/support-axle-flex | New `Peak.dynamic.minEndOfPeakSoe` parameter reserves SoE for post-peak niv-chase. Used in time-to-empty calc as `dischargeable_soe = soe − min_end_of_peak_soe`, creating slack so the dynamic HOLD-on-LONG branch can actually fire instead of always falling through to forced full discharge. Default 0 (legacy behaviour). +5 unit tests (v2.1.1) | | 2026-05-08 | TBD | feature/support-axle-flex | Multi-peak support in priceCurveAlgo: new `peaks: [...]` list form alongside legacy `peak: ...` (mutually exclusive). Enables dispatch into multiple price-elevated windows per day. Backward-compat verified — existing single-peak fixture summary unchanged within tolerance (v2.1.0) | | 2026-04-30 | TBD | feature/profile-filter-and-nameplate | Opt-in profile anomaly filter (`maxEnergyPerIntervalKwh`) + display-only `nameplateKwp` metadata field on Profile (v2.0.4) | diff --git a/pyproject.toml b/pyproject.toml index e96c793..07e5d98 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "skypro" -version = "2.1.1" +version = "2.2.0" description = "Skyprospector by Cepro" authors = ["damonrand "] license = "MIT" From 3502780e2cc74d3e72fc4acae6f4ce1b92fa7343 Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sun, 10 May 2026 00:31:33 +0100 Subject: [PATCH 6/7] =?UTF-8?q?docs:=20v2.2.0=20release=20notes=20?= =?UTF-8?q?=E2=80=94=20schema=20reference=20+=20proposal=20status?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - CHANGELOG.md: Added v2.2.0 entry with the three schema additions (enabled:false, per-flow imbalanceDataSourceOverride, multi-finals), YAML examples, and compatibility notes. - CLAUDE.md: Added a "simulate.yaml schema reference (post-v2.2.0)" cheatsheet under Key Concepts, summarising all three features with example YAML and the symmetry rules (apply override to live AND final; finals is mutually exclusive with final). - docs/proposals/per-flow-imbalance-source-override.md: Marked as ✅ IMPLEMENTED in v2.2.0, commit ca77995. - docs/proposals/multi-final-rates.md: Marked as ✅ IMPLEMENTED in v2.2.0, commit 2188e35 — with note that the shipped fan-out approach differs from the brief's column-multiplex design (deferred — user prioritised YAML ergonomics over compute saving). --- CHANGELOG.md | 83 +++++++++++++++++++ CLAUDE.md | 45 ++++++++++ docs/proposals/multi-final-rates.md | 12 ++- .../per-flow-imbalance-source-override.md | 6 +- 4 files changed, 144 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 233d4e6..4437123 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,89 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [2.2.0] - 2026-05-10 + +### Added + +- **`enabled: false` annotation on simulations**: Scenarios marked + `enabled: false` in `simulate.yaml` are now dropped before strict + schema validation, letting externally-managed scenarios (e.g. + Monte-Carlo runs whose outputs are produced outside skypro) coexist + with live scenarios in the same yaml. Removes the need for callers + (e.g. the rebuilder in skypro-service) to write a temp staged yaml + to scrub them out. + +- **Per-flow `imbalanceDataSourceOverride` on `RatesFiles`**: Each flow + in a rates block can now override the block-level `imbalanceDataSource` + via a new dict shape: + + ```yaml + rates: + final: + imbalanceDataSource: *imbSrc_default # block-level default + files: + gridToBatt: # legacy list shape — inherits default + - dno_import.json + - supply_import.json + solarToGrid: # new dict shape with override + rates: + - dno_export.json + - supply_statkraft_export.json + imbalanceDataSourceOverride: *imbSrc_plain + ``` + + Closes the structural Axle-premium leak on two-MPAN BSCP550 sites + where the BESS and site MPANs settle against different imbalance + signals. Backward-compatible — legacy list shape on flows continues + to parse. + +- **Multi-`finals` per simulation**: Declare N settlement variants in + one simulation. Mutually exclusive with the legacy `final: ` + field (same precedent as `peak`/`peaks` in priceCurveAlgo): + + ```yaml + rates: + live: *ratesLive_basecase + finals: + fullfcl: *ratesFinal_fullfcl + trio_imbflex: *ratesFinal_trio_imbflex + ``` + + Fanned out at parse time into one expanded `SimulationCase` per + variant (sim name `.`). CSV paths get an automatic + variant suffix when they don't use `$_SIM_NAME`. The optimiser runs + N times — accepted trade-off for YAML ergonomics; the + single-dispatch / multi-settlement-column variant is deferred (see + `docs/proposals/multi-final-rates.md`). + +### Changed + +- `RatesFiles` flow fields are now `FlowFilesType` (a `FlowFiles` + dataclass) rather than `List[PathType]`. Schema parsing accepts + either list or dict shape transparently. **No migration needed for + existing YAML configs.** Internal callers that previously accessed + e.g. `rates_files.grid_to_batt[0]` need to use + `rates_files.grid_to_batt.rates[0]`. Only consumer affected was + `parse_vol_rates_files_for_all_energy_flows`, which has been + updated. + +- `parse_vol_rates_files_for_all_energy_flows` accepts an optional + `flow_imbalance_pricings: Dict[str, pd.Series]` keyword argument + for per-flow override pricings. Cache key now + `(file_list_str, id(pricing))` so flows with the same files but + different overrides don't share rate instances. + +### Compatibility + +- Legacy single-`final` configs unchanged. Verified by integration + tests (`integrationTestPriceCurve`, `integrationTestPriceCurveMultiPeak`, + `integrationTestPerfectHindsightLP`) — bit-identical LP output + within tolerance `0.01`. +- Legacy list-shape rate files continue to parse and behave + identically. +- `ratesDB` source rejects per-flow overrides with a clear error + (override only supported with the YAML `files` source). + ## [2.0.5] - 2026-05-07 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 3e0d1dc..871b7fc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -123,6 +123,51 @@ TWINE_USERNAME=__token__ TWINE_PASSWORD=$(cat ~/.simt/testpypi.token) python -m - **Market** - actual cashflows with suppliers - **Internal** - opportunity cost for optimization +### `simulate.yaml` schema reference (post-v2.2.0) + +Three small ergonomic features added in v2.2.0. Authoritative +reference: `CHANGELOG.md` v2.2.0 entry; design history: +`docs/proposals/`. + +**`enabled: false`** on a simulation drops it pre-schema — useful for +externally-managed scenarios that may use unsupported strategy keys: +```yaml +simulations: + hmce.202512.imb.mc-reopt: + enabled: false # outputs preserved as-is, schema doesn't see this scenario + strategy: { monteCarloAlgo: ... } +``` + +**Per-flow `imbalanceDataSourceOverride`** for two-MPAN BSCP550 sites +(or anywhere flows in a single rates block need different imbalance +signals). Flow can be either a list (legacy) or a dict carrying an +override: +```yaml +files: + gridToBatt: [ a.json, b.json ] # legacy — inherits block-level + solarToGrid: # dict shape with override + rates: [ a.json, b.json ] + imbalanceDataSourceOverride: *imbSrc_plain +``` +Apply override **symmetrically** to live AND final blocks — MPAN +imbalance treatment is structural, not algo-only. + +**Multi-`finals`** declares N settlement variants per simulation, +fanned out at parse time. Mutually exclusive with `final:` (same +precedent as `peak`/`peaks`). Sim names become `.`; +CSV paths get a variant suffix when not using `$_SIM_NAME`: +```yaml +rates: + live: *ratesLive_basecase + finals: + fullfcl: *ratesFinal_fullfcl + trio_imbflex: *ratesFinal_trio_imbflex +``` +Each variant runs the optimiser independently — accepted trade-off +for YAML ergonomics. The single-dispatch / multi-column variant +(loop `_process_final_rates`, OSAM-mutation-safe via deepcopy) is +deferred — see `docs/proposals/multi-final-rates.md`. + ### OSAM (P395) On-site Allocation Methodology for calculating final demand levies. Runs in parallel with Skypro's own methodology; discrepancies reported as Notices. diff --git a/docs/proposals/multi-final-rates.md b/docs/proposals/multi-final-rates.md index 47cf3e9..d772772 100644 --- a/docs/proposals/multi-final-rates.md +++ b/docs/proposals/multi-final-rates.md @@ -2,7 +2,17 @@ **Audience:** Skypro engineer (Python core) **Author:** Damon Rand -**Status:** Brief — design + implementation TBD +**Status:** ✅ IMPLEMENTED in v2.2.0 (commit `2188e35` on `feature/multi-and-per-flow-rates`) +**— with a different design.** The shipped implementation is a +**parse-time fan-out**: `rates.finals: {: Rates, ...}` expands +into one independent `SimulationCase` per variant, each with its own +output CSV. The optimiser runs N times. The single-dispatch / +multi-settlement-column variant described below (loop +`_process_final_rates`, suffix output columns with +`.final[]`) is **deferred** — user priority was YAML +ergonomics over compute saving. See `CHANGELOG.md` v2.2.0 for the +final YAML reference. This brief is preserved as the design record +for if/when the optimised variant lands. ## Background — why this matters diff --git a/docs/proposals/per-flow-imbalance-source-override.md b/docs/proposals/per-flow-imbalance-source-override.md index 520d4a2..b45f552 100644 --- a/docs/proposals/per-flow-imbalance-source-override.md +++ b/docs/proposals/per-flow-imbalance-source-override.md @@ -2,7 +2,11 @@ **Audience:** Skypro engineer (Python core) **Author:** Damon Rand -**Status:** Brief — design + implementation TBD +**Status:** ✅ IMPLEMENTED in v2.2.0 (commit `ca77995` on `feature/multi-and-per-flow-rates`). +The implementation matches this brief — schema lives inside `RatesFiles` +as a new `FlowFiles` dataclass with optional `imbalanceDataSourceOverride`. +Legacy list shape continues to parse. See `CHANGELOG.md` v2.2.0 for the +final YAML reference. ## Background — why this matters From 89c65c0d883a7ba8cbf02f07f5c9245223641712 Mon Sep 17 00:00:00 2001 From: Damon Rand Date: Sun, 10 May 2026 21:46:28 +0100 Subject: [PATCH 7/7] fix: drop unused import in test_parse_config Leftover from an earlier draft that used copy.deepcopy. The current test file uses SimpleNamespace mocks throughout. Caught by ruff in CI. --- src/tests/unit/skypro/commands/simulator/test_parse_config.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/tests/unit/skypro/commands/simulator/test_parse_config.py b/src/tests/unit/skypro/commands/simulator/test_parse_config.py index 0c3d10f..5080de1 100644 --- a/src/tests/unit/skypro/commands/simulator/test_parse_config.py +++ b/src/tests/unit/skypro/commands/simulator/test_parse_config.py @@ -8,7 +8,6 @@ End-to-end behaviour (`enabled:false` drop, schema validation) is exercised via the existing integration test in `test_integration_simulator.py`. """ -import copy import unittest from types import SimpleNamespace