Enhance Caller.py and CaseStudy with logging, error handling, and functions

.claude/rules/claude-md-maintenance.md

@@ -0,0 +1,23 @@
+---
+description: Rules for keeping CLAUDE.md and README.md files up to date after code changes in InOutModule/
+---
+
+After making any code changes in `InOutModule/`, update the relevant documentation files.
+
+**`CLAUDE.md`** — context for Claude that is hard to derive from reading the code: non-obvious patterns, gotchas, architectural decisions, naming conventions, and implementation constraints.
+
+**`README.md`** — human-facing documentation: usage instructions, API overview, key concepts, and examples.
+
+**Rules:**
+- Update the file(s) closest to the code that changed.
+- If a change also affects `LEGO/` or root-level code, update the corresponding docs there too (see root `.claude/rules/claude-md-maintenance.md` for the full index).
+- Do not duplicate content between `README.md` and `CLAUDE.md` — if something is in `README.md`, `CLAUDE.md` should reference it, not repeat it.
+- Use prose references (`See README.md for ...`) to point to large human-facing docs, not `@`-imports.
+- Delete stale entries rather than commenting them out.
+
+**Index of documentation files in InOutModule/:**
+
+| File                    | Audience | Contents                                                                |
+|-------------------------|----------|-------------------------------------------------------------------------|
+| `InOutModule/CLAUDE.md` | Claude   | CaseStudy read order, ExcelReader/Writer patterns, SQLiteWriter, Caller |
+| `InOutModule/README.md` | Humans   | CaseStudy API, SQLiteWriter usage, Caller, Excel file format            |

CLAUDE.md

@@ -0,0 +1,53 @@
+# CLAUDE.md — InOutModule
+
+This file provides guidance to Claude Code when working with code in this directory.
+See `README.md` for usage, key concepts, and data structure.
+
+## Architecture Notes
+
+### CaseStudy
+
+- Sequential reads happen first (`Global_Parameters`, `Global_Scenarios`, `Power_Parameters`), because subsequent file selection depends on `pEnable*` flags.
+- All remaining files are read in parallel via `ThreadPoolExecutor` — order of assignment is non-deterministic, so no file read may depend on another parallel read.
+- `dPower_WeightsRP` is **computed** from `dPower_Hindex` (counting occurrences per `rp`); if a `Power_WeightsRP.xlsx` file also exists, it is read and compared — a mismatch triggers a warning but uses the **file** value, not the computed one.
+- `merge_single_node_buses()` preserves the `z` (zone) column as a sorted unique union string of all merged zones (e.g. `"R1_R2"`). This is documented in root `CLAUDE.md` as well.
+- `merge_generators()` collapses all generators sharing the same `(tec, i)` into one representative generator with ID `"{i}_{tec}"`. It must be called after construction (scaling is not required). VRESProfiles and Inflows are merged **before** VRES so that the original per-generator `MaxProd` weights are available for the capacity-factor weighted average. Generators in VRESProfiles/Inflows that have no matching entry in `dPower_VRES` (left-join miss) are grouped under `(tec=NaN, i=NaN)` — filter to a single scenario first to avoid mixing scenarios in the groupby.
+- `filter_zone(zone)` keeps only buses whose `z` column exactly matches the given zone name(s). Cascade order: BusInfo → Network (both endpoints must survive) → ThermalGen / VRES / Storage (by `i`) → Demand (by `i` index level) → VRESProfiles (by surviving VRES gen IDs) → Inflows (by union of surviving VRES + Storage gen IDs) → ImportExport (by `i` index level). `z` is an exact-match filter; merged-bus zone strings like `"R1_R2"` are not matched by `"R1"` alone.
+- `shift_transition_matrix(positions, inplace=True, seed=42)` cyclically shifts every row of `rpTransitionMatrixAbsolute` right by `positions` (via `np.roll`, negative shifts left), normalizes, then rebuilds Hindex by Markov chain sampling from the shifted row distributions — anchoring the first period of each scenario to its original RP assignment. Recomputes `dPower_WeightsRP` and all three TM attributes from the new Hindex. The RNG seed defaults to 42.
+- `perturb_transition_matrix(randomness, inplace=True, seed=42)` interpolates each row between its original distribution and a random draw: `new_prob = (1-randomness)*orig_prob + randomness*random_draw`, where `random_draw` is `n` uniform values normalized to sum to 1. `randomness=0.0` leaves the matrix unchanged; `randomness=1.0` replaces it fully. Rebuilds Hindex identically to `shift_transition_matrix` (Markov chain sampling, first period anchored, missing-RP correction loop). Uses a single RNG instance for both the random draws and the chain sampling.
+- `CaseStudy.copy()` is a full `deepcopy` — safe to modify independently.
+- `to_full_hourly_model()` processes each enabled scenario from `dGlobal_Scenarios` independently. It is a no-op (returns unchanged) if all scenarios are already hourly (no two p-values share the same 'k' within any scenario). The produced Hindex uses consecutive h0001…hN p-labels and k0001…kN k-labels per scenario, all mapped to rp01 with weight 1.
+- Transition matrices (`rpTransitionMatrixAbsolute`, `rpTransitionMatrixRelativeTo`, `rpTransitionMatrixRelativeFrom`) are computed in the constructor and attached as attributes.
+
+### ExcelReader
+
+- Excel sheets whose name starts with `~` are silently skipped (used to disable scenarios in a multi-sheet file without deleting them).
+- All Excel files have a version specifier in cell `C2` of each sheet. `check_LEGOExcel_version()` warns (or raises, if `fail_on_wrong_version=True`) on mismatch — wrong version can cause silent column misreads.
+- The reader uses `calamine` engine (fast), not `openpyxl`.
+
+### ExcelWriter
+
+- All cell styles, column definitions, and table layouts are declared in `TableDefinitions.xml`, not in Python code. When adding a new output table, define its columns there first.
+- `ExcelWriter.__init__()` parses the XML once and stores resolved objects (`self.columns`, `self.cell_styles`, etc.). Avoid re-instantiating per row.
+
+### SQLiteWriter
+
+- `model_to_sqlite()` automatically calls `add_objective_decomposition_to_sqlite()` and `add_dual_values_to_sqlite()` — these do not need to be called separately.
+- `add_run_parameters_to_sqlite()` stores all run configuration in the `run_parameters` table. Evaluation scripts should read from this table (more reliable than filename parsing).
+- Pyomo component types not handled by the writer emit a `printer.warning()` and are skipped silently — add new `case` branches if new Pyomo types need to be stored.
+
+### Utilities
+
+- `inflowsToCapacityFactors()` joins inflows onto `vresProfiles_df` by dividing by `MaxProd`; generators with missing or zero `MaxProd` are dropped with a warning.
+- `capacityFactorsToInflows()` is the inverse; the `remove_Inflows_from_VRESProfiles_inplace` flag modifies the input DataFrame in place when set.
+- `plot_transition_matrix()` emits **two** figures per call — a row-normalised ("from" perspective) and a column-normalised ("to" perspective) version — via the inner `_render()` helper. When `output` is given, `os.path.splitext` inserts a `-rowNorm` / `-colNorm` suffix before the extension (e.g. `MK-foo.png` → `MK-foo-rowNorm.png`, `MK-foo-colNorm.png`). Colour intensity and the per-cell percentage encode the respective normalisation; absolute counts and the row/col/grand totals are identical between the two. Cell text flips from black to white where the Blue fill is too dark to read on (perceived luminance `0.299·R + 0.587·G + 0.114·B < 0.5`). It passes `bbox=[0, 0, 1, 1]` to `ax.table()` so the table is forced to fill the entire Axes — without it, matplotlib sizes table rows from font metrics rather than the Axes height, leaving large dead space above/below when the figure is taller than the table's natural size. `fig_w`/`fig_h` are tuned per-cell-inch (not just a generic min-size heuristic) since they now directly determine the rendered cell aspect ratio.
+
+### Printer
+
+- `Printer` is a singleton — obtain the instance with `Printer.getInstance()`, never call the constructor directly.
+- `set_logfile(path)` redirects all subsequent output to a file (appending). Set to `None` to stop logging.
+
+### Caller
+
+- `Caller.py` is a parallel job runner reading from a text file. It uses sentinel files (`.finished{n}`, `.error{n}`) for barrier synchronization across parallel workers.
+- Lines containing only `---` act as barriers — workers wait until all prior jobs are complete before continuing past the barrier.

Caller.py

@@ -0,0 +1,187 @@
+import argparse
+import ctypes
+import datetime
+import os
+import subprocess
+import sys
+import time
+
+if __name__ == "__main__":
+    sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from InOutModule.printer import Printer
+
+printer = Printer.getInstance()
+
+
+def _tail_file(filepath, n=20):
+    """Return the last n lines of a file, or all if fewer."""
+    try:
+        with open(filepath, 'r', encoding='utf-8', errors='replace') as f:
+            lines = f.readlines()
+    except OSError:
+        return "(could not read log file)"
+    if not lines:
+        return "(no output)"
+    if len(lines) <= n:
+        return ''.join(lines)
+    return f"... ({len(lines) - n} lines omitted)\n" + ''.join(lines[-n:])
+
+
+parser = argparse.ArgumentParser(description='Calls the exact lines from the given file, can be called multiple times.')
+
+parser.add_argument('jobs', type=str, help='Path to the text-file containing the commands to be called.')
+parser.add_argument("--spawn", type=int, help='Number of jobs to spawn (if this is specified, it will call itself multiple times)', nargs='?', default=0)
+args = parser.parse_args()
+printer.information(f"Using jobs from '{args.jobs}'")
+
+if args.spawn >= 1:
+    printer.information(f"Spawning {args.spawn} parallel jobs")
+    for i in range(args.spawn):
+        subprocess.Popen([
+            "cmd", "/c", "start", f"Caller {i}: {args.jobs}", "cmd", "/k",
+            f"set POST_ACTIVATE_COMMAND=python {os.path.abspath(__file__)} {args.jobs} && call Conda-Activation-Scripts/activate_environment_windows.bat"
+        ])
+    printer.information(f"Spawned {args.spawn} parallel jobs, exiting... ")
+    exit(0)
+
+barrier_waited = {}  # tracks total seconds waited per barrier line index; resets when barrier passes or a job is picked up
+
+
+def _all_previous_done(jobs_file, lines, barrier_index):
+    return all(
+        os.path.exists(f"{jobs_file}.finished{j}") or os.path.exists(f"{jobs_file}.error{j}")
+        for j in range(barrier_index) if lines[j].strip() not in ("---", "")
+    )
+
+
+def _any_previous_unclaimed(jobs_file, lines, barrier_index):
+    return any(
+        not os.path.exists(f"{jobs_file}.started{j}")
+        and not os.path.exists(f"{jobs_file}.finished{j}")
+        and not os.path.exists(f"{jobs_file}.error{j}")
+        for j in range(barrier_index) if lines[j].strip() not in ("---", "")
+    )
+
+
+while True:
+    with open(args.jobs, 'r', encoding='utf-8') as f:
+        lines = f.readlines()
+
+    found_one = False
+    restart = False
+    for i, line in enumerate(lines):
+        started_job_flag = f"{args.jobs}.started{i}"
+        finished_job_flag = f"{args.jobs}.finished{i}"
+        error_job_flag = f"{args.jobs}.error{i}"
+        if line.strip() == "---":
+            dot_i = 0
+            found_unclaimed = False
+            last_len = 0
+            while not _all_previous_done(args.jobs, lines, i):
+                if _any_previous_unclaimed(args.jobs, lines, i):
+                    found_unclaimed = True
+                    break
+                barrier_waited[i] = barrier_waited.get(i, 0) + 3
+                dots = "." * ((dot_i % 5) + 1)
+                msg = f"Barrier '---' at line {i}: Checking every 3s, waited {barrier_waited[i]}s already{dots}"
+                sys.stdout.write(f"\r{msg}{' ' * max(0, last_len - len(msg))}")
+                sys.stdout.flush()
+                last_len = len(msg)
+                dot_i += 1
+                time.sleep(3)
+            if last_len:
+                sys.stdout.write('\n')
+                sys.stdout.flush()
+            if found_unclaimed:
+                restart = True
+                break  # restart outer while loop to pick up the unclaimed job
+            barrier_waited.pop(i, None)
+            continue  # barrier cleared, keep scanning for next job
+
+        try:
+            fd = os.open(started_job_flag, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
+        except FileExistsError:
+            continue  # Another worker already claimed this job
+        if os.path.exists(finished_job_flag) or os.path.exists(error_job_flag):
+            os.close(fd)
+            continue
+
+        if barrier_waited:
+            barrier_waited.clear()
+        start_datetime = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+        with os.fdopen(fd, 'w', encoding='utf-8') as f:
+            f.write(f"Command: {line.strip()}\n")
+            f.write(f"Started at: {start_datetime}")
+        found_one = True
+        log_file = f"{args.jobs}.log{i}"
+        try:
+            printer.information(f"Executing job {i} from '{args.jobs}': {line.strip()}")
+            ctypes.windll.kernel32.SetConsoleTitleW(f"Job {i} from '{args.jobs}': {line.strip()}")  # type: ignore[attr-defined] - it is Windows-only anyway, so we can ignore the fact that this attribute doesn't exist on other platforms
+
+            start_time = time.time()
+            with open(log_file, 'w', encoding='utf-8') as log_f:
+                log_f.write(f"Command: {line.strip()}\n")
+                log_f.write(f"Started at:  {start_datetime}\n")
+                log_f.write(f"{'=' * 60}\n")
+                log_f.flush()
+
+                proc = subprocess.Popen(
+                    line.strip(),
+                    shell=True,
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.STDOUT,
+                )
+                try:
+                    for raw_line in proc.stdout:
+                        decoded = raw_line.decode(errors='replace').replace('\r\n', '\n').replace('\r', '\n')
+                        sys.stdout.write(decoded)
+                        sys.stdout.flush()
+                        log_f.write(decoded)
+                except KeyboardInterrupt:
+                    # CTRL+C also signals Gurobi, which does a graceful shutdown and saves
+                    # results. Drain remaining output so the log is complete, then let the
+                    # subprocess finish — do not re-raise here.
+                    try:
+                        for raw_line in proc.stdout:
+                            decoded = raw_line.decode(errors='replace').replace('\r\n', '\n').replace('\r', '\n')
+                            sys.stdout.write(decoded)
+                            sys.stdout.flush()
+                            log_f.write(decoded)
+                    except KeyboardInterrupt:
+                        remaining = proc.stdout.read()
+                        if remaining:
+                            decoded = remaining.decode(errors='replace').replace('\r\n', '\n').replace('\r', '\n')
+                            sys.stdout.write(decoded)
+                            log_f.write(decoded)
+                proc.wait()
+            end_time = time.time()
+
+            if proc.returncode != 0:
+                raise RuntimeError(
+                    f"Command exited with code {proc.returncode}. "
+                    f"See log: {log_file}\n"
+                    f"Last output:\n{_tail_file(log_file, 20)}"
+                )
+
+            with open(finished_job_flag, 'w', encoding='utf-8') as f:
+                f.write(f"Command: {line.strip()}\n")
+                f.write(f"Started at:  {start_datetime}\n")
+                f.write(f"Finished at: {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
+                f.write(f"Execution time: {end_time - start_time:.2f} seconds (= {(end_time - start_time) / 60 / 60:.2f} hours)\n")
+                f.write(f"Log file: {log_file}\n")
+
+            printer.information(f"Finished job {i} from '{args.jobs}' after {end_time - start_time:.2f} seconds (= {(end_time - start_time) / 60 / 60:.2f} hours).")
+        except Exception as e:
+            printer.error(f"Error while executing job {i}: {e}")
+            with open(error_job_flag, 'w', encoding='utf-8') as f:
+                f.write(f"Command: {line.strip()}\n")
+                f.write(f"Error while executing job {i} from '{args.jobs}': {e}\n")
+                f.write(f"Started at:  {start_datetime}\n")
+                f.write(f"Occurred at: {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
+                f.write(f"Log file: {log_file}\n")
+        break
+
+    if not found_one and not restart:
+        printer.information(f"No more jobs to execute in '{args.jobs}', exiting.")
+        break