Enhance Caller.py and CaseStudy with logging, error handling, and functions - #36
Merged
Conversation
Fix type-annotation for parameter files
Fix bug in calculating PowerWeights_RP
Add white text for sections which are otherwise too dark
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands InOutModule’s documentation and introduces several usability/robustness enhancements across the CaseStudy loader, SQLite export, utilities, and a new parallel job-runner (Caller.py).
Changes:
- Added/expanded documentation (
README.md,CLAUDE.md, and.claude/rules/…) describing architecture, APIs, and doc maintenance rules. - Added new functionality: a
Caller.pyparallel job runner, new transition-matrix plotting utility, and additionalCaseStudydata-manipulation methods. - Hardened data handling: ensured Excel index columns are read as strings; updated SQLite export typing and solver-statistics recording.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Utilities.py | Adds utility fixes and a new plot_transition_matrix() helper (introduces plotting dependency considerations). |
| SQLiteWriter.py | Tightens model_to_sqlite typing and refactors solver-stat export to pull from a LEGO instance. |
| README.md | Adds high-level docs, file index, CaseStudy API overview, and usage examples. |
| printer.py | Adds note(), rule(), and timestamp() helpers. |
| ExcelReader.py | Forces index columns to string for both pivoted and non-pivoted reads. |
| CLAUDE.md | Adds architecture notes / design constraints for key modules. |
| CaseStudy.py | Adds scenario filtering option, generator merge utilities, transition-matrix perturb/shift utilities, and refactors weight computation. |
| Caller.py | New parallel job runner supporting barriers, per-job logs, and error handling (needs platform guards). |
| .claude/rules/claude-md-maintenance.md | Adds doc maintenance rules specific to InOutModule. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import typing | ||
| from typing import TYPE_CHECKING, Literal, Dict | ||
|
|
||
| import matplotlib.pyplot as plt |
Comment on lines
8
to
10
| from InOutModule.printer import Printer | ||
| from LEGO.LEGO import LEGO | ||
|
|
Comment on lines
+228
to
+234
| if not self.do_not_filter_unused_scenarios: | ||
| self.dGlobal_Scenarios = self.dGlobal_Scenarios[self.dGlobal_Scenarios['relativeWeight'] != 0] # Drop rows in dGlobal_Scenarios where relativeWeight is 0 | ||
| if len(self.dGlobal_Scenarios) == 0: | ||
| raise ValueError("No scenarios are present in the 'Global_Scenarios' table. Please check if the file exists and contains valid data.") | ||
| elif len(self.dGlobal_Scenarios) == 1: | ||
| self.filter_scenario(self.dGlobal_Scenarios.index[0], inplace=True) # Filter case study to only include actually present scenarios | ||
|
|
Comment on lines
+1211
to
+1213
| sc = hindex_flat[hindex_flat['scenario'] == scenario].copy() | ||
| sc = sc.sort_values(['p']) | ||
| n_ks_per_rp = len(self.dPower_WeightsK['scenario'] == scenario) |
Comment on lines
+38
to
+46
| if args.spawn >= 1: | ||
| printer.information(f"Spawning {args.spawn} parallel jobs") | ||
| for i in range(args.spawn): | ||
| subprocess.Popen([ | ||
| "cmd", "/c", "start", f"Caller {i}: {args.jobs}", "cmd", "/k", | ||
| f"set POST_ACTIVATE_COMMAND=python {os.path.abspath(__file__)} {args.jobs} && call Conda-Activation-Scripts/activate_environment_windows.bat" | ||
| ]) | ||
| printer.information(f"Spawned {args.spawn} parallel jobs, exiting... ") | ||
| exit(0) |
Comment on lines
+119
to
+121
| printer.information(f"Executing job {i} from '{args.jobs}': {line.strip()}") | ||
| ctypes.windll.kernel32.SetConsoleTitleW(f"Job {i} from '{args.jobs}': {line.strip()}") # type: ignore[attr-defined] - it is Windows-only anyway, so we can ignore the fact that this attribute doesn't exist on other platforms | ||
|
|
Comment on lines
+73
to
+77
| for i, line in enumerate(lines): | ||
| started_job_flag = f"{args.jobs}.started{i}" | ||
| finished_job_flag = f"{args.jobs}.finished{i}" | ||
| error_job_flag = f"{args.jobs}.error{i}" | ||
| if line.strip() == "---": |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces comprehensive documentation and several improvements to the
InOutModule, clarifying both developer and user-facing aspects. It also adds minor code fixes and enhancements to data handling and the parallel job runner. The most important changes are summarized below.Documentation and Maintenance Rules
.claude/rules/claude-md-maintenance.mdto specify clear rules for updatingCLAUDE.mdandREADME.mdafter code changes, including guidance on avoiding duplication and maintaining up-to-date documentation forInOutModule.CLAUDE.mdwith detailed architecture notes for Claude, documenting non-obvious design decisions, data flows, and implementation constraints for core modules likeCaseStudy,ExcelReader,ExcelWriter,SQLiteWriter, andCaller.README.mdwith a high-level overview, file index, API documentation forCaseStudy, usage examples forCallerandSQLiteWriter, and a description of the Excel input format.New Features and Enhancements
Caller.pyscript: a robust parallel job runner supporting job barriers, per-job logging, error handling, and Windows terminal spawning. This tool enables batch execution and synchronization for experiments.Bug Fixes and Data Handling
ExcelReader.py, ensured index columns are always converted to strings in both pivoted and non-pivoted file readers to prevent subtle bugs with integer indices. [1] [2]SQLiteWriter.py, clarified the type annotation formodel_to_sqliteto require aConcreteModel, improving type safety.