Clean up input plant database

[atpham88]

Summary

This PR cleans up NEMS input plant database to remove most columns from the existing database to reduce its size and make it easier to track.

Implementation notes

• The NEMS unit database is reprocessed upstream (Clean up EIA-NEMS processing ReEDS_Input_Processing#6) to:
  • Remove all the scripts that assign resources to rsc units in NEMS (this step is now done in main ReEDS repo after Adding technology classes #12)
  • Clean up NEMS by removing all columns that are not used in ReEDS or not informative
  • Include minor update in geospatial mapping script for more accurate nearest distance calculation
  • Remove # character in T_UID
• Nuclear retirement year columns are removed. Nuclear retirement years are now determined in process_unitdata.py based on nukeretscen switch.
• Note: Although unit capacity and dates do not change, VOM and FOM slightly change due to removal of reeds_ba from the process of adjusting VOM and FOM for certain units and replacing it with FIPS.
• Put back script in copy_files.py that reads # as comment when reading unitdata https://github.com/ReEDS-Model/ReEDS/blob/main/reeds/input_processing/copy_files.py#L457

Switches added/removed/changed

retscen -> nukeretscen

Validation, testing, and comparison report(s)

~0 changes in unit capacity and online/retirement dates:

Comparison report (full here)

Checklist for author

Details to double-check

• Charge code provided to reviewers
• Included comparison reports for appropriate test cases
• Documentation updated if necessary
• If input data added/modified:
  • Dollar year recorded and converted to 2004$ for GAMS
  • Timeseries are in Central Time
  • Units are specified
  • Preprocessing steps have been documented and committed to ReEDS_Input_Processing
  • New large data files handled with .h5 instead of .csv
  • If new parameters are added to d_objective.gms, they are included in objective_function_params.yaml for completeness checking
  • If spatially resolved inputs are modified, the following visualizations for each file are included in the PR description (time-averaged if the inputs are time-resolved):
    • Map of absolute values before
    • Map of absolute values after
    • Map of differences: (after - before) or (after / before)
  • If entries are added/removed/changed in the EIA-NEMS unit database:
    • Changes have been committed to ReEDS_Input_Processing
    • hourlize/resource.py was rerun to regenerate the existing/prescribed VRE capacity data
• Code formatting standardized
• Reusable functions used where possible instead of copy/pasted code

General information to guide review

• Zero impact on results of default case
• No large data file(s) added/modified
• No substantive impact on runtime for full-US reference case
• No substantive impact on folder size for full-US reference case
• No change to process flow (runreeds.py, reeds/core/solve/solve.py)
• No change to code organization
• No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

No

Tag points of contact here if you would like additional review of the relevant parts of the model

[patrickbrown4]

Nice! One minor thought - what would you think about removing the four alternative nuclear retirement columns (NukeRefRetireYear, Nuke60RetireYear, Nuke80RetireYear, NukeEarlyRetireYear) and instead adding a simple modification to RetireYear for nuclear plants (using the retscen switch) to process_unitdata.py? It seems like a waste to have 4 columns for every unit when the data only vary for a few nuclear plants, and the code-based approach would be easier to understand and edit (i.e., if you want to create a new nuclear retirement scenario, you wouldn't need to modify the whole unit database; you'd only need to add 2 lines of code).

[patrickbrown4]

A few other notes if you're thinking of downsizing more:

• battery_duration is just energy_capacity_MWh / summer_power_capacity_MW so could also be removed and determined in process_unitdata.py
• county and T_STATE should both be able to be determined from FIPS in process_unitdata.py using inputs/zones/county_state.csv
• the nuclear retirement columns noted above

[atpham88]

Thanks for the suggestions, Patrick. I removed battery_duration since we don't use it in the model. I can add back if people like to have it but as you said it's just energy cap/power cap. county and state are now added in process_unitdata.py. And I keep only one RetireYear column now, which is updated for nuclear retirement based on nukeretscen (formerly retscen), which is now just number of years after startyear (default = 80).

I can rerun comparison report but this shouldn't change anything.

[patrickbrown4]

Looks great, thank you! You can actually see all the columns on one screen now, which is nice for inspection.

There are a few that could be rounded to fewer decimals upstream (T_VOM, T_FOM, T_CC*) to cut down file size further, but not a big deal.

[atpham88]

Good point. They are rounded to 2 decimals upstream now.

[patrickbrown4]

Thanks for doing this! A couple minor cleanups but otherwise looks good from my end. It'd still be good to get Wesley's quick look though.

Actually, one important check: Could you run input_plots.plot_units_existing() before and after (on the main branch and on this one) and add that to the compare report, just to make sure everything looks right?

[patrickbrown4]

Looks great, thank you! You can actually see all the columns on one screen now, which is nice for inspection.

There are a few that could be rounded to fewer decimals upstream (T_VOM, T_FOM, T_CC*) to cut down file size further, but not a big deal.

[patrickbrown4]

Unclear what's happening here - if it's a renaming, can you do it in a single dictionary above (part of .rename(columns={'county_name':'county'}))? Explicit dictionary renaming seems safer than order-based renaming.

I don't think the column order should matter; if it does, it'd be safer to rework the code downstream that depends on a particular order to avoid that.

[atpham88]

I just put county and state right next to FIPS column here for easy reading/checking location (so basically reorder county and state to the FIPS.index + 1 position)

[atpham88]

Thanks for doing this! A couple minor cleanups but otherwise looks good from my end. It'd still be good to get Wesley's quick look though.

Actually, one important check: Could you run input_plots.plot_units_existing() before and after (on the main branch and on this one) and add that to the compare report, just to make sure everything looks right?

I added the two existing tech maps to the comparison report. Confirmed that they look identical.

[wesleyjcole]

Why is there a difference in capacity in the 2020 solve year?

Also, it looks like the "cleanedNEMS" existing capacity map has an additional hydropower unit in Western Washington, so I don't think they are exactly the same. It looks like there are also some solar changes in New York between the two maps. There is also a PV plant north of New York (in both maps, so doesn't need fixed here, but should looked at for your plant database update PR that will come later).

[wesleyjcole]

Suggested change

	nukeretscen,Nuclear retirement scenario (years from startyear),int,80,
	nukeretscen,Nuclear retirement scenario (years from plant online year),int,80,

Right?