Skip to content

Conversation

@ktehranchi
Copy link
Collaborator

@ktehranchi ktehranchi commented Sep 22, 2025

Resolves conflicts from #631 :

I closed the previous pull request and made a new one (I couldn't figure out how to change the branch I was merging into develop) - I made a local branch from develop, then cherry-picked the five wus-integration commits as suggested. Hopefully it works this time? Let me know if there's anything I need to fix!

Changes and their purpose

Changes made to

  • build_renewable_profiles.py: if cf_source is set to WUS, it will substitute in the WUS capacity factor timeseries for wind and solar for the closest lat/lon into the profile for all the buses in the system after the it does the land use calculations, etc. using the ERA5 data/atlite
  • build_electricity.smk: changed calls for build_renewable_profiles to have it generate multiple profiles and list the capacity factor source in the generated profile name, changed calls for add_electricity so it calls the multiple profiles
  • add_electricity.py: changed the code where it puts the renewable profiles in and just copies over the same profile year for each planning horizon - this concatenates the profiles generated (in order) and attaches them to the buses in the network
  • add_demand.py, add_fuel_prices.py: change the way snapshots are called because of the new way that snapshots are defined in the config file (see below)

There are also changes to retrieve_databundles.py to facilitate zipfile downloads on windows, that has nothing to do with the WUS stuff.

Config File Changes

I'm not sure if it updates or shows the config files but just in case:

In your run's config file (i.e. the config.default.yaml if that's where you're starting from), the renewable weather year information needs to look like this:

renewable_weather_years: [2012,2019,2023] # need to be historical

snapshots:
  '2019':
    start: "2019-01-01"
    end: "2020-01-01"
    inclusive: "left"
  '2012':
    start: "2012-01-01"
    end: "2013-01-01"
    inclusive: "left"
  '2023':
    start: "2023-01-01"
    end: "2024-01-01"
    inclusive: "left"

And in the config.common.yaml, you need to add:

cf_source: "WUS" # WUS or ERA5
cf_path: "T:/WRFDownscaled/ec-earth3-veg_r1i1p1f1_ssp370_bc/Annual_Solar_Wind" 

If you're looking to use the WUS data, let me know and I can help facilitate getting you access to the capacity factor data; you'll need to change the cf_path to wherever you store the data. If you just want to use the multiple weather profile capability, just set cf_source equal to ERA5 and it will use the ERA5 weather data for the years you specify for the renewable weather years, no need to worry about the cf_path.

trevorb1 and others added 26 commits April 13, 2025 21:11
Adds downstream ch4 tracking and updates associated configuration documentation.
* update dag (PyPSA#517)

Update DAGs for documentation site

* added for loop around the solve_network. New myopic and perfect foresight if statements in the solve_network function to pull the foresight type from a config file. Foresight type option still needs to be added to the repo_data/config/ files, as it is just manually added into the workflow/config currently to avoid init_pypsa_usa.sh reuse. With precommit fixes.

* fixed errors with pulling the myopic foresight from config file, model running, gives same results as without myopic since we are not pulling from prior timesteps

* moved for loop inside of the solve_network, removed the add_myopic constraint

* first working version, no bugs, runs fully. needs checks to see if output matches expectations. output does combine first year with the existing infrastructure (need to fix). added in the time dependent components. added a limited part of c in the c.iterate_components, to just include what was in the previous timestep.

* fixed the limited c.df to be for build years below the investment horizon, previously was years equal to the investment horizon

* made it so that the links are properly removed, since their build year is 0

* first successful version to have results for all decades, needs first level checks but is giving reasonable results for myopic output. Key change was creating a deep copy of the c.pnl and adding all rather than just input time variables back in after removal

* fixed the loop to be inclusive for c_lim rather than less than the planning horizon

* updated the configtables documentation

* Fix clim, other fix for perf foresight.

* RPS updates

* fixed the retirement issue for one time horizon

* updated to m.add to carry over the custom fields

* added the storage state_of_charge rollover, but this is leading to infeasible solve, need to fix

* changed state of charge to just be the final snapshot, now feasible

* updated the config files in repo_data, and the docs to show that myopic is available

* n.madd for generators so that the attributes carry over. this is a fix that likely isnt ideal in the long run, but n.add doesn't carry these attributes over

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Trevor Barnes <[email protected]>
Co-authored-by: ktehranchi <[email protected]>
Co-authored-by: Jones <[email protected]>
Co-authored-by: Kamran Tehranchi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Add lifetimes and build rates for sector technologies
* update dag (PyPSA#517)

Update DAGs for documentation site

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/pycqa/isort: 5.13.2 → 6.0.0](PyCQA/isort@5.13.2...6.0.0)
- [github.com/asottile/pyupgrade: v3.19.0 → v3.19.1](asottile/pyupgrade@v3.19.0...v3.19.1)
- [github.com/psf/black-pre-commit-mirror: 24.10.0 → 25.1.0](psf/black-pre-commit-mirror@24.10.0...25.1.0)

* new linting

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.4 → v0.9.9](astral-sh/ruff-pre-commit@v0.1.4...v0.9.9)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.9 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.9...v0.9.10)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.0](astral-sh/ruff-pre-commit@v0.9.10...v0.11.0)

* [pre-commit.ci] pre-commit autoupdate (PyPSA#579)

updates:
- [github.com/snakemake/snakefmt: v0.10.2 → v0.11.0](snakemake/snakefmt@v0.10.2...v0.11.0)
- [github.com/astral-sh/ruff-pre-commit: v0.11.0 → v0.11.5](astral-sh/ruff-pre-commit@v0.11.0...v0.11.5)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* correct notebooks for precommit

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kamran Tehranchi <[email protected]>
Co-authored-by: ktehranchi <[email protected]>
* fix filter components for single horizon

* store erm duals

* write tests and refactor

* fix tests, still errors on reserves

* fix filter components for single horizon

* store erm duals

* write tests and refactor

* fix tests, still errors on reserves

* Functionalize brownfield for myopic

* Fix ERM and test for ERM

* remove defunct AI writen tests

* ERM link support, all timesteps, rm duplicate function

* Address comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs to run test

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Adds REM Unit Test

* Adds RPS unit tests

* Adds TCT unit test
* fix missing SD zone, and pudl path bug

* rm unnec conditional
* Refactor RPS to enable REC sharing

* Move build_powerplants out of run dependency
* Trim Network with external imports

* Fix plotting issues with split

* Fixes Policy Opts for Myopic foresight
* add sorting for diff

* split data query functions, and sort outputs for diffs

* Adds Planned and Retired Generators to BPP, change filtering logic in add_ele

* Bump Pudl version to v2025.5.0

* Fix plants filtering dates
* globalconstraint hotfix

* change co2 constraint

---------

Co-authored-by: tmb8 <[email protected]>
* `v0.6.0` Release Candidate (PyPSA#594)

* fix plot bug

* Update mem reqs, fix TCT bug

* update scenario comparison

* update get_region_buses to include reeds zone id

* dr by carrier components

* dr constraint update

* correct conditional

* docs update

* fix docs

* dr sector hotfix

* Adds population weighting strategy for simplify and cluster

* update default config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove virtual buses

* update tct logic

* tct note in docs

* update comment

* add assertion

* typo

* dr implementation

* plotting runs

* carrier added

* dr constraint templated

* power sector runs

* plotting updates

* config updates

* update dr docs

* update TCT constraint name

* cluster config updates

* typo

* update dependencies

* fix scenario comparison

* Update pre-commit config to use Ruff for formatting and linting

* update linting options

* Update pre-commit config to use Ruff for formatting and linting

* ruff re-formating

* new linting

* rm isort conf

* rm old docs page

* reorganize repo_data/geospatial

* clean repo_data

* Add UV install support! PyPSA#483

* update uv env

* add uv pyproj

* add lock file

* update docs for UV

* fix bug from refactor

* bug fix from refactor

* minor corrections

* remove unused config, and set dynamic memory resource request by input file size

* cop fixed

* natural gas done

* sector costs

* stock data

* constants updated

* corrects eulp

* more fixes

* dynamically set memory allocation by file size

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eia fix

* update solve elec rule mem

* demand updated

* valiadation plots linted

* fix type hints

* add uv and ruff badges

* sector plots update

* env updates

* update dev dependency

* minor fixes

* dev dependency update

* fix sankey file extension

* sector costs hotfix

* fix nan in efficiency2

* typo

* handle no-data scenarios

* Modify battery efficiencies to align with NREL ATB

* Reference PUDL s3 Parquests, Add PUDL versioning

* refactor PRM

* remove SAFE, change SAFER name to PRM

* co2 constraints updated

* update constraint name

* tct update

* correct pudl path

* PRM fixes

* docs

* docs

* refactor OSW implementation (PyPSA#577)

* conditional for multi-period (PyPSA#581)

* Sector Marginal Cost Updates (PyPSA#557)

* eia fuel cost extraction updated

* trn production plots update

* furnace efficiency update

* central marginal cost stores

* dynamic cost scaling

* methane tracking always on

* correct marginal cost conversions

* update lpg vehicle efficiency

* ev policy implemented for endogenous investment

* plotting color fix

* conditional for multi-period (PyPSA#582)

* minor plotting updates (PyPSA#586)

* sector RPS (PyPSA#575)

* Correct Linepack Calculation and CECS Build Years (PyPSA#589)

* correct linepack calc

* correct commercial water heaters

* Downstream Methane Leaks (PyPSA#590)

Adds downstream ch4 tracking and updates associated configuration documentation.

* Issue 518 - Myopic Implementation (PyPSA#554)

* update dag (PyPSA#517)

Update DAGs for documentation site

* added for loop around the solve_network. New myopic and perfect foresight if statements in the solve_network function to pull the foresight type from a config file. Foresight type option still needs to be added to the repo_data/config/ files, as it is just manually added into the workflow/config currently to avoid init_pypsa_usa.sh reuse. With precommit fixes.

* fixed errors with pulling the myopic foresight from config file, model running, gives same results as without myopic since we are not pulling from prior timesteps

* moved for loop inside of the solve_network, removed the add_myopic constraint

* first working version, no bugs, runs fully. needs checks to see if output matches expectations. output does combine first year with the existing infrastructure (need to fix). added in the time dependent components. added a limited part of c in the c.iterate_components, to just include what was in the previous timestep.

* fixed the limited c.df to be for build years below the investment horizon, previously was years equal to the investment horizon

* made it so that the links are properly removed, since their build year is 0

* first successful version to have results for all decades, needs first level checks but is giving reasonable results for myopic output. Key change was creating a deep copy of the c.pnl and adding all rather than just input time variables back in after removal

* fixed the loop to be inclusive for c_lim rather than less than the planning horizon

* updated the configtables documentation

* Fix clim, other fix for perf foresight.

* RPS updates

* fixed the retirement issue for one time horizon

* updated to m.add to carry over the custom fields

* added the storage state_of_charge rollover, but this is leading to infeasible solve, need to fix

* changed state of charge to just be the final snapshot, now feasible

* updated the config files in repo_data, and the docs to show that myopic is available

* n.madd for generators so that the attributes carry over. this is a fix that likely isnt ideal in the long run, but n.add doesn't carry these attributes over

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Trevor Barnes <[email protected]>
Co-authored-by: ktehranchi <[email protected]>
Co-authored-by: Jones <[email protected]>
Co-authored-by: Kamran Tehranchi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add lifetimes and build rates for sector studies (PyPSA#593)

Add lifetimes and build rates for sector technologies

* Corrects `v0.6.0` merge conflicts  (PyPSA#595)

* update dag (PyPSA#517)

Update DAGs for documentation site

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/pycqa/isort: 5.13.2 → 6.0.0](PyCQA/isort@5.13.2...6.0.0)
- [github.com/asottile/pyupgrade: v3.19.0 → v3.19.1](asottile/pyupgrade@v3.19.0...v3.19.1)
- [github.com/psf/black-pre-commit-mirror: 24.10.0 → 25.1.0](psf/black-pre-commit-mirror@24.10.0...25.1.0)

* new linting

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.4 → v0.9.9](astral-sh/ruff-pre-commit@v0.1.4...v0.9.9)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.9 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.9...v0.9.10)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.0](astral-sh/ruff-pre-commit@v0.9.10...v0.11.0)

* [pre-commit.ci] pre-commit autoupdate (PyPSA#579)

updates:
- [github.com/snakemake/snakefmt: v0.10.2 → v0.11.0](snakemake/snakefmt@v0.10.2...v0.11.0)
- [github.com/astral-sh/ruff-pre-commit: v0.11.0 → v0.11.5](astral-sh/ruff-pre-commit@v0.11.0...v0.11.5)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* correct notebooks for precommit

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kamran Tehranchi <[email protected]>
Co-authored-by: ktehranchi <[email protected]>

---------

Co-authored-by: ktehranchi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kamran Tehranchi <[email protected]>
Co-authored-by: Wei Ai <[email protected]>
Co-authored-by: Lyssa Freese <[email protected]>
Co-authored-by: Jones <[email protected]>
# Conflicts:
#	uv.lock
#	workflow/envs/dev.yaml
#	workflow/repo_data/config/config.common.yaml
#	workflow/rules/build_electricity.smk
#	workflow/run_slurm.sh
#	workflow/scripts/add_electricity.py
#	workflow/scripts/add_extra_components.py
#	workflow/scripts/build_fuel_prices.py
#	workflow/scripts/build_powerplants.py
#	workflow/scripts/simplify_network.py
#	workflow/scripts/solve_network.py
#	workflow/snakemake_profiles/slurm/config.yaml

* [pre-commit.ci] pre-commit autoupdate (PyPSA#597)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.11.5 → v0.11.8](astral-sh/ruff-pre-commit@v0.11.5...v0.11.8)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (PyPSA#600)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.11.8 → v0.11.10](astral-sh/ruff-pre-commit@v0.11.8...v0.11.10)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Enable state-level power sector modeling. State level transmission data generated from reeds_ba level data aggregation (see issue 609).

---------

Co-authored-by: Trevor Barnes <[email protected]>
Co-authored-by: ktehranchi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kamran Tehranchi <[email protected]>
Co-authored-by: Lyssa Freese <[email protected]>
Co-authored-by: Jones <[email protected]>
@ktehranchi ktehranchi marked this pull request as ready for review September 22, 2025 04:23
@ktehranchi
Copy link
Collaborator Author

Outstanding issues with this:

  1. The weather_year wildcard causes a cutout to be repeatedly downloaded if the weather year is the same per planning horizon ie [2012, 2012, 2012]
  2. Reconciliation with @asiazzzhang godeeep method for tracking renewable profile years. the configuration file is messy and things need to be combined
  3. The fix for the EFS zipdeflate works on Mac (by bypassing it) but should be tested on windows

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants