Skip to content

Make the new-format templater output set granularity-invariant - #125

Merged
nick-gorman merged 7 commits into
mainfrom
granularity-invariant-templater-output
Aug 23, 2026
Merged

Make the new-format templater output set granularity-invariant#125
nick-gorman merged 7 commits into
mainfrom
granularity-invariant-templater-output

Conversation

@nick-gorman

@nick-gorman nick-gorman commented Jun 24, 2026

Copy link
Copy Markdown
Member

Two related but separable changes to the new-format templater. The first is the substance of the PR; the second would be better separated but I think the commits are entangled so I'm just leaving combined (sorry).

1. Granularity-invariant output set

Previously the new-format templater produced a different set of tables depending on regional granularity: the custom-constraint tables were written only at sub_regions, since coarser granularities collapse the flow paths they reference. That forced create_ispypsa_inputs to track a granularity-dependent output list and left downstream consumers checking for tables that may or may not exist.

This makes the output set identical at every granularity. Where there's nothing to constrain, the custom-constraint tables are output header-only ("all columns, no rows") rather than omitted, so the column shape is invariant and only the row contents differ.

2. Wire timeslices templating in

timeslices.py and its tests were already on main but nothing called it. This adds the call in create_ispypsa_inputs_template_template_timeslices(load_timeslice_calendar(version), manual_tables["reference_year_sequence"]) — and registers timeslices in the output list. Templater tests patch load_timeslice_calendar with a two-window stub calendar; the CLI test pins the decoded row count against the shipped 7.5 calendar (content is covered by test_timeslices.py::test_shipped_calendar_decodes).

src/ispypsa/templater/
├── create_template.py                  → header-only cc tables at coarse granularity;
│                                          wire in timeslices
└── custom_constraints_from_plexos.py   → empty_custom_constraint_tables() + shared column constants
tests/
├── test_cli/test_create_ispypsa_inputs_new_table_formats.py  → verify against list_templater_output_files;
│                                                                 timeslices row-count pin
└── test_templater/test_create_ispypsa_inputs_template.py     → set-equality + header-only checks;
                                                                 timeslice stubs

🤖 Generated with Claude Code

@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/ispypsa/templater/create_template.py 93.02% <100.00%> (ø)
...spypsa/templater/custom_constraints_from_plexos.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nick-gorman and others added 7 commits August 17, 2026 15:27
The new-format templater now emits the same table set at every
granularity: timeslices and costs_connection are wired in, and the
custom-constraint tables are emitted header-only at nem_regions /
single_region instead of being absent. This reverses the gating from
9366945 — the PLEXOS constraints are sub-regional export-group limits
with no meaningful representation once sub-regions are collapsed, but
writing "all columns, no rows" tables means list_templater_output_files
needs no granularity awareness and downstream consumers never check for
missing tables.

costs_connection is also now tracked in the output list — it was being
written but untracked, the same class of bug 9366945 fixed for the
constraint tables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The three _stub_* helpers built their DataFrames by hand (dict-of-lists),
the outliers in a file that otherwise uses the csv_str_to_df convention.
Convert them so the fixtures read as row-oriented tables — the timeslice
calendar especially, since test_timeslices.py already writes that exact
shape with csv_str_to_df. The helpers are module-level, so the fixture is
threaded in as a parameter rather than imported.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nem_regions and single_region tests both assert the new-format output
at a collapsed granularity, but had diverged: single_region only checked
the custom-constraint tables had *some* columns where nem_regions pinned
the exact column set, and neither asserted timeslices at all. timeslices is
granularity-invariant (decoded from the calendar + reference_year_sequence,
independent of regional_granularity), so omitting it left the invariance
this PR establishes unchecked at the coarse granularities. Pin the exact
custom-constraint column sets in single_region and assert timeslices (two
patterns, same as sub_regions) in both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new-format CLI test pins a row count plus referential integrity for
every output table except custom_constraints, which only checked that the
LHS/RHS rows carried no orphan constraint_ids. Add a principled constraint
count (one per CONSTRAINT_NAMES in scripts/extract_plexos_constraints.py)
and a drift-detection count for the LHS coefficient rows, so a regression
in the populated sub_regions output is caught here rather than only in the
per-module tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
empty_custom_constraint_tables() declared the three tables' column sets as
module constants, but the populated builders still hardcoded their own
copies of the same lists. Nothing coupled the two, so a future change to a
builder's output columns would silently diverge from the header-only tables
emitted at coarse granularities, breaking the granularity-invariance this
branch establishes. Project every builder onto the shared constants so the
populated and empty tables have one definition, and split the CLI test's
empty-case assertion so a failure names the offending table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The same explanation now lives in empty_custom_constraint_tables()'s
docstring, which the else-branch calls.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rebase onto main brought generators_new_entrant / storage_new_entrant
into _NEW_FORMAT_TEMPLATE_OUTPUTS, and nothing tied the templater's returned
keys to what list_templater_output_files declares — a wrong conflict
resolution would silently untrack written files. The new-format templater
tests now assert set equality, and the CLI test verifies the on-disk set
against the same source list instead of keeping its own copies.

The coarse-granularity tests compare against empty_custom_constraint_tables()
directly rather than re-spelling its columns, and the "why header-only"
rationale now lives once in that function's docstring, with the other sites
reduced to pointers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@nick-gorman
nick-gorman force-pushed the granularity-invariant-templater-output branch from adbf3c8 to 7a5f0ee Compare August 17, 2026 06:16
@EllieKallmier EllieKallmier added module: templater Covers contents of `templater` module type: feature New feature or request labels Aug 19, 2026

@EllieKallmier EllieKallmier left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep all makes sense to me, no notes :)

@nick-gorman
nick-gorman merged commit d0e7d74 into main Aug 23, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: templater Covers contents of `templater` module type: feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants