Fail the calibration build on targets it drops or cannot fit

[MaxGhenis]

Do not merge on its own. Once approved on its gates, this lands with the d833 uk-data batch on Max's go. It is part of workstream W3 of the Autumn Budget 2026 CI plan: the required-target-coverage gate.

Integration. This branch merges main (a43e8c51), #604 (29ca7542) and #557 (473adafc), and its last commit drops the known gaps those two close. Land it after both, in the order #604, #557, #595.

Why

create_target_matrix builds the national half of the calibration loss. It let targets go in four ways, and none of them fails a build:

• a target whose name another target already has is dropped without a log line. Main's registry drops 159 targets this way (see "Duplicate names" below);
• a target whose year does not resolve is skipped, also without a log line (Calibration loss under-penalises overshoots and silently drops targets #456). On 1.58.1 this drops all 15 dwp/uc/two_child_limit/* targets, which are dated 2026 against the calibration year 2025;
• a target whose column raises is skipped with a warning;
• a target whose column is zero for every household enters the matrix, where the weights cannot move it (Nineteen calibration targets have no signal on 1.58.1: £1m+ income bands, ages 85-90 and others #593). On 1.58.1 there are 19 of these:
  • ten HMRC £1m+ income bands;
  • ONS ages 85–90 (two targets);
  • three VOA band-H regions;
  • SLC postgraduate repayments;
  • three UC payment bands for couples without children.

Duplicate names

The registry keeps the first target of each name, and the matrix keeps the first across national, region and country levels. On main's registry 159 targets are dropped this way. With #557 merged, 3 are:

• 156 HMRC SPI income-band projections. hmrc_spi emits each band twice: the 2023-24 outturn with a 2024 value only, then the incomes_projection.csv copy for 2025–2029. The matrix keeps the outturn, so calibration at 2025 carries the 2024 outturn forward and never sees the projection. Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557 fixes this.
• 3 ONS targets (ons_demographics emits three target names twice, and one copy of ons/uk_population is 1,000 times too small #605, filed for this PR). The matrix keeps the national ons/uk_population and the region Scottish targets, so the 2025 matrix is unaffected. The dropped copies are:
  • a region ons/uk_population read from demographics.csv 1,000 times too small;
  • NRS country copies of the two Scottish targets, with identical values.

What changes

• build_target_matrix records every offered target's status:

  • in_matrix;
  • duplicate, a target dropped because an earlier one has its name;
  • no_value_for_year;
  • no_column;
  • error;
  • non_finite;
  • zero_signal, meaning a non-zero target with no household contributing.

  It reads every target the sources emit (registry.emitted_targets) and keeps the same targets in the same order as before. So known gaps and duplicates' winners still enter the matrix. Every offered target ends with exactly one status. The status is computed after the column enters the matrix, and a column whose status cannot be computed (a non-numeric column) leaves it as an error, where main kept it.

• create_target_matrix (require_coverage=True by default, as on the Enhanced FRS build paths in create_datasets.py and the local-area calibrate modules) fails with TargetCoverageError on any target not in the matrix, unless targets/known_gaps.yaml lists it under that same status, with a reason and an issue URL.

  • Errors and non-finite values are never excused, whatever a gap says.
  • A gap applies to every build unless builds limits it to testing (TESTING=1, as pull request builds set) or release. The build kind is read from TESTING, as create_datasets reads it.
  • A testing-only gap may name fnmatch patterns instead of targets. Release gaps must name each target.
  • A name may be listed under two statuses: the ten SPI £1m+ bands are both without signal and duplicated.
  • A listed target that no longer has its status is logged as stale, not failed, so an improvement never breaks a build. A pattern is stale only if it matches no target, because what it excuses varies between testing builds.
  • Stale gaps and problems are also written once each to the GitHub Actions job summary ($GITHUB_STEP_SUMMARY), so they don't get buried in a multi-hour log.

• utils/loss.get_loss_results (a diagnostic of any dataset) and utils/reweight (only the CPS-based PolicyBench transfer dataset) opt out with require_coverage=False.

• targets/known_gaps.yaml lists:

  • the 15 targets with no 2025 value (Calibration loss under-penalises overshoots and silently drops targets #456);
  • the 3 ONS duplicates (ons_demographics emits three target names twice, and one copy of ons/uk_population is 1,000 times too small #605). The 156 SPI duplicates are gone, because Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557, merged here, emits each band once.
  • the 17 with no signal on the released 1.58.1 build. The ages 85–90 entries are gone, because Match ONS age targets to the FRS top code of 80 #604, merged here, uses 80-and-over targets that match the top code. impute_student_loan_plan never assigning POSTGRADUATE is permanent.
  • one testing-only pattern, dwp/uc_payment_dist/* under zero_signal, for UC payment bands whose signal flickers between pull request builds (Builds of the same code calibrate to two different weight sets #598). It replaces the three single-claimant bands this PR listed before.

• python -m policyengine_uk_data.targets.coverage <dataset.h5> <year> [out.json] reports every target's status, problems and stale gaps, for the build kind TESTING sets. It outputs names, statuses and counts only.

UC payment bands in pull request builds

The gate failed twice on this PR:

• run 38726993359 failed on the three SINGLE bands;
• run 38732018053, after they were listed, failed on COUPLE_WITH_CHILDREN_annual_payment_27_600_to_28_800.

The final calibration logs of 21 pull request builds (8–15 October) show which bands had a national estimate of exactly zero:

Band	Builds with zero estimate (of 21)
COUPLE_NO_CHILDREN 27,600–28,800	20
COUPLE_NO_CHILDREN 28,800–30,000	21
COUPLE_NO_CHILDREN 30,000+	21
SINGLE 24,000–25,200	20
SINGLE 25,200–26,400	20
SINGLE 27,600–28,800	21
COUPLE_WITH_CHILDREN 27,600–28,800	3

Listing these bands one by one would keep failing pull requests, so the testing-only pattern covers the family. Release builds still fail on any UC band without signal, except the three COUPLE_NO_CHILDREN bands listed by name. Pull request and release builds both set PE_UK_DATA_OA_CLONES=1, so the clone count is not a difference between them. The TESTING differences are the epochs (32, not 512), the SPI income-model sample (10,000 records, not 100,000) and the SLC snapshot in place of a live fetch.

Evidence

On the released uk-data 1.58.1 enhanced_frs_2024_25.h5 (sha256 056e80fd), with the locked policyengine-uk 2.122.2 at 2025:

• at 6e03d4c, with Match ONS age targets to the FRS top code of 80 #604 and Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557 merged: 661 targets offered: 626 in_matrix, 3 duplicate, 15 no_value_for_year, 17 zero_signal. There are no problems and no stale gaps, run as a release build and again as a testing build. The output is in coverage-1.58.1-2025-integration-{release,testing}.json.
• at a590573, before the integration: 819 offered: 626 in_matrix, 159 duplicate, 15 no_value_for_year, 19 zero_signal, with no problems and no stale gaps. The 660 targets that are not duplicates had the same statuses as in the run before eb247c6.

The data test test_built_data_has_no_target_outside_the_matrix_and_the_known_gaps runs the same check wherever the built dataset is present, as in CI's build-and-test job.

Not yet checked: a release build of current main. No non-TESTING pre-calibration dataset built from current main has been run through the gate. The evidence above covers the released 1.58.1 file, which older code built, plus the TESTING builds in CI. So the next release build is the gate's first full-size run. A release build fails safe: if it meets a target the gaps don't cover, it stops at the calibration step.

Invariants (tested)

• column_status:
  • non-finite beats everything;
  • a zero column is zero_signal only when its target is non-zero.
• Hypothesis over random statuses, named and pattern gaps, and build kinds:
  • the problems are exactly the targets out of the matrix that no gap for this build excuses under their own status;
  • errors and non-finite values are never excused, even by a hand-built gap that lists those statuses;
  • stale gaps are exactly the named targets without their status, and the patterns that match no target;
  • order doesn't matter.
• Differential, Hypothesis over random source modules: require_coverage=False builds exactly the matrix and values that main's create_target_matrix and get_all_targets build from the same sources, over random names, levels, years, values, columns and country scopes. These include duplicate names within and across levels, unresolved years, missing and raising columns, and NaN and infinite values. Every target offered at a matrix level gets exactly one status.
• Through create_target_matrix on a fake simulation:
  • a dropped target or a raising column fails, with the reason in the message;
  • a name two sources emit fails as duplicate, and so does a regional target named like a national one, which loses to the national one even when emitted first;
  • a column that cannot enter the matrix, or whose status cannot be computed, leaves one error status and no column;
  • a gap entry excuses only its own target and status;
  • a non-finite column is never excused.
• Build kinds:
  • a testing-only gap, named or pattern, excuses nothing in a release build;
  • a release-only gap excuses nothing in a testing build, and isn't stale there;
  • the shipped file excuses any UC band without signal in testing builds, only the three listed bands in release builds, and never an error.
• Known-gap file:
  • every entry has an excusable status, a reason and an issue;
  • no target is listed twice under one status;
  • patterns are testing-only and must contain a wildcard;
  • the shipped file has the 15 + 17 + 3 named entries measured above and the one UC pattern.
• test_no_duplicate_names counts what the sources emit, before the registry drops anything. It fails on any duplicate not listed in known_gaps.yaml. Before Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557 it reported all 159 without the entries; with Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557 merged, the three ONS names remain.

Interactions

• Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557 (merged here) emits each SPI band once, and its 156 duplicate entries are removed. The ten £1m+ bands keep their zero_signal entries, because Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557 gives them no signal.
• Match ONS age targets to the FRS top code of 80 #604 (merged here) replaces ons/{female,male}_85_90, and those two entries are removed.
• ons_demographics emits three target names twice, and one copy of ons/uk_population is 1,000 times too small #605 tracks the ONS duplicates.
• This doesn't change the loss itself; Calibration loss under-penalises overshoots and silently drops targets #456's log-ratio proposal is separate.
• Each SPI PR (Rebase SPI income draws to the FRS survey year #532, Draw SPI incomes at each donor's FRS rank within their cell #551, Calibrate SPI income bands to their projected values, and map the 2023-24 outturn to 2023 #557) may give the £1m+ bands signal. The gap then shows as stale in the build log and the job summary, and should be removed from known_gaps.yaml.
• A PR that adds a target the build can't fit, or reuses a name, now fails until the target is fixed or listed. That is the intended effect.

Tests

At 6e03d4c, run serially:

• pytest policyengine_uk_data/tests/test_target_coverage.py policyengine_uk_data/tests/test_obr_universal_credit_target.py: 39 passed, 1 skipped (needs the built dataset).
• pytest policyengine_uk_data/tests/test_target_registry.py: 17 passed (live source downloads).

At a590573, before the integration:

• Full suite (pytest policyengine_uk_data/tests, no built datasets locally): 2247 passed, 132 skipped, 1 xfailed. Without the opt-outs of get_loss_results and the CPS transfer reweighting, the PolicyBench transfer test fails, because a synthetic dataset has hundreds of zero-signal columns. (That last point was observed at an earlier head and not re-run at a590573.)

axiom: n/a: calibration build gate; no policy change

🤖 Generated with Claude Code