Build OBR targets only from hash-verified EFO workbooks

[MaxGhenis]

Do not merge on its own. Once approved on its gates, this lands with the d833 uk-data batch on Max's go. It is part of workstream W3 of the Autumn Budget 2026 CI plan: Budget-day OBR source ingestion with hashes, vintage, units, geography and year coverage.

Why

The OBR targets came from whatever obr.uk served, falling back to two committed workbooks matched by URL substring. Four things could go wrong without anything failing:

• The vintage label came from sources.yaml, not from the bytes. Pointing sources.yaml at the October 2026 tables while the download failed would have labelled the March 2026 bytes as October 2026. obr.uk refuses scripted downloads (403 to Actions runners per 11e41f9, and to a developer machine on 2026-10-15), so the fallback is the normal path.
• Year columns were fixed letters. C meant 2024-25 on four tables and D did on the current-receipts table. An EFO whose tables start at 2025-26 would have been read one year off in every target.
• Missing rows were logged and dropped. get_targets caught every receipts or expenditure exception and continued, so a renamed row, a formula in a value cell or a failed parse calibrated without those targets.
• Unit and coverage were assumed. Values were multiplied by 1e9 without reading the unit, and no year coverage was checked.

What changes

• targets/source_manifests/obr_march_2026.yaml names the two committed workbooks with:

  • SHA-256, size, title (Contents!B2), canonical URL and provenance;
  • unit (£ billion) and financial years (2024-25 to 2030-31);
  • the 25 targets they must yield, with the self-employed NICs line as either the combined row or Classes 2 and 4 separately.

  sources.yaml names the manifest in use.

• The manifest is bound to the bytes.

  • The vintage, its label and both workbook titles must agree: march_2026, "March 2026", "March 2026 Economic and fiscal outlook … receipts".
  • Each title must name exactly that vintage (no other month and year), call itself the Economic and fiscal outlook's "detailed forecast tables", and end with its kind.
  • Unicode spaces and dashes, reordered words, a closing note in brackets ("(revised)") and trailing punctuation are accepted, so a plausible October title format doesn't stop Budget-day ingest.
  • Each workbook's title is read from its verified bytes, so the bytes decide the vintage.

• targets/obr_sources.py.

  • Builds read only verified bytes and never download.
  • Each table's year columns come from its header row, which must cover exactly the manifest's years.
  • Its unit label must match the manifest's unit. A row whose label states a unit of its own is refused, in any spelling:
    • any £, % or "per cent";
    • a unit word in brackets ("(m)", "(000s)", "(£ millions)");
    • a unit word after a final comma (", £ million").
  • So is a row under a sub-heading that gives another unit ("Memo: £ million"). A sub-heading that restates the table's own unit ("£ billion", "£bn", "£ billion unless otherwise stated") changes nothing and is accepted.
  • None of the 26 rows the March 2026 build reads is refused. The TV licence target reads the "BBC licence fee receipts" row only.
  • workbook_targets fails unless:
    • every required target is built with every year;
    • every workbook target carries the vintage;
    • nothing undeclared is built;
    • nothing is built twice.
  • A target's forecast_vintage is the manifest's vintage, so it always names the bytes it came from.

• get_targets no longer catches. A calibration without the OBR targets is never correct.

• python -m policyengine_uk_data.targets.obr_sources ingest adopts a vintage on Budget day:

  1. Download the two workbooks in a browser and pass their paths, with --receipts-url and --expenditure-url, the pages they came from. Every target cites those pages, so each must be https on obr.uk or a subdomain of it, with nothing else in the address but port 443.
  2. It refuses an ingested vintage, an earlier ingest's leftovers and a page off obr.uk, then checks each workbook's title names the vintage.
  3. It parses them in a staging folder. Only then does it move them to storage/obr_efo/<vintage>/ and write the manifest, each under a temporary name that is renamed into place. An interrupted ingest leaves a .partial file, which the next attempt names, never a half-written file under the real name. An ingested vintage is never replaced.
  4. It parses them with the build's own parsers and requires the current manifest's target set.
  5. It writes obr_<vintage>.yaml and prints every target's old and new value.

  The manifest records each workbook's SHA-256, size and download time, with downloaded_at_basis saying how that time is known:

  • mtime: the file's modification time at ingest. The default. Copying a file without keeping its times, or unzipping it, changes that time, so pass --downloaded-at for such a file.
  • given: stated with --downloaded-at.

  A person's browser download is therefore hash-bound and timed, and the manifest says how sure the time is. The March 2026 manifest records unrecorded and no time. Its workbooks entered git in 11e41f9, so they were downloaded no later than 2026-07-21 21:06:50 +01:00, but nothing recorded when. Git keeps no file times, so the time is neither an mtime nor a given one. The parser requires mtime or given when there is a downloaded_at, and unrecorded when there isn't.

  --dry-run writes nothing. It checks everything and prints the titles, hashes, sizes, download times with their basis, and every target's change. It also checks what the real ingest would refuse: an ingested vintage, an earlier ingest's leftovers, or a download page missing or not on obr.uk. It prints each as "would refuse: …", returns them, and exits 1. A real ingest raises exactly those refusals. Any other problem (a title, a year, a missing target) raises in a dry run as in a real ingest.

  On the active vintage it says whether the bytes match the committed workbooks. A dry run of the active vintage on the bytes its manifest names is a rehearsal of the Budget-day path, so that manifest is not a refusal:

  uv run python -m policyengine_uk_data.targets.obr_sources ingest --vintage march_2026 --label "March 2026" \
    --receipts ~/Downloads/<receipts>.xlsx --expenditure ~/Downloads/<expenditure>.xlsx \
    --receipts-url https://obr.uk/download/march-2026-economic-and-fiscal-outlook-detailed-forecast-tables-receipts/ \
    --expenditure-url https://obr.uk/download/march-2026-economic-and-fiscal-outlook-detailed-forecast-tables-expenditure/ \
    --first-year 2024 --last-year 2030 --dry-run

  Run on copies of the committed March workbooks, it:

  • reports both as "the same bytes as the committed March 2026 workbook";
  • lists 175 unchanged target-years;
  • says the manifest "already names these bytes: a rehearsal, nothing to write";
  • exits 0.

  Without the two URLs it prints "would refuse: give the obr.uk https page each workbook was downloaded from (--receipts-url, --expenditure-url)" and exits 1. With other bytes under the March name it would refuse the ingested vintage.

  Adopting is then a one-line sources.yaml change. verify checks the active manifest.

• Removed: the download, retry and fallback path (_download_workbook, _fallback_workbook) and its tests.

Effect on the data

None. On the committed March 2026 bytes, all 29 OBR targets are identical before and after the change: names, variables, every value, vintage and reference URL. The 25 workbook targets plus 4 static ones were compared directly, and test_header_read_years_match_the_fixed_columns_on_march_2026 pins the year mapping against the old column letters. Calibration inputs are unchanged, so no release numbers move. This round's changes leave the 25 workbook targets (175 target-years) identical, field by field, to those built at the previous head, f1faff2f.

Invariants (tested)

• Bytes. Targets are built only when both workbooks match their manifest's SHA-256, size and title.
  • Hypothesis: any flipped byte, truncation or appended bytes is refused.
  • A file with matching hash but another vintage's title is refused.
• Years follow headers. Hypothesis over header row, start column, first year, length and column order: values are keyed by their header's year, and declaring the range a year early or late fails.
• Coverage.
  • Hypothesis: deleting any one year of any target, or any required target, produces exactly that problem.
  • A renamed row, a formula in a value cell and a lost self-employed line each fail.
  • A built target missing from the manifest fails.
• Differential. The header-read year columns equal the old fixed letters on every March 2026 table, and representative targets equal the values read with those letters.
• Ingest round trip. Ingesting the March 2026 workbooks into an empty storage and manifests folder reproduces every target value.
  • A label the workbook titles don't carry is refused.
  • So are years the tables don't head, leaving no files behind.
  • So is the HTML page obr.uk serves to scripts.
  • A real ingest without the download pages is refused, and so is one whose page is off obr.uk.
  • A write that fails while the workbooks or the manifest are being moved into place leaves no half-written file under a real name. If the manifest write fails after the workbooks are in place, their complete folder stays under its final name with no manifest, and the retry refuses it and names it.
• Dry run parity.
  • A dry run with leftovers and a page off obr.uk returns both refusals, still checks and diffs every target, and writes nothing. The real ingest then raises exactly "ingest: " + "; ".join(refusals).
  • An ingested vintage that isn't the active manifest is a refusal.
  • On the active manifest, a dry run on its own bytes has no refusals. A re-saved receipts workbook (same title, other bytes) is refused, as a real ingest refuses any ingested vintage.
  • The CLI rehearsal exits 0 with the URLs, and 1 with "would refuse:" without them.
  • These tests and the CLI rehearsal run on a copy of the committed manifest and workbooks, made the active one, so none can touch the committed files.
• Download pages.
  • Hypothesis over hosts built from labels and suffixes: a page is accepted if and only if it is https and its host is obr.uk or ends in .obr.uk.
  • 20 examples, among them obr.uk.example.com, notobr.uk, obr.uk@example.com, example.com\.obr.uk, example.com\@obr.uk, user:password@obr.uk, obr.uk:8443, -.obr.uk and http://obr.uk.
• Manifest round trip.
  • Hypothesis over basis (mtime, given, unrecorded) and offset-aware times: a manifest's YAML parses back to the same files and targets.
  • A dry run returns the manifest a real ingest writes and reads back.
  • Five basis/time mismatches are refused.
• Units and titles (examples):
  • 14 spellings of a row-level unit are each refused, and so are 4 sub-headings giving another unit;
  • 5 sub-headings restating the table's own unit leave the row's values unchanged, and another unit in the same cell is refused;
  • 5 plausible title formats are accepted;
  • titles naming another vintage, report or kind, or not "detailed forecast tables" ("Economic and fiscal outlook – October 2026: supplementary fiscal tables – receipts"), are refused;
  • a target built twice fails coverage.

Interactions with open PRs

Sol's review checked these with git merge-tree:

• Stop calibrating OBR total NICs on ni_employee #537 (stop calibrating obr/ni on ni_employee) conflicts in a test, and removes a target this manifest requires. Whichever lands second must edit targets.required; coverage fails until it does, which is the intended check.
• Calibrate DWP and OBR welfare targets on the countries their source covers #542 (GB scope) conflicts in imports; its geography changes keep the workbook loading and vintage.
• Calibrate households' SDLT to the OBR total less corporate SDLT #569 (SDLT less corporate) conflicts in the parser signature. Its corporate_sdlt_receipts() defaults to _FY_COL_TO_YEAR.values(), which this PR removes. Use active_manifest().years instead (noted on Calibrate households' SDLT to the OBR total less corporate SDLT #569).

Tests

Run serially:

• test_obr_sources.py at this head (133632af): 102 passed.
• The other OBR test files (test_obr_council_tax.py, test_obr_nic_signal.py, test_obr_nics.py, test_obr_receipts_sheet.py, test_obr_universal_credit_target.py) and the two OBR fixture files test_esa_jsa_targets.py and test_dwp_housing_benefit_targets.py: 42 passed, 6 skipped. The skipped tests need the enhanced FRS dataset, which isn't built locally.
• test_target_registry.py, which builds every target, and test_changelog_fragment_check.py: 105 passed.

The last two runs were at 159db44e, which differs from this head only in a docstring.

The full suite (pytest policyengine_uk_data/tests, no built datasets locally) last ran at the previous head, f1faff2f: 2244 passed, 131 skipped, 1 xfailed, 1 failed. The failure was test_target_registry.py::test_savings_interest, a live ONS download that failed during a network outage. It passes on re-run, and it passed in the registry run above. The full suite has not been re-run at this head.

axiom: n/a: calibration-target ingestion infrastructure; no policy change

🤖 Generated with Claude Code