feat(scripts): refresh a vendored host tree from its source repository

[TarikGul]

The host trees under hosts/ are snapshots and their sources keep moving. A
refresh runs at cutover, then for as long as a source repository keeps a release
line of its own. It was a procedure nobody had written down.

scripts/refresh-host-import.sh status ios
scripts/refresh-host-import.sh refresh ios

Both ways it goes wrong have already happened, so both are encoded. A tree
replace drops work in one direction or the other, so the new tree is taken whole
and adaptations reapplied as a three-way patch. git add honours ignore rules
the source never had, which cost an earlier import 57 files, so paths are staged
with --force and every one is compared against the source by blob hash. A
difference no adaptation accounts for is reported as unexplained, and conflicts
are left uncommitted.

hosts/imports.json records the source and revision per host. status is what
confirms the recorded ones are right: Android differs from its source only by the
.gitignore adapted at import, iOS by the workflows moved to the root plus the
manifest and signing adaptations.

Part of #726.

[github-actions]

CI Status: 18 required jobs green, 12 passed and 6 skipped by path filter.

All job results

job	result
android-bindings	skipped
changes	success
changeset-guard	skipped
cli-package	skipped
codegen	success
e2e	skipped
explorer	success
ios-bindings	success
ios-swift	skipped
licenses	success
playground	success
release-guard	success
rust	success
ts-client	success
ts-debugger	success
ts-host	success
wasm-provider	success
workflow-lint	skipped

_{Commit 9ba806fc · run log}

[TarikGul]

@lore-bot-app review

[lore-bot-app]

Reading the diff and checking what the record says. Back in a few minutes.

[lore-bot-app]

TL;DR
Adds scripts/refresh-host-import.sh plus hosts/imports.json so hosts/ios and hosts/android can be re-imported from upstream with this repo's adaptations re-applied as a three-way patch. 3 blocking, 4 minor. As written, refresh aborts on its first git call, so the refresh path cannot have run end to end.

Summary
The script records each vendored host's source repo, branch and revision in hosts/imports.json. status fetches the recorded revision and compares every tracked path under hosts/<host> against it by blob hash. refresh builds the recorded upstream tree under the hosts/<host>/ prefix, diffs it against HEAD to extract this repo's adaptations, replaces the tree with the new upstream revision via git archive, force-stages it, re-applies the adaptations with git apply --3way, compares the result against upstream, and updates the manifest. A Python helper does the NUL-delimited comparison and fails when a difference has no adaptation behind it. README and CLAUDE.md gain layout entries and a usage section.

What the record says

• Issue #726 prescribes this exact procedure: merge rather than tree replace, three-way patch, blob-hash verification, confirm no files dropped by ignore rules. It was rehearsed by hand with 161 commits of drift. This PR is the automation of that.
• PR #703 is the 57-file incident. The rules were token* and *.js in the vendored tree's own hosts/android/.gitignore, fixed by anchoring them. The README's "this repository's ignore rules are not the source's" is slightly off: they were the source's rules, which in the source never applied because the files were already tracked.
• PR #774 refreshed hosts/ios by 200+ commits by hand, with conflicts in the network suffix logic and Package.resolved resolved manually. It is not on this branch. See question 1.
• PR #690 imported iOS at develop 844e1b9, and PR #703 imported Android. Both match the manifest.
• PR #787 sets the iOS cutover for Thursday 17 September, so this script is expected to run tomorrow.
• docs/design/host-monorepo.md:173 says the import commit is where the source revision is recorded. The manifest is now a second record and the design doc does not mention it.
• Tarik Gul owns this area per who_knows; they are the author.

Concerns

Blocking:

1. scripts/refresh-host-import.sh:82-83. mktemp creates a 0-byte file, and git read-tree with GIT_INDEX_FILE pointing at an existing empty file dies with "index file smaller than expected". Git only tolerates a missing index file. With set -e every refresh stops here before touching the tree. Fix: rm -f "$index" before line 83, or mktemp -u. I could not execute in this environment. Check with:

   GIT_INDEX_FILE=$(mktemp) git read-tree --prefix=x/ HEAD

2. scripts/refresh-host-import.sh:198. After a conflicted --3way apply, git add --all --force clears the unmerged index entries and stages the conflict markers as content. git status then shows ordinary staged changes, git commit accepts them, and scripts/lib/compare-host-import.py:81 counts the path as explained because it is in the adapted list. The README's "left staged, a conflict is a human decision" does not hold. Fix: run line 198 only when apply succeeded, or exclude git diff --name-only --diff-filter=U paths and print them.
3. scripts/refresh-host-import.sh:193-197 with scripts/lib/compare-host-import.py:81. git apply is all-or-nothing when any single patch is rejected outright rather than sent to 3-way: a binary hunk from a diff made without --binary, or an adapted path that upstream has since deleted or renamed ("does not exist in index"). Nothing is applied, line 196 says "conflicts left", the tree equals upstream, and the comparison prints "every difference is accounted for" because the check is one-sided. It flags a difference with no adaptation but never an adaptation with no difference. The current iOS adaptation set has 32 paths, none binary and none renamed, measured from this checkout's root, so it is latent today. One upstream rename of a workflow file we delete triggers it. Fixes: git diff --binary at line 177, apply per adapted path so one rejection does not drop the rest, and report adapted paths whose blob now equals upstream.

Minor:

4. hosts/imports.json:4,9. Abbreviated SHAs are not valid fetch refspecs, so line 71 always fails and line 72 fetches the remote HEAD wholesale. That works only while the recorded commit is reachable from the remote's default branch. A force-push or a non-default branch makes refresh die at line 75. manifest_set_ref writes full SHAs, so formats mix after the first run. Record full SHAs now.
5. scripts/refresh-host-import.sh:186. rm -rf deletes ignored developer files: hosts/android/local.properties (hosts/android/.gitignore:10), hosts/ios/source_packages and .build/ (hosts/ios/.gitignore:35-36). The dirty check at line 154 does not see ignored files. git rm -r -q hosts/<host> then git read-tree --prefix=hosts/<host>/ -u <target>^{tree} replaces lines 186-189, touches only tracked files, bypasses ignore rules without --force, and avoids git archive, which would drop any future upstream export-ignore path (none today). AGENTS.md: leave unrelated local changes alone.
6. README.md:265-288. The new section sits between the ### Working on the iOS host heading and its body. That heading is now empty and the iOS build steps appear under "Refreshing a vendored host tree". Also point docs/design/host-monorepo.md:173 at the manifest.
7. scripts/refresh-host-import.sh:10-20 and README.md:280-282 narrate history ("both have already happened", the 57 files). CLAUDE.md: comments describe current code only, no migration or history narration. #703 already holds the story. One line stating the rule is enough.

Questions for the author

1. #774 moves hosts/ios about 200 commits past 844e1b99 and is not on this branch. Which PR updates the iOS ref? If #774 lands first, the next refresh treats those 200 upstream commits as adaptations.
2. compare-host-import.py is the first Python file in scripts/lib/. Every other helper is .mjs with a sibling test that npm run test:scripts runs (package.json:17). Is Python deliberate, and where does its test live?
3. Has refresh run end to end against real drift, for example replaying #774's range? The commit message's "which files differ" claim is obtainable from status alone.

Next: run the one-liner in concern 1.

---

🤖 Reviewed by Lore (Parity knowledge base) · 47 agent turns · 633.0s · knowledge as of 2026-09-16

[TarikGul]

All seven addressed in 6fdac54. Three were real breakage and I had not caught
them because I only ever ran status, which does not touch the refresh path.

The empty index. Confirmed exactly as described: GIT_INDEX_FILE pointing at
an existing zero byte file gives "index file smaller than expected", so every
refresh died before touching the tree. mktemp then rm -f before the
read-tree.

Staging a conflicted apply. Correct, and it defeated the whole claim. Staging
now happens only when the apply succeeded. On conflict the unmerged paths are
listed and left unmerged, so git refuses the commit.

All or nothing apply. The patch is made with --binary, and a failed apply
with no unmerged paths is now a hard error rather than a note, since that is the
case where the tree silently becomes plain upstream with every adaptation gone.
The comparison also checks the other direction now: an adaptation that left no
difference is reported. It is not fatal, because the source adopting an
adaptation produces the same signal, and #774 is a live example of that.

rm -rf taking ignored files. Replaced with git rm -r then
git read-tree --prefix -u, as suggested. Verified by planting a
hosts/android/local.properties, which git status --porcelain does not see,
and running a refresh: it survives.

Abbreviated SHAs. Both now full.

The orphaned heading. Correct, the section split ### Working on the iOS host from its body. Moved above it. The design doc now points at the manifest.

History narration. Removed from the header and the README. The rule is
stated without the incident.

A real refresh of hosts/android now runs end to end: 25 commits forward, 498
files staged, the one .gitignore adaptation reapplied cleanly, 0 missing and 0
extra against the source.

Not done: per-path apply so one rejection cannot drop the rest. The outright
rejection error covers the case where it matters, and per-path apply loses the
three-way merge base that makes the rest of this work. Worth revisiting if a
rename ever actually trips it.

[TarikGul]

@lore-bot-app review

[lore-bot-app]

Reading the diff and checking what the record says. Back in a few minutes.

[lore-bot-app]

Since last review

• Resolved: empty-index crash (rm -f "$index" at scripts/refresh-host-import.sh:84), staging after a conflicted apply (now only on success, :193-195), --binary on the diff (:178), full SHAs in hosts/imports.json, git rm + read-tree instead of rm -rf (:188-189), README section placement, design doc pointer, trimmed history narration.
• Partly resolved: all-or-nothing git apply. The outright-rejection case is now detected and dies (:201-205), and inert adaptations are reported (compare-host-import.py:86-91). There is still no way to complete a refresh once one adapted path was deleted or renamed upstream.
• Still open: iOS ref vs #774, Python helper without a test.
• New: one blocking problem introduced by the rm -rf fix, plus two minor ones.

TL;DR
Automates the host-tree refresh from #726 with a manifest, a three-way re-apply and a blob-hash comparison. 1 blocking, 3 minor.

Summary
hosts/imports.json records source, branch and revision per vendored host. status fetches the recorded revision and compares every tracked path by blob hash. refresh builds the recorded upstream tree under hosts/<host>/, diffs it against HEAD to extract this repo's adaptations, replaces the tree with the target revision via git rm and git read-tree -u, re-applies the adaptations with git apply --3way, compares the result against the target in both directions, and records the new revision. Conflicts are left unmerged. A patch rejected outright aborts.

What the record says

• Issue #726 prescribes exactly this procedure: three-way patch rather than tree replace, blob-hash verification, and a check that no file is dropped by ignore rules. This PR is that automation.
• PR #811 (this PR) records an end-to-end refresh of hosts/android with 498 files staged and no missing or extra paths against the source. That answers the earlier "has it run" question for Android. No iOS run is recorded.
• PR #774 refreshed hosts/ios by hand past 844e1b99 and is described as complete with CI passing. It is not in this branch: git log -- hosts/ios here ends at #706, and no commit references #774. Whichever lands second must update the iOS ref, or the next refresh treats 200 upstream commits as adaptations.
• PR #703 is the 57-file drop caused by the vendored hosts/android/.gitignore. The read-tree path now sidesteps it for tracked content, which is the right fix. The --force that the earlier revision used to work around it is what causes the concern below.
• Tarik Gul owns this area per who_knows and is the author.

Concerns

Blocking:

1. scripts/refresh-host-import.sh:195. git add --all --force hosts/<host> after a clean apply stages every ignored file under the tree. --force exists to add otherwise-ignored files, and git apply --3way already updated the index, so the line adds nothing tracked. What it does add is whatever the developer has locally: hosts/ios/.build/ and source_packages (hosts/ios/.gitignore:35-36), hosts/android/.gradle/ and build/ (hosts/android/.gitignore:2-4), and the secret patterns polkadot-app/env-vars.sh, GoogleService-Info.plist (hosts/ios/.gitignore:89-91), /token*, .env* (hosts/android/.gitignore:56-57). The comparison lists them under "here but not in the source" but does not fail, and the script's last line still says "review it, then commit". In a public repo that is a credential leak one git commit away. Fix: delete line 195. The recorded Android run saw no extras because that checkout had no ignored files under hosts/android at the time.

Minor:

2. scripts/refresh-host-import.sh:193,205. Detecting the outright rejection is an improvement, but a single adapted path that upstream deleted or renamed, or a pure-rename adaptation (3-way declines renames with no content change), now makes refresh unusable for that host with no knob to continue. The die also leaves the plain upstream tree staged in the index with no recovery hint. At minimum print git reset --hard HEAD in the message. A --skip <path> option, or applying the patch per file so one rejection does not block the rest, would let the operator finish.

3. scripts/refresh-host-import.sh:214. The comparison's exit status is discarded and the final lines always say the refresh is ready for review. The "inert adaptation" and "UNEXPLAINED" verdicts scroll past in the middle. Either propagate the status or print the verdict after the manifest note, so the last line of output is the one that matters.

4. scripts/lib/compare-host-import.py. Still the only Python under scripts/lib/ and the only helper there without a test, while npm run test:scripts covers every .mjs sibling (package.json:17, ci.yml:482). The inert and unexplained branches are the logic the whole procedure relies on and are cheap to cover with two fixture files.

Questions for the author

1. Which PR updates the iOS ref in hosts/imports.json after #774 lands, and to which upstream commit?
2. Of the iOS adaptation paths, has any been deleted or renamed on upstream develop since 844e1b99? If so, the first iOS refresh hits concern 2 immediately.
3. Is Python deliberate for the comparison helper, given the .mjs plus test convention in scripts/lib/?

Next: delete scripts/refresh-host-import.sh:195 and re-run the Android refresh with an ignored file placed under hosts/android/ to confirm it stays unstaged.

---

🤖 Reviewed by Lore (Parity knowledge base) · 36 agent turns · 415.5s · knowledge as of 2026-09-16 · re-review

[TarikGul]

All three fixed.

The forced add. You are right and it is the serious one. --3way has
already updated the index for every path it touched, so the line did no work,
and what it did do was stage everything ignored under the tree, which on these
hosts includes GoogleService-Info.plist, env-vars.sh, /token* and .env*.
Deleted.

Verified rather than reasoned about: planted hosts/android/.env.local and
hosts/ios/polkadot-app/env-vars.sh, confirmed both are ignored and invisible
to git status --porcelain, then ran a real refresh. Neither is staged, and the
504 staged paths are all tracked content.

Recovery hint. The outright rejection message now ends with
git reset --hard HEAD, since the die leaves the upstream tree staged.

The verdict scrolling past. Moved after the manifest note and made fatal. A
refresh that drops work no adaptation accounts for now dies rather than
finishing with "review it, then commit".

Not done: --skip or per-file apply. The recovery hint covers getting out, and
per-file apply loses the merge base the rest depends on. Worth doing when a
rename actually trips it rather than in anticipation.

[filvecchiato]

approving to unblock, but small nit

[TarikGul]

Fixed, and it is a good catch: I moved the listings to -z in the first round
and left the adaptation list on --name-only, so the two sides disagreed about
exactly the path the -z comment was written for.

Confirmed there is precisely one non-ASCII tracked path under hosts/:

raw    hosts/ios/polkadot-app/Assets.xcassets/color/assetUSDС.colorset/Contents.json
quoted "hosts/ios/polkadot-app/Assets.xcassets/color/assetUSD\320\241.colorset/Contents.json"

Took the -z option rather than core.quotePath=false, since the rest of the
comparison is already NUL delimited and a literal newline in a path would still
defeat the other one.

Verified by running the comparison against that path both ways: with the quoted
list it exits 1 and the refresh would die on "dropped work"; with the NUL list
it exits 0 and the difference is accounted for. A real hosts/android refresh
still ends clean, 0 missing and 0 extra.