ci: build only the Zephyr boards that have the changed module - #11316
ci: build only the Zephyr boards that have the changed module#11316lynt-smitka wants to merge 7 commits into
Conversation
|
Now has merge conflicts. |
ci_set_matrix.py cannot ask make which Zephyr boards compile a module, so a change to any module built all 29 of them, and a frozen library update too although the port has no frozen modules. The module table of a Zephyr board is only known to its build, which writes it to autogen_board_info.toml. Keep it: build_release_files.py copies the file the push build generated to bin/zephyr-modules/<ref>/ <board>.toml, from where the existing bin/ upload puts it on S3 next to mpy-cross and the stubs. The scheduler fetches it for the PR's base ref and builds a board when the table says the module is enabled, or when the table is missing, unreadable or does not know the module. frozen/ changes build no Zephyr boards; supervisor/ changes still build all. Until the first push build after this lands there is no table and every Zephyr board is built, as today. ZEPHYR_MODULES_URL overrides the location for testing.
97ad665 to
05384ce
Compare
|
Rebased, it should works now. |
|
I'd rather not push an up to date copy to S3 because it puts state in another place. Instead, just base this on the checked-in version. Stale for a little while will be fine. To keep the checked in version updated, we can add an action triggered by main merges that rebuilds them and makes a PR. They'll feel similar to the translate PRs and be easy to review. |
The board's build already commits autogen_board_info.toml and its header says other scripts use it, so read that instead of putting a fresh copy on S3. build_release_files.py is untouched again. The table now decides whether a board is built, so a stale one costs coverage instead of nothing. The build's warning about that sits in one board job's log among 29; as an annotation it shows on the run and next to the file in the pull request. The board builds once per language, so only its first build annotates. The scheduler also names the boards it left out.
# Conflicts: # tools/ci_set_matrix.py
Flips synthio to false although the board has audiobusio. To be reverted once the run shows what a stale table looks like in a pull request.
|
Done - it reads the committed Originally I picked S3 because only the build knows a board's module set, and the committed file can go stale. I tried three other ways:
S3 also refreshed itself on every push. The committed copy doesn't. In practice the tables are regenerated in the same commit that changes the modules, so it rarely matters. A stale So the warning is an annotation now, and the scheduler names the boards it left out: The last commit makes native_sim's table stale on purpose to test what that looks like. I'll revert it after the run. |
This reverts commit 341055d.
|
Example of the warning annotation from the test run: https://github.com/adafruit/circuitpython/actions/runs/34487178939/job/102905278441 |
tannewt
left a comment
tannewt
left a comment
There was a problem hiding this comment.
Thanks for the PR. Please simplify this new code.
One predicate per board, zephyr_board_has_module(), and a loop at the call site instead of set algebra and a helper returning a list. Same boards selected for a module, frozen and supervisor change.
tannewt
left a comment
tannewt
left a comment
There was a problem hiding this comment.
I still think it's a bit more complicated than it needs to be.
No sorted(), board_to_port[] for the port check, and no log of the boards left out.
3 participants
ci_set_matrix.py cannot ask make which Zephyr boards compile a module, so any module change builds all 29 of them, and so does a frozen library update although the port has no frozen modules.
The module table of a Zephyr board is only known to its build, which writes autogen_board_info.toml; the committed copy can lag, CI only warns when it is out of date. So build_release_files.py copies the table the push build generated to bin/zephyr-modules//.toml, the existing bin/ upload puts it on S3 next to mpy-cross and the stubs, and the scheduler fetches it for the PR's base ref. A board with a missing table is built. Since the module set only changes through files that build every Zephyr board anyway, the tables refresh themselves on the next push.
Until the first push build after this lands every Zephyr board is built, as today.