The coreml_static_int8 half of test-backend-macos (operators suite) has never completed. It runs to the 180 minute job cap and gets cancelled, so it produces no signal at all.
Measured on a recent main commit:
| flow / suite |
duration |
| coreml / models |
12 min |
| coreml / operators |
11 min |
| coreml_static_int8 / models |
25 min |
| coreml_static_int8 / operators |
181 min, cancelled at the cap |
Both flows are built by the same _create_coreml_flow in backends/test/suite/flows/coreml.py. coreml_static_int8 is the identical models + operators suite with quantize=True and minimum_deployment_target=iOS17. So the cost is a full PT2E prepare / calibrate / convert pass on every single test case, not larger weights.
#21609 moved this flow to the nightly schedule, so it no longer consumes the shared macOS pool on every commit. That limits the damage but does not fix it: the nightly run will keep hitting the same cap, and the int8 flow will keep reporting nothing.
Possible directions, roughly in order of effort:
- Shard the operators suite across several jobs so each one finishes inside the cap.
- Reuse a single quantized artifact across cases where the operator under test allows it, instead of quantizing per case.
- Trim the int8 case list to the operators where quantization actually changes the lowering.
Until one of those lands, the int8 operators coverage should be treated as absent rather than passing.
The
coreml_static_int8half oftest-backend-macos(operators suite) has never completed. It runs to the 180 minute job cap and gets cancelled, so it produces no signal at all.Measured on a recent main commit:
Both flows are built by the same
_create_coreml_flowinbackends/test/suite/flows/coreml.py.coreml_static_int8is the identical models + operators suite withquantize=Trueandminimum_deployment_target=iOS17. So the cost is a full PT2E prepare / calibrate / convert pass on every single test case, not larger weights.#21609 moved this flow to the nightly schedule, so it no longer consumes the shared macOS pool on every commit. That limits the damage but does not fix it: the nightly run will keep hitting the same cap, and the int8 flow will keep reporting nothing.
Possible directions, roughly in order of effort:
Until one of those lands, the int8 operators coverage should be treated as absent rather than passing.