Fix test-samsung-models-linux: accelerate dep and the datasets/torchvision collate

[psiddh]

Fixes test-samsung-models-linux, red on main since the job started running on Sep 14.

The bug

test_mobilebert_qat_a8w8 constructs a transformers TrainingArguments, which requires
accelerate>=1.1.0. The job installs only evaluate, so:

ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=1.1.0`

It is the sole error in the suite, identically, on every affected commit:

Commit	Result
f3c08fcf35	Ran 12 tests ... FAILED (errors=1)
4798a676c3	Ran 12 tests ... FAILED (errors=1)
500849ba5b	Ran 12 tests ... FAILED (errors=1)
cfc96cc0e2	Ran 12 tests ... FAILED (errors=1)
026ca3fff3	Ran 12 tests ... FAILED (errors=1)

accelerate appears nowhere in the job logs, so it is absent rather than stale — but the pin is
still the right form, because transformers raises the same error for a too-old version.

Why it matters beyond the test

The non-zero exit propagates: unittest exits 1 → the script runs under set -ex →
run_with_env_secrets.py raises → the job fails → the trunk workflow concludes failure. trunk
is in update-viablestrict's requires list, so the commit is disqualified and viable/strict has
not advanced since Sep 14. This is one of two blockers; the other is the parallel_for empty-range
bug, fixed separately in #22869. Both are needed — each independently reddens trunk.

Verification

Honest scope: I verified the diagnosis, not the fix. The sole error is identical across five
commits and accelerate is never installed in the job. I cannot run this job — it needs the
Samsung device and SAMSUNG_AI_LITECORE_KEY — so the one-liner itself is untested. trunk.yml
parses and lintrunner is clean.

One note for reviewers: the [CDI: ERROR] only support axis as 1 / OP LOGSOFTMAX lines in the log
appear after FAILED (errors=1). They are teardown noise from a passing test, not a second
failure.

cc @digantdesai

---

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22870

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4750a57 with merge base 1f4ea61 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

🟢 Approval recommended

The required test dependency is added with no unresolved review issues.

Pull request overview

Adds the missing accelerate>=1.1.0 dependency required by Samsung MobileBERT QAT tests.

Changes:

• Installs accelerate alongside evaluate.
• Documents the dependency requirement.

File summaries

File	Description
.github/workflows/trunk.yml	Updates Samsung model-test dependencies.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[psiddh]

Update: fix confirmed working, but a second failure was hiding behind it

The ciflow/trunk run exercised this for real. The accelerate ImportError is gone — the test
now gets past TrainingArguments, into trainer.train(), and all the way into the dataloader
before failing on something else:

datasets/formatting/torch_formatter.py:80, in _tensorize
    from torchvision.io import VideoReader
ImportError: cannot import name 'VideoReader' from 'torchvision.io'

So the diagnosis in this PR was right and the change is still required — it just isn't the whole story.

The second issue

Not a version-pin problem. From torchvision's own io/__init__.py:

# In fbcode, import from the fb-only location
# For OSS, these imports would fail (video_reader not available)
try:
    from pytorch.vision.fb.io import (..., VideoReader)
except ImportError:
    pass

VideoReader exists only in Meta-internal torchvision builds; in OSS torchvision 0.29 it silently
does not exist. Meanwhile datasets does this unconditionally inside its guard:

if config.TORCHVISION_AVAILABLE and "torchvision" in sys.modules:
    from torchvision.io import VideoReader      # assumes it exists

where TORCHVISION_AVAILABLE = importlib.util.find_spec("torchvision") is not None. I checked
datasets 3.6.0, 4.0.0, 4.1.0, 4.2.0 and 4.3.0 — all of them do this, so upgrading datasets
does not help. torchvision is in sys.modules because timm (from requirements-examples.txt)
imports it.

The trigger on our side is examples/samsung/scripts/mobilebert_finetune_QAT.py:143:

tokenized_datasets.set_format(type="torch", columns=["input_ids", "attention_mask", "label"])

Options, none of them obviously right

1. Drop set_format(type="torch", ...) and let the Trainer's collator convert. Most surgical, but
   the script also builds raw DataLoaders from tokenized_datasets (get_dataset, ~line 250)
   that may depend on torch formatting — needs someone who knows this script.
2. Pin torchvision <= 0.25, the last release carrying OSS VideoReader. Conflicts with the
   image's torch nightly; heavy.
3. Skip test_mobilebert_qat_a8w8 in CI. It is a ~3h HF finetune inside a required workflow, which
   is arguably the real problem.

I'd lean 1, or 3 if the runtime cost is itself unwanted — but this is an owner call, so I have not
guessed at it. @digantdesai

Correction

I earlier stated that this PR plus #22869 were "necessary and sufficient" to unstick viable/strict.
That was wrong on this half — it held for the two persistent blockers I could see, but this third
failure was invisible behind the accelerate error. #22869 is unaffected and still verified.

[psiddh]

Forward fix pushed (13dfd56) — no skip needed

Rather than skipping test_mobilebert_qat_a8w8, this now fixes the second failure properly.

datasets takes this branch whenever torchvision merely imports:

if config.TORCHVISION_AVAILABLE and "torchvision" in sys.modules:
    from torchvision.io import VideoReader

TORCHVISION_AVAILABLE is just find_spec("torchvision") is not None, and timm puts torchvision
in sys.modules. The import then fails because OSS torchvision has no VideoReader — its
io/__init__.py only gets it from the fb-internal path:

# For OSS, these imports would fail (video_reader not available)
try:
    from pytorch.vision.fb.io import (..., VideoReader)
except ImportError:
    pass

This dataset is text only — input_ids, attention_mask, label — and never decodes an image or
a video, so that branch is dead weight. Turning the flag off before set_format keeps the tensors
and skips it.

Why not the alternatives

• Skip the test. Loses real MobileBERT QAT coverage for an upstream bug that isn't ours.
• Upgrade datasets. Doesn't help — I checked 3.6.0, 4.0.0, 4.1.0, 4.2.0 and 4.3.0; all do the
  same unguarded import.
• Drop set_format(type="torch"). I initially thought this was the surgical option; it isn't.
  get_dataset does batch["label"].tolist() and the QAT loop does batch["input_ids"].to(device),
  both of which need tensors, so it would mean writing a collate_fn for two call sites.
• Pin torchvision ≤ 0.25. Conflicts with the image's torch nightly.

Verification

Reproduced locally against datasets==3.6.0 with a torchvision whose io has no VideoReader,
matching CI:

[1] reproduce the CI failure
  as-is:    ImportError: cannot import name 'VideoReader' from 'torchvision.io'
[2] forward fix: datasets.config.TORCHVISION_AVAILABLE = False
  with fix: OK -> input_ids torch.int64 (1, 3); label [1]

So the exact error is reproduced and cleared, and the formatter still produces the int64 tensors the
DataLoaders need. The full job is what settles it — ciflow/trunk is running.

Both commits here are needed: accelerate gets the test to trainer.train(), and this gets it
through collation.

[Copilot]

🔵 Needs a closer look

Scope or restore the process-wide torchvision configuration mutation.

Review details

Suppressed comments (1)

examples/samsung/scripts/mobilebert_finetune_QAT.py:151

• This mutates datasets.config.TORCHVISION_AVAILABLE process-wide and leaves it disabled after load_CSV_dataset returns. Any later caller in the same process that uses a datasets image/video column will silently lose torchvision decoding, and concurrent dataset consumers can race with this flag. Please scope the workaround to this pipeline's DataLoader/Trainer lifetime (or otherwise avoid a global config mutation) and restore the prior value during cleanup.

        datasets.config.TORCHVISION_AVAILABLE = False

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[psiddh]

Both fixes confirmed working — test_mobilebert_qat_a8w8 now passes

The ciflow/trunk run on 13dfd56 cleared the whole MobileBERT chain. VideoReader appears zero
times in the log and the test no longer errors.

The job is still red, but on a different test:

FAIL: test_mv2_fp16 (test_mobilenet_v2.TestMilestoneMobilenetV2)
AssertionError: Output 0 does not match reference output.
    Given atol: 0.02, rtol: 0.001.
    Difference: max: 0.02465033531188965, mean abs error: 0.00578

A numerical tolerance miss on the NPU, 23% over atol — not an import or dependency problem.

Is this fallout from these commits?

History across every run I have logs for:

Run	test_mobilebert_qat_a8w8	test_mv2_fp16
6 runs before these fixes	ERROR	pass
this run	pass	FAIL

I can't call it unrelated flake outright. There is an indirect path: unittest discover runs
test_mobilebert_qat before test_mobilenet_v2, and mobilebert previously aborted early. It now
runs a full QAT finetune and device export first, so the NPU has done substantially more work by
the time mv2 runs. Nothing in these commits touches mv2's code, but what precedes it on the device
did change.

The alternative reading is simply that test_mv2_fp16 is marginal at atol=0.02 and this is its
first observed flake.

I have re-run the failed jobs to discriminate. If mv2 passes on the re-run, it is flake and these
commits are complete. If it fails again at a similar magnitude, the ordering/device-state
hypothesis needs investigating and test_mv2_fp16's tolerance is probably too tight for fp16 on
hardware.

cc @digantdesai

[psiddh]

Correction to an earlier comment of mine

I described test_mobilebert_qat_a8w8 as "a ~3h HF finetune inside a required workflow" and used
that to argue the job should be de-gated. The runtime figure was wrong.

	suite wall time
6 runs where mobilebert aborted immediately	10,301–11,582s (mean ~3.0h)
the run where it completed	12,047s (~3.35h)

The finetune costs roughly 20 minutes. The ~3 hours is the other 11 tests in the suite, which
were always running. My de-gating argument leaned on the wrong number and was weaker than I made it
sound.

For the record on cadence, since I had that vague too: this job does not run per-PR. trunk
only triggers on PRs touching .ci/docker/ci_commit_pins/pytorch.txt, .ci/scripts/** or
zephyr/** — it ran here because I applied ciflow/trunk. On main it is gated by
_ci-run-decision.yml's deterministic 25% depth sample; there were 3 trunk runs on main in the
last 24h.

A separate issue worth a look

While checking the above I noticed the job's device guard:

if [[ "${DEVICE_ACQUIRED:-0}" != "1" ]]; then
  echo "::warning::Skipping tests - no Samsung device available"
  exit 0
fi

When no Samsung device is free this exits 0 — the job reports success having run no tests, and
trunk is required by update-viablestrict. So viable/strict can advance on a commit where this
suite verified nothing. Filing separately if you'd prefer; flagging here since it came out of this
investigation. @digantdesai

[psiddh]

Re-run result: not flake — deterministic, and it is a test-isolation bug

The re-run failed identically. Not "similarly" — bit for bit:

run 1: max: 0.02465033531188965, abs: 0.029329299926757812, mean abs error: 0.0057808648757636544
run 2: max: 0.02465033531188965, abs: 0.029329299926757812, mean abs error: 0.0057808648757636544

A flaky fp16 hardware comparison would vary in the last digits. Identical output twice means the
computation is deterministic, so flake is ruled out.

Both fixes in this PR remain confirmed: test_mobilebert_qat_a8w8 passes, VideoReader appears
zero times, FAILED (failures=1) with test_mv2_fp16 the only failure.

What this means

test_mv2_fp16 passed in all six runs where mobilebert aborted early, and fails deterministically
in both runs where mobilebert ran to completion. The suite runs in a single process via
python -m unittest discover, and test_mobilebert_qat sorts before test_mobilenet_v2.

So the most likely explanation is that the MobileBERT QAT test leaves global state behind that
changes the eager reference computation for the test that follows it — it calls prepare_pt2e,
convert_pt2e, allow_exported_model_train_eval and move_exported_model_to_train, any of which
can mutate process-wide torch state. The assertion compares ExecuTorch output against an eager
reference, so perturbing the reference is enough to produce a fixed offset like this.

I have not proven that mechanism — I cannot run the Samsung device. What is proven is that it is
deterministic and correlates exactly with mobilebert completing.

Where that leaves this PR

The two commits here are correct and I would not change them. But they do not make the job green,
and I do not think this PR should be the place that fixes the isolation problem — that needs
someone who can run the hardware and can say whether the right answer is test isolation, a tolerance
change, or splitting the QAT test out of the suite.

@digantdesai — this is worth an Arm/Samsung owner. The narrow question: does
test_mobilebert_qat_a8w8 leave global torch state that perturbs test_mv2_fp16?

[psiddh]

Root cause found — test_mv2_fp16 depends on global RNG state

Proven from source, no device required. The chain:

1. MV2Model.get_example_inputs() is return (torch.randn(1, 3, 224, 224),) — an unseeded draw from the global torch RNG (examples/models/mobilenet_v2/model.py:28).
2. HF Trainer calls set_seed(self.args.seed) in both __init__ and train() (trainer.py:425, trainer.py:2080), default seed 42.
3. set_seed calls torch.manual_seed(seed) — the global RNG (trainer_utils.py:105).
4. unittest discover runs everything in one process, and test_mobilebert_qat sorts before test_mobilenet_v2.

So once mobilebert actually reaches Trainer, it reseeds the process RNG to 42, and test_mv2_fp16 subsequently draws a different input tensor than it used to. Different input, different NPU-vs-eager error, and the marginal atol=0.02 tips over.

This accounts for every observation:

Observation	Explained by
bit-identical failure across two runs	seed is fixed at 42, so the "different" input is the same every time
passed in all 6 runs where mobilebert aborted early	those aborted before Trainer, so the RNG was never reseeded
fails in both runs where mobilebert completed	Trainer ran, RNG reseeded
magnitude only just over tolerance (0.02465 vs 0.02)	a different random draw, not a systematic error

What this means

The commits in this PR did not break test_mv2_fp16. They exposed that it was depending on whatever RNG state happened to be left by the tests before it. Note also that the six previous passes each used a different random input, so atol=0.02 holds for most draws but not for the one seed 42 produces — the test was passing partly by luck.

The fix is not mine to pick

Two parts, and the second needs hardware:

1. Make the input deterministic — seed the RNG in the test (or in SamsungTester) so it no longer depends on execution order. Unambiguously correct hygiene.
2. Decide the tolerance. Once the input is pinned, atol=0.02 either holds for that input or it does not, and I cannot measure that without the NPU. Picking a seed that happens to pass would be cherry-picking; the honest version is to pin the seed and set a tolerance that holds across a few of them.

I have not pushed either, because part 2 determines part 1 and both need someone who can run the device. Happy to implement whichever you choose.

@digantdesai — this is now a concrete, fully-explained bug with a stable repro, rather than a mystery flake.

[psiddh]

Pushed the isolation fix (4750a57) and rebased onto current main

Third commit fixes the test_mv2_fp16 failure at its source, and the rebase clears the three
unittest-release failures this branch was carrying from before #22869 landed.

The fix

setUp snapshots Python, NumPy and torch generators; tearDown restores them. That fixes the test
that leaks rather than the one that noticed, and leaves test_mv2_fp16 seeing the same stream it
saw in the six runs where it passed.

Verified the mechanism locally — the device is not needed to show the leak or the repair:

baseline (no mobilebert)      : -3.4798994064331055
after Trainer set_seed(42)    :  2.7247946262359620   -> DIFFERS (bug reproduced)
with setUp/tearDown restore   : -3.4798994064331055   -> MATCHES baseline (fix works)

Why this test only started failing now

It did not regress. test_mobilebert_qat_a8w8 used to abort before ever reaching Trainer, so the
reseed never happened. The first two commits here fixed its dependencies, it began running to
completion, and the pre-existing leak became visible. The six historical passes and the two
bit-identical failures are all consistent with that and with nothing else.

Still open, and deliberately not addressed here

test_mv2_fp16 draws a fresh random input every run, so atol=0.02 holds for most draws and
not for others — it was passing partly on luck, independently of this PR. Pinning a seed that
happens to pass would be cherry-picking. That wants an owner with the hardware to choose a tolerance
that holds across several draws, or to seed the input deliberately. Worth a separate issue.

@digantdesai — the narrow ask on this PR is just a review of the three commits; the tolerance
question above is a pre-existing one I would rather not settle blind.

[Copilot]

🔵 Needs a closer look

CUDA RNG state is not restored on CUDA-capable runners, allowing the test to affect later tests.

Review details

Suppressed comments (2)

backends/samsung/test/models/test_mobilebert_qat.py:39

• The new comment has a grammatical error in the draw seed 42 produces lands just outside it, which makes the explanation harder to follow. Please rephrase this sentence.

    # unseeded torch.randn and compares against atol=0.02, and the draw seed 42
    # produces lands just outside it. Snapshot the generators here and restore

backends/samsung/test/models/test_mobilebert_qat.py:44

• The cleanup only snapshots/restores the CPU generator. get_finetune_mobilebert() selects CUDA when it is available, and Transformers' set_seed also reseeds CUDA generators, so on a CUDA runner this test still changes the CUDA RNG consumed by later tests. Snapshot torch.cuda.get_rng_state_all() in setUp and restore it in tearDown when CUDA is available.

        self._torch_rng_state = torch.get_rng_state()

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[psiddh]

Follow-up needed: test_mv2_fp16 uses an unseeded random input against a tight tolerance

Recording this here so it is not lost. It is not caused by this PR and is not fixed by it —
this PR only restored the RNG stream the test used to see. Leaving it written down because the
next change that perturbs global RNG will break it again the same way.

The problem

backends/samsung/test/models/test_mobilenet_v2.py does:

example_input = MV2Model().get_example_inputs()   # -> (torch.randn(1, 3, 224, 224),)
...
.run_method_and_compare_outputs(inputs=example_input, atol=0.02)

MV2Model.get_example_inputs() (examples/models/mobilenet_v2/model.py:28) returns an unseeded
torch.randn. So every run compares NPU output against the eager reference on a different input,
with a fixed atol=0.02 and rtol=0.001.

That makes the test's outcome a function of (a) whatever RNG state the preceding tests left behind
and (b) luck.

Evidence it is marginal rather than robust

Runs	Input	Result
6 runs before this PR	fresh random draw each time	passed
2 runs after test_mobilebert_qat began completing	the draw following torch.manual_seed(42)	failed, bit-identical both times

The failing magnitude was max: 0.02465033531188965 against atol: 0.02 — about 23% over, i.e. the
same order as the tolerance, not a gross error. Mean abs error was 0.00578. So the model is
behaving normally; the tolerance simply does not cover the spread across random inputs.

The bit-identical repeat is the tell: the computation is deterministic for a given input, so the six
earlier passes were six different draws that happened to land under the bar.

How to reproduce deliberately

import torch
torch.manual_seed(42)          # what transformers' Trainer leaves behind
# then run test_mv2_fp16 -- it draws the failing input

Or run the suite with test_mobilebert_qat preceding it and the RNG restore from this PR removed.

Options, for whoever picks this up

1. Pin the input and set a tolerance that holds for it. Cheapest. The risk is choosing a seed
   because it passes, which re-hides the spread rather than measuring it.
2. Pin the input and widen atol to cover the observed spread. Needs a handful of draws
   measured on the device to pick a defensible number. 0.02465 is one known data point; nobody has
   sampled the distribution.
3. Use a fixed, representative input (a real image tensor rather than randn) so the comparison
   is against something meaningful instead of noise. Arguably the most correct for an accuracy test,
   and the most work.

Option 2 or 3 is the honest fix. Option 1 alone would leave the same latent fragility.

This needs someone who can run the Samsung NPU — I cannot measure the spread without it, and
picking a number blind would be guessing.

@digantdesai for routing to the right owner.

[psiddh]

trunk / test-samsung-models-linux / linux-job (push)Successful in 217m Job passes with this fix. Merging now