Skip to content

test(example): agent-device visual regression PoC for Surface - #5115

Draft
JKobrynski wants to merge 6 commits into
mainfrom
poc/agent-device-visual
Draft

test(example): agent-device visual regression PoC for Surface#5115
JKobrynski wants to merge 6 commits into
mainfrom
poc/agent-device-visual

Conversation

@JKobrynski

@JKobrynski JKobrynski commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Motivation

Our component tests assert style props and tree snapshots; they can't tell whether a component looks right. This is a proof of concept for screenshot-based visual regression tests using agent-device against the existing example-app screens, scoped to Surface only since most other components are mid-rework.

It answers one question: does agent-device capture deterministically enough to diff Surface against a baseline, and does that diff catch a realistic regression?

Result: yes on both platforms, with one catch. Noise floor is 0 px on iOS and Android across warm captures and a relaunch. A one-level shadow regression (elevation 1 rendered as 2) is caught at --threshold 0.02 — bit-identical across three captures per platform — but at agent-device's default threshold (0.1) it is reported as a perfect match on both. Full numbers in example/visual/README.md; every number resolves to a file under example/visual/evidence/.

Changes

  • example/src/Examples/SurfaceExample.tsx — testID on the two List.Sections so screenshot --crop-on can target them. Only app change; no library code touched.
  • example/visual/README.md — results, caveats, hand-run CLI commands to reproduce, and what a real runner would have to handle.
  • example/visual/env.json — the pinned devices and versions the baselines are valid for.
  • example/visual/__baselines__/{ios,android}/ — four PNGs.
  • example/visual/evidence/ — the diff screenshot JSON behind every table, the diff images, 19 agent-device dogfooding issues, and two small accessibility-tree excerpts.

Scope

This is a PoC and stays one. Three review rounds grew a reproduction script into a 1,200-line runner and the evidence into 29k lines of tree dumps; both were cut back in e59fd2eb1 and preserved unchanged on branch poc/agent-device-visual-runner. The lessons from that work (relaunch for a fresh bundle, persisted navigation state, dev-client chrome inside the crop, --force-full, Node 20 entry guard, per-story capture sizes) are in the README as follow-ups, not code.

Related issue

None. Follows the discussion about replacing style-prop assertions with visual tests.

Test plan

Commands in the README's "Reproducing by hand", verified on 2026-09-10/11 on the pinned iPhone 17 Pro simulator and Pixel_10_Pro emulator. Expect differentPixels: 0 against the committed baselines; apply the one-line Surface break described there, relaunch, and expect ~10,179 px (iOS) / ~9,336 px (Android) at 0.02 and match: true at 0.1.

Out of scope

Web (--crop-on is refused there), CI, other components, dark theme, cross-platform comparison, any runner script.

@JKobrynski
test(example): add agent-device visual regression PoC for Surface
32391c4 
Proof of concept for screenshot-based visual regression testing using
agent-device against the existing example screens, scoped to Surface.

- example/visual/run.mjs: one command per platform (iOS simulator,
  Android emulator) that opens the example app, navigates to the Surface
  screen, crops the "Elevated surface" / "Flat surface" sections, and
  diffs them against committed baselines. Exits 1 on regression;
  --update re-baselines.
- example/visual/README.md: results. Noise floor is 0 px on both
  platforms across warm captures and a relaunch. agent-device's default
  diff threshold (0.1) misses a realistic one-level shadow regression on
  both platforms; 0.02 catches it with noise still at 0.
- example/visual/env.json: pinned device/OS/density/versions the
  baselines are valid for.
- SurfaceExample: testID + accessible on the two List.Sections so
  `screenshot --crop-on` can target them.

Twelve agent-device dogfooding issues were recorded locally for filing.
@JKobrynski
test(example): address review on the visual regression PoC
e4f0311 
- Drop `accessible` from the two List.Sections. With it, iOS collapsed each
  section into a single accessibility element (XCUITest tree evidence in
  evidence/a11y/). `testID` alone resolves `screenshot --crop-on` on both
  platforms with identical rects, so the example app's accessibility tree
  is unchanged by the hook.
- Rename the test ids to surface-example-{elevated,flat} so they cannot be
  confused with the library defaults removed in #5088 / #5099. Baselines
  renamed to __baselines__/<platform>/<story>.png on both platforms.
- Commit the evidence: raw diff JSON behind every table, break diff images,
  issues.md, the a11y trees, and the final script runs.
- run.mjs: env.json mismatch and crop-size mismatch are hard failures
  (exit 2 / 3) unless --force; --update can create a missing baseline;
  always relaunch so the bundle is fresh (Fast Refresh was seen to stop
  reaching Android); handle the Expo dev menu and dev launcher; because
  the example app persists navigation state, go Back to the list root and
  press the Surface row by position, not the header by label; --force-full
  on every snapshot; summary.json written on every exit; --out.
- Re-measure the realistic break three times per platform at pre-declared
  thresholds: bit-identical (iOS 10,179 px, Android 9,336 px at 0.02;
  0 at 0.1) and identical to the original single captures.
- README: status, evidence links, a11y section, re-measurement section,
  issues 13-18, updated Running-it.
@JKobrynski
test(example): address round-2 review on the visual regression PoC
63e1699 
- Entry point: replace `import.meta.main` (Node >= 24.2 only) with a
  realpath comparison of argv[1] and the module URL. On Node 20/22 the
  script previously printed nothing and exited 0. Verified on 20.20.2.
- `--update` skips the capture-size check and prints the captured
  dimensions, so a new story can get a baseline; regression test added.
- `--force` covers the device check only; a size mismatch in diff mode
  has no override.
- Remove the dev-launcher geometric fallback, the unreachable size-check
  branch, the duplicate iOS version comparison and unused exports.
- List-root detection requires the "Examples" title; argument errors
  write a summary.json too.
- Strip npm_config_* from the spawned environment so the nested
  `npx agent-device` works when the script itself runs under `npx -p`.
- run.test.mjs (node:test, 6 cases) and evidence/summarize.mjs, the
  committed generator behind evidence/summary.md.
- Evidence: drop the 12 wait.json files and two full-tree snapshots;
  add the round-2 runs on Node 24 and Node 20.
- README: Android section no longer credits `accessible`; Node
  requirement, test and generator documented; env.json drops the
  parent-commit pin.
@JKobrynski
test(example): address round-3 review on the visual regression PoC
e9245dc 
- Evidence citations: every path in evidence/issues.md now resolves to a
  committed file. Issue 15 re-captured (evidence/runner/ios-*), issues 4
  and 14 rewritten onto committed evidence; the two lost logs are stated
  as not kept.
- Crop size is checked against the baseline PNG's own header, per story,
  so `--update` on a new story is followed by a passing run; env.json
  drops the per-platform crop pins and totalPixels.
- Drop the arg-error summary machinery; an argument error prints and
  exits 2.
- Device observers are injected; the --force test no longer shells out.
  `yarn example test:visual` runs the seven node:test cases; README says
  it is manual, not Jest/hook/CI. README Node version is 20+ throughout.
- Guardrails triggered for real: exit 2 on a UDID that is not attached,
  exit 3 on an 804x428 capture against the 3x baseline
  (evidence/runs/ios/round3-exit*).
- New: the Expo dev-client's floating "Tools" button inside the crop
  produced a deterministic false FAIL on Android with src clean
  (issue 19). The runner now detects it, disables it via the dev menu,
  or stops with exit 2; evidence in evidence/runner/android-* and
  evidence/runs/android/round3-*.
@JKobrynski
test(example): cut the visual regression PoC back to PoC scope
e59fd2e 
Three review rounds turned the reproduction script into a 1,200-line
runner and the evidence folder into 29,000 lines of accessibility-tree
dumps. Neither is what a proof of concept needs. This commit keeps:

- the results (README), the pinned environment, the four baselines and
  the two test ids;
- the diff JSON every table is computed from, the diff images, and the
  19 agent-device issues with their evidence;
- two small excerpts replacing the full accessibility-tree snapshots,
  each saying how to regenerate the full tree;
- hand-run CLI commands to reproduce, instead of a script.

Removed from this PR: run.mjs, run.test.mjs, evidence/summarize.mjs and
summary.md, evidence/runs, evidence/runner, the full evidence/a11y
trees, and the example/package.json test script. All of it is preserved
unchanged on branch poc/agent-device-visual-runner, and the README
lists what a real runner would have to handle, learned from it.
@JKobrynski
test(example): collapse PoC evidence into one results table
7917bdf 
96 per-threshold diff JSON files, each ~20 lines, become one
evidence/results.csv with 72 rows (platform, capture, threshold,
pixels, regions, match, source file). Keep one diff image per platform
for the realistic and gross breaks plus the Tools-button false FAIL.
Raw JSON and the other images stay on poc/agent-device-visual-runner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JKobrynski