Skip to content

Playground: clearer pixel-diff failure diagnostics#1775

Merged
bkaradzic-microsoft merged 3 commits into
BabylonJS:masterfrom
bkaradzic-microsoft:improve-validation-pixel-diff-diagnostics
Jul 10, 2026
Merged

Playground: clearer pixel-diff failure diagnostics#1775
bkaradzic-microsoft merged 3 commits into
BabylonJS:masterfrom
bkaradzic-microsoft:improve-validation-pixel-diff-diagnostics

Conversation

@bkaradzic-microsoft

Copy link
Copy Markdown
Member

Summary

Makes the Playground validation runner's pixel-comparison failures faster to triage, and flags the common transient network/font-load flake (e.g. Parse GUI json with unicode), which is a playgroundId snippet test whose GUI text renders with the fallback font until droidsans.ttf finishes loading over the network.

Changes (Apps/Playground/Scripts/validation_native.js)

  • Fix a bug in the "First pixel off" line. It printed the blue channel as renderData[index] + 2 (red + 2) instead of renderData[index + 2], so the reported "Value" blue was wrong. It now prints the true value and the offending pixel's x,y coordinates.
  • Report the diff ratio vs. the allowance. Pixel difference: now shows N / total pixels (X.XXX%, per-channel threshold T); allowed errorRatio R%, so it's immediately clear how close a diff is to the pass line.
  • Actionable failure diagnostics. On a pixel-comparison failure the runner prints the rendered/diff PNG paths, and for playgroundId (snippet) tests only a note that the scene + its GUI/assets/fonts load over the network (so the diff is often a transient async font/asset-load flake) plus the exact isolation re-run command. Local scene-file tests get only the PNG paths — a diff there is a real regression, not a flake.

Example output

Passing snippet test (new ratio line + fixed blue channel + coords):

First pixel off at 277192 (pixel 69298 @ x=298, y=115): Value: (81, 81, 100) - Expected: (51, 51, 76)
Pixel difference: 2031 / 240000 pixels (0.846%, per-channel threshold 25); allowed errorRatio 2.5%.
Test 'Parse GUI json with unicode' validated

Failing snippet test (new hint):

Test 'Parse GUI json with unicode' failed (pixel comparison)
  Rendered result: .../Results/parseGuiJsonWithUnicode.png
  Diff overlay:    .../Errors/parseGuiJsonWithUnicode.png
  Note: this test loads playgroundId #ERVGT5#0 from the snippet server and pulls GUI/assets/fonts over the network, so a pixel diff is often a transient async asset/font-load timing flake.
  Re-run in isolation to confirm a real regression:
    Playground --headless --once --test "Parse GUI json with unicode" app:///Scripts/validation_native.js

Validation

  • node --check passes.
  • Ran Parse GUI json with unicode in isolation — new output verified, test still passes (ran=1 passed=1).
  • Diagnostics-only change; no effect on pass/fail logic.

Improve the validation runner's output so a pixel-comparison failure is
faster to triage, and flag the common transient network/font-load flake:

* Fix the "First pixel off" line, which printed the blue channel as
  renderData[index] + 2 (i.e. red + 2) instead of renderData[index + 2];
  it now shows the true value and the pixel's x,y coordinates.
* "Pixel difference" now reports the diff ratio and the allowed errorRatio
  (e.g. "2031 / 240000 pixels (0.846%, per-channel threshold 25); allowed
  errorRatio 2.5%"), so it's obvious how close a diff is to the pass line.
* On a pixel-comparison failure, print the rendered/diff PNG paths, and for
  playgroundId (snippet) tests a note that the scene plus its GUI/assets/
  fonts load over the network -- so the diff is often a transient async
  font/asset-load flake -- with the exact isolation re-run command. Local
  scene-file tests get only the PNG paths (a diff there is a real
  regression). Canonical example: 'Parse GUI json with unicode' renders GUI
  text with the fallback font until 'droidsans' finishes loading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 9, 2026 15:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves the Babylon Native Playground validation runner’s pixel-diff failure output to make image-comparison failures faster to diagnose, especially for snippet-based (playgroundId) tests that can flake due to async network font/asset loading.

Changes:

  • Adds a dedicated pixel-comparison failure diagnostics helper that prints output PNG paths and (for snippet tests) an isolation re-run command plus a transient network/font-load hint.
  • Fixes incorrect “First pixel off” reporting (blue channel) and augments it with pixel index and x/y coordinates.
  • Enhances “Pixel difference” output to include diff ratio vs. the allowed error ratio and the per-channel threshold.

Comment thread Apps/Playground/Scripts/validation_native.js
Addresses review feedback: testWidth / getHardwareScalingLevel() can be
non-integer when hardware scaling != 1, producing fractional x/y in the
"First pixel off" line. Round it so the logged coordinates are integers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comment thread Apps/Playground/Scripts/validation_native.js Outdated
Comment thread Apps/Playground/Scripts/validation_native.js
Address review feedback: convert the logFailureDiagnostics and "Pixel
difference" console.log calls from string concatenation to template
literal interpolation, matching the style used elsewhere in the file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bkaradzic-microsoft bkaradzic-microsoft merged commit 7e7e5db into BabylonJS:master Jul 10, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants