Playground: clearer pixel-diff failure diagnostics#1775
Merged
bkaradzic-microsoft merged 3 commits intoJul 10, 2026
Merged
Conversation
Improve the validation runner's output so a pixel-comparison failure is faster to triage, and flag the common transient network/font-load flake: * Fix the "First pixel off" line, which printed the blue channel as renderData[index] + 2 (i.e. red + 2) instead of renderData[index + 2]; it now shows the true value and the pixel's x,y coordinates. * "Pixel difference" now reports the diff ratio and the allowed errorRatio (e.g. "2031 / 240000 pixels (0.846%, per-channel threshold 25); allowed errorRatio 2.5%"), so it's obvious how close a diff is to the pass line. * On a pixel-comparison failure, print the rendered/diff PNG paths, and for playgroundId (snippet) tests a note that the scene plus its GUI/assets/ fonts load over the network -- so the diff is often a transient async font/asset-load flake -- with the exact isolation re-run command. Local scene-file tests get only the PNG paths (a diff there is a real regression). Canonical example: 'Parse GUI json with unicode' renders GUI text with the fallback font until 'droidsans' finishes loading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Improves the Babylon Native Playground validation runner’s pixel-diff failure output to make image-comparison failures faster to diagnose, especially for snippet-based (playgroundId) tests that can flake due to async network font/asset loading.
Changes:
- Adds a dedicated pixel-comparison failure diagnostics helper that prints output PNG paths and (for snippet tests) an isolation re-run command plus a transient network/font-load hint.
- Fixes incorrect “First pixel off” reporting (blue channel) and augments it with pixel index and x/y coordinates.
- Enhances “Pixel difference” output to include diff ratio vs. the allowed error ratio and the per-channel threshold.
Addresses review feedback: testWidth / getHardwareScalingLevel() can be non-integer when hardware scaling != 1, producing fractional x/y in the "First pixel off" line. Round it so the logged coordinates are integers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ryantrem
reviewed
Jul 9, 2026
ryantrem
reviewed
Jul 9, 2026
Address review feedback: convert the logFailureDiagnostics and "Pixel difference" console.log calls from string concatenation to template literal interpolation, matching the style used elsewhere in the file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bghgary
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the Playground validation runner's pixel-comparison failures faster to triage, and flags the common transient network/font-load flake (e.g.
Parse GUI json with unicode), which is aplaygroundIdsnippet test whose GUI text renders with the fallback font untildroidsans.ttffinishes loading over the network.Changes (
Apps/Playground/Scripts/validation_native.js)renderData[index] + 2(red + 2) instead ofrenderData[index + 2], so the reported "Value" blue was wrong. It now prints the true value and the offending pixel'sx,ycoordinates.Pixel difference:now showsN / total pixels (X.XXX%, per-channel threshold T); allowed errorRatio R%, so it's immediately clear how close a diff is to the pass line.playgroundId(snippet) tests only a note that the scene + its GUI/assets/fonts load over the network (so the diff is often a transient async font/asset-load flake) plus the exact isolation re-run command. Local scene-file tests get only the PNG paths — a diff there is a real regression, not a flake.Example output
Passing snippet test (new ratio line + fixed blue channel + coords):
Failing snippet test (new hint):
Validation
node --checkpasses.Parse GUI json with unicodein isolation — new output verified, test still passes (ran=1 passed=1).