Skip to content

[CI, material_ui, cupertino_ui] Gold testing - #10753

Open
Piinks wants to merge 2 commits into
flutter:mainfrom
Piinks:goldTesting
Open

Piinks wants to merge 2 commits into
flutter:mainfrom
Piinks:goldTesting

Conversation

@Piinks

@Piinks Piinks commented Jan 8, 2026

Copy link
Copy Markdown
Contributor

Adds Skia Gold integration (flutter_goldens) to the flutter/packages repository (targeting the flutter-packages Gold instance) to support golden file testing across packages such as material_ui and cupertino_ui.

Temporary Rollout Behavior (TODO(Piinks)): While the flutter-packages Gold dashboard (https://flutter-packages-gold.skia.org) is being stood up, FlutterPostSubmitFileComparator and FlutterPreSubmitFileComparator upload images and results to Skia Gold so the backend can ingest initial data, while logging any SkiaExceptions and returning true to avoid blocking CI until the instance frontend is live.

Nearly identical to flutter/flutter instance, adapting to unique aspects of CI between repos and naming conventions/directory structure for packages.

Part of flutter/flutter#172933

Pre-Review Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

Footnotes

  1. Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. 2 3

@github-actions github-actions Bot added p: two_dimensional_scrollables Issues pertaining to the two_dimensional_scrollables package triage-framework Should be looked at in framework triage labels Jan 8, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new internal package, flutter_goldens, to enable Skia Gold testing for packages within the flutter/packages repository. It adds the necessary client logic, environment-aware comparators, and test configurations, with two_dimensional_scrollables serving as the initial implementation example. My review has identified some critical and high-severity issues, including a syntax error that will prevent compilation, incorrect Skia Gold configuration, and fragile logic for naming golden files. I have also noted several medium-severity issues related to code duplication, debugging artifacts, and dependency management that should be addressed to improve maintainability. Overall, this is a significant and valuable addition, and my feedback aims to ensure its correctness and robustness.

Comment thread script/flutter_goldens/lib/flutter_goldens.dart Outdated
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread .ci.yaml
Comment thread script/flutter_goldens/README.md Outdated
Comment thread script/flutter_goldens/lib/flutter_goldens.dart Outdated
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread script/flutter_goldens/pubspec.yaml Outdated
@github-actions github-actions Bot added p: material_ui and removed p: two_dimensional_scrollables Issues pertaining to the two_dimensional_scrollables package labels Feb 24, 2026
@Piinks

Piinks commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

Note self for tomorrow: Use the SDK path environment variable, this appears more stable cross CI platforms -flutter +packages.

@Piinks Piinks added the autosubmit Merge PR when tree becomes green via auto submit App label Mar 11, 2026
@auto-submit auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Mar 11, 2026
@auto-submit

auto-submit Bot commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

autosubmit label was removed for flutter/packages/10753, because - The status or check suite Linux_web web_dart_unit_test_wasm_shard_2 master has failed. Please fix the issues identified (or deflake) before re-applying this label.

@Piinks Piinks added the CICD Run CI/CD label Apr 16, 2026
@github-actions github-actions Bot removed the CICD Run CI/CD label Apr 16, 2026
@Piinks Piinks added the CICD Run CI/CD label Apr 17, 2026
@github-actions github-actions Bot removed the CICD Run CI/CD label Apr 27, 2026
auto-submit Bot pushed a commit that referenced this pull request May 8, 2026
…the design migration (#11649)

This branches off of #10753 and strips it down to just the skipping comparator. This means when golden file testing is used with this version of flutter_goldens, `matchesGoldenFile` will return true in all environments without executing image comparison.
The goal of this PR is to join the material_ui and cupertino_ui mega-migration PR (cc @justinmc ) to disable golden file testing for now in material_ui and cupertino_ui so we can get a clearer signal from CI about any remaining blockers.
Full golden file testing support will likely land in #10753 as a separate step given the enormity of these PRs.

cupertino_ui and material_ui are set `publish_to: none` here, which is intentional. This allows us to split up these changes, and wait to publish until everything lands and is working as we want it to.

## Pre-Review Checklist

**Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

[^1]: Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling.
creatorpiyush pushed a commit to creatorpiyush/packages that referenced this pull request Jun 10, 2026
…the design migration (flutter#11649)

This branches off of flutter#10753 and strips it down to just the skipping comparator. This means when golden file testing is used with this version of flutter_goldens, `matchesGoldenFile` will return true in all environments without executing image comparison.
The goal of this PR is to join the material_ui and cupertino_ui mega-migration PR (cc @justinmc ) to disable golden file testing for now in material_ui and cupertino_ui so we can get a clearer signal from CI about any remaining blockers.
Full golden file testing support will likely land in flutter#10753 as a separate step given the enormity of these PRs.

cupertino_ui and material_ui are set `publish_to: none` here, which is intentional. This allows us to split up these changes, and wait to publish until everything lands and is working as we want it to.

## Pre-Review Checklist

**Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

[^1]: Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling.
@Piinks Piinks changed the title WIP - Gold testing [CI, material_ui, cupertino_ui] WIP - Gold testing Jun 23, 2026
@Piinks Piinks added triage-design Should be looked at in design triage and removed triage-framework Should be looked at in framework triage labels Aug 17, 2026
@github-actions github-actions Bot removed p: material_ui triage-design Should be looked at in design triage labels Aug 25, 2026
@Piinks Piinks added the CICD Run CI/CD label Sep 17, 2026
@Piinks
Piinks marked this pull request as ready for review September 17, 2026 22:51
@Piinks Piinks changed the title [CI, material_ui, cupertino_ui] WIP - Gold testing [CI, material_ui, cupertino_ui] Gold testing Sep 17, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates Skia Gold into the flutter/packages repository's CI and local testing environments. It adds the goldctl dependency to .ci.yaml targets and introduces several golden file comparators along with a SkiaGoldClient to handle authentication, initialization, and image uploads. Feedback on the changes includes fixing an invalid Dart list literal syntax, correcting the path to result-state.json inside the temp directory, adding HTTP status code checks before processing network responses, handling potential missing environment variables, and catching io.IOException instead of specific socket exceptions for more robust error handling.

Comment thread script/flutter_goldens/lib/flutter_goldens.dart
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread script/flutter_goldens/lib/skia_client.dart
Comment thread script/flutter_goldens/lib/skia_client.dart
Comment thread script/flutter_goldens/lib/skia_client.dart Outdated
Comment thread script/flutter_goldens/lib/flutter_goldens.dart
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant