DEV Community

mavoryl
mavoryl

Posted on

A simpler way to write golden tests in Flutter — golden_matrix

#flutter #testing #golden #dart

I want to share a Flutter package I made. It is called golden_matrix. It helps you write golden tests when your app has many themes, locales, and devices.

The problem

My app has 5 locales, light and dark themes, and many device sizes. I also need to test different text scales for accessibility.

If I want to test one button, I need many golden tests:

testGoldens('button - en - light - small phone', ...);
testGoldens('button - en - dark - small phone', ...);
testGoldens('button - ru - light - small phone', ...);
testGoldens('button - ru - dark - small phone', ...);
// ...30 more tests
Enter fullscreen mode Exit fullscreen mode

This is too much copy-paste. Most people give up and only test one combination. Then bugs come from the combinations they did not test.

The idea

With golden_matrix, you write one test:

matrixGolden(
  'MyButton',
  scenarios: [
    MatrixScenario('default', builder: () => const MyButton(label: 'OK')),
    MatrixScenario('disabled', builder: () => const MyButton(label: 'OK', enabled: false)),
  ],
  axes: MatrixAxes(
    themes: [MatrixTheme.light, MatrixTheme.dark],
    locales: [Locale('en'), Locale('ru'), Locale('ar')],
    textScales: [1.0, 2.0],
    devices: [MatrixDevice.phoneSmall, MatrixDevice.tablet],
  ),
);
Enter fullscreen mode Exit fullscreen mode

This makes 48 golden files. One declaration. No loops.

The package knows that Arabic is RTL, so it sets that for you. Each file has a clear name like goldens/default/dark_ar_rtl_2x_tablet.png.

What if 48 tests is too many?

You can use smaller test sets:

Smoke — only a few key combinations:

matrixGolden('Widget', scenarios: [...], axes: axes,
  sampling: MatrixSampling.smoke); // about 5 tests
Enter fullscreen mode Exit fullscreen mode

Pairwise — covers all pairs of values with the smallest number of tests:

matrixGolden('Widget', scenarios: [...], axes: axes,
  sampling: MatrixSampling.pairwise); // about 12 tests
Enter fullscreen mode Exit fullscreen mode

Most bugs come from how two settings work together (like dark theme + Arabic). Pairwise testing finds these bugs with much fewer tests.

Presets — for common cases:

matrixGolden('Widget', scenarios: [...], preset: MatrixPreset.componentSmoke);
Enter fullscreen mode Exit fullscreen mode

A nice extra feature

Golden tests check pixels. But sometimes a widget is broken even when pixels look the same.

For example: your row has an icon, text, and a button. It looks fine. But on a small phone with large text, the button goes off the screen.

A RenderFlex overflowed by 613 pixels on the right.
Enter fullscreen mode Exit fullscreen mode

Flutter cuts off the extra part, so the image looks "okay". The pixel test passes. But the user sees a broken button.

golden_matrix catches these errors. It listens for Flutter warnings during the test. You get a report like this:

{
  "scenario": "default",
  "device": "phoneSmall",
  "textScale": 2.0,
  "status": "passed",
  "warnings": [
    "A RenderFlex overflowed by 613 pixels on the right."
  ]
}
Enter fullscreen mode Exit fullscreen mode

The test passes, but you see the warning. You can fix the bug before users find it.

HTML reports

After tests run, the package makes one HTML file. You open it and see all your golden images on one page. You can filter by theme, status, or scenario. You can click an image to see it bigger.

This is much faster than opening files one by one.

How to start

Add the package:

# pubspec.yaml
dev_dependencies:
  golden_matrix: ^0.6.1
Enter fullscreen mode Exit fullscreen mode

Set up fonts (do this once):

// test/flutter_test_config.dart
import 'dart:async';
import 'package:golden_matrix/golden_matrix.dart';

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
  await loadAppFonts();
  return testMain();
}
Enter fullscreen mode Exit fullscreen mode

Write a test:

import 'package:golden_matrix/golden_matrix.dart';

void main() {
  matrixGolden(
    'MyButton',
    scenarios: [
      MatrixScenario('default', builder: () => const MyButton(label: 'OK')),
    ],
    preset: MatrixPreset.componentSmoke,
  );
}
Enter fullscreen mode Exit fullscreen mode

Run:

flutter test --update-goldens
flutter test
Enter fullscreen mode Exit fullscreen mode

That is all.

Testing full screens

For screens with navigation or special setup, use screenMatrixGolden. You give it a function that builds your app:

screenMatrixGolden(
  'LoginScreen',
  appBuilder: (combination) => MaterialApp(
    theme: combination.theme.resolve(),
    locale: combination.locale,
    home: LoginScreen(
      errorMessage: combination.scenario.name == 'error' ? 'Invalid' : null,
    ),
  ),
  states: [
    MatrixScenario('default', builder: () => const SizedBox.shrink()),
    MatrixScenario('error', builder: () => const SizedBox.shrink()),
  ],
  preset: MatrixPreset.screenSmoke,
);
Enter fullscreen mode Exit fullscreen mode

You control the app. The package handles the matrix.

Why I made this

Other packages did not work for me:

  • golden_toolkit — not updated since 2023
  • alchemist — good for one case, but no matrix idea

So I made golden_matrix. It is MIT license, no extra dependencies, 108 tests, and on pub.dev.

It is not perfect. It may not fit every project. But if you have many themes and devices to test, give it a try.

If something does not work or you want a new feature, open an issue. I will help.

Links:

Top comments (0)