DEV Community: Samuel Abada

Optimizing Flutter Test Execution Time: A Comprehensive Guide

Samuel Abada — Wed, 10 Apr 2024 15:11:18 +0000

Testing is a critical phase in the software development lifecycle, ensuring that your application meets its quality standards. However, as a project grows, so does its test suite, potentially leading to longer test execution times. For Flutter projects, there are effective strategies to optimize these times, ensuring efficient and rapid testing processes that fit seamlessly into your development workflow. This article explores practical approaches to cutting down Flutter test execution time and coverage optimization.

Understanding the Challenge

Flutter's testing framework is robust, allowing developers to write unit, widget, and integration tests to cover various aspects of their applications. However, with the expansion of test suites, developers often face increased execution times, particularly when running tests with coverage collection enabled. This not only slows down the CI/CD pipeline but can also hamper developer productivity, especially in agile environments where quick feedback loops are crucial.

Strategies for Optimization

Parallel Test Execution/Concurrency

One of the first strategies to consider is concurrency. By running tests concurrently, you can significantly reduce the overall time it takes to run your entire test suite. Flutter supports this through the --concurrency flag in the flutter test command, allowing you to specify the number of concurrent tests processes to run.

flutter test --concurrency=4

This doesn't mean each individual test function is executed in parallel with others within the same test file. Rather, it allows multiple test files to be run at the same time, each in its own Dart VM process. However, the actual performance gain depends on your machine's capabilities and the nature of the tests. Note: The --concurrency flag is ignored for integration tests because they often require a more controlled environment, potentially involving real or simulated devices.

The Wrapper Test Method/Test Bundling

A more nuanced approach involves grouping your tests into a single test suite, significantly reducing the performance penalty associated with coverage collection. This method involves creating a "wrapper" test file that imports and runs all individual tests. The advantage here is the reduction in the overhead of initializing the Dart VM for each test file and the optimization of coverage data collection.

Implementing the Wrapper Test

Create a Wrapper Test File: Place a new Dart file in your project's test directory. Name it to reflect its purpose, such as all_tests_wrapper.dart. The naming however, is irrelevant.
Group Your Tests: Import your individual test files and use the group function from flutter_test to encapsulate them within the main function of your wrapper test file.

// Example: all_tests_wrapper.dart
import 'package:flutter_test/flutter_test.dart';
import 'widget_tests.dart' as widget_tests;
import 'integration_tests.dart' as integration_tests;
void main() {
  group('Widget Tests', widget_tests.main);
  group('Integration Tests', integration_tests.main);
}

Run the Wrapper Test: Execute your tests through the wrapper to benefit from optimized test execution and coverage collection.

flutter test --coverage test/all_tests_wrapper.dart

Test Sharding

While parallel execution increases efficiency, test sharding takes it a step further by splitting your test suite into smaller segments (shards) and running each segment in parallel. This can be particularly effective in CI/CD environments where multiple executors or containers are available to run tests concurrently.

Flutter's built-in sharding mechanism divides the test suite into multiple shards and allows each shard to be run independently. This is achieved by specifying the total number of shards and the index of the current shard being run.

Implementing Test Sharding in CI/CD

Determine Total Shards

Decide on the total number of shards you want to split your tests into. This could be based on the total number of tests, the structure of your project, or the resources available in your CI/CD environment.

Modify CI/CD Pipeline

Adjust your CI/CD pipeline to run the test command multiple times, each time with a different --shard-index, ranging from 0 to --total-shards minus one.

CI/CD Pipeline Integration (GitHub Actions Example)

When integrating test sharding into a CI/CD pipeline like GitHub Actions, you can leverage the matrix strategy to parallelize test execution across multiple runners, each running a different shard of tests as a separate job.

name: CI with Test Sharding and Coverage

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard-index: [0, 1, 2, 3] # Define the number of shards here
    steps:
      - uses: actions/checkout@v2
      - uses: subosito/flutter-action@v1
        with:
          flutter-version: '3.19.x'
      - name: Run Tests in Shards
        run: flutter test --coverage --total-shards=4 --shard-index=${{ matrix.shard-index }}

This configuration sets up four separate jobs, each running a portion of the tests. The --total-shards=4 flag indicates the tests are divided into four shards, and --shard-index specifies the index of the shard to run in each job. You should ensure to merge the coverage reports before using the report for your needs.

Caution: When to Use Test Sharding

While test sharding is a powerful tool for optimizing test execution times in Flutter projects, it's essential to apply it judiciously. Here are key considerations to keep in mind:

Overhead of Initialization

Each test shard runs in its own separate process, requiring a separate initialization phase. This includes starting up the Flutter test environment, which can introduce overhead, especially if the number of tests in each shard is relatively small.

CI/CD Resource Utilization

While sharding can parallelize test execution, each shard typically runs on its own runner or container in CI/CD environments. If your CI/CD platform has limited parallel jobs or if the setup and teardown times for each job are significant, you might not see the expected decrease in total test time. In fact, for smaller test suites, the overhead of managing multiple parallel jobs might result in longer overall execution times compared to running all tests in a single job.

When to Consider Sharding

Large Test Suites: For projects with a substantial number of tests, where running the entire suite in a single job becomes prohibitively time-consuming.
Sufficient CI/CD Resources: If your CI/CD platform supports a high number of parallel jobs and the overhead of setting up additional jobs is minimal.
Balanced Shard Distribution: When you can evenly distribute tests across shards to ensure that no single shard becomes a bottleneck.

Best Practices for Implementing Test Sharding

Evaluate Your Needs: Start by analyzing your current test execution times and CI/CD resource availability. Implement sharding incrementally, beginning with a small number of shards to measure the impact on total execution time.
Monitor and Adjust: Continuously monitor the performance of your test execution times after implementing sharding. Be prepared to adjust the number of shards based on the results and any changes in your test suite or CI/CD environment.
Balance Test Distribution: Aim for an even distribution of tests across shards to avoid imbalances that could lead to some shards finishing much earlier than others, thereby not fully utilizing parallel execution.

Test sharding in Flutter can be a double-edged sword. While it offers the potential for significantly reduced test execution times, especially for large and complex projects, it can also introduce overhead that may outweigh the benefits for smaller projects or in resource-constrained CI/CD environments. Careful consideration and continuous monitoring are key to leveraging test sharding effectively.

Very Good Cli

The Very Good CLI, developed by Very Good Ventures, introduces an innovative approach to optimizing Flutter test execution. The CLI's very_good test command supercharges Dart and Flutter apps testing with performance optimizations and developer experience improvements.

Integrating Very Good CLI into your development workflow is straightforward. Install it globally via Dart, and use the very_good test command to run your tests. The command's flags and arguments, such as --coverage, --recursive, and --min-coverage, provide granular control over how your tests are executed and how coverage is collected.

# Install Very Good CLI
dart pub global activate very_good_cli

# Run optimized tests with Very Good CLI
very_good test --coverage --min-coverage 100

CI/CD Integration

Integrating these strategies into your CI/CD pipeline can further enhance efficiency. Use CI/CD capabilities to leverage parallel execution and test sharding, and consider the wrapper test method for large, complex projects that suffer from significant delays in test execution.

Benchmarking and Comparison

To truly appreciate the impact of these optimization strategies on Flutter test execution times, it's beneficial to conduct a benchmarking exercise. This section compares regular flutter test execution times against optimized runs using the strategies discussed above, including running tests with coverage.

Benchmark Setup

To ensure a fair comparison, tests should be run under similar conditions and on the same hardware. For the purpose of this benchmark, we'll consider some Flutter project with a mix of unit and widget tests distributed across several features.

Running Standard Flutter Tests

First, run the entire test suite using the standard flutter test command:

time flutter test

Then, run the tests again with coverage enabled:

time flutter test --coverage

Record the total execution time for each run.

Running Optimized Tests

Next, apply the optimization strategies:

Parallel Test Execution: Use the --concurrency flag with an appropriate value based on your machine's CPU cores.
The Wrapper Test Method: Combine all tests into a single suite using a wrapper test file.

Run the optimized tests and the optimized tests with coverage, recording the execution time for each:

# For The Wrapper Test Method
time flutter test --coverage test/all_tests_wrapper.dart

Results and Comparison

After running the tests using both standard and optimized methods, we compare the execution times. Its observed that there is significant reductions in total test runtime, especially for the optimized tests with coverage. The exact performance gains will depend on factors like the number and complexity of tests, project size, and your CI/CD environment's capabilities.

Optimized Tests with Coverage vs. Flutter Test with Coverage

When comparing optimized tests with coverage against standard flutter test runs with coverage, the difference can be stark. Coverage collection often adds considerable overhead, but by aggregating tests and minimizing the initialization overhead, the optimized approach can drastically reduce this penalty, resulting in faster CI/CD pipelines and more efficient development cycles.

Conclusion

Optimizing test execution time in Flutter projects is essential for maintaining a fast, efficient development pipeline. Through parallel execution, test sharding, and innovative approaches like the wrapper test method, developers can significantly reduce testing times. This not only accelerates the CI/CD process but also improves developer productivity by providing quicker feedback on the quality of the codebase. Implementing these strategies ensures that your Flutter project remains agile, robust, and ready for rapid iteration.

The goal of optimization is to find the sweet spot where the benefits of reduced execution time outweigh the costs of increased complexity and resource usage.

Happy testing 🧪

Resources

Test-Driven Development In Flutter

Samuel Abada — Tue, 25 Feb 2020 18:31:49 +0000

Test Driven Development Flutter

Test Driven Development(TDD), is a discipline in software development that prohibits us from writing our code/implementation before writing the test first.

According to Uncle Bob, there are 3 rules for TDD:

1. You are not allowed to write any production code unless it is to make a failing unit test pass.

2. You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.

_3. You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

Uncle Bob (_ http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd)

It is a cycle we have to iterate over. We write a test on something that we want to implement such that when the test is run, the result will be “fail” then we write an implementation that is just enough so that the test pass. We then fix our code quality which we may have overlooked since we were focusing on making the test pass without making the test fail.

Some try to write the implementation first, the test later and then committing only the test. This is now how one should go about TDD. You should note that you are only allowed to implement just enough code so that your test pass(rule 3 according to Uncle Bob).

TDD In Flutter

To understand this concept in Flutter, let’s jump into a practical example. For the purpose of this tutorial, I will be using the PiggyX application. The codes are available on Github. Click here to access the codes.

We would create and test a couple of simple functionalities: EmailFieldValidator , PasswordFieldValidator . Let’s separate the login logic by creating a new file call loginpage.dart and implement as such:

class EmailFieldValidator {
  static String _validate_(String value) {
    return false;
  }
}

class PasswordFieldValidator {
  static String _validate_(String value) {
    return false;
  }
}

Let’s write tests for the above validators. We head over to the test folder and delete the defaultwidget_tests.dart and add a new file called fieldvalidators_tests.dart . We implement the test as shown below:

import 'package:PiggyX/ui/loginpage.dart';
import 'package:flutter\_test/flutter\_test.dart';

void main() {

  test('empty email returns error string', () {

    final result = EmailFieldValidator._validate_('');
    expect(result, 'Email can\'t be empty');
  });

  test('non-empty email returns null', () {

    final result = EmailFieldValidator._validate_('email');
    expect(result, null);
  });

  test('empty password returns error string', () {

    final result = PasswordFieldValidator._validate_('');
    expect(result, 'Password can\'t be empty');
  });

  test('non-empty password returns null', () {

    final result = PasswordFieldValidator._validate_('password');
    expect(result, null);
  });
}

In the above, we expect the following cases:

i) The email field must not be empty

ii) The password field must not be empty

We can extend the test cases if we want to but for the purpose of this tutorial and this app, we would be sticking with these two.

When we try running flutter test before the implementation and you might see +0 -4: Some tests failed.This tells us that 0 test passes (+) and 4 tests have failed (-)😪 😪 😪.

Let’s fix them now in our TDD mode!

Head over to loginpage.dart and fix EmailFieldValidator with:

static String _validate_(String value) {
  return value.isEmpty ? 'Email can\'t be empty' : null;
}

Run the test again and you should see +2 -2 .

Now let’s head over to loginpage.dart and fix PasswordFieldValidator with:

static String _validate_(String value) {
  return value.isEmpty ? 'Password can\'t be empty' : null;
}

Running the tests again, you should +4: All Tests Passed! 💃 💃 💃

You have successfully tested your app in a TDD manner with Flutter! You deserve some accolades. Now, it’s safe to complete our cycle since the objective of our feature is now completed (only making the UI)!

The codes are available in this GitHub repo PiggyX.

Bonus Points

For mocking dependencies, you can make use of the mockito package. For example, we need to be able to mock data from an app needs in order to authenticate against an API instead of relying on the live API, emulate a live web service or database to return specific results depending on the situation, operations with firebase.e.t.c. All we need to do is to create an alternate implementation of a class by either hand-coding or using the mockito package. You could follow this guide on mocking.

If you have any questions, feel free to leave a comment 🙂.

References:

Flutter Testing

Testing Flutter Apps

Mocking Dependencies Using Mockito

Mockito Package

Configuring Travis CI and Coveralls with Flutter

Samuel Abada — Mon, 24 Feb 2020 06:20:10 +0000

Flutter et Travis CI

In my previous article, I talked about test-driven development(TDD) with Flutter which you can read here. This time around, we would try to set up for Travis-CI and integrate Flutter app with Coveralls, which will give us feedback on test coverage in your app.

I hope this guide helps you out ** 😀 **.

Requirements

The Flutter app should be hosted on GitHub, GitLab or Bitbucket.
Travis CI account
Flutter SDK

Create a Github/BitBucket/GitLab Project

Create your flutter application and push it to GitHub/BitBucket/GitLab. For the purpose of this tutorial, I will be using the PiggyX application which is to demonstrate the concept of test-driven development in Flutter. The codes are available on Github. Click here to access the codes. You can read the previous article on test-driven development here.

Integrating with Travis CI

You need to first register your GitHub project/repository on Travis-CI dashboard. This way, travis can detect changes in your repository. Go to https://travis-ci.org/ if your repository is public, or https://travis-ci.com/ if it is private.

In Travis, you implement the pipeline under the YAML schema, defined on a file named .travis.yml. To get things going on Travis all you need is add the correct .travis.yml to the root of your repository and tell Travis to start a continuous build of your repository by following their instructions. Here I’ll only explain what goes in the .travis.yml file. Below is the configuration I settled on:

language: dart
dist: xenial
addons:
  apt:
    packages:
      - lib32stdc++6
install:
  - git clone https://github.com/flutter/flutter.git -b stable
  - ./flutter/bin/flutter doctor
script:
  - ./flutter/bin/flutter test
cache:
  directories:
    - $HOME/.pub-cache

Language

This tailors the CI environment for a specific language. Leaving out this part tells travis to use the default language Ruby.

Dist

This specifies the Ubuntu base version that gets used. I’m specifying Xenial Xerus which is supported until 2021.

Addons

This lets you have additional packages installed on the OS. In my experimentation, tests still pass without lib32stdc++6 but flutter doctor complained so I'm including it.

Install

This should call Dart’s pub get but Flutter complains so let’s just override that and install Flutter instead.

Script This is where we run the tests. So we provide the means of running the test here.

Cache

Here is the final part. This tells Travis to cache the contents of the default Dart package cache. This should make it faster for Flutter to install the app’s dependencies.

This configuration runs well and takes under 2minutes 30seconds to complete.

Adding Github Project to Coveralls

To add a project to Coveralls, logon to https://coveralls.io/ and Sign In with your GitHub/GitLab/BitBucket account, go to Add Repos and pick repository you would like to integrate. Then, when you go to Repos, you should see your project.

coveralls.io

Updating .travis.yml

To call coveralls on Travis build you need to change .travis.yml file. All we need to do is tweak it a little to generate and send coverage report.

language: dart
dist: xenial
addons:
  apt:
    packages:
      - lib32stdc++6
install:
  - git clone https://github.com/flutter/flutter.git -b stable
  - ./flutter/bin/flutter doctor
  - gem install coveralls-lcov
script:
  - ./flutter/bin/flutter test --coverage
after\_success:
  - coveralls-lcov coverage/lcov.info
cache:
  directories:
    - $HOME/.pub-cache

We have added two more lines and edited the script:

Install : we added gem install coveralls-lcov to install coveralls

Script : we changed the script to ./flutter/bin/flutter test — coverage to generate coverage report on tests.

after_success : we added coveralls-lcov coverage/lcov.info here to send the coverage report.

So that is it, you have now configured travis-ci and coveralls with flutter and should be able to see your test coverage 😄 😄 😄.

Bonus Points

You can add a badge to the project readme to allow people to know how the project is tested. To add a coverage/coveralls badge, sign in and open the project on coveralls, click on settings.

Project board coveralls.io

In the repo settings, you would see an Embed dropdown button beside a readme badge. Clicking on this button would provide you with a variety of badge option. For this project, I used the Markdown variety and added to the project README.md

Repo settings coveralls.io

To add travis build status badge, sign in and open the project on travis. You should see a badge with a build status on the project dashboard. The build status varies and could be unknown, failed, passed, error as the case may be. Click on this build status badge.

project dashboard travis-ci

Some sort of popup should appear on your screen providing you options for the status image. For this project, I integrated on the master branch and used the markdown status image.

status image travis-ci

And that’s it! Now after proper build on Travis, you will be able to see your test coverage and some shiny badge in your project💃 💃 💃. The full example is available here.

https://github.com/Mastersam07/PiggyX/blob/master/README.md

If you have any questions, feel free to leave a comment 🙂.

Setting Up CI/CD For Flutter With Codemagic

Samuel Abada — Sat, 18 Jan 2020 07:29:26 +0000

CI/CD For Flutter Apps

Building good software and writing code quickly to achieve business goals seems like some dilemma as claimed by some programmers. Developers want to build products faster and also write good software at the same time. With CI/CD (Continuous Integration/Continuous Delivery), this is no longer an issue and yes, everyone wins 😊 😊 😊.

The purpose of CI/CD is to continuously merge working product versions to the main development branch and frequent build execution to discover potential bugs and resolution of integration problems. CI improves on the stability of the product with each development cycle since each time code is pushed, automatic builds are triggered and integration tests are run to ensure that the new units being added do not “break” the previously successfully built application. If the test fails, the build fails hence ensuring that the app remains stable. When it comes to preparing an app for release, that is “Continuous Delivery” as “Continuous Deployment” is responsible for automatic deployment to production. I guess it’s clear now 😉 😉 😉.

Today, there are a good number of tools for setting up CI/CD for many programming languages. I will be walking us through setting up CI/CD for a Flutter application with CodeMagic.

https://blog.codemagic.io/getting-started-with-codemagic/

CodeMagic is a CI/CD tool and like other CI/CD tools, it starts with a Git repository or some decentralized repository at least. When code is pushed to this repository, it triggers a new build process. If the build fails at any point, it will halt any further progress of the CI/CD process and give logs so we know where exactly the issue came from. Successful builds deliver the application artefacts for iOS and Android for installation on the respective devices. Codemagic allows to test and release Flutter apps without issues and with no configuration. As a developer, you can run custom scripts and create custom workflows for your apps. Cool right 😎😎😎? Lets Start…

Requirements

Setting up your Flutter app on Codemagic, you need a couple of things.

Obviously, your Flutter app with some tests.
The Flutter app should be hosted on GitHub, GitLab or Bitbucket.
Optional code signing details, like certificates and provisioning profiles, if you want to publish to App Store or Play Store.
Optional Slack workspace for sending build reports and artefacts.

Now, we are ready to add CI/CD to our Flutter app. However, this writeup will not cover code signing and publishing.

Create a Github Project

Create your flutter application and push it to GitHub. For the purpose of this tutorial, I will be using the WhatsApp status saver built in my previous article. The codes are available on Github. Click here to access the codes. You can read the previous articles here and here and follow the development process.

Register For CodeMagic

Visit codemagic.io and sign up using your GitHub, Bitbucket or GitLab account. You can connect all of these repositories with one Codemagic account, so you can have all of your apps in one place.

Your First Build

Once you sign in, you will see the Codemagic dashboard with the app repositories. You can filter the list or search apps to find your Flutter app.

Application Overview

For apps which you are configuring for the first time, you can start by clicking “Start your first build” button. In the above screenshot, we have a not applicable button as this app has been built previously.

The Build Workflow

When you start a build, Codemagic will create a workflow for your Flutter project. You can also customize it to suit your needs.

Build Workflow

From the above, we can see that the default workflow will start the build with the following steps — preparing the build machine, fetching app sources(checking out the source code), installing Flutter dependencies, running tests, building the apps and publishing the artefacts.

Build Report

Below is an overview of the details when the build is finished.

Build Report

An email will also be sent to you with all these details. You can also set up your Slack channel to receive the build reports. The mail you receive should look like this:

Yes, we have just added CI/CD support for our Flutter app without any configuration 😋 😋 😋. This is the magic in Code-magic 😁😁😁.

But wait, what if your build fails? 😪 😪 😪. Do not worry, it will not end in tears 😉.

Build Failures

Well, not all builds are successful. I have had my own fair share of failed builds 😪 😪 😪 but then we can always resolve it as Codemagic provides very detailed logs of each step regardless of whether it passes or fails 😃 😃 😃. Here we can see a failing unit test and all of the details are logged in the build failure.

blog.codemagic.io

As seen above, integration tests and widget tests are passing, but the unit test is failing. We can then fix the problem to fix the build. When tests fail, you will also get an email about the test failure with link to the build logs. I will cover integration, unit and widget tests in another article 🤓 🤓 🤓.

Default Workflow And Custom Workflow

Default flow covers all the basics needed for the CI/CD but then, you will want to add some custom things as part of your build process. You may want to send build status to some third party communication channel such as Slack for example. Let’s dive into customizing a workflow.

Build Phase

Codemagic build phase of Flutter apps has various options to define the build process. You can start by selecting which branches to build and how to trigger builds.

You can also select other categories like Flutter version, build configuration, platform, Xcode version and custom build arguments.

Above is the default workflow settings. You can customize the build process by changing the default behaviour. Also, you can add a custom script before every phase. It's easy to setup. Just click on the + sign on top of each phase. See the below image for what you are expected to see.

You can add any command in the pre-build script say you want to install some dependencies.

Test Phase

Here, you can enable or disable execution. This phase has two sections, one for the unit and widget tests and the other is for the integration tests. The test settings are kind of straightforward. You set how the tests affect the build. You can enable Flutter analyzer, enable Flutter test which runs the unit tests in your application. You can deactivate the automatic build failures resulting from failed tests here, though this is not a recommended option for builds going out to production at least. Always save the settings for each individual section since there is no universal button to save all settings. Codemagic automatically recognizes the widget, unit and integration tests and enables them by default as you can see below.

we can use custom test commands by disabling the default behaviour. For example, say we want to run just UI tests from the command line, we can add a pre-test phase with the command:

$flutter drive --target=test_driver/main.dart

Publish Phase

By default, only the email is configured if you have provided it with GitHub/Bitbucket/GitLab as we have to explicitly configure any other aspects of the Publishing phase, including:

Uploading the code signing assets for Android and iOS
Setting up slack
Providing Google Play and App Store Connect details for publishing

Even with these, you can also write some custom scripts to deploy the artefacts to third-party services. With the power of scripting, we can configure custom workflows in Codemagic to define a stronger build process for Flutter apps.

Multiple workflows

Codemagic allows us to create multiple workflows to meet different requirements of various configurations and software versions. Create a new workflow by duplicating the existing one and renaming it. To do this, follow these easy steps:

Go to app settings.
In the Workflow settings section, click Duplicate workflow. Select the workflow and rename it.

Build Status Badge

You want to include a build status badge to your GitHub repo so the status of your build is shown? Say no more. Click on the settings icon in your Codemagic build. A screen as shown below should appear.

At the right-hand side, we can see a build badge. For builds that passed, colour is green and for failed builds, the colour is red. You could choose to make the builds available publicly. You can also add the badge to your repo by either a markdown or a badge URL which you could add to your README.md. Click on the arrow/dropdown button beside the badge and you should see the badge URL and markdown. Copy it and use where necessary. We should have a shiny badge as shown below in the Github repository.

https://github.com/Mastersam07/wa_status_saver

And just like that, your Workflow is set and all the triggers are in place 😍 😍 😍. When you push code, make a pull request, it will trigger the magic and boom your app will be delivered and deployed to all interested parties through the preset channels.

Head to codemagic.io and try it out. The user experience is 💥 💥 💥.

You can also visit Codemagic blog — https://blog.codemagic.io/getting-started-with-codemagic/