DEV Community: TheNerdyGeek

How to run parallel tests with CircleCI

TheNerdyGeek — Thu, 17 Mar 2022 19:50:38 +0000

More often than not, testers have to deal with a large codebase comprising hundreds or even thousands of tests. Most tests validating website or app functionality must be rerun on different platforms (devices, browsers, operating systems).

However, running so many tests on different platforms can be immensely difficult, not to mention tedious and error-prone, without the right strategy and tech. The best and easiest way to test under such conditions is to use parallel testing.

This article will explore how to run parallel tests with a particularly popular and useful CI/CD tool like CircleCI.

But, first thing’s first.

What is parallel testing?
How to run tests in parallel with CircleCI
How to set up parallelism in CircleCI
How to split test files using CircleCI CLI
Globbing the tests
Splitting the tests
Split By Time
How to split tests using environment variables
How to run split tests

What is parallel testing?

In parallel testing, you test different modules or applications on multiple browser-device-OS combinations simultaneously rather than one after another. In a scenario where we have two versions of software available, and both must be examined to ensure stability and compatibility, it is easier to run tests on two versions simultaneously, get results and detect bugs faster

Parallel testing reduces execution time and effort, resulting in faster time to delivery. It is particularly useful in the case of cross browser testing, compatibility testing, localization, and internalization testing.

How to run tests in parallel with CircleCI

Before exploring parallel testing with CircleCI, it’s important to place a quick note on why it’s important to run parallel tests on real browsers and devices.

While it may seem easier to download and test on emulators and simulators, they are riddled with inadequacies that prevent them from providing a reliable test environment. For example, they cannot mimic low network conditions, low battery, or incoming calls. In general, they cannot replicate all features of real browsers and devices, making the results of tests run on them prone to serious error.

Now, on to CircleCI.

CircleCI provides us with capabilities to run tests in parallel and achieve faster completion.

When you run your tests with CircleCI, they usually run on a single VM. Now, the more tests, the more time it will take to complete on one machine. In order to reduce this time, you can run tests in parallel by distributing them across multiple separate executors.

In order to do this, you need to specify the parallelism level to define how many separate executors should be spun up for your job. After that, for separating the test files, you can either use CircleCI CLI or environment variables to configure each machine individually.

How to set up parallelism in CircleCI

Test suites are usually defined at the job level in your .circleci/config.yml file. So let’s first set up the parallelism level on your config file.

This parallelism key defines how many independent executors CircleCI will create to run your tests. To do this, you just need to add this key before calling the steps to run your tests.

Depending on your CircleCI’s pricing plan, the parallelism value can be increased to match your requirements.

To run a job’s steps in parallel, set the parallelism key to a value greater than 1.

# ~/.circleci/config.yml

version: 2

jobs:

  test:

    docker:

      - image: cimg/<language>:<version TAG>

        auth:

          username: yourdockerhub-user

          password: $DOCKERHUB_PASSWORD  

    parallelism: 4

Now, in order to split the actual tests, the next step is to use the CircleCI CLI (Command Line Interface) to call additional commands to split the tests.

How to split test files using CircleCI CLI

CircleCI supports automatic test allocation across your containers. The allocation is based on filename or class name, depending on the requirements of the test-runner we are using. It requires the CircleCI CLI, which is automatically injected into your build at run-time.

The commands that we will use for splitting the test files are glob and split.

A glob is basically a pattern that will match a list of filenames to a specific pattern that you specify.
The split command, by default, will split your tests according to a list of filenames, but you can also use other strategies such as splitting it by timing data or file size. By using these two commands, you can split your tests into multiple executors that are independent of each other as you can see from the image above.

Globbing the tests

In order to assist in defining your test suite, the CircleCI CLI supports globbing test files using the following patterns:

* matches any sequence of characters (excluding path separators)
** matches any sequence of characters (including path separators)
? matches any single character (excluding path separators)
[abc] matches any character (excluding path separators) against characters in brackets
{foo, bar,…} matches a sequence of characters if any of the alternatives in braces matches So to glob test files, pass one or more patterns to the CircleCI tests glob command.

Now, let’s look at how we can achieve this in our configuration file.

circleci tests glob "tests/unit/*.java" "tests/functional/*.java"

You can use the glob command in your config file with the echo command as shown below, to check the results of pattern match:

# ~/.circleci/config.yml

version: 2

jobs:

  test:

    docker:

      - image: cimg/<language>:<version TAG>

        auth:

          username: mydockerhub-user

          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference

    parallelism: 4

    steps:

      - run:

          command: |

            echo $(circleci tests glob "foo/**/*" "bar/**/*")

            circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo

Splitting the tests

The split command, by default, will split your tests according to a list of filenames but you can also use other strategies such as splitting it by timing data or file size. Let’s discuss all of the splitting techniques.

Split By Name

This is the default splitting technique. So in case you don’t specify a method using the –split-by flag, CircleCI expects a list of filenames/classnames and splits them alphabetically by test name.

There are a few ways to provide this list:

Create a text file with test filenames

circleci tests split test_filenames.txt

Provide a path to the test files

circleci tests split < /path/to/items/to/split

Pipe a glob of test files

circleci tests glob "test/**/*.java" | circleci tests split

Once the list of names has been provided, CLI looks up the number of available containers, along with the current container index, and then it uses the deterministic splitting algorithms to split the test files across all available containers.

Split By Time

This is the best way to optimize your test suite across a set of parallel executors. This ensures that the tests are split in the evenest way, leading to overall reduced test time.

In order to split by test timings, use the –split-by flag with the timings split type. The available timings data will then be analyzed and your tests will be split across the parallel-running containers as evenly as possible.

circleci tests glob "**/*.go" | circleci tests split --split-by=timings

On each successful run of a test suite, the timings data is saved from the directory specified by the path in the store_test_results step. This timings data mentions how long each test took to complete per filename or classname, depending on the language used.

One thing to be considered is if you do not use store_test_results, there will be no timing data available for splitting your tests.

The CLI expects both filenames and classnames to be present in the timing data produced by the test suite. By default, splitting defaults to the filename, but you can specify classnames by using the –timings-type flag.

cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classname

For partially found test results, we automatically assign a random small value to any test we did not find timing data for. You can override this assigned value to a specific value with the –time-default flag.

circleci tests glob "**/*.rb" | circleci tests split --split-by=timings --time-default=10s

Additionally, If you need to manually store and retrieve timing data, use the store_artifacts step.

Splitting by filesize

You can also split tests by filesize. To do this, use the –split-by flag with the filesize split type.

circleci tests glob “**/*.go” | circleci tests split –split-by=filesize

How to split tests using environment variables

CircleCI provides us with two environment variables that can be used in place of the CLI to configure each container individually.

They are CIRCLE_NODE_TOTAL **and **CIRCLE_NODE_INDEX.

**CIRCLE_NODE_TOTAL **is the total number of parallel containers being used to run your job.
**CIRCLE_NODE_INDEX **is the index of the specific container that is currently running. The environment variables ensure us full control over parallelism.

How to run split tests

Now after globbing and splitting the tests, it’s time to run them. To combine test grouping with test execution, you can consider saving the grouped tests to a file and then passing this file to your test runner.

circleci tests glob "test/**/*.rb" | circleci tests split > /tmp/tests-to-run

bundle exec rspec $(cat /tmp/tests-to-run)

The contents of the file /tmp/tests-to-run will be different in each container, based on $CIRCLE_NODE_INDEX and $CIRCLE_NODE_TOTAL.

Conclusion

Applying test parallelism using CircleCI regardless of your underlying technical approach is always beneficial because it will speed up your overall execution time and in some cases, can even reduce the time by half.

Utilize the steps in this article to run parallel tests in CircleCI, thus getting faster results and integrating speed into your CI/CD pipeline with ease.

How to run integration tests on Flutter apps

TheNerdyGeek — Mon, 07 Mar 2022 05:26:21 +0000

Flutter, designed by Google, is a free and open-source framework for building visually appealing, natively compiled, and multi-platform applications from a single codebase.

Flutter has become quite popular due to its ability to create multi-platform apps. It enables users to build, deploy and test web, mobile, desktop, and embedded applications from a single codebase.

In this article, let’s explore how to run integration tests on Flutter apps.

Introduction To Flutter
Features of Flutter Framework
Testing In Flutter Apps
Integration Testing In Flutter Apps
Migrating From Flutter Driver
Integration_Test Package
Steps To Perform Integration Testing In Flutter Apps
Integration Tests In Flutter Apps With BrowserStack

Introduction To Flutter

Flutter is an open-source mobile UI framework that can create Android and iOS apps (among other platforms) from a single codebase.

Generally, frameworks offer different features to develop mobile applications. For example, Android provides a framework based on Java and Kotlin to develop mobile apps, whereas iOS uses a framework based on Objective-C or Swift language.

As a result, Devs had to use two different languages and frameworks to develop applications for both OS. Naturally, a cross-platform framework like Flutter makes life simpler for them by saving resources, time, and effort.

Features of Flutter Framework

Facilitates fast development
Multi-Platform support
Provides a modern and reactive framework
Uses the easy to learn Dart programming language
Fluid and intuitive user interfaces
Vast widget catalog
Runs the same UI for multiple platforms
Stable and Reliable

Testing In Flutter Apps

Flutter supports three types of tests:

Unit Test
Widget Test
Integration Test

A unit test verifies the behavior of a method or class.
A widget test checks the behavior of Flutter widgets without having to run the app.
An integration test (also called end-to-end testing or GUI testing) runs the entire application.

Integration Testing In Flutter Apps

Unit tests and Widget tests can test individual classes, widgets, or functions. However, they don’t verify how individual components work together or monitor application performance on real devices. This is where integration tests come in.

Integration tests in Flutter are written using the integration test package, provided by the SDK. This is Flutter’s version of Selenium WebDriver (generic web), Protractor (Angular), Espresso (Android), or Earl Gray (iOS). The package internally uses flutter driver to drive the test on a device.

Tests written with the integration_test package can:

Run directly on the target device, enabling tests on numerous Android or iOS devices.
Run using flutter test integration_test.
Use flutter_test APIs, which makes the creation of integration tests similar to creating widget tests.

Migrating From Flutter Driver

When Flutter was introduced, integration tests were written using flutter_driver, which enabled testers to maintain programmatic control of a Flutter app. On running the futter drive command, it initiates 2 processes:

A “driver” process on the host machine (i.e. the developer’s laptop) that sends instructions to and receives data from the app.
The app is configured to identify connections coming from the driver process.
This allowed the developer to write tests in plain Dart that could interact with the app. However, it came with significant disadvantages:
The driver code couldn’t share any code with the app or unit tests. In fact, it couldn’t leverage any package: flutter dependencies.
It relied on strongly-typed APIs because it could not import Key, Widget, or MyCustomWidget. find.byType(‘EvnetsPage’) was often mistyped and misread.

Integration_Test Package

The integration_test package was released to fix some of the issues mentioned above. Here, the code for tests runs in the same isolate as the app code itself. In other words, it can access the same memory. This essentially resolves the previous issues, and offers a few other benefits:

Same API as component tests.
Can share code with the app.
Internal app state is totally visible to tests, and runApp is called inside the test.
Since tests are built into the app executable, they can now run on physical devices.

Steps To Perform Integration Testing In Flutter Apps

This example will demonstrate how to test a counter app. This demonstrates how to set up integration tests, how to verify if a specific text is being displayed by the app, how to tap specific widgets, and how to run integration tests.

In order to perform the Integration test, follow the below steps:

Create an app to test.
Add the integration_test dependency.
Create the test files.
Write the integration test.
Run the integration test.

Create an app to test

First, create an app for testing. This test will use a counter app produced by the flutter create command. This app will let a user tap on a button to increase one count.

import 'package:flutter/material.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {

  const MyApp({Key? key}) : super(key: key);

  @override

  Widget build(BuildContext context) {

    return const MaterialApp(

      title: 'Counter App',

      home: MyHomePage(title: 'Counter App Home Page'),

    );

  }

}

class MyHomePage extends StatefulWidget {

  const MyHomePage({Key? key, required this.title}) : super(key: key);

  final String title;

  @override

  _MyHomePageState createState() => _MyHomePageState();

}

class _MyHomePageState extends State<MyHomePage> {

  int _counter = 0;

  void _incrementCounter() {

    setState(() {

      _counter++;

    });

  }

  @override

  Widget build(BuildContext context) {

    return Scaffold(

      appBar: AppBar(

        title: Text(widget.title),

      ),

      body: Center(

        child: Column(

          mainAxisAlignment: MainAxisAlignment.center,

          children: <Widget>[

            const Text(

              'You have pushed the button this many times:',

            ),

            Text(

              '$_counter',

              // Provide a Key to this specific Text widget. This allows

              // identifying the widget from inside the test suite,

              // and reading the text.

              key: const Key('counter'),

              style: Theme.of(context).textTheme.headline4,

            ),

          ],

        ),

      ),

      floatingActionButton: FloatingActionButton(

        // Provide a Key to this button. This allows finding this

        // specific button inside the test suite, and tapping it.

        key: const Key('increment'),

        onPressed: _incrementCounter,

        tooltip: 'Increment',

        child: const Icon(Icons.add),

      ),

    );

  }

}

Source

Add the integration_test dependency

Now, use the integration_test, flutter_driver, and **flutter_test **packages for writing integration tests.

Add these dependencies to the dev_dependencies section of the app’s pubspec.yaml file. The location of the package should be the Flutter SDK.

# pubspec.yaml

dev_dependencies:

  integration_test:

    sdk: flutter

  flutter_test:

    sdk: flutter

Create the test files

Create a new directory, integration_test, with an empty app_test.dart file:

counter_app/

  lib/

    main.dart

  integration_test/

    app_test.dart

Source

Write the integration test

Now, write the integration test with the steps below:

Initialize IntegrationTestWidgetsFlutterBinding, a singleton service that runs tests on a physical device.
Interact and test widgets via the WidgetTester class.
Test the necessary scenarios.

import 'package:flutter_test/flutter_test.dart';

import 'package:integration_test/integration_test.dart';

import 'package:introduction/main.dart' as app;

void main() {

  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  group('end-to-end test', () {

    testWidgets('tap on the floating action button, verify counter',

        (WidgetTester tester) async {

      app.main();

      await tester.pumpAndSettle();

      // Verifies that the counter starts at 0.

      expect(find.text('0'), findsOneWidget);

      // Finds the floating action button to tap on.

      final Finder fab = find.byTooltip('Increment');

      // Emulates a tap on the floating action button.

      await tester.tap(fab);

      // Triggers a frame.

      await tester.pumpAndSettle();

      // Verifies if the counter increments by 1.

      expect(find.text('1'), findsOneWidget);

    });

  });

}

Source

Run the integration test

The process of executing integration tests depends greatly on the platform used. If the platform so permits, You can test on mobile devices or web browsers.

Mobile

To test on a real iOS / Android device, first, connect the device and run the following command from the root of the project:

flutter test integration_test/app_test.dart

Or, specify the directory to run all integration tests:

flutter test integration_test

This command runs the app and integration tests on the target device.

To test on a web browser (Chrome, in this example), Download ChromeDriver.

Next, create a new directory named test_driver containing a new file named integration_test.dart:

import 'package:integration_test/integration_test_driver.dart';

Future<void> main() => integrationDriver();

Launch WebDriver, for example:

chromedriver --port=4444

From the root of the project, run the following command:

flutter drive \

  --driver=test_driver/integration_test.dart \

  --target=integration_test/app_test.dart \

  -d web-server

Source

DEV Community: TheNerdyGeek

How to run parallel tests with CircleCI

Table of Contents

What is parallel testing?

How to run tests in parallel with CircleCI

How to set up parallelism in CircleCI

How to split test files using CircleCI CLI

Globbing the tests

Splitting the tests

Split By Name

Split By Time

Splitting by filesize

How to split tests using environment variables

How to run split tests

Conclusion

How to run integration tests on Flutter apps

Table of Contents

Introduction To Flutter

Features of Flutter Framework

Testing In Flutter Apps

Integration Testing In Flutter Apps

Migrating From Flutter Driver

Integration_Test Package

Steps To Perform Integration Testing In Flutter Apps

Create an app to test

Add the integration_test dependency

Create the test files

Write the integration test

Run the integration test