How to Identify and Mitigate Flaky Tests: Best Practices and Strategies.

Enhancing Test Reliability and Efficiency in CI/CD Pipelines

A flaky test is a test that sometimes passes and sometimes fails without any changes to the code being tested. These tests can be particularly troublesome because they undermine the reliability of the test suite.

Consider your CI/CD pipeline is configured such that only after the build is passed, only if your code passes a set of predefined test cases.

In an ideal situation, you must have set the priority for each test case and assume the latest code base to pass at least some percentage of cases.

But due to the flaky test cases, which keep on failing, as they might be stale or the use case is changed your test case fails and merging the pull request becomes a nightmare. Instead of reducing the percentage of passing cases, we should consider revamping those test cases.

Reason for understanding Flaky Test.

1. Unpredictable Test Results: Flaky tests cause unpredictability by sometimes passing and other times failing, even though the code hasn’t changed. This randomness can make it difficult to trust test outcomes.

2. Complex Debugging: Tracking down the root cause of a flaky test can be challenging because the issue may not reproduce consistently, making it hard to identify and fix.

3. Wasted Time and Resources: Developers can spend a significant amount of time rerunning tests, investigating false positives, and debugging issues that aren’t actually related to the code’s functionality.

4. Impact on Continuous Integration (CI): Flaky tests can disrupt continuous integration pipelines, leading to unnecessary build failures and reducing the overall efficiency of automated testing processes.

5. False Confidence or Distrust: Flaky tests can either create false confidence when they pass sporadically or cause distrust in the test suite when they fail unpredictably, making it harder to rely on test results.

Ways to mitigate Flaky test cases.

1. Best Practices to Mitigate: To reduce flaky tests, developers can mock external dependencies, use deterministic data, ensure tests are isolated, and avoid relying on timing or order of execution.

2. Automated Detection: Implementing automated tools that detect flaky tests by running tests multiple times and comparing results can help identify and address flakiness early in the development cycle.

3. Test Isolation: Ensuring that each test runs in complete isolation, without relying on shared states or external factors, can significantly reduce the chances of flakiness.

4. Regular Maintenance: Regularly reviewing and refactoring the test suite to remove or fix flaky tests helps maintain the integrity and reliability of the testing process over time.

Different strategies and tools to mitigate flaky test cases

1. Jenkins, CircleCI, Travis CI: Continuous Integration/Continuous Deployment (CI/CD) tools like these can be configured to rerun tests that fail, helping to identify flaky tests. They often have plugins or built-in support for handling flaky tests.

2. Docker: Companies use Docker to create isolated environments for running tests. This ensures that tests have a consistent and clean environment each time they are executed, reducing flakiness caused by environmental differences.

3. Virtual Machines (VMs): Similar to Docker, VMs can be used to ensure tests run in a controlled and isolated environment, minimizing interference from other processes or dependencies.

4. Statistical Analysis using Machine Learning: Some advanced systems use machine learning to analyze test results and identify patterns indicative of flaky tests. This can help in proactively identifying and addressing flakiness.

5. Code Review Policies and Version Control Hooks: Implementing strict code review policies that include checks for potential sources of flakiness can prevent flaky tests from being introduced.
   Using pre-commit hooks or other version control mechanisms to run tests in a controlled manner before changes are merged can catch flaky tests early.

Strategies by some of the big organisations

1. Google:

• Rerun Failed Tests: Google has a policy where they rerun tests that fail to determine if the failure is consistent. This helps identify flaky tests. They also have internal tools and infrastructure to manage and mitigate flakiness across their extensive test suites.

• Test Isolation: Google emphasizes the importance of test isolation to ensure that tests do not interfere with each other, which is critical in reducing flakiness.

1. Microsoft:

• Test Analytics and Reporting: Microsoft uses detailed test analytics and reporting tools to track flaky tests. By analyzing test results over time, they can identify patterns and pinpoint flaky tests.

• Quarantining Flaky Tests: Microsoft sometimes quarantines flaky tests, separating them from the main test suite until they are fixed to prevent them from affecting the overall test results.

3. Facebook:

• Detox: Facebook developed an open-source library called Detox to test their mobile apps. Detox ensures that tests are run in a consistent state and environment, reducing flakiness caused by asynchronous operations and other timing issues.

• Continuous Testing: Facebook integrates continuous testing into their development process, using tools to automatically rerun tests and identify flaky behavior early in the development cycle.

4. Netflix:

• Chaos Engineering: Netflix employs chaos engineering practices to test the resilience of their systems. By intentionally introducing failures and disruptions, they can identify flaky tests and improve the robustness of their tests and systems.

• Automated Retrying: Netflix uses automated retry mechanisms within their CI/CD pipelines to rerun tests that fail intermittently, helping to identify and manage flaky tests.

5. LinkedIn:

• Flaky Test Management Tools: LinkedIn has developed tools specifically for managing flaky tests. These tools help track flaky tests, provide visibility into their occurrence, and prioritize their resolution.

• Test Environment Standardization: LinkedIn focuses on standardizing test environments to reduce variability and ensure that tests run under consistent conditions, which helps mitigate flakiness.

About The Author

Apoorv Tomar is a software developer and blogs at **Mindroast. You can connect on social networks. Subscribe to the **newsletter for the latest curated content.