Flaky tests degrade confidence in CI/CD pipelines, delay releases, and waste engineering cycles. Managing retries, test isolation, and worker parallelism strategically allows teams to control execution speed while ensuring test reliability.
Here is a practical guide to handling test flakiness, configuring retries, and establishing actionable stability thresholds for Playwright and automated test suites.
Core Recommendations for Test Stability
Retry Policy: Keep auto-retries low (1–2 attempts max). Use retries only for transient network blips and explicitly track retry attempts in execution logs.
Test Isolation: Ensure every test scenario is stateless. Clean up shared artifacts (database records, downloaded files, local storage) before and after each execution.
Stable Selectors: Prefer dedicated test attributes (e.g., data-test-id) over fragile layout-dependent CSS or XPath locators.
Controlled Parallelism: Start CI runs with limited workers (e.g., workers: 2) to avoid resource exhaustion on runner nodes. Scale parallelism up only after flakiness stays consistently low.
Flakiness Thresholds & Escalation Matrix
To keep test suites healthy, establish clear operational thresholds based on a rolling 7-day window:
- Acceptable (< 1% intermittent failures): Standard operational state; monitor trends routinely.
- Warning (1% – 5% intermittent failures): Trigger an immediate investigation and root-cause analysis sprint task.
- Critical (> 5% intermittent failures): Block release deployments until infrastructure and test design issues are resolved.
Observability & Reporting
- Emit Metrics: Include retry counts and flaky test flags directly inside generated test execution reports (e.g., Allure or JSON reporters).
- Track Dashboards: Maintain a weekly flakiness dashboard tracking total tests executed against failure rates and retry counts to catch degrading tests early.
Top comments (0)