Mihir Shinde

Posted on Apr 17 • Originally published at kleore.com

How to Find and Fix Flaky Tests in pytest

#pytest #testing #python #cicd

pytest is the gold standard for Python testing. But its powerful fixture system and Python's dynamic nature create unique flakiness patterns that only surface in CI.

This guide covers the five most common causes of flaky pytest tests and gives concrete fixes for each.

Why pytest tests become flaky

Database state leaking between tests — Tests share a DB and don't isolate transactions
Time-dependent assertions — Tests use datetime.now() directly
Network calls to real services — Real HTTP requests to external APIs
File system conflicts — Tests write to the same paths with pytest-xdist
Import side effects — Modules that execute code at import time

How to identify flaky pytest tests

pytest-randomly shuffles test order to surface ordering dependencies. pytest-repeat stress-tests a suspected flake by running it N times. pytest-rerunfailures automatically reruns failed tests — tests that pass on rerun are flaky by definition.

The 5 patterns and fixes

Pattern 1: Database state leaking

Use autouse=True fixtures that wrap every test in a transaction rollback.

@pytest.fixture(autouse=True)
def reset_db(db):
    yield
    db.session.rollback()
    db.session.remove()

Pattern 2: Time-dependent tests

Mock time with freezegun. Never let tests depend on the wall clock in CI.

from freezegun import freeze_time

@freeze_time("2026-01-15 10:00:00")
def test_token_expiry():
    token = create_token(expires_in=3600)
    assert not token.is_expired()

Pattern 3: Network calls

Mock HTTP with the responses library. Tests should never reach real external APIs.

import responses

@responses.activate
def test_fetch_user():
    responses.add(responses.GET, "https://api.example.com/user/1",
                  json={"id": 1, "name": "Alice"}, status=200)
    result = fetch_user(1)
    assert result["name"] == "Alice"

Pattern 4: File system conflicts

Use pytest's tmp_path fixture for isolated per-test directories.

def test_write_report(tmp_path):
    report_file = tmp_path / "report.json"
    write_report(report_file)
    assert report_file.exists()

Pattern 5: Import side effects

Patch module-level initialization and reset global state in fixture teardown.

@pytest.fixture(autouse=True)
def reset_global_config():
    original = config.copy()
    yield
    config.clear()
    config.update(original)

CI configuration tips

Set PYTHONDONTWRITEBYTECODE=1 to prevent .pyc conflicts in parallel runs. Set PYTHONHASHSEED=0 for deterministic dictionary ordering.

Quarantining while you fix

pytest-quarantine marks known-flaky tests so they don't block CI. For automated detection, Kleore analyzes your GitHub Actions history and ranks every flaky test by failure rate and dollar cost.

DEV Community