Pytest - Production test Coverage

Production-Ready Pytest

You've written solid tests — fixtures keep setup DRY, parametrization makes edge cases explicit, and temporary directories prevent leaked artifacts. Now the question is: how do you run these tests in a way that gives you fast feedback, readable output, and reliability in CI? This final post covers logging configuration, splitting unit and integration tests with markers, code coverage done honestly, and a few principles I've learned from maintaining test suites for data pipelines that process millions of transactions.

Running Tests with Colorful Logs and the Right Verbosity

Logs are invaluable when a test fails and you're trying to understand why. To see your application's logs (and any print statements) during the test run, use:

pytest -v --log-cli-level=INFO --color=yes

• -v gives verbose output, listing each test name.
• --log-cli-level=INFO sends logs at INFO level and above to the console. Set it to DEBUG if you need more detail.
• --color=yes ensures the output uses colours, making it much easier to scan.

If you need to capture logs inside a test (for example, to assert that a certain warning was emitted), use the caplog fixture that pytest provides. It captures log messages emitted by your code under test, letting you assert on their presence and content without setting up complex logging handlers:

def test_warning_on_negative_values(caplog):
    with caplog.at_level("WARNING"):
        clean_amount(-50.0)
    assert "Negative value received" in caplog.text

Splitting Tests by Markers: Unit vs. Integration

Not all tests are equal. Some are pure unit tests that run in milliseconds; others are integration tests that need a real database, an S3 bucket, or cloud credentials. You don't want those running on every CI push if the environment isn't set up for them.

Custom Markers

Define markers in pytest.ini or pyproject.toml:

# pytest.ini
[pytest]
markers =
    integration: marks tests that need external resources (deselect with '-m "not integration"')

Tag your integration test:

@pytest.mark.integration
def test_glue_catalog_connection():
    client = boto3.client("glue")
    # … test that requires real AWS credentials

Now, in your CI pipeline, run only unit tests:

pytest -m "not integration"

Locally, you can run the full suite when you need it.

Auto‑Skipping with a Fixture

For extra safety, create an autouse fixture in conftest.py that automatically skips integration tests unless an environment variable is set:

import os
import pytest

@pytest.fixture(autouse=True)
def skip_integration_tests(request):
    if request.node.get_closest_marker("integration") and not os.getenv("RUN_INTEGRATION"):
        pytest.skip("Set RUN_INTEGRATION=1 to run integration tests")

Now your CI stays green even if someone forgets to filter with -m, and you have an explicit opt‑in mechanism for the heavy tests.

Code Coverage: What It Tells You and What It Doesn't

Code coverage measures which lines of your production code are executed during a test run. It's a useful signal — but only if you treat it as a floor, not a ceiling. A line being covered means it ran, not that it ran correctly. I've seen pipelines with 95% coverage that still produced wrong numbers because the assertions were weak.

Setting Up pytest-cov

Install pytest-cov and run it alongside your tests:

pip install pytest-cov
pytest --cov=my_pipeline --cov-report=term-missing

The --cov flag points to your source package. --cov-report=term-missing prints a table showing which lines in each file were not executed, so you can scan for gaps immediately.

For a more detailed view, generate an HTML report:

pytest --cov=my_pipeline --cov-report=html

Open htmlcov/index.html in a browser and you can click through each file to see exactly which branches and lines were missed, highlighted in red.

The Hidden Files Problem

Here’s a trap that inflates coverage numbers: pytest-cov only tracks files that are actually imported during the test run. If you have a module deep in your package that no test touches — not even an import — it won’t appear in the report at all. Your coverage percentage will look great simply because invisible files aren’t counted.

To get an honest picture, make sure every module you care about is at least imported during testing. For those you can’t unit‑test easily, a minimal smoke test that imports the module and checks for basic sanity (e.g., “can the module be loaded without errors”) brings it into the coverage report and prevents silent blind spots.

If you need to enforce a strict coverage threshold across the whole source tree, consider using the --cov-fail-under flag, but only after you’ve verified that all relevant modules appear in the report. Otherwise you’re measuring the coverage of a subset, not the whole.

What to Aim For

Line coverage is the simplest metric, but branch coverage is more honest. A function like this:

def clean_amount(raw):
    if raw is None:
        return 0.0
    return float(raw)

If you only test with a non‑null value, line coverage reports 100% — all three lines ran. But branch coverage sees that the None path was never taken. Run with --cov-report=term-missing --cov-branch to get both.

For data engineering pipelines, I aim for:

• 80–90% line coverage on transformation logic. Pure functions that map, filter, aggregate, and clean data should be thoroughly tested.
• 100% branch coverage on any function that contains conditional logic — null handling, validation rules, currency conversion branches. These are exactly the places where silent bugs hide.
• Meaningful coverage for orchestration glue. DAG definition files and Airflow operator instantiation are harder to unit‑test in isolation, but that doesn’t mean they deserve zero attention. At minimum, a test that imports the DAG and verifies that it parses without error and that the task dependency structure makes sense will catch typos and broken references. That gets the module into the coverage report and gives you a real safety net. Cover these areas where possible; don’t write them off entirely.

Saving Evidence of What Ran

In regulated environments — banking, insurance, anything with audit requirements — you often need to prove that tests passed. Generate a machine‑readable report with --junitxml and store it alongside the coverage HTML as a build artifact:

pytest --junitxml=test-results.xml --cov=my_pipeline --cov-report=html

Your CI pipeline can archive both test-results.xml and the htmlcov/ directory. This gives you a timestamped, versioned record of exactly which tests passed and what code they exercised. When an auditor asks “how do you know this pipeline was tested before deploying?” you point them at the artifact store, not your memory.

The Coverage Trap

Don't let coverage become a vanity metric. I've watched teams write tests that call a function and then assert nothing meaningful — or worse, assert True — just to turn a line green. A covered line with a weak assertion is worse than an uncovered line because it gives a false sense of safety.

The remedy is to pair coverage checks with mutation testing occasionally. Tools like mutmut make small changes to your code (flipping a < to <=, removing a not) and re-run your tests to see if any fail. If a mutation survives, your test didn't actually verify that behavior. It's too heavy to run on every commit, but running it once a sprint on critical modules surfaces exactly where your tests are paper‑thin.

Coverage is a compass, not a destination. Use it to find the code you forgot to test, not to hit a number.

Best Practices for a Maintainable Test Suite

Looking back across everything we've covered, a few principles stand out:

• Tests are code too. Apply the same standards you'd apply to production code: single-responsibility test functions, clear naming, and no copy-paste. A fixture is just a function — keep it small and focused.
• Isolate the pure logic from the infrastructure. If a function mixes transformation logic with API calls, refactor it before testing. The transformation should be testable without any mocking at all.
• Test edge cases explicitly. Parametrization with named IDs makes it obvious what you've thought about — nulls, empty lists, boundary values. If it's not listed in a parametrize block, did you really test it?
• Keep the test suite fast. The difference between a test suite that runs in five seconds and one that takes five minutes is the difference between running tests before every commit and running them only in CI. Use session‑scoped fixtures for expensive resources and skip integration tests by default.
• Clean up after yourself. tmp_path, yield in fixtures, and autouse cleanup hooks prevent state from leaking between tests. A test that passes in isolation but fails when run after another test is a debugging nightmare.
• Measure coverage honestly. Coverage tells you what code ran, not what was verified. Make sure all the modules you care about actually appear in the report, save artifacts for audit trails, and test orchestration code where it’s practical. Strong assertions over high percentages, every time.

Conclusion

Testing data pipelines is not a luxury — it's what separates a one‑off script from maintainable, production‑grade code. Across this series, we've covered the Arrange‑Act‑Assert pattern, mocking with @patch, reusable fixtures with scoped lifetimes, parametrization for edge cases, temporary file cleanup, log verbosity, marker‑based test splitting, and code coverage as a genuine signal rather than a vanity metric. Each piece builds on the last to give you a test suite that's fast, expressive, and robust enough to survive schema changes, cloud provider upgrades, and team turnover.

The next time you're about to write a 200‑line transformation, ask yourself: how would I test this? Build the tests right alongside the code, and you'll thank yourself when the schema changes at 2 a.m.