Pytest pt2 - Mastering Pytest Fixtures

Mastering Pytest Fixtures

Fixtures are the backbone of pytest testing. They set up objects — like a SparkSession, a sample DataFrame, or a database connection — that your tests can use. Instead of constructing the same input data in ten different test functions, you define a fixture once and inject it where needed. In the previous post we covered the fundamentals of testing and mocking; now it's time to eliminate duplication and make your test suite faster and easier to maintain.

This post covers everything you need to know about fixtures: how to define them, the different scopes available, how to use yield for setup and teardown, when to reach for autouse=True, and how conftest.py acts as a shared repository for fixtures across your entire test suite.

Defining a Fixture and Scoping It

A fixture is just a function decorated with @pytest.fixture. You can request it in a test by including its name as a parameter:

import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")
def spark():
    return (SparkSession.builder
            .master("local[2]")
            .appName("test")
            .getOrCreate())

Notice the line continuation uses parentheses instead of backslashes — it's cleaner and avoids accidental white‑space issues.

The scope parameter tells pytest how long the fixture should live:

• function (default): created anew for each test function. Safest when a test might mutate the fixture.
• class: created once per test class. Good when you have several tests inside a class that share setup.
• module: created once per test file.
• package: created once per test package (a directory with __init__.py or conftest.py).
• session: created once for the entire test run. Use for very expensive setup like a SparkSession.

Scopes let you trade isolation for speed. I use session for the SparkSession because starting a new Spark context for every test is painfully slow. But if a test writes to a temporary table or changes a config, that state leaks to later tests — a conflict. For mutable fixtures, stick with function or module scope and clean up after yourself via a yield or a finalizer.

Setup and Teardown with yield

Fixtures aren't just about creating objects — they can also run cleanup after a test finishes. Use yield to separate setup from teardown:

@pytest.fixture(scope="function")
def temp_table(spark):
    spark.sql("CREATE OR REPLACE TEMP VIEW test_view AS SELECT 1 AS id")
    yield
    spark.sql("DROP VIEW IF EXISTS test_view")

Everything before yield runs before the test. Everything after yield runs after the test, even if the test fails. This is the cleanest way to ensure resources get released.

Autouse Fixtures: Implicit Setup and Teardown

Sometimes you want every test to begin in a known state without having to explicitly request a fixture in each test. That's exactly what autouse=True is for. An autouse fixture runs automatically for every test in its scope, making it ideal for tasks like cleaning up files, resetting environment variables, or preparing test data.

In the example below, every test starts with a clean slate by ensuring a log file doesn't already exist. After the test finishes, the same fixture removes the log file again so it can't affect any subsequent tests.

from pathlib import Path
import pytest

LOG_FILE = Path("test.log")

@pytest.fixture(autouse=True)
def remove_test_log():
    # Setup: start each test with no log file
    LOG_FILE.unlink(missing_ok=True)

    yield

    # Teardown: remove any log file the test created
    LOG_FILE.unlink(missing_ok=True)

The code before yield is the setup phase. It runs before each test and deletes test.log if it's present, guaranteeing that every test starts with a clean environment. Once the test completes, execution resumes after yield. This is the teardown phase, which removes the log file again in case the test created or modified it. Because the fixture uses autouse=True, you don't need to include it as a parameter in your test functions — it runs automatically for every test, helping keep your test suite isolated and deterministic.

Sharing Fixtures with conftest.py

The conftest.py file is the central repository for fixtures and plugin configuration. Any fixture defined in a conftest.py at the root of your test directory is automatically available to all tests in that directory and its subdirectories — no imports needed. I use conftest.py for shared Spark sessions, test data generators, and mock helpers.

For example, a conftest.py that provides a DataFrame of sample transactions:

import pytest
from pyspark.sql import Row

@pytest.fixture(scope="module")
def sample_transactions(spark):
    data = [
        Row(id=1, amount=100.0, currency="USD"),
        Row(id=2, amount=200.0, currency="EUR"),
        Row(id=3, amount=0.0, currency="USD"),
        Row(id=4, amount=None, currency=None),
    ]
    return spark.createDataFrame(data)

Because the spark fixture is session‑scoped, it will be created once and reused across all tests in the session. The sample_transactions fixture is module‑scoped, so it's created once per test file that uses it.

Fixture Best Practices

A few guidelines I follow after watching test suites spiral out of control:

• Keep fixtures close to where they're used. Put shared, general-purpose fixtures in conftest.py. Put specific fixtures in the test module that needs them. Don't turn conftest.py into a dumping ground.
• Match scope to the cost of creation. A SparkSession is expensive; make it session‑scoped. A list of numbers is cheap; function scope is fine and safer.
• Use yield for any fixture that acquires a resource. Files, database connections, Spark temporary views — all should be cleaned up after the test, even if it fails.
• Watch for fixture conflicts. A session‑scoped fixture that gets mutated by one test will affect every subsequent test. If you need mutation, drop the scope or reset the state in a teardown step.

Conclusion

Fixtures let you stop copy‑pasting setup code and start composing reusable, well‑scoped building blocks. You've seen how to define them, control their lifetime with scopes, separate setup from teardown with yield, and share them across your project using conftest.py. Combined with the mocking fundamentals from the previous post, you now have the tools to structure a clean, maintainable test suite.

What's still missing is a way to test the same logic across many inputs without writing nearly‑identical test functions. In the next post, we'll cover parametrization and how to handle temporary files properly — two techniques that will make your tests more expressive and keep your filesystem clean.