DEV Community

Deependra Rajput
Deependra Rajput

Posted on

Testing AI Pipelines Without Calling the Model Every Time

Tests that call a real LLM are slow, cost money on every run, and fail randomly because the output changes. So many teams quietly skip testing their AI features altogether.

The fix is to notice that most of an "AI pipeline" isn't the model at all. Building the prompt, retrieving context, parsing the reply and deciding what to do next are ordinary, deterministic code, and ordinary code is easy to test.

1. Put the model behind an interface

Make your pipeline depend on a small interface instead of a specific SDK:

from typing import Protocol


class LLM(Protocol):
    async def complete(self, prompt: str) -> str: ...


class FakeLLM:
    def __init__(self, reply: str) -> None:
        self.reply = reply
        self.prompts: list[str] = []

    async def complete(self, prompt: str) -> str:
        self.prompts.append(prompt)
        return self.reply
Enter fullscreen mode Exit fullscreen mode

Production passes in the real client. Tests pass in FakeLLM, which returns whatever you tell it to and records what it was asked.

2. Test the code around the model

Now you can check that the right information reaches the prompt and that replies are handled correctly, with no network calls:

import pytest


@pytest.mark.asyncio
async def test_extracts_order_id_from_reply() -> None:
    llm = FakeLLM('{"order_id": "A-1042", "intent": "refund"}')

    result = await classify_message("Where is my refund for A-1042?", llm=llm)

    assert result.order_id == "A-1042"
    assert "A-1042" in llm.prompts[0]
Enter fullscreen mode Exit fullscreen mode

(These examples use the pytest-asyncio plugin for async tests.)

3. Test the ugly replies

Real models sometimes return empty strings, plain text instead of JSON, or JSON that's cut off halfway. Those are exactly the cases that crash production, and they're trivial to test with a fake:

@pytest.mark.asyncio
@pytest.mark.parametrize("reply", ["", "not json", '{"intent": "refund"'])
async def test_bad_model_output_falls_back(reply: str) -> None:
    result = await classify_message("hello", llm=FakeLLM(reply))
    assert result.intent == "unknown"
Enter fullscreen mode Exit fullscreen mode

4. Keep a few live tests, but run them separately

You still want some tests against the real model, to catch prompt regressions and provider changes. Mark them and keep them out of the normal run:

@pytest.mark.live
@pytest.mark.asyncio
async def test_live_refund_classification(real_llm) -> None:
    result = await classify_message("I want my money back", llm=real_llm)
    assert result.intent == "refund"
Enter fullscreen mode Exit fullscreen mode

CI runs pytest -m "not live" on every commit. The live suite runs on a schedule or before a release.

5. Aim coverage at the deterministic code

With the model behind an interface, prompt building, parsing, retrieval and routing can all reach high coverage, because none of them needs a network call. That's where most bugs live anyway.

The result

Fast tests on every commit, a small live suite for real-model behaviour, and confidence when you change a prompt or a parser. AI features don't have to be the untested corner of your codebase.

How do you test your LLM features? I'm curious what's working for other teams.

Top comments (0)