Testing Code That Calls an LLM: How I Mock the OpenAI Client in pytest (Without Burning Requests)

I'm Aman Kumar. I build an OpenAI-compatible gateway, so I spend a lot of time reading other people's integration code. One pattern shows up constantly: a test suite that calls the real model on every run. It's slow, it's flaky, it eats into whatever request budget you have, and when it fails you can't tell whether your code broke or the model just answered differently today.

This is the setup I use instead. It works with the official openai Python package pointed at any OpenAI-compatible base URL, and none of it depends on which provider you use.

The rule: tests check your code, not the model

Most of what goes wrong in LLM features isn't the model. It's the code around it: building the messages list, picking the right model ID, parsing JSON out of the reply, handling an empty choices list, retrying on a 429, counting tokens. All of that is deterministic and testable if you stop the network call from happening.

So I split tests into two groups:

1. Unit tests (the vast majority). The client is replaced with a fake. They run in milliseconds and never touch the network.
2. Live smoke tests (a handful). They hit the real endpoint, are marked so they're skipped by default, and check only shape, not exact wording.

Step 1: give the client a seam

The hardest code to test is code that creates its client at import time:

# hard to test
from openai import OpenAI
client = OpenAI()

def summarize(text):
    r = client.chat.completions.create(model="gpt-4o-mini", messages=[...])
    return r.choices[0].message.content

I pass the client in instead, with a small factory for production:

# llm.py
import os
from openai import OpenAI

def make_client():
    return OpenAI(
        base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
        api_key=os.environ["LLM_API_KEY"],
        timeout=60,
        max_retries=2,
    )

def summarize(client, text, model):
    r = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "Summarize in 2 sentences."},
            {"role": "user", "content": text},
        ],
        temperature=0.2,
    )
    if not r.choices:
        raise ValueError("empty response")
    return (r.choices[0].message.content or "").strip()

Model ID and base URL come from config, which also means switching providers never touches the function.

Step 2: a tiny fake client

You don't need a mocking library for this. The OpenAI client is accessed as client.chat.completions.create(...), so a fake only has to provide that path and return objects with .choices[0].message.content.

# tests/fakes.py
from types import SimpleNamespace as NS

def completion(content, finish_reason="stop", prompt_tokens=10, completion_tokens=5):
    msg = NS(content=content, role="assistant", tool_calls=None)
    return NS(
        choices=[NS(index=0, message=msg, finish_reason=finish_reason)],
        usage=NS(prompt_tokens=prompt_tokens, completion_tokens=completion_tokens,
                 total_tokens=prompt_tokens + completion_tokens),
    )

class FakeCompletions:
    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = []
    def create(self, **kwargs):
        self.calls.append(kwargs)
        reply = self.replies.pop(0)
        if isinstance(reply, Exception):
            raise reply
        return reply

class FakeClient:
    def __init__(self, *replies):
        self.chat = NS(completions=FakeCompletions(replies))

Two things make this useful beyond "returns a string":

• calls records every request, so you can assert on what your code sent.
• A reply can be an exception, so you can script failure sequences.

Step 3: tests that actually catch bugs

# tests/test_llm.py
import pytest
from llm import summarize
from tests.fakes import FakeClient, completion

def test_sends_expected_request():
    fake = FakeClient(completion("Short summary."))
    out = summarize(fake, "long text", model="my-model")
    assert out == "Short summary."
    call = fake.chat.completions.calls[0]
    assert call["model"] == "my-model"
    assert call["messages"][0]["role"] == "system"
    assert call["messages"][1]["content"] == "long text"

def test_none_content_becomes_empty_string():
    fake = FakeClient(completion(None))
    assert summarize(fake, "x", model="m") == ""

def test_empty_choices_raises():
    fake = FakeClient(completion("x"))
    fake.chat.completions.replies[0].choices = []
    with pytest.raises(ValueError):
        summarize(fake, "x", model="m")

The None content case is real: when a model returns only a tool call, or a response is filtered, message.content can be None. I've seen production code crash on .strip() there more than once.

Testing retries and errors

If you wrap calls in your own retry logic, script the failures. The real exception classes need a response object, so I usually raise my own lightweight stand-in in unit tests and test the mapping separately:

class FakeRateLimit(Exception):
    pass

def call_with_retry(client, attempts=3, **kw):
    for i in range(attempts):
        try:
            return client.chat.completions.create(**kw)
        except FakeRateLimit:
            if i == attempts - 1:
                raise

def test_retries_then_succeeds():
    fake = FakeClient(FakeRateLimit(), FakeRateLimit(), completion("ok"))
    r = call_with_retry(fake, model="m", messages=[])
    assert r.choices[0].message.content == "ok"
    assert len(fake.chat.completions.calls) == 3

In real code, catch openai.RateLimitError and openai.APITimeoutError, and inject the exception type or the sleep function so tests don't actually wait.

Testing JSON parsing

If your feature asks the model for JSON, the parser is where most bugs live. Feed it the ugly outputs models actually produce:

import json, re

def parse_json_reply(text):
    text = text.strip()
    m = re.search(r"```

(?:json)?\s*(.*?)

```", text, re.S)
    if m:
        text = m.group(1)
    return json.loads(text)

@pytest.mark.parametrize("raw", [
    '{"a": 1}',
    '```

json\n{"a": 1}\n

```',
    '  {"a": 1}\n',
])
def test_parse_variants(raw):
    assert parse_json_reply(raw) == {"a": 1}

Collect real bad replies from your logs and add them as cases. That list becomes the most valuable test file in the project.

Streaming

For streamed responses, the fake returns an iterable of chunks shaped like chunk.choices[0].delta.content:

def stream_chunks(*parts):
    for p in parts:
        yield NS(choices=[NS(index=0, delta=NS(content=p), finish_reason=None)])

Then assert your code joins them correctly and handles a chunk where delta.content is None (the first and last chunks often are).

Step 4: a few live smoke tests, on purpose

Fakes can drift from reality, so I keep a few real calls, skipped unless asked for:

# tests/test_live.py
import os, pytest
from llm import make_client, summarize

live = pytest.mark.skipif(not os.environ.get("RUN_LIVE"), reason="set RUN_LIVE=1")

@live
def test_live_summary_shape():
    client = make_client()
    out = summarize(client, "The cat sat on the mat. It was warm.",
                    model=os.environ["LLM_MODEL"])
    assert isinstance(out, str) and 0 < len(out) < 500

Notice the assertion: type and rough length, never exact text. I run these before a release or after changing providers, base URL or model ID, not on every commit. When pointed at a gateway, this is also how I confirm the model ID string is right, since a wrong ID usually fails with a 404 or a "model not found" error right here instead of in production.

Example environment for a gateway run:

RUN_LIVE=1 LLM_BASE_URL=https://apiclaw.biz/v1 LLM_API_KEY=... LLM_MODEL=deepseek-v4-flash pytest tests/test_live.py

What I'd skip

• Snapshot-testing model output. It breaks whenever the model changes and tells you nothing about your code.
• Recording real responses for every test with cassette tools. It works, but the cassettes go stale and hide shape changes. I use recordings only for a couple of complex tool-calling flows.
• Mocking requests/httpx underneath the SDK. It couples tests to SDK internals. Mock at the client boundary instead.

Summary

Pass the client in, fake chat.completions.create, assert on what you sent as much as what you got back, script failures, and keep a small, opt-in set of live tests that check shape. Your suite gets fast, deterministic and free to run, and the live tests catch the config mistakes that fakes can't.

Disclosure: I build APIClaw, an OpenAI-compatible gateway with flat monthly pricing, which is the base URL in the live-test example. Everything here works the same with any OpenAI-compatible provider.