DEV Community: MikiBuilder

I added one object and broke 25 tests without changing a single assertion

MikiBuilder — Sun, 20 Sep 2026 15:59:25 +0000

Last month I was finishing a contribution to Symfony AI: redacting secrets and personal data from recorded HTTP cassettes before they get committed. The design had been agreed in the issue, reviewed, and approved. The platform test suite was green — 957 tests.

Then CI came back with 25 failures in a job I had never had to think about.

The failure

Example "mistral/structured-output-math.php" produced output differing from its recorded fixture.

--- Expected
+++ Actual
-MathReasoning {#475
+MathReasoning {#477
   +steps: array:4 [
-    0 => Step {#491
+    0 => Step {#493
       +explanation: "Start with the given equation: 8x + 7 = -23."
       +output: "8x + 7 = -23"

Read it twice. Every asserted value is identical. Same explanation, same output, same number of steps.

What changed is {#475} versus {#477}.

What those numbers are

{#475} is VarDumper's object handle — a spl_object_id() value. When you dump() something, Symfony's dumper labels each object with an id that comes from a process-wide counter.

The test suite compares a full dump against a committed .out file. Those files contain the ids.

Which means: any object constructed anywhere earlier in the process shifts every id that comes after it.

My change added one line to a constructor:

$this->redactor = $redactor ?? new BodyRedactor();

One object per cassette. Two cassettes in an example, ids shift by two. Four cassettes, they shift by four. The deltas in the failure log matched exactly.

The fix that was not a fix

The obvious move is to not build the object until you need it:

private ?BodyRedactor $redactor;

public function __construct(
    private readonly string $path,
    ?BodyRedactor $redactor = null,
) {
    $this->redactor = $redactor;
}

private function redactor(): BodyRedactor
{
    return $this->redactor ??= new BodyRedactor();
}

A cassette that only replays never redacts anything, so it never needs a redactor. This is better code regardless of the test problem — you do not instantiate a rule set nobody asked for.

It took the failures from 25 to 21, and the shift from +2 to +1.

Better. Not fixed.

The remaining instantiation came from the verification path: when a live request does not match the recorded signature on the first attempt, the code retries against the redacted body, and that retry needs the redactor.

The part that is actually interesting

At that point I stopped trying to be clever, because the problem was never the initialisation strategy.

A fixture that captures object ids couples your test to things you are not testing.

Those 251 fixtures assert, implicitly and without anyone deciding it, that the number of objects PHP allocates before the dump will never change. Not the values. Not the structure. The allocation order of the entire process.

Nobody wrote that assertion. It came for free with dump(), and it sat there until someone added a collaborator to a class three layers down.

That is the same failure mode I spend my time on in llm-vcr: a recording that captures more than the thing under test. A cassette that stores a timestamp fails tomorrow. One that stores a request id fails on the next run. A fixture that stores object ids fails when anyone allocates an object upstream.

The recording is too faithful. Fidelity sounds like a virtue until it starts asserting things you never meant to assert.

What I did about it

I could have forced my way through. Make the built-in rules static functions and only construct a BodyRedactor when someone passes one in explicitly. The ids would stay put and CI would go green.

That would also have thrown away the design the maintainer and I had agreed on: an optional injectable collaborator, so a project can plug in its own redaction rules. Making the default path object-free to satisfy a fixture would have been solving the wrong problem loudly.

So I wrote it up instead — the measurements, the cause, and three ways out with what each one costs:

Re-record the fixtures. Needs API credentials for nine providers. I have none of them, and guessing at model output is not re-recording.
Make the comparison ignore object ids. A change to the test harness, not to my feature, and it protects the suite from every future collaborator rather than just mine.
Keep the default path object-free. Preserves the ids, gives up the design.

Then I said I would pick 2, explained why, and offered to do the work either way.

How it ended

The next morning, Christopher Hertel landed a separate PR:

The {#123} handles VarDumper prints next to an object are spl_object_id() values, so they shift whenever anything allocates one more object earlier in the process. (...) Instead of resyncing the ids, bootstrap.php now dumps with withRefHandles(false), so the handles never enter the output.

One line in the bootstrap. Twenty-eight fixture files regenerated from the existing cassettes — no re-recording, no cassette touched, and every changed line was a handle line.

Two things in that PR description are worth sitting with.

It was not my change that broke it. main was already red. Adding Capability::REALTIME_SESSION in an unrelated feature had been enough to break the same 21 examples days earlier. I had not caused the problem; I had walked into it, and my diff happened to be holding the bag when the CI ran.

And the fix was the boring one. Not clever, not defensive, not a workaround in my class. One flag, in the one place that owns the output format.

My contribution merged three days later.

What I would take from this

If your test compares a dump, strip what you are not asserting. Object ids, memory addresses, timestamps, autoincrement keys. Anything the runtime assigns rather than your code producing.

A test that fails for a reason unrelated to its name is worse than no test. Twenty-five red builds told me nothing about whether redaction works. They told me about PHP's allocator.

And when the fix belongs to someone else's file, say so. I spent two CI cycles trying to make the problem disappear from inside my own diff. The useful move was to measure it, write it down, and hand the decision to the people who own those fixtures.

That last one took me longer to learn than it should have.

The contribution is symfony/ai#2487. The fix for the fixtures is symfony/ai#2531, by Christopher Hertel. I maintain llm-vcr, a record and replay library for testing AI features in PHP.

How to test AI features in PHP without spending a cent

MikiBuilder — Wed, 29 Jul 2026 07:53:24 +0000

I have been writing PHP for twenty years. For the last two I have been putting language models into real applications: ticket classifiers, document analysis, conversational assistants.

And there is one question I could not answer well: how do you test this?

The test that does not work

Say you have a service that classifies support tickets. You give it text, it returns JSON with a category, a sentiment and an urgency level. You want a test.

public function testClassifiesAnAccessProblem(): void
{
    $result = $this->analyzer->analyze('I cannot access my account');

    $this->assertSame('access', $result->category);
}

Looks reasonable. It is not.

First: it is not deterministic. The same prompt returns different text on every run. Today "access", tomorrow "Account access", the day after "access_problem". Your test fails randomly, and a test that fails randomly is worse than no test at all: it teaches the team to ignore red builds.

Second: it costs money. Every run burns tokens. Multiply by the number of tests, by every push, by every developer on the team.

Third: it is slow. Between 200 ms and 3 seconds per call. A suite with 200 AI tests takes ten minutes. And a ten-minute suite stops being run.

Fourth: it needs network access and a production API key in CI. If your provider has an incident, your build turns red without you breaking anything.

And then there is the problem nobody talks about

Those four are annoying. The fifth one wakes you up at three in the morning.

Your classifier returns this:

{"category": "access", "sentiment": "frustrated", "urgency": 4}

And you, being tidy, map it to a typed DTO:

final readonly class TicketDto
{
    public function __construct(
        public string $category,
        public string $sentiment,
        public int $urgency,     // ← here
    ) {}
}

On some random Tuesday, the provider updates the model. Nobody tells you: from their side it is a minor version bump. And the model starts returning this:

{"category": "access", "sentiment": "frustrated", "urgency": "high"}

urgency went from int to string. Your DTO blows up in production.

And you never touched a single line of code.

Here is the important part: no test catches this. The mocks you wrote by hand still return 4, because you froze that value six months ago. Your suite is green while production burns.

What already exists

Before writing anything, I looked at what was out there. This matters, because the gap might not be a gap.

Hand-written mocks. Symfony AI ships InMemoryPlatform and MockPlatformFactory, and they are good. But they return values you made up. They are the right tool for testing your own logic — what happens if the model fails, if it returns empty — and the wrong tool for knowing what the model actually returns. By definition, they can never detect drift.

php-vcr. Excellent library, over three million installs. It records HTTP requests and replays them. The catch is that it matches requests by exact body, and a real prompt carries timestamps, UUIDs, order IDs and RAG context that change on every run. The cassette is invalidated the moment a comma moves.

Eval packages. There are several on Packagist and they are useful, but they answer a different question: is the answer good?. They do not solve determinism, cost or drift. They are complementary.

Promptfoo, DeepEval, Langfuse. The whole mature ecosystem lives in Node and Python. They need a separate runtime and do not plug into PHPUnit.

I searched Packagist for llm cassette record replay. Zero results.

And something I found later

While preparing this article I went back and looked at the Symfony repositories, not just Packagist. There is work in progress:

symfony/ai#2129 — a RecordingProvider that records cassettes at the provider level
symfony/ai#2128 — cassettes at the HTTP boundary
symfony/symfony#63781 — a RecorderHttpClient in php-vcr style

All three are open drafts at the time of writing, and I only found them after building my own. Honestly, that felt like a good sign rather than a bad one: two core contributors arriving independently at the same conclusion means the problem is real.

They are also solving a different layer. Quoting #2129 directly:

Interactions are matched by a signature over (model, input, options) and consumed FIFO.

Result metadata and token usage are not preserved in this first version.

Exact-signature matching brings back the php-vcr problem: a prompt with a different timestamp does not match. And none of the three does what I needed most — telling me when the provider's behaviour changed.

The idea

A recorder. Like php-vcr, but one that understands prompts.

Locally it records real responses into versionable JSON files.
In CI it replays them from disk: no network, no API key, no cost.
Every night it replays them against the real provider and warns you if anything changed.

$platform = new RecordingPlatform(
    inner: new GroqPlatform($apiKey),   // your real provider
    cassetteDir: __DIR__ . '/cassettes',
    mode: Mode::fromEnv(),              // record locally, replay in CI
);

Your code does not change: RecordingPlatform implements the same interface. It is a decorator, not a fork.

Three decisions that matter

1. Matching is semantic, not a hash

This is what kills php-vcr for this use case. A real prompt looks like this:

Incident 445566 from 2026-01-10: I cannot access my account.

And on the next run, like this:

Incident 998877 from 2026-07-25: I cannot access my account.

Semantically they are the same case. To a hash they are different universes.

The fix: normalise the volatile noise (<id>, <date>, <uuid>) and compare using cosine similarity over a bag of words. No embeddings, no network, no dependencies. The two prompts above score 0.89 and match.

If you want something stricter, you can declare which parts vary:

new PlaceholderMatcher([
    'order_id' => '/ORD-\d+/',
    'amount'   => '/\d+\.\d{2} ?€/',
]);

Still an exact, deterministic comparison — but immune to what you mark as variable. If something you did not declare changes, it does not match. And that is the correct behaviour.

2. Cassettes get committed, so they get redacted

This one is not negotiable. If cassettes go into git and contain real prompts, they contain real data: emails, phone numbers, national ID numbers, sometimes an API key someone pasted by mistake.

That is why redaction is on by default. Not opt-in.

{
  "role": "user",
  "content": "I'm <REDACTED:EMAIL>, tel <REDACTED:PHONE>, cannot log in."
}

Secure defaults matter more in development tooling than almost anywhere else. Nobody reads the configuration reference of a dev dependency. If the safe behaviour is something you have to switch on, most people will never switch it on — and the failure is silent until it is public.

Building this I hit a nice bug. The cassette stores <REDACTED:EMAIL>, but the incoming request carries the original email. If you compare the raw prompt against the sanitised cassette, they never match, and it fails precisely on the requests that contain personal data. The fix: run the matching on already-redacted text, so both sides live in the same space.

3. Drift is detected by comparing shape, not text

Back to the int that became a string.

Comparing textual similarity is not enough: {"urgency":4} and {"urgency":"high"} look fairly similar. You have to compare the shape of the JSON: which keys exist and what type each one has.

🔴 CRITICAL   sim 0.79   type change in "urgency": int -> string
                         | new field: "confidence" (float)
🟢 OK         sim 1.00   no schema changes
🟡 MEDIUM     sim 0.60   no schema changes

One command, one nightly cron, and an issue opened automatically when something moves. It exits with a non-zero code, so it breaks the build.

What it feels like to use

In PHPUnit, a trait and assertions that talk about the problem:

use InteractsWithLlm;

public function testClassifiesAnAccessProblem(): void
{
    $platform = $this->recordLlm(GroqPlatform::fromEnv());

    $result = $platform->invoke('llama-3.1-8b-instant', [
        ['role' => 'system', 'content' => 'Classify tickets. Respond with JSON.'],
        ['role' => 'user',   'content' => 'I cannot access my account.'],
    ]);

    $this->assertNoLiveLlmCalls();
    $this->assertLlmJsonShape([
        'category' => 'string',
        'urgency'  => 'int',
    ], $result);
}

Notice what is not asserted: nowhere does it say the category must be exactly "access". It asserts the contract. An LLM's value is not deterministic, but its shape has to be.

assertNoLiveLlmCalls() is my favourite. Put it in your suite and CI will tell you the day someone adds a test that escapes to the real API.

In Pest, the same thing with expectations:

expect($platform)->toHaveMadeNoLiveCalls();
expect($result)->toBeLlmJson()
               ->toMatchLlmShape(['category' => 'string', 'urgency' => 'int']);

And in Symfony, the panel

This is where you see it at a glance:

Metric	Value
From cassette	4
Live calls	0
Tokens saved	60
Time saved	1,280 ms

Zero API calls. Almost a second and a half saved on a single request. And in the same toolbar you can read that the whole page took 3 ms.

The badge turns red if live calls were made while in replay mode, which usually means a cassette is missing.

The numbers, honestly

Measured against cassettes recorded from Llama 3.1 on Groq, with real latencies of 437, 227 and 235 ms:

	Without llm-vcr	With llm-vcr
API calls per run	3	0
Tokens consumed	426	0
Time waiting for the API	~900 ms	0 ms
Cost in CI	provider-dependent	0

One caveat worth stating: the time saved depends on your real latency. If your provider answers in 200 ms, you save 200 ms per call. The number that matters is not a spectacular multiplier — it is this: zero network and zero tokens in CI, every time, deterministically.

Getting started

Groq gives you a free API key with no credit card, so you can try the whole thing without spending anything.

composer require --dev mikibuilder/llm-vcr

$platform = new RecordingPlatform(
    inner: GroqPlatform::fromEnv(),
    cassetteDir: __DIR__ . '/cassettes',
    mode: Mode::fromEnv(default: Mode::Replay),
);

Run once with LLM_VCR_MODE=record, commit the cassettes, and from then on CI runs for free.

What I learned building it

The tests found bugs I would never have seen. The redacted-text matching one, for instance, only showed up on requests containing personal data. Without a test that deliberately included an email address, it would have shipped.

Actually installing it found the ones the tests could not. I had 112 green tests and the Symfony Profiler panel never registered. The cause: the default value arrived as the unresolved string "%kernel.debug%", and comparing it to true was always false. My tests missed it because every single one passed the parameter explicitly. It only surfaced when I created a fresh Symfony project and installed the package like any user would.

Testing on another operating system found more. The suite took 0.8 seconds on Linux and 25 seconds on Windows: fifteen Symfony kernels writing the container to disk. Rewritten on top of ContainerBuilder, the full suite dropped to 2.5 seconds on the same machine.

The lesson, if there is one: correct code and a usable product are not the same thing, and the only way to tell them apart is to use it.

Where it lives

Packagist: mikibuilder/llm-vcr
GitHub: github.com/MikiBuilder/llm-vcr
License: MIT

It is version 0.3.x. It works, it has 116 tests and level 9 static analysis, but there is road ahead: embedding-based matching with a cache, streaming and tool call support.

If you try it and something does not fit your case, open an issue. Especially if the way you build prompts breaks the matcher — that is exactly what I need to know.

Twenty years in, PHP is still the language I reach for first. It is worth having the same testing tools everyone else takes for granted.