I added one object and broke 25 tests without changing a single assertion

Last month I was finishing a contribution to Symfony AI: redacting secrets and personal data from recorded HTTP cassettes before they get committed. The design had been agreed in the issue, reviewed, and approved. The platform test suite was green — 957 tests.

Then CI came back with 25 failures in a job I had never had to think about.

The failure

Example "mistral/structured-output-math.php" produced output differing from its recorded fixture.

--- Expected
+++ Actual
-MathReasoning {#475
+MathReasoning {#477
   +steps: array:4 [
-    0 => Step {#491
+    0 => Step {#493
       +explanation: "Start with the given equation: 8x + 7 = -23."
       +output: "8x + 7 = -23"

Read it twice. Every asserted value is identical. Same explanation, same output, same number of steps.

What changed is {#475} versus {#477}.

What those numbers are

{#475} is VarDumper's object handle — a spl_object_id() value. When you dump() something, Symfony's dumper labels each object with an id that comes from a process-wide counter.

The test suite compares a full dump against a committed .out file. Those files contain the ids.

Which means: any object constructed anywhere earlier in the process shifts every id that comes after it.

My change added one line to a constructor:

$this->redactor = $redactor ?? new BodyRedactor();

One object per cassette. Two cassettes in an example, ids shift by two. Four cassettes, they shift by four. The deltas in the failure log matched exactly.

The fix that was not a fix

The obvious move is to not build the object until you need it:

private ?BodyRedactor $redactor;

public function __construct(
    private readonly string $path,
    ?BodyRedactor $redactor = null,
) {
    $this->redactor = $redactor;
}

private function redactor(): BodyRedactor
{
    return $this->redactor ??= new BodyRedactor();
}

A cassette that only replays never redacts anything, so it never needs a redactor. This is better code regardless of the test problem — you do not instantiate a rule set nobody asked for.

It took the failures from 25 to 21, and the shift from +2 to +1.

Better. Not fixed.

The remaining instantiation came from the verification path: when a live request does not match the recorded signature on the first attempt, the code retries against the redacted body, and that retry needs the redactor.

The part that is actually interesting

At that point I stopped trying to be clever, because the problem was never the initialisation strategy.

A fixture that captures object ids couples your test to things you are not testing.

Those 251 fixtures assert, implicitly and without anyone deciding it, that the number of objects PHP allocates before the dump will never change. Not the values. Not the structure. The allocation order of the entire process.

Nobody wrote that assertion. It came for free with dump(), and it sat there until someone added a collaborator to a class three layers down.

That is the same failure mode I spend my time on in llm-vcr: a recording that captures more than the thing under test. A cassette that stores a timestamp fails tomorrow. One that stores a request id fails on the next run. A fixture that stores object ids fails when anyone allocates an object upstream.

The recording is too faithful. Fidelity sounds like a virtue until it starts asserting things you never meant to assert.

What I did about it

I could have forced my way through. Make the built-in rules static functions and only construct a BodyRedactor when someone passes one in explicitly. The ids would stay put and CI would go green.

That would also have thrown away the design the maintainer and I had agreed on: an optional injectable collaborator, so a project can plug in its own redaction rules. Making the default path object-free to satisfy a fixture would have been solving the wrong problem loudly.

So I wrote it up instead — the measurements, the cause, and three ways out with what each one costs:

1. Re-record the fixtures. Needs API credentials for nine providers. I have none of them, and guessing at model output is not re-recording.
2. Make the comparison ignore object ids. A change to the test harness, not to my feature, and it protects the suite from every future collaborator rather than just mine.
3. Keep the default path object-free. Preserves the ids, gives up the design.

Then I said I would pick 2, explained why, and offered to do the work either way.

How it ended

The next morning, Christopher Hertel landed a separate PR:

The {#123} handles VarDumper prints next to an object are spl_object_id() values, so they shift whenever anything allocates one more object earlier in the process. (...) Instead of resyncing the ids, bootstrap.php now dumps with withRefHandles(false), so the handles never enter the output.

One line in the bootstrap. Twenty-eight fixture files regenerated from the existing cassettes — no re-recording, no cassette touched, and every changed line was a handle line.

Two things in that PR description are worth sitting with.

It was not my change that broke it. main was already red. Adding Capability::REALTIME_SESSION in an unrelated feature had been enough to break the same 21 examples days earlier. I had not caused the problem; I had walked into it, and my diff happened to be holding the bag when the CI ran.

And the fix was the boring one. Not clever, not defensive, not a workaround in my class. One flag, in the one place that owns the output format.

My contribution merged three days later.

What I would take from this

If your test compares a dump, strip what you are not asserting. Object ids, memory addresses, timestamps, autoincrement keys. Anything the runtime assigns rather than your code producing.

A test that fails for a reason unrelated to its name is worse than no test. Twenty-five red builds told me nothing about whether redaction works. They told me about PHP's allocator.

And when the fix belongs to someone else's file, say so. I spent two CI cycles trying to make the problem disappear from inside my own diff. The useful move was to measure it, write it down, and hand the decision to the people who own those fixtures.

That last one took me longer to learn than it should have.

---

The contribution is symfony/ai#2487. The fix for the fixtures is symfony/ai#2531, by Christopher Hertel. I maintain llm-vcr, a record and replay library for testing AI features in PHP.