Two identical runs, two different conversations

Matías Hernández — Thu, 06 Aug 2026 15:39:05 +0000

What "reproducible" means when the system you're measuring has memory.

Turn 3 of a scripted conversation. The question was "What's my brother's name?".

Run A answered "His name is Nicolás." Run B answered "Nicolás."

Same fact, same configuration, same pinned clock, same seed, temperature zero. Two processes that were supposed to be identical, and from that turn on they weren't, because my system stores every response as a memory and that memory feeds the next turn. From turn 3 onward, the two conversations lived different lives.

I had gotten there to verify something I thought was obvious, that my measuring instrument was reproducible. What I verified was something else: that the reproducibility I was looking for is physically impossible, and that defining what "the same" means was going to be the real work.

What I was measuring

I build a memory system for a conversational agent. It remembers what you tell it, consolidates it, and retrieves the relevant context on every turn. It runs on Node, against hosted models.

After months of building I wanted to start measuring. There are questions you can't avoid once the system stops being a toy. How much does each turn cost as the conversation grows? Does the sophisticated memory ranking earn its place over plain cosine similarity? Those are experiment questions, and an experiment needs an instrument.

That's where I set the first rule of the process: calibrate the instrument before measuring with it. The instrument here isn't the system in the abstract, it's the harness — the simulator that runs a scripted conversation, the pinned clock that makes two runs comparable, and the traces that record what the system did on each turn. None of it had ever been put under measurement.

Before running a single experiment, I audited the harness.

Before measuring, the instrument was lying

The central finding wasn't a suspicion, it was a query against the production trace database: 49 of 208 chat turns, 23.6%, carried the exact signature of a code fallback.

The component that classifies user intent has a fallback path. If the model call fails, it returns "this is chit-chat, do nothing". As a defense that seems reasonable to me. What isn't reasonable is that it's silent: in nearly one out of every four turns, the system retrieved no memory, stored nothing, executed no action, and not because the user was making small talk but because a call had failed.

"Silent" is the exact word. The trace recorded confidence: 0, which is exactly what you see when the classifier actually runs and isn't sure. Nothing distinguished "the classifier hesitated" from "the classifier never ran". I had been staring at those traces for months.

I could only detect it statistically, from the same triple of fields repeating across messages of 4 characters and of 22,000, across greetings and long reflections. No real classifier lands on the exact same point a quarter of the time over inputs that heterogeneous. And of those 49 turns, 22 never even completed a response: the whole request fell over, and in the trace that looked normal too.

My first hypothesis about the cause was elegant and it was wrong. I ran it against the real API and it refuted itself. The actual mechanism was different and considerably dumber: the classifier received recent context without budgeting it. There was a guardrail, the query pulls at most the last fifty messages, but a message cap is not a token cap. And it isn't a threshold the conversation crosses once it gets long, it's a ceiling that's always in place: any conversation past fifty messages inside the 72-hour window hands the classifier exactly that maximum, unbudgeted. With long messages that overflows the model's window, the API returns a 400, and a catch block turned it into "the user was just making small talk".

How many of those 49 turns fell over for exactly that reason, I don't know and I never will, because the trace didn't record the fallback's cause. That field was part of the fix, not part of the diagnosis.

The second finding was in the harness itself. The simulator had eight assertions about memory consolidation, gated on a trace field that no code ever wrote. The condition was never true, so all eight skipped silently and the simulator reported success with exit code 0. The instrument I was planning to measure with had eight checks that never checked anything.

The audit found seven blocking defects across twenty-seven findings. And the lesson I liked learning least is that observability that lies is worse than no observability, because on top of everything it charges you confidence for the service.

The fixes, and the bet

Seven fixes, all surgical, none of them re-architecture. The character of the list says a fair amount about what kind of problem this was: unify the clocks so a single timestamp governs the whole turn, label every fallback with its cause so it stops being indistinguishable from a genuine result, cost and latency accounting per call instead of only the last one, pinnable sampling parameters, honest exit codes.

Then came the bet, which was the exit gate. Run the same scripted conversation twice with everything pinned — temperature zero, fixed seed, pinned clock, a clean database per run, against real models and real embeddings — and verify that the traces came out equivalent.

I expected them to come out identical. I'm writing that down knowing how it sounds.

"The same" doesn't exist

Turn 3, "His name is Nicolás." against "Nicolás."

The seed the provider offers is best-effort by contract, it doesn't guarantee byte-for-byte identity even at temperature zero. I knew that, in fact it was written down as a footnote before I ran anything. What I hadn't internalized is what happens next.

Divergence compounds, in the compound-interest sense: each divergence becomes the base for the next one. The mechanic is three steps — the response is persisted as a memory, that memory enters the next turn's context, and the two processes stop receiving the same input. From turn 3 on, runs A and B were no longer looking at the same world. The divergence wasn't in wording but in conversational state, and by that I mean something concrete: the set of memories each run was accumulating. A stored "His name is Nicolás." and B stored "Nicolás.". That difference reaches the next turn unmediated, because the recent-context block carries the last exchanges exactly as they were stored. And it comes back later through another door: once that memory falls out of the recent window, they are two texts with different embeddings, competing differently to get back into the prompt. From that point on the two runs didn't just say different things, they had different material to decide with.

What I found interesting is how long the system held. The intent classifier absorbed about ten turns of textual drift without changing behavior, because the inputs were different but not different enough to cross any decision boundary. The first hard difference showed up only at turn 13, and it was barely a tremor in the classifier's confidence, 0.9 against 0.95, with the same intent in both runs. Behavior proper diverged only at turn 22, when one run decided to retrieve memory and the other didn't.

From there it cascaded: different intents dispatch different actions, which write different memories, which change the next turn's context. I ended up with 34 hard differences by the end of the run — by hard difference I mean a difference in a field that should never vary, like which intent was classified or which actions were executed. They were spread across 12 turns, and several were the same event seen from different fields: a single routing flip showed up at once as intent, confidence, action count and executed actions.

The comparator's formal verdict was NOT EQUIVALENT, exit code 1.

And here comes the number that matters most to me, which isn't the one I planned to use. The genuinely clean turns, with identical inputs in both runs, were two out of thirty-three: drift starts far too early and leaves no margin. Saying "zero hard differences in clean turns" sounds forceful, but on a base of two turns it doesn't prove much.

What does prove something is a different measurement. There were 18 turns where the context assembled for each run was demonstrably different — the prompt block lengths and the router's character budget differ, and that's measured — and every field that must not vary still agreed: intent, confidence, temporal scope, policy, action dispatch, model-call sequence. Eighteen real opportunities to fail, eighteen agreements. That is what licenses the claim that the deterministic layer always does the same thing given the same input, and that where it stopped agreeing, it was because the input was already different. What doesn't license it is the absence of failures in two turns where nothing could have failed.

Redefining "the same" without cheating

Here's the part of the method that matters to me, because it's where you can cheat without anyone noticing. The easy way out was to lower the bar by feel and say "34 differences, sure, but they all come from a wording difference at turn 3, so they're cascade, so they're noise". The problem is that without a rule written in advance, that sentence doesn't distinguish a genuine cascade from a defect I find convenient to call a cascade. I don't think that qualifies as a criterion, it's a hunch in the shape of one.

What I did, before looking at a single data point, was classify every trace field into three tiers.

Exact equality required, for the deterministic spine: which intent was classified, which temporal policy applied, which actions were dispatched, which sequence of model calls occurred.
Difference tolerated, but only with attribution to one of two noise mechanisms named in advance: the model's sampling variance, and embedding jitter (the same string embedded twice doesn't always return the same vector, and that's enough to flip two candidates sitting on a tie).
Excluded by design, the things that can never match: identifiers, latencies, durations.

The governing rule was zero unattributed differences. A global tolerance, the "it passes if 95% of fields agree" kind, lets a real defect hide behind the noise budget. Attribution doesn't allow that, because it demands every difference belong to a mechanism I already understand.

I also added a cascade rule. From the turn where textual drift starts, in this case turn 3, any hard difference appearing afterwards gets flagged separately, because a benign explanation is already available: the inputs stopped being equal. It still counts, but it's kept apart from a difference in a turn whose inputs really were identical, which is the only place where a divergence has no possible explanation outside the system itself.

Now the part that was hard to write.

The classification was pre-registered and it worked as designed. All 34 hard differences landed after turn 3, so the cascade rule applied to every one of them; every tolerated difference was attributed to one of the two named noise mechanisms; and nothing forced me to invent a new category after seeing the data, which is the real test of a pre-registered taxonomy.

What I amended after seeing the results was the pass criterion. The original said any hard difference fails, and with that the gate was impossible to pass for any multi-turn conversation against a hosted model. I wrote a dated addendum redefining it in one line: it passes if no hard difference appears in a turn whose inputs were identical, and every residual is attributed to a named noise mechanism.

I'm aware that amending the bar after seeing the data is exactly the move that invalidates an experiment. That's why the addendum carries a date and says out loud that it came after the run, because the distinction between what was registered before and what was decided after is part of the data. And I think the justification stands on its own: a criterion no system can satisfy tells you nothing about that system.

If you build agents

Two things I'm taking away that I think transfer.

Instrument first, experiment second: audit your traces against production before you believe them.

And the one that surprised me most, which is that divergence compounds through persisted state. In a system with memory, two "identical" runs don't just diverge in wording but in conversational state, and from there in behavior. A system holding for several turns before it shows isn't robustness, it's delay.

I'll also state what none of this claims. It doesn't claim text-to-text reproducibility, in fact it demonstrates that's impossible. And it doesn't validate any experiment yet, it only validates that the instrument that will measure them doesn't lie about its own behavior.

What's next

With the instrument calibrated, the experiments begin, to understand which memory features earn their place and what each one costs.

If you build agents and measured something similar, or something different, or reached the opposite conclusion, I'd like to compare notes. Especially if you found a way to bound that compounding effect that I might be missing.

DEV Community: Matías Hernández