We built a multi-turn harness for our support agent after a bug that a single-turn eval could not have caught. A customer asked about one account, got a good answer, then switched to a second account halfway through. The agent kept resolving the first one. The failure does not exist until there is a "before", and a single-turn case has no before.
It is the same shape as the document-switch bug I wrote about in July, where a user introduced an amendment partway through and the agent kept answering from the original contract. Different product, same failure, and the fact that I had already written it up once and still shipped a harness that could not see it should have told me something.
The harness itself: an LLM prompted to act like a customer, looped against the agent, transcript scored at the end. About forty lines. It ran, it went green, and it stayed green for six weeks while real conversations kept going sideways.
The prompt was fine. The scoring was fine. The problem was the for loop.
# simulated_user, agent and judge are all ours; the loop is the whole harness
history = []
for turn in range(6):
user_msg = simulated_user(history)
agent_msg = agent(user_msg, history)
history += [user_msg, agent_msg]
score = judge(history)
Six, because six felt like a conversation. That number is the least examined line in most harnesses I have seen, including three of ours, and it decides more than it looks like it does.
What a fixed turn budget destroys
Two conversations. In the first, the simulated customer got what they needed on turn two and the remaining four turns were padding: thank-yous, a follow-up question invented to fill the budget, a goodbye. In the second, the customer was still going at turn six, still trying to repair the misunderstanding, and the loop cut them off.
Both produce a six-turn transcript. Both go to the same judge, on the same rubric.
The first scores well, partly for four turns of work nobody needed. The second is our actual failure mode, and it scores as a slightly awkward but otherwise complete exchange. A truncated conversation does not look finished, exactly. It looks slightly awkward, and slightly awkward is well inside the noise of a transcript rubric that was never asked where the conversation stopped or why.
A turn budget is a timeout, and a timeout tells you nothing about the thing it interrupted.
Three ways to end a simulated conversation
The budget. Stop at N. Cheap, deterministic, and on its own it produces no information, because it fires on every single run and a signal with no contrast is not a signal. It has one legitimate use: a backstop against a runaway loop. Keep it for that.
A sentinel from the simulated user. Give the simulated user a way to say it is finished, a magic string it can emit or a tool it can call, and break the loop when you see it.
This is better because the stop now carries a claim: the party with a goal believes the goal is met. It is also the mechanism most likely to flatter you, and it took us a while to see why. The simulated user is a model that has been asked to behave like a satisfied customer, and models are agreeable. Ours declared itself satisfied on turn three of a conversation where the agent had answered a question it was never asked. That signal is genuine information about the simulated user rather than about the agent.
- A controller that returns a decision and a reason. Instead of a boolean, a small object: should we stop, and why. The "why" is a string, and it goes into the record.
I would like to claim we designed this. We did not. We built a worse version, then found the same shape already shipped in an open-source harness, which is roughly how every idea in this post arrived.
What made it pay was not the object. It was that a reason field forced us to enumerate the ways a conversation can end before we ran anything. Our list came out at five: the goal was met, the agent explicitly could not help, the agent looped, the user gave up, and the backstop fired. Writing that list took twenty minutes and was worth more than the harness change it justified.
Four of those five are assigned by a model reading the exchange. The loop check is the exception and is deterministic: two consecutive agent turns carrying no information the transcript did not already contain.
What we look at now
The transcript score is still there. Underneath it is a distribution over those five reasons, and the distribution is what we read first.
A run where "goal met" dominates is a good run. "Backstop fired" is now a residual rather than the default, since four other reasons get a chance first, so when its share climbs it means conversations are genuinely running longer, which for us has usually meant the agent started asking clarifying questions it did not use to ask. Any meaningful share of "agent looped" is a bug report, and it is one the transcript score had been averaging into the middle of the range for the whole six weeks, because a looping agent produces polite, well-formed, on-topic text.
The account-switching bug we started with now lands under "the user gave up", at turn four, where the budget had previously just run it out to six.
One caveat I want to be honest about: I cannot show you the reason labels are well calibrated, since four of the five come from the same class of model that plays the user. What I can say is that they are checkable in a way a scalar is not. When the controller says "agent looped" I can open the transcript and confirm or refute it in half a minute. When it says 0.71 there is nothing to open.
Objections I'd accept, and ones I wouldn't
Accepted: this is harness work, not agent work. None of it made the agent better. A human read a transcript, understood the account-switching bug, and fixed it. The harness surfaced the transcript sooner, which is worth something and is worth less than the section above might imply.
Accepted: the reason taxonomy is a maintenance surface. Five categories today, and the honest forecast is nine in a year, several overlapping, with an argument about the boundaries. Every classification scheme I have shipped has gone that way. I still prefer it to a number.
Not accepted: "we set the budget high, so nothing gets truncated." Setting it to twenty converts a truncation problem into a padding problem. You then score fourteen turns of a model being polite to another model, and a judge that weights the whole transcript gets less signal per token as the budget grows.
Not accepted: "the score already reflects this." Test it in an afternoon. Take your last hundred simulated transcripts, hand-label why each one ended, then group the scores by that label and compare the distributions. Ours sat on top of each other. That is evidence rather than proof, since a strong separation would not vindicate the score either (looping conversations are plausibly worse conversations, so the two could move together for reasons that have nothing to do with stopping). But a flat result is hard to explain away.
And note what the hand-labelling proves on its own: the stop reason is recoverable from the transcript. Nobody is hiding it. The scalar simply throws it away, because no default rubric asks.
Where I'd push back on this
The strongest case against me is that I have replaced one arbitrary decision with a slightly larger one. Six was arbitrary; so is a five-category taxonomy invented by three engineers in a room on a Tuesday. I have not shown that our five categories are the right five, or that a different team would land anywhere near them, and if the categories are wrong then the distribution I keep looking at is a well-organised way to be confidently wrong.
What I will defend is narrower. Whatever ends your simulated conversation is a decision you have already made. It is either written down somewhere you can inspect, or it is a literal in a for loop, and only one of those can be argued with.
If you do nothing else from this, go and find the number in your own harness. It is there. Somebody typed it.
Top comments (0)