Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task.
That is embarrassing onstage. In produ...
For further actions, you may consider blocking this person and/or reporting abuse
The temperature-0 point is the one most people miss: greedy decoding fixes how a token is picked, not the distribution it is picked from, so a near-tie can still flip on a hosted endpoint. Pass^k next to Mean@k should be standard.
We see the same thing in screen-control agents (I'm on the Auten team, it lets coding agents drive a real computer and Android phone). Our answer for repeated UI tasks is blunter than guidelines: once a run succeeds, the steps are saved and replayed without a model call at all, and the model only comes back in when the screen no longer matches. That removes the flat decisions on the happy path, but it moves the risk to the "does this screen still match" check.
Question: in your traces, how many flip-prone steps came from ambiguous observations (two similar notes, two matching search results) rather than from the reasoning itself? It reads like Guideline 2 is exactly that kind, and I'm curious whether observation-side flips respond to guidelines as well as strategy flips like the regex one.
Great question. We did not classify the flagged steps as observation ambiguity versus reasoning, so I don’t have a reliable count. In this example, Guideline 2 is observation-side and Guideline 1 is strategy-side, but they were evaluated together, so we cannot isolate their individual effects. Your Auten approach is interesting—how do you determine when the screen match is reliable enough to replay safely? And how do you address consistency at that decision point? Is the answer to “does this screen still match?” consistent across repeated runs?
Thanks, that's a fair caveat on Guidelines 1 and 2 being tested together.
On the match check: it isn't a model call, so for an identical screen it returns the same answer every run. The variance comes from the screen itself: a page that hasn't finished loading, a popup, a second item that matches the same description. Our rule is that anything short of a clear match hands the step back to the model rather than clicking. So we accept a slower, model-assisted step in exchange for never replaying blind. That's the trade-off.
Honestly, we don't have a Pass^k-style number for that check yet. Your post made me want one: replay the same task k times on the same machine and count how often the match decision differs. Ambiguous observations like your two-similar-notes case are exactly where I'd expect it to wobble.
Got it. The matcher is deterministic and errs on the side of caution. However, you may encounter the same consistency issue once the step is handed back to the model. Also, when the model successfully executes, do you update the saved replay path or replace it? That seems like a useful opportunity to learn.
Right, the consistency question doesn't go away, it just moves to fewer steps. The model only sees the one step that didn't match, with the rest of the run already fixed, so there are fewer places for a near-tie to flip. But that step is exactly as flip-prone as you'd expect.
On your second question: we update the saved task. When the model gets past a step that no longer matched and the run finishes, that step is written back, so the next replay uses the new version. That's what lets it keep working after an app changes its UI.
The risk you're pointing at is real. A step that worked once by luck gets saved too, so one lucky run becomes the default. Running it k times before saving would be the obvious guard, and your Pass^k framing is a good way to set the threshold.
That makes sense: updating the saved task after recovery provides adaptation, while a Pass^k-style promotion gate could prevent one lucky recovery from becoming the new default. But what about the side effects of running the same task—for example, sending the same message multiple times? Do you retain the previous step for rollback when a replacement is written?
One thing I'd watch with distilled guidelines: they're fitted to one model's trajectories. I swap models per job a lot, and a guideline that pinned a flip-prone step for one model can create a new near-tie for the next one, because now it's arguing with that model's defaults. I'd treat the guideline set like a lockfile tied to a model version: rerun the Pass^k check on every model or prompt change and drop guidelines that stop earning their keep, otherwise the file only grows.
Did any guidelines make consistency worse on the similar-task split?
Good point—I like the lockfile analogy. We have observed that guidelines tend to be better and more suitable when the generation model is the same as the agent’s model. On the similar-task split, Pass^5 improved overall, although we observed regressions in a few cases. Because we did not run per-guideline ablations, we cannot attribute those regressions to specific guidelines.
The gap between Mean@5 and Pass^5 is the number I wish more teams reported. A 77 percent average sounds shippable. Knowing that only about half the tasks pass on all five runs tells you what users will actually feel.
Reid's split between input ambiguity and plan branching seems like the right first cut. Ambiguity is often a requirements problem, and it can be fixed by narrowing what the agent is asked to do. Plan branching is closer to what your guidelines target. Auten's approach of replaying a saved successful path and handing back to the model only when nothing matches is a practical middle ground for repeated tasks.
On client work we have started writing acceptance as passes on every run for a fixed set of named cases, agreed before the build. It changes the conversation early, because the owner has to say which cases matter most, and the flip prone ones show up during scoping.
Did the analyzer find that most flips clustered around a few step types?
I like the idea of agreeing on named cases and all-runs acceptance before the build. I think thats the way to go with AI systems and making sure they are production ready. We did not aggregate flip-prone steps into a taxonomy, so I can’t say that most clustered around a few types. Qualitatively, recurring patterns included API selection and parameters, and pagination. Separating input ambiguity from plan branching would be a useful next analysis. are you choosing k for client acceptance cases?
The sampling-temperature fix is what catches people off guard: greedy decoding pins down token selection, but if the underlying logits shift even slightly across revisions or hardware targets, a near-tie still flips. Pass^k exposes that variance much better than Mean@k ever did.
One split we found in our agent traces was whether the flip came from ambiguous input state or from branching in the plan itself. When a tool returns two nearly identical records, guidelines helped keep the choice consistent. But when the prompt just gave the model too much runway to restructure its own steps, guidelines felt like patching over a design that should have been a deterministic state machine instead.
Agreed—that’s an important boundary. We detect unstable decision steps, but did not classify them as observation ambiguity versus plan branching. Guidelines make sense where judgment remains necessary. I’d be very interested to hear more about your use case and what you observed in your evaluations.
Pass^k belongs next to cost. A setup that passes one run in three costs three runs for each success, and Mean@k hides that. Cost for each passing task makes the flaky setup show its price.
Agreed. Pass^k measures dependability; cost per successful task exposes the real cost of retries. That’s something worth considering in our next set of experiments.