I’ve seen quite a few “multi-model agents” lately.
Usually there’s a dropdown in the corner of the screen that allows switching from one model to another. Which is useful. I also enjoy using dropdowns. It makes everything feel very configurable.
But then, when you click on that dropdown…
The model may return an answer that is in a different format than you asked for, or which calls into tools that either omit or include one that was previously not called at all.
The dropdown worked perfectly.
The agent, unfortunately, has developed a new personality.
The model changed. Everything around it noticed.
Imagine a small agent that reads a support ticket, looks up the customer’s account, and returns a suggested reply.
One model might return exactly what you asked for:
{
"priority": "high",
"action": "refund",
"reply": "I’ve issued a refund..."
}
Another may return the same information with a short introduction:
`Here is the recommended response:
{
"priority": "high",
"action": "refund",
"reply": "I’ve issued a refund..."
}`
A third may decide recommended_action is a much nicer field name than action.
To a person, these answers are basically the same. To the next function in the pipeline, they may be three completely different situations.
This is where “supports multiple models” starts becoming less about access and more about everything wrapped around the model.
Tool use is not identical either
The differences become more obvious when the agent has tools.
One model may look up the customer account immediately. Another may ask a follow-up question first. A third may write a confident response without checking the account at all, which is efficient in the same way skipping all your tests is efficient.
Even when models have access to the same tools, they do not always make the same decisions about when to use them.
That means the harness has to do more than pass messages back and forth. It needs to check whether the required steps actually happened.
Did the model call the account tool before recommending a refund? Did it receive a valid result? Did it finish the task, or did it simply announce that it was finished?
Apparently, “done” is also a matter of opinion.
A failed agent does not always return an error
The annoying failures are rarely the dramatic ones.
A timeout is easy to notice. A rate-limit error is clear. You can retry or switch providers.
The stranger case is when the API returns 200 OK, the model gives you a polished answer, and the workflow has absolutely no idea what to do next.
The request succeeded. The task did not.
So a useful multi-model agent needs its own definition of success. The output may need to pass a schema check. Certain tools may need to appear in the trace. A value may need to fall within an expected range. If those checks fail, the harness can retry, repair the response, or move the task to another model.
Without that layer, model switching is mostly optimistic copy-pasting.
The less photogenic part of multi-model systems
We’ve been running into this while working on TokenBay.
Connecting several models through one API is the neat part you can show in a demo. The less photogenic part is making sure the rest of the workflow does not panic every time the model changes.
In practice, that means normalizing outputs, checking tool calls, handling different failure patterns, and keeping the application’s internal contract stable even when the model is not.
The models do not need to behave identically. They probably never will.
The workflow just needs to know what to do when they don’t.
A model dropdown can give users a choice. A real multi-model agent needs enough plumbing behind that choice to make switching boring.
And boring, in production, is usually a compliment.
Top comments (1)
I think the biggest misconception is assuming API compatibility means behavioral compatibility. Two models can expose the same endpoint, support the same tools, and still produce very different workflows. That’s why I see the harness less as a provider abstraction layer and more as a contract enforcement layer between the application and the model.