The more models I test, the less comfortable I am answering the question: Which LLM should I use?
A model can look like the obvious choice in a benchmark and still be wrong for a real product. Maybe it is too slow. Maybe the price stops making sense at scale. Maybe it follows instructions well in a clean test but falls apart on the messy inputs users actually send.
And even when you make a good choice, it may only stay good for a few months. I have been thinking about this a lot while working on TokenBay, where we connect several models through one OpenAI-compatible API. I spend a lot of time comparing model behavior, pricing, latency, and the friction involved in switching providers. The closer I look, the harder it is to believe in one universally “best” model.
There is only the model that fits your current task and constraints.
So I am curious: How are teams making this decision in practice?
Four strategies I keep running into
Most model-selection discussions seem to lead back to one of four approaches.
Use different models for different jobs
This is the most flexible setup. A team might use one model for writing, another for code, and a smaller, cheaper model for classification or tagging.
It makes sense because these tasks do not reward the same strengths. The downside is that a multi-model architecture creates more work around fallbacks, error handling, monitoring, and billing.
You stop searching for one model that does everything, but now you have to maintain the routing system that decides what does what.
Start cheap and escalate when needed
Another approach is to send a request to a cheaper model first, then escalate when the task is difficult or the result does not meet a quality threshold.
I like the idea, but the phrase “when needed” hides the hardest part. How do you know that a response is not good enough without paying another model to judge it? A confidence score is useful for some structured tasks, but much less clear for writing, research, or open-ended reasoning.
The routing logic can become more complicated than the original integration.
Choose one provider and stay there
This is often described as lock-in, but sometimes it is a completely rational decision.
If a provider has already passed a company’s security and compliance review, adding another one may create more risk and work than it removes. A slightly better model is not automatically worth a new legal review, another billing relationship, and a second set of operational failures.
For these teams, the best model may simply be the one they are allowed to use and can audit properly.
Keep testing whatever is new
Some teams treat model evaluation as an ongoing process. When a new release appears, they run it against their own tasks and switch if the improvement is meaningful.
This sounds ideal in a fast-moving market, but constant experimentation has a cost too. Prompts behave differently, tool calls break in new ways, and a better benchmark score can still produce a worse user experience.
At some point, staying current can turn into chasing every release.
None of these strategies is obviously correct. They optimize for different things: flexibility, cost, stability, or speed of adoption.
The questions I still struggle with
When is model lock-in actually a problem?
Model-agnostic architecture sounds safer. If one provider goes down, raises its prices, or changes model behavior, you have somewhere else to go.
But abstraction is not free. Providers handle tool calling, structured output, rate limits, caching, and errors differently. Even when APIs look compatible, applications still end up depending on provider-specific behavior.
So where do you draw the line? Do you build for portability from day one, or wait until switching becomes a real need?
How do you compare models in production?
Offline evals are useful, but they miss things users feel immediately: latency, consistency across a conversation, tone, and the strange edge cases that never made it into the test set.
Do you shadow traffic to a second model? Route a small percentage of users to it? Keep each user on the same model so the experience stays consistent? What do you measure besides thumbs-up and thumbs-down?
I am especially interested in what people actually do here, because production evaluation is usually much messier than the diagrams make it look.
What happens when a new model launches?
Do you test it immediately, wait for other developers to find the problems, or ignore it until your current setup gives you a reason to change?
Testing every release can consume a surprising amount of time. Ignoring new releases can leave real improvements on the table. I still do not know what a healthy evaluation cadence looks like for a small team.
Which matters most: cost, latency, or quality?
“Pick two” is a simplification, but most products still have to favor one or two of these constraints.
For a coding assistant, users may wait longer for a better answer. For autocomplete or live chat, an extra second can make the product feel broken. For high-volume classification, a small price difference can matter more than a quality gain that users never notice.
The answer changes with the task, which is another reason a company-wide “default model” can be misleading.
The decision that matters after the decision
My current view is that choosing a model matters less than knowing how you will revisit that choice.
Before committing to one, I would want to know:
Do we have a small set of real prompts that represent our product?
Can we compare cost, latency, and output quality on those prompts?
How much provider-specific behavior have we built around?
What would have to happen for us to switch?
Could we test another model without rewriting the whole application?
That does not mean every product needs an elaborate router or five providers. It just means “we use Model X” should not become an architectural fact nobody is allowed to question.
Working on a multi-model platform has made me more skeptical of permanent model choices. The model that fits a workflow today may not be the one that fits it six months from now. The useful skill is not predicting the winner. It is building a process that lets you notice when your answer has changed.
So I would love to hear how other developers handle this:
What is your current LLM setup, and what would make you switch?
If you have migrated a production workflow between models, I am especially curious about what was harder than you expected.
Top comments (0)