Workshops at this year's AI Engineer World's Fair shifted noticeably away from RAG and prompt engineering toward evals and open models. This transition highlights a broader change in our industry — an increased focus on how to measure and trust AI outputs. It's an important problem to tackle.
The more immediate issue is that most engineers simply refuse to consider any model other than the latest and most powerful frontier for their day-to-day tasks. I spend an inordinate amount of time trying to convince people that frontier models are not always necessary. Engineers tend to default to them even for trivial tasks like checking the weather. There's a change in developer behavior and belief that needs to occur before fast models can succeed.
We don't collectively seem ready to trust AI yet. The tendency to default to the most powerful option is a safe hedge. But today's fast models are equivalent to what the frontier was six months ago. Sonnet 4.6 performs comparably to Opus 4.1, Gemini Flash 3.5 competes with Gemini Pro 3.1, and GPT-5.4 Mini matches the performance of GPT-5.1. Fast models are a fraction of the cost, and they're substantially faster than waiting for a max-thinking response.
As models continue to improve, we're evaluating and optimizing for correctness when we might not need to. This might just be my hill to die on, but there's something that irks me about developers' unwillingness to use anything other than the frontier.
If you were happy with Opus's weather results six months ago, today's Sonnet can probably handle the job. The next time you're checking the weather, maybe think about a fast model. You might not miss the frontier.
Top comments (8)
Maybe the trick could be as simple as a few changes in the UI:
vs the current typical GUI which comes with a barely noticeable dropdown you have to remember to click every time before hitting send, or the typical TUI where you have to /model it in the most unnatural way you can think of.
I like the instinct, but imo surfacing 3 model buttons just moves the guess to the user at the worst possible moment. The reason people default to the frontier isn't that the dropdown is hidden, it's that they can't tell upfront which prompts are the easy ones (Nazar's fuzzy-middle point above). Give them 3 buttons and most will just hammer the biggest one every time, same hedge w/ a nicer UI. You might even entrench frontier-defaulting by making the powerful option a one-tap thing.
What I'd actually want is the opposite: don't make me pick, default to fast and make the escalation visible. Cheap model runs by default, and either it auto-bumps when the output fails a quick check, or I get a one-tap "redo on the big one" when I'm not happy. That way the human only spends a decision on the misses instead of pre-judging every prompt. Routing made visible beats model-selection made manual, tbh, the choice you make best is the one you make after seeing the answer, not before.
The real change in the industry is MoE. Take Gemini 3.5 Flash, the performance difference to Gemini 3 Flash, is the difference MoE makes... Less active parameters, same param count = higher throughput and lower hallucination, because the model is completely focused.
The weather example is easy to agree with, but I think the reason people reach for the frontier is they can't tell upfront which tasks are the easy ones. Checking the weather is obviously trivial, but most real prompts sit in a fuzzy middle where you can't tell if a small model will nail it until you've already seen the answer. That's why a default of frontier everywhere feels safer even when it's wasteful. The thing that actually changes the habit is probably routing rather than willpower, where a cheap model takes the first pass and only escalates when its own confidence is low. Have you seen that work in practice, or does the escalation overhead eat the savings?
The default-to-frontier habit is real. Worth naming what it hedges against. It is not "will this model get it right" hedge. It is "when this goes sideways, can I blame the model or do I have to have caught it myself" hedge. Frontier default is a verification-shifting move, not a capability move. Fast models don't lose to Opus on weather. They lose to it on who takes the fall.
The "trust gap" piece is the one that doesn't get enough airtime. Engineers default to frontier models not because they need the raw capability, but because a worst-case failure with a frontier model feels defensible while a worst-case failure with a fast model feels like a bad call. The interesting shift happens when you decouple the model choice from the validation layer â if you have good evals and persistent state, you can run a smaller model ten times and pick the best result cheaper than paying for one frontier call. The bottleneck isn't really the model anymore, it's the infrastructure to trust the output.
Your point is the crux: the weather's obviously trivial, but most tasks sit in the fuzzy middle, so routing beats willpower. The catch is what you route ON. If you escalate based on the model's own confidence, you're trusting the exact thing small models are worst at, since they'll hand you a wrong answer with full confidence.
What's held up for me is gating escalation on a cheap deterministic check of the OUTPUT, not the model's self-report: did it parse, does it pass the schema, does it satisfy a rule or a quick validator. For structured work (extraction, classification, SQL, anything with a checkable shape) that check costs almost nothing, so the cheap-first-pass savings survive. The fast model takes most of the volume, the validator catches the misses, and only the failures escalate to the frontier. Where it breaks down is open-ended generation, where "is this good" isn't cheaply checkable, and that's exactly where the escalation overhead can eat the savings.
Which is probably why the Fair drifted toward evals. Routing without an eval is just a vibe, you have to measure which slice the fast model actually nails before you trust a router to send it there.
Thanks for sharing this informative post! It gave me a better understanding of the current trends in AI and why choosing the right model for the right task is important.