The short version
- To pick a cost-effective LLM, score your prompt's outputs on your best model, then on cheaper and smaller models, and use the cheapest one whose average still clears your quality bar.
- Run the prompt 5 to 10 times on each model and average the scores, because outputs vary from run to run and a model that scored 98 once can score 85 on the next run.
- Set the minimum score per task before you test, for example 90 out of 100. When a model drops below it in most runs, go back up one tier.
- Freeze the prompt before comparing models. Every model has to be scored on the same prompt and inputs, or the averages can't be compared.
- Most business tasks, like summaries and pulling fields out of documents, run fine on small, cheap models. Keep frontier models for work that needs real reasoning.
To choose the cheapest LLM that still does a task well, run your prompt 5 to 10 times on your best model and score every output out of 100. Then run the same prompt the same number of times on cheaper and smaller models, and keep going down until the average score drops below the bar you need. The model you use is the cheapest one that still clears that bar.
Below is that process step by step, with the spreadsheet I track it in and the mistakes that make the scores lie. The video above covers the same thing.
Why most teams overpay for tokens
Companies are spending ridiculous amounts on LLM tokens without much to show for it. The Information got hold of an internal Meta leaderboard showing employees used more than 60 trillion tokens in 30 days. At Claude Opus list prices that's around $900 million. Meta's real bill isn't public, and cached tokens cost far less, but it gives you the scale. Uber's CTO said the company went through its whole 2026 budget for AI coding tools in about four months.
A lot of that comes from mandates. When a company tracks how many tokens each employee uses and pushes them to use more, people run frontier models on everything, including summarizing emails and drafting PRs. A small, cheap model like Gemma 4 handles those jobs fine.
There are other ways to cut AI costs, like better prompts and caching (I cover how prompt caching works in another video). This page is only about picking the model, which is the one most teams skip.
Step 1: pick one task and write the prompt
You need one task you're automating and the prompt that does it. That's the prompt you'd feed to the LLM or the agent in production.
Do this per task, not per company. "Summarize this support ticket" and "plan a database migration" need very different models, and a single company-wide default is how you end up paying frontier prices for ticket summaries.
Step 2: score your best model first
Take the prompt to the most capable model you have, whether that's from Anthropic, OpenAI or anyone else. Run it 5 to 10 times and score every output from 0 to 100. Then average the scores.
One run tells you nothing. LLM outputs vary from run to run, and a model that scored 98 once can score 85 the next time on the same input. The average over several runs is the number you compare.
This first round gives you the ceiling. If your best model averages 97, that's roughly the best this prompt can do, and it's what every cheaper model gets measured against.
You'll also find problems with the prompt while you do this. Repeated runs show you the edge cases the model trips on and the instructions you forgot to write down. Fix the prompt as you go, with more specific instructions and a couple of good examples for the edge cases. Once you're happy with it, freeze it and re-score the frontier model with the final version, because every model has to be tested on the same prompt or the comparison means nothing.
Step 3: move down to cheaper models
Now pick a cheaper, smaller model and do exactly the same thing. Same prompt, same inputs, 5 to 10 runs, every output scored out of 100, then the average.
Keep stepping down. At first the scores barely move. The frontier model might give you 97, 98, 99, and the next model down might give you 97, 95, 94. At some point you hit a model where the outputs get noticeably worse, and that's the point you're looking for.
Step 4: decide where to stop
Before you start, set the minimum score this task needs. For a lot of business tasks I'd say 90 out of 100. A model that averages below that is out, no matter how cheap it is.
When a model starts falling below your bar in most of its runs (scores like 85, 83, 82), go back up one tier to the slightly smarter model before it. That's the model for this task. If the cheapest model you've tried still clears the bar, try one that's cheaper still.
Leave yourself some margin. If a model averages 91 and its runs swing between 86 and 96, it will fail your bar in production on a bad day. I'd rather pick a model that clears the bar by more than its usual swing.
The spreadsheet
You don't need any tooling for this. I use a spreadsheet with one row per model:
| Task | Prompt | Model | Scores | Average |
|---|---|---|---|---|
| Summarize a support ticket | v3 | Frontier model | 97, 95, 98, 99, 97 | 97.2 |
| Summarize a support ticket | v3 | Mid-tier model | 96, 94, 95, 97, 93 | 95.0 |
| Summarize a support ticket | v3 | Small model | 93, 91, 94, 90, 92 | 92.0 |
| Summarize a support ticket | v3 | Smallest model | 85, 83, 88, 82, 86 | 84.8 |
These numbers are made up to show the shape. In the video I score Claude Sonnet at 97, 95, 98, 99 and 97, the same as the first row. With a bar of 90, the small model wins here: it averages 92 and its worst run is 90. The smallest model is out.
Put the prompt version in its own column. When you change the prompt, the old scores no longer count, and you want to see that at a glance.
How to score without fooling yourself
Scoring out of 100 by feel works, but it drifts. After 30 outputs you'll grade the 31st differently from the first. Before you start, write down what a 100 looks like and what costs points. For a ticket summary that might be "names the customer's problem and includes the order number, in under 80 words".
Use the same set of inputs for every model, and pick inputs that look like real traffic, awkward ones included. If you can, hide the model name while you score. It's easy to give the expensive model the benefit of the doubt.
When the outputs are open-ended and there are a lot of them, you can have another LLM score them against your written criteria. That's usually called LLM-as-a-judge. It saves time, but the judge costs tokens too, and you should check a sample of its scores against your own before you trust it.
Also compare models on what a finished task costs, not just the per-token price. A cheap model that writes three times as many tokens, or that fails often enough that you have to re-run it, can cost more per useful output than the model above it.
Which tasks can run on small models
When you do this for a while, you'll find that most business tasks run fine on small, cheap models. Summaries, first drafts of emails and PR descriptions, pulling fields out of a document, sorting messages into categories: none of these need a frontier model.
Save the expensive models for tasks that need real reasoning, like planning a system or debugging something that spans a whole codebase. Using a frontier model to summarize emails is like hiring a senior engineer to rename files.
Re-run the test when a new model comes out or prices change. A provider can also update a model you picked and make it worse at your task without telling you. Keep the spreadsheet and your test inputs, and re-scoring takes an afternoon.
Is it worth the manual work?
I'll be honest, this is so obvious that you might wonder why I made a video about it. The answer is that most companies don't do it. They pick one model, usually the smartest one, and run everything through it.
It's a lot of manual work, for sure. But it drives costs down hard, and you can keep using all these models without bankrupting your company, or your own wallet if you're paying for this yourself. If you'd rather have someone set this up for your workflows, that's the kind of thing I do in AI automation projects.
Originally published on hassandev.me.
Top comments (0)