DEV Community

Grain64
Grain64

Posted on Originally published at clearbar.grain64.com

Five checks before you swap to a cheaper LLM (with the code for each)

#llm #ai #openrouter #testing

A cheaper model that looks fine on ten examples can cost more per finished task than the one you have. These five checks take a few lines each. The worked numbers come from one public run: 150 banking questions with 77 possible labels, ten models, a 70% pass bar.

1. Add up every attempt, then divide by the successes. Your bill includes the retries, and so should the price you compare.

const costPerSuccess = attempts.reduce((sum, a) => sum + a.cost, 0) / successes;
Enter fullscreen mode Exit fullscreen mode

On that run DeepSeek V4.1 Flash made 32 retries and spent 31.3% of its money on them; 11 of its final answers were still unreadable. Mistral Large 4 made 106 retries (44.6% of its spend) and ended at $0.002274 per success, above the $0.001272 of the baseline, GPT-6.1 Sol.

2. Judge by the lower edge of the interval, not the pass rate. On 150 rows, 77.3% is not "clears 70%".

function wilsonLow(k, n, z = 1.96) {
  const p = k / n, d = 1 + z * z / n, c = p + z * z / (2 * n);
  const m = z * Math.sqrt(p * (1 - p) / n + z * z / (4 * n * n));
  return (c - m) / d;
}
// wilsonLow(116, 150) -> 0.7000
Enter fullscreen mode Exit fullscreen mode

Qwen 3.8 Flash passed 116 of 150 (77.3%), and its interval's lower edge is exactly 70.0%, the bar: borderline. Gemini 3.5 Flash Lite (113 of 150) and Llama 4 Maverick (107 of 150) read the same way, above 70% as a point estimate and not clearly above it as an interval. GPT-6 Luna, at 123 of 150, has a lower edge of 75.1% and clears.

3. Count unreadable replies apart from wrong ones. A reply you cannot parse is a different failure from a reply that parses to the wrong label: the first gets retried and billed, the second is final. Keep two counters (invalid_final, wrong) and look at both. In the run above, DeepSeek V4.1 Flash had 11 unreadable finals, Qwen 3.8 Flash 8, Mistral Large 4 45; the other seven models had none.

4. Fix the bar before you look at any results. Write the pass rate and the margin you need in a file, commit it, then run. If you pick the bar after seeing the table, it will land under whichever model you were already hoping for.

5. Price the cheapest model that clears the bar against what you run today, on the same rows. On this run GPT-6 Luna is the cheapest model that clears the bar, at $0.00009 per success against $0.001272 for Sol, about 14 times less. It passes 82.0% against Sol's 86.0%, and on the same rows it fails 8 that Sol passes and passes 2 that Sol fails. Whether that trade matters is your call, and only your own examples can answer it.

All 150 rows, the prompt, the prices and the re-run script are on the verdict page, so you can repeat the run with your own OpenRouter key: https://clearbar.grain64.com/verdicts/banking77-intent/?utm_source=devto&utm_campaign=devto-five-checks

This run covers one case: 150 rows of single-label classification, where a row either matches its label or it does not. If your task is summarisation or free-text generation, where "pass" is a judge's call, would you still set the bar on the lower edge of the interval, or would you widen it for the judge's own disagreement with your labels, and by how much?

From Clearbar, by Grain64.

Top comments (0)