Model selection is an architecture decision

Stop Asking “Which LLM Is Best?” – Ask Which Model Fits This Workload

Last quarter I was elbow‑deep in a fintech product that had to turn noisy bank statements into clean expense reports. My first instinct was to grab the “biggest” LLM, fire‑off GPT‑4, Claude‑2, Gemini‑1.5 and hope it would magically understand every column, currency and weird abbreviation. After a weekend of latency checks, cost tallies and exploding error logs I realized the “best” model on every public leaderboard still missed our real KPI: cost per successful parsing.

The fix was to stop treating LLMs like shiny toys and start treating them as replaceable services. I listed the dimensions that actually mattered:

• capability (financial jargon, tables)
• reasoning (multi‑step extraction without hallucination)
• latency (<300 ms for a smooth UI)
• context length (statements can be 10 k tokens)
• modality (sometimes we get a PDF image, so OCR + generation matters)
• cost (every API call eats our margin)

Then I built a tiny benchmark that mirrors the exact workflow: feed a real statement, ask for JSON line items, measure success (JSON parses, totals match). I ran it against three services: OpenAI gpt‑3.5‑turbo‑16k, AWS Bedrock Claude‑3 Opus, and Google Gemini‑1.5‑Flash.

Model	Accuracy	Cost / 1k‑tok	Latency
gpt‑3.5‑turbo	92 %	$0.012	420 ms
Claude‑3 Opus	95 %	$0.018	280 ms
Gemini‑Flash	88 %	$0.006	150 ms

Cost‑per‑successful‑task = (cost per call) ÷ (success rate).

• GPT‑3.5: $0.013
• Claude‑Opus: $0.019
• Gemini‑Flash: $0.007

Suddenly Gemini‑Flash became the clear winner for our budget‑tight pipeline, even though its raw accuracy looked lower on paper.

The trick is to version‑track the exact model you use (e.g., gpt-3.5-turbo-0613) and re‑run the benchmark whenever a new upgrade lands. In a later ed‑tech project I swapped Claude‑2 for Claude‑3‑Sonnet after a single test showed a 30 % latency drop and a 5 % reasoning bump, shaving $0.003 per student quiz from the cost‑per‑task metric.

So the next time someone asks “Which LLM is best?” answer with a checklist and a cost‑per‑successful‑task chart. It turns brag‑rights into real engineering decisions.

What’s the weirdest metric you’ve used to pick a model? Drop a comment, I’m curious about your war stories!

If you are someone who loves to know the technical work and architecture design I have shared details based on my experience on this here:

https://github.com/SalmonJoy/My_guide_for_building_AI_systems/blob/main/Model_selection_is_an_architecture_decision.md