We are not a software company. We sell refurbished laptops and iPhones.But we run software. A lot of it. And over eighteen months of integrating LLM APIs into our operations — product description generation, customer query routing, inventory grading assistance, return reason classification — we went from "this is affordable" to "this is genuinely threatening our margins."Here is the honest breakdown of what happened, what we found when we actually looked at the numbers, and the three changes that brought our monthly LLM spend down by 71% without meaningfully degrading quality.
How We Got Into This Situation
It started with product descriptions. We list hundreds of refurbished devices. Writing accurate, specific, SEO-optimised descriptions for each one manually is expensive in time. We built a pipeline that takes device specs, condition grade, and battery health and generates listing copy. It worked well. We expanded it.
Then we added a customer query classifier. Then a return reason analyser. Then a draft email responder for common support tickets. Each addition felt incremental. Each one made sense in isolation.
Six months later our monthly LLM API spend had grown from £180 to £1,340.
For a physical products business operating on 30–35% gross margins, £1,340/month in API costs is not a rounding error. It is a meaningful percentage of margin on a significant portion of our monthly revenue. We had not modelled it that way when we built each feature. We had modelled each feature in isolation.
The Audit — What We Actually Found
LLM API pricing varies by more than 600x across major providers — from $0.05 to $30 per million input tokens depending on model and provider. We had not been paying attention to where in that range we were sitting.
When we audited our usage properly, we found three problems:
Problem 1 — We were using flagship models for tasks that didn't need them.
Our product description pipeline was running on GPT-4o. At the time that was $2.50 per million input tokens and $10 per million output tokens. We were generating descriptions for devices with specs that fit neatly into a structured template. The task required consistent formatting and accurate spec integration — not sophisticated reasoning. A flagship model was complete overkill.
Problem 2 — Our system prompts were enormous and we were paying for them on every call.
Our product description system prompt was 2,400 tokens long. It included detailed formatting instructions, brand voice guidelines, SEO principles, and example outputs for every device category. We were paying to send those 2,400 tokens on every single API call — hundreds of times per day. We had never implemented prompt caching.
Every major provider now offers prompt caching where frequently reused system prompts are stored server-side and charged at a fraction of the normal input rate — OpenAI offers 90% savings on cached reads, Anthropic charges just 10% of base input price for cache hits. WPS Office
We were paying full price for the same 2,400 tokens hundreds of times daily. That was fixable in an afternoon.
Problem 3 — We had no routing logic. Everything went to the same model.
Customer support query classification — figuring out whether an incoming message was a return request, a shipping query, a product question, or something else — was running on the same flagship model as our most complex tasks. Classification is a simple task. It does not require a model that costs $10 per million output tokens.
The Three Fixes
Fix 1 — Model tiering based on task complexity
A typical enterprise distribution routes 70% of queries to a budget model, 20% to a mid-tier model, and 10% to a premium model — this tiered approach reduces average per-query cost by 60–80% compared to routing all traffic through a single premium model. Laptop Mag
We categorised every LLM task in our stack by complexity:.
`TASK_ROUTING = {
# Simple classification, templated output
"query_classifier": "gpt-4.1-nano", # $0.10/$0.40 per MTok
"return_reason_classifier": "gpt-4.1-nano",
"email_category": "gpt-4.1-nano",
# Structured generation with moderate complexity
"product_description": "gpt-4.1-mini", # $0.40/$1.60 per MTok
"condition_summary": "gpt-4.1-mini",
# Complex reasoning, customer-facing quality required
"support_draft": "gpt-4.1", # $5/$15 per MTok
"dispute_analysis": "gpt-4.1",
}`
The classification tasks — our highest volume workload — moved from $10/MTok output to $0.40/MTok output. A 96% cost reduction on those calls.
Fix 2 — Implement prompt caching properly
For OpenAI this required structuring prompts so static content appears first:
messages = [
{
"role": "system",
"content": STATIC_SYSTEM_PROMPT # 2,400 tokens — cached after first call
},
{
"role": "user",
"content": f"Generate description for: {device_specs}" # Variable per call
}
]
For Anthropic's API, explicit cache control headers are required:
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": STATIC_SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": f"Generate description for: {device_specs}"
}
]
}
]
After implementing caching, the cost of our static system prompt dropped from full input token pricing to 10% of that — on every subsequent call. On our description pipeline running 300+ calls per day, this change alone reduced monthly spend by approximately £280.
Fix 3 — Batch processing for non-latency-sensitive workloads
Not every LLM task needs to return in real time. Our return reason classification runs on submitted return requests — not on the customer-facing interface. A 30-minute processing delay is completely acceptable.
Batch API offers 50% discount on all models at both OpenAI and Anthropic for non-latency-sensitive workloads.
We moved return reason classification, condition summary generation, and overnight inventory description updates to batch processing. Half price on all of them.
# Instead of real-time API calls for return classification:
client.batches.create(
input_file_id=batch_file_id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"job_type": "return_classification"}
)
The Numbers After Three Changes
Support email drafts showed the smallest saving because those genuinely require a higher-quality model — the output is customer-facing and quality matters. We kept that on GPT-4.1 and accepted the cost. The other three workloads moved without any meaningful quality degradation.
What I'd Have Done Differently From The Start
Build a cost model before shipping each LLM feature. Not a rough estimate — an actual calculation based on expected call volume, token counts, and the model you're planning to use. We treated LLM costs the way early cloud adopters treated compute costs — as something to worry about later. Later arrived.
Default to the cheapest model that can do the job. Start with the budget tier and work up to a more capable model only if the output quality is genuinely insufficient. We did the opposite — defaulted to the flagship and optimised down only when the bill arrived.
Implement caching from day one. There is no reason not to. For applications with consistent system prompts, prompt caching can reduce input costs by 70–90% on the cached portion. It takes an afternoon to implement and the savings are permanent. Laptop Mag
Separate latency-sensitive from batch-eligible workloads at architecture time. The question "does this need to return in under 3 seconds?" should be asked for every LLM call before it's built. If the answer is no — batch it.
The Broader Point
LLM API prices dropped approximately 80% between early 2025 and early 2026 — which is genuinely good news. But prices dropping doesn't protect you from cost growth if you're scaling usage faster than prices are falling, or if you're using the wrong model tier for the task.
The cost problem in LLM integration is not usually a pricing problem. It is an architecture problem. Flagship model where a budget model would do, uncached system prompts, synchronous calls where batch would work — these are architectural decisions made at build time that compound into significant monthly costs at scale.
The fix is not complicated. It just requires actually looking at the numbers before the bill arrives.
Exact Solution sells professionally refurbished MacBooks, iPhones, and laptops across the UK, Poland, and Europe — tested, graded, warranty backed.
exactsolution.com
Top comments (0)