---
title: "Token Budget Optimization: Cut Your LLM API Bill by 70% Without Losing Quality"
published: true
description: "Practical techniques for reducing LLM API costs at scale: tiktoken auditing, LLMLingua prompt compression, pgvector semantic caching, and request batching."
tags: api, architecture, performance, postgresql
canonical_url: https://mvpfactory.co/blog/token-budget-optimization-llm-api-costs
---
## What We Are Building
A four-layer cost reduction pipeline for production LLM workloads. Apply all four layers to Claude or GPT-4o at scale and you are looking at **60–75% cost reduction with no measurable quality regression**. Let me show you a pattern I use in every project that handles serious API volume.
The average production prompt carries 35–50% redundant tokens you are paying for and the model largely ignores. Most teams treat this as a billing problem — they cap spend and call it done. The waste is structural.
---
## Prerequisites
- Python 3.9+ (for tiktoken and LLMLingua)
- PostgreSQL with the `pgvector` extension installed
- A backend service making calls to Claude or GPT-4o
---
## Layer 1: Audit Before You Compress
Here is the minimal setup to get this working. Before touching anything, know your baseline with `tiktoken`:
python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
INPUT_COST_PER_1M_TOKENS = 5.00 # USD — verify at your provider's pricing page
def audit_prompt(system: str, user: str) -> dict:
system_tokens = len(enc.encode(system))
user_tokens = len(enc.encode(user))
total = system_tokens + user_tokens
return {
"system": system_tokens,
"user": user_tokens,
"total": total,
"estimated_cost_usd": (total / 1_000_000) * INPUT_COST_PER_1M_TOKENS
}
Run this across one week of production logs. Teams are consistently shocked to find system prompts have ballooned to 800–1,200 tokens through iterative editing — often three times the original size. Find your top 10 by token count. Most can be trimmed 30% in a single editing pass.
---
## Layer 2: Compress Retrieved Context with LLMLingua
Microsoft's LLMLingua uses a small local model to strip low-information tokens while preserving semantic content. Compression ratios of 2–4x with under 5% quality degradation on structured tasks are realistic.
python
from llmlingua import PromptCompressor
compressor = PromptCompressor(
model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank"
)
compressed = compressor.compress_prompt(
context,
rate=0.5,
force_tokens=["\n", ".", "?"]
)
| Rate | Reduction | Quality Drop | Use Case |
|------|-----------|--------------|----------|
| 0.7 | 30% | <1% | High-stakes generation |
| 0.5 | 50% | 3–5% | Standard API workflows |
| 0.3 | 70% | 8–12% | RAG context chunks |
Apply compression **only to retrieved context**, never to structured instructions. The docs do not mention this explicitly, but compressing instruction content introduces unpredictable regressions that are difficult to catch at query time.
---
## Layer 3: Semantic Caching with pgvector
Exact-match caching misses the point. "Summarize Q3 results" and "give me a Q3 summary" should hit the same cache entry. Store embeddings of previous prompts in PostgreSQL and run a nearest-neighbor lookup before every API call:
sql
SELECT response, 1 - (embedding <=> $1::vector) AS similarity
FROM prompt_cache
WHERE 1 - (embedding <=> $1::vector) > 0.92
ORDER BY similarity DESC
LIMIT 1;
A threshold of **0.92** is your reliable starting point. Drop below 0.88 and you start serving semantically adjacent but meaningfully different responses — a quality risk not worth the savings.
Cache hit rates in document-heavy workflows (legal, finance, content pipelines) regularly reach 40–60% after a few days of warm-up. This is your highest-leverage optimization.
**Tie invalidation to your document pipeline, not just the clock.** A stale semantic cache hit is worse than a cache miss — it returns confident-sounding answers from outdated context. Set a TTL of 24–72 hours and invalidate explicitly on document update or re-ingestion events.
---
## Layer 4: Request Batching
If your backend fires individual LLM calls per user event, you are leaving efficiency on the table. Batch from a 200–500ms window:
typescript
const batchPrompt =
Process each item and return a JSON array in the same order:
${items.map((item, i) =>).join('\n')};
In production, batching 10–20 items per call reduces per-item latency by 40–60% and amortizes fixed network and cold-start overhead across the batch.
---
## Gotchas
**Compressing instructions, not just context.** LLMLingua is for RAG chunks and long-form context — not for your structured system prompt. Compressing instructions introduces subtle quality regressions that appear in edge cases, not benchmarks.
**TTL-only cache invalidation.** A cache that expires on time alone will happily serve outdated financial or medical summaries with full confidence. Tie invalidation to your ingestion pipeline.
**Batch JSON parsing.** One malformed item can poison the whole batch response. Validate structure before processing and implement per-item fallback retry logic.
**Skipping the audit step.** Jumping straight to compression without a baseline means you cannot measure what you actually saved. The audit takes an afternoon and pays for itself immediately.
---
## Putting It Together
Full pipeline: **audit with tiktoken → compress context with LLMLingua → check semantic cache → batch remaining requests → store response with embedding for future cache hits.** Each layer reduces load on the next — which is why the compound savings outpace any single technique alone.
Start with auditing. Layer in semantic caching — it requires the least architectural change and delivers fastest ROI. Then add compression, targeted only at retrieval chunks. Speaking of sustained focus during long optimization sessions: I use [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) for break reminders and desk exercises during the kind of deep backend work this pipeline requires — staying sharp across a long refactor matters.
The 60–75% reduction is real. Each step funds the next. Start this week.
Top comments (0)