How I Cut LLM Token Usage by 99% — 3 Production Engineering Steps

#rag #llm #python #ai

At Idyllic Services, I was working on an automated candidate sourcing system running LLM calls at scale. One iteration consumed 200,000+ tokens (costing ₹15–20 per run) and took 6 minutes.

I redesigned the pipeline architecture to cut token consumption by ~99% down to 5,000 tokens and reduced execution time to 90 seconds. Here is the exact 3-step breakdown of how I did it.

1. RAG Integration, Semantic Routing & Caching

The biggest inefficiency was dumping full candidate histories and long job descriptions into every single prompt.

RAG Memory Retrieval: I integrated Qdrant vector database to store conversation history and candidate data. Instead of sending 15,000 tokens of raw history, nearest-neighbor search retrieved only the top 2–3 relevant snippets (~400 tokens) per step.
Semantic Routing: Instead of sending every request to expensive LLMs, I implemented semantic routing to direct deterministic tasks to fast regex or local Python scripts.
SHA-256 Response Caching: For repeat queries (same JD, same candidate profile), I cached responses using prompt content hashes — making duplicate runs cost 0 tokens.

2. TOON Format (Token-Oriented Object Notation)

JSON has massive syntactic overhead (braces, quotes, repeated keys). When sending structured arrays of candidate data to LLMs, this overhead is pure waste.

I switched the prompt payload format to TOON (Token-Oriented Object Notation), which declares keys once as a header row:

# JSON (before) — 847 tokens for 5 candidates:
[
  {"name": "Rahul Sharma", "skills": ["Python", "AWS"], "yoe": 4},
  {"name": "Priya Patil",  "skills": ["React", "Node"],  "yoe": 2}
]

# TOON (after) — 290 tokens for 5 candidates:
{name | skills | yoe}
Rahul Sharma | Python, AWS | 4
Priya Patil  | React, Node | 2

Result: 30–40% baseline token reduction on all structured payloads with zero information loss.

3. Removing LLMs from Loops · Batch API Calls

The original codebase called the LLM once per candidate inside a for loop (50 candidates = 50 separate API calls, each carrying full instructions).

I replaced the sequential loop with a single structured batch prompt:

# BEFORE: LLM inside a loop (50 API calls = 200K tokens)
for candidate in candidates:
    score = llm.invoke(f"Score candidate:\nJD: {full_jd}\nCandidate: {candidate}")

# AFTER: Single batched call (1 API call = ~5K tokens)
batch_prompt = build_batch_prompt(jd_toon, candidates_toon)
all_scores = llm.invoke(batch_prompt)

Summary of Results

Optimization	Impact
RAG + Routing + Caching	-95% memory tokens, 0 tokens on repeats
TOON Format	-30% to -40% payload overhead
Batch API Calls	50 calls → 1 call (Runtime: 6 min → 90 sec)