How I Shipped Document QA for $0.27/M Tokens — A CTO's Diary

How I Shipped Document QA for $0.27/M Tokens — A CTO's Diary

I still remember the Slack message from our finance lead in late 2025. Our document Q&A bill had crossed five figures for the month, and the burn rate was only accelerating. We were paying $2.50 per million input tokens and $10.00 per million output tokens on what was, frankly, a glorified retrieval pipeline. That's when I knew we had a problem, and that's when the real architecture work began.

This isn't a textbook walkthrough. This is what I actually did, what blew up in my face, and the exact numbers I ran to get our document Q&A infrastructure from a runaway cost center to something that could scale to our Series A projections without giving the board a heart attack. I'm writing this in first person because honestly, the docs out there are too clean. Real CTO life is messier.

The Vendor Lock-In Trap Nobody Warns You About

When we first built our document Q&A feature, we did what most startups do. We picked the obvious default, plumbed everything through a single provider's SDK, and shipped to production. Fast iteration, great. The trap is what happens six months later when your usage is real, your customers are paying, and suddenly you're staring at an invoice that doesn't make sense.

The problem isn't just cost. It's that you've baked assumptions into your codebase that make swapping providers a multi-week migration. Your prompts are tuned for one model's behavior. Your error handling assumes one provider's failure modes. Your entire RAG pipeline expects specific function-calling semantics. Vendor lock-in isn't just pricing — it's accumulated technical debt that gets worse every quarter you ignore it.

The fix I landed on was simple in concept but annoying in execution: route everything through an OpenAI-compatible interface so the underlying provider becomes a config change, not a refactor. That's why my entire stack now points at https://global-apis.com/v1. One base URL, 184 models, and I can A/B test between DeepSeek, Qwen, and whatever new model drops next Tuesday without touching application code.

If you take one thing from this entire article, take this: the abstraction layer is the most important architectural decision you'll make. Not the model, not the vector database, not the chunking strategy. The abstraction layer.

The Numbers That Made Me Switch

I ran a proper cost analysis before moving a single request. Here's the table that lived in my Notion for two weeks while I kept poking holes in it:

Model	Input ($/M)	Output ($/M)	Context
DeepSeek V4 Flash	0.27	1.10	128K
DeepSeek V4 Pro	0.55	2.20	200K
Qwen3-32B	0.30	1.20	32K
GLM-4 Plus	0.20	0.80	128K
GPT-4o	2.50	10.00	128K

The price spread is roughly 9x on input and 12.5x on output between the cheapest and most expensive options on this list. And these aren't scrubs — DeepSeek V4 Pro gives you 200K context, which is meaningfully more than what we were paying premium prices for. GLM-4 Plus at $0.20 input is genuinely absurd from a pure cost perspective.

When I projected our document Q&A workload at scale, the numbers told a clear story. Going from GPT-4o to a mix of DeepSeek V4 Flash for simple lookups and DeepSeek V4 Pro for the complex multi-hop reasoning questions gave us a 40-65% cost reduction. Not theoretical — projected off our actual usage telemetry over 60 days. The same questions, the same documents, the same user patterns. Just better routing.

Quality didn't crater either. We held 84.6% average benchmark score across the mix versus our previous baseline. For a startup that hasn't raised a Series C, that tradeoff is obvious.

My Actual Production Stack

I won't bore you with the parts of the architecture that aren't AI-specific. But for the LLM layer specifically, here's the shape of it.

Document ingestion: PDFs, DOCX, and HTML come in. We chunk at 800 tokens with 200 token overlap. Embeddings go to a vector store (not the interesting part of this story).

Retrieval: Top-10 semantic search, reranked to top-4, fed into context. Standard.

The LLM call: This is where the architecture decisions actually matter. We run a tiered system. Simple factual lookups — "what's the cancellation policy in section 4?" — go to the cheapest viable model. Multi-document synthesis and reasoning tasks route to the heavier model. Classification and extraction tasks go to whatever's fastest. No single model handles everything because no single model needs to.

The crucial piece: every model call goes through the same OpenAI-compatible interface. Same client library. Same function signature. The only thing that changes is a string in a config file.

Code: What I Actually Wrote

Here's the Python snippet that ended up being roughly 90% of our LLM integration code. The whole thing, more or less:

import openai
import os

client = openai.OpenAI(
    base_url="https://global-apis.com/v1",
    api_key=os.environ["GLOBAL_API_KEY"],
)

def answer_document_question(question: str, context_chunks: list[str]) -> str:
    context_block = "\n\n".join(context_chunks)
    prompt = f"""You are a document Q&A assistant. Use only the provided context.

Context:
{context_block}

Question: {question}

Answer concisely and cite relevant sections when possible."""

    response = client.chat.completions.create(
        model="deepseek-ai/DeepSeek-V4-Flash",
        messages=[
            {"role": "system", "content": "You answer questions based only on provided context."},
            {"role": "user", "content": prompt},
        ],
        temperature=0.1,
        max_tokens=500,
    )
    return response.choices[0].message.content

That's it. That's the whole integration. Because we went OpenAI-compatible, the SDK is the standard one, the call signature is the standard one, and migrating models is literally changing a string. When we want to A/B test DeepSeek V4 Pro against GLM-4 Plus for the reasoning tier, that's a config flag. When a new model drops, I can have it in production within an hour.

The cost-aware version looks like this:

def route_question(question: str, context_chunks: list[str]) -> str:
    is_simple = len(question) < 80 and not requires_synthesis(question)

    model = "deepseek-ai/DeepSeek-V4-Flash" if is_simple else "deepseek-ai/DeepSeek-V4-Pro"

    return call_with_model(model, question, context_chunks)

In practice, about 60% of our traffic ends up hitting the Flash tier. That alone was worth the engineering time. At our volume, routing 60% of requests to a $0.27 input / $1.10 output model instead of a $2.50 / $10.00 model is a six-figure annual difference. The math is not subtle.

The Real Cost Optimization Playbook

Here's what actually moved the needle for us, in order of impact:

1. Aggressive caching. Our cache hit rate sits around 40% in steady state. Documents get queried repeatedly — people ask the same questions about their contracts, their HR docs, their onboarding guides. A simple Redis cache keyed on a hash of (document_id, question_embedding) eliminated two-fifths of our LLM calls overnight. The ROI on caching is always positive at our scale. Always.

2. Streaming everything. I had a junior engineer push back on this once, arguing it added complexity. I had to remind them that perceived latency matters more than actual latency. Users hate waiting for a spinner. They tolerate text appearing word by word. Streaming dropped our perceived latency by about 60% even though actual latency barely changed. UX is an ROI lever, not just a polish thing.

3. Tiered model routing. Already covered this above, but it's worth repeating. Use the cheapest model that can handle the task. We route 60% of traffic to DeepSeek V4 Flash at $0.27/M input, and only the genuinely hard stuff hits the Pro tier. If you're sending every single request to your most expensive model, you're not thinking about the architecture — you're just writing checks.

4. Output token discipline. This one I see people miss constantly. Every output token costs money. We had prompts that said "explain in detail" and our outputs ballooned to 1,500 tokens for what should have been 200-token answers. Adding "answer in 2-3 sentences unless asked otherwise" to the system prompt cut our average output tokens nearly in half. Same quality, half the cost. The system prompt is product surface area.

5. Monitoring and feedback loops. We track user satisfaction via thumbs-up/thumbs-down buttons on every response. We track hallucinations via a separate eval pipeline. We track cost per question in a dashboard. Without measurement, you're guessing, and guessing at scale burns money fast.

6. Graceful degradation on rate limits. Every provider will rate-limit you eventually. We have fallback logic that retries with exponential backoff, then degrades to a cached response, then to a "sorry, try again" message. The user never sees a 500. That's a production-ready system, not a demo.

Throughput and Latency: What Production Actually Looks Like

Our average latency hovers around 1.2 seconds end-to-end for a typical document question. That includes retrieval, reranking, and the LLM call. Throughput clocks in at roughly 320 tokens per second for the Flash tier, which is more than enough for our use case. The Pro tier is slower but we route less to it.

Those numbers aren't magic — they're what you get when you stop overengineering and start measuring. The fastest architecture is the one with the fewest unnecessary steps. Every abstraction that doesn't earn its complexity cost should be deleted.

I'll be honest: the first version of our system had about 40% more latency. We were doing redundant embedding calls, re-ranking twice, and pulling too many chunks. I spent a week stripping it down and latency dropped from 2.1 seconds to 1.2. The lesson is the same lesson I keep learning: simplicity wins at scale.

Vendor Lock-In Avoidance: The Long Game

I want to come back to vendor lock-in because it's the thing I think about most as a CTO. The models will keep changing. New providers will emerge, prices will shift, capabilities will move around. The startup that survives the next three years is the startup that can adapt to those changes without rewriting their product.

Going through an OpenAI-compatible endpoint layer means we're not betting the company on any single model provider. If DeepSeek raises prices, we move to Qwen. If OpenAI releases something dramatically better, we route to it. If a brand new model drops tomorrow and the benchmarks look insane, we have it in production by end of week.

That's not paranoia. That's just good architecture. The cost of building the abstraction layer is maybe a week of engineering time. The cost of NOT having it is giving up optionality, and optionality is the most