DEV Community

Sir Max
Sir Max

Posted on

3 Things I Wish I Knew Before Building with LLM APIs

#api #ai #webdev #tutorial

3 Things I Wish I Knew Before Building with LLM APIs

I've spent the last year shipping features backed by LLM APIs — GPT, Claude, Gemini, and a handful of open-source models via Ollama. Along the way I burned through a lot of credits, broke production a couple of times, and learned things the hard way.

Here are three lessons I wish someone had told me before I started.

1. Token counting is not optional — it's the whole job

Early on, I treated token limits as a "nice to have" — something you throw a rough character count at and hope for the best. That worked fine until a user pasted a 40KB log file into a chat window and my backend silently truncated their context, giving answers that looked confident but were completely wrong.

What I do now: Every request goes through a token counter before it touches the LLM.

import tiktoken

def count_tokens(text: str, model: str = "gpt-4") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))
Enter fullscreen mode Exit fullscreen mode

For multi-model setups, I built a small abstraction that maps model names to their tokenizers. claude-* uses Anthropic's claude-tokenizer, Gemini uses google-generativeai, and for Ollama models I fall back to a conservative estimate based on character count.

The real insight: You need two limits — a hard cutoff for the full context window, and a softer "reserved output" budget so the model has room to actually respond. I set aside 25% of the context window for output tokens. If your input + reserved output > context limit, trim the input — not the other way around.

This one change eliminated about 90% of the "why is the AI giving incomplete answers?" bug reports.

2. Streaming is table stakes, but structured output is the hard part

Everyone knows you should stream LLM responses — users don't want to stare at a spinner for 15 seconds. That's the easy part.

The hard part is what to do with the stream. If you're calling an LLM from a backend that needs to return JSON to a frontend, you can't just stream raw tokens. You need structured output on the backend side, and you need it fast.

The pattern that works for me:

import asyncio
import json

async def stream_and_parse(prompt: str, schema: dict):
    """Stream LLM response, parse JSON at the end."""
    buffer = []

    async for chunk in llm_stream(prompt):
        buffer.append(chunk)
        # Send raw text to frontend via WebSocket
        await ws.send(chunk)

    full_text = "".join(buffer)
    # Parse the structured result from completed text
    return json.loads(full_text)
Enter fullscreen mode Exit fullscreen mode

But here's the gotcha: not all LLMs are equally good at JSON output. Claude is great at it. Gemini 2.0 Flash is fast but occasionally drops closing braces. GPT-4o handles it well but sometimes wraps JSON in markdown fences that you need to strip.

My rule of thumb: Always validate the parsed JSON against your schema. If parsing fails, retry with a system prompt that says "Return ONLY valid JSON, no markdown fences, no explanations." That second attempt fixes 95% of failures.

For production, I also log every JSON parse failure with the raw response — nothing is more frustrating than a silent parse error that turns into a NoneType has no attribute three callers later.

3. Fallback is not a feature — it's architecture

My first LLM integration looked like this:

user → my API → OpenAI → response
Enter fullscreen mode Exit fullscreen mode

When OpenAI had an outage (and it did, twice in six months), my entire feature was dead. Users got timeout errors with no explanation.

What I run now:

user → my API → primary provider (Claude)
              → fallback #1 (GPT-4o) — different provider
              → fallback #2 (Gemini Flash) — cheap, always available
Enter fullscreen mode Exit fullscreen mode

But it's not enough to just chain fallbacks. You need circuit breakers:

import time
from functools import wraps

class CircuitBreaker:
    def __init__(self, failure_threshold=3, reset_timeout=60):
        self.failures = 0
        self.threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.last_failure_time = 0

    def record_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()

    def is_open(self):
        if self.failures >= self.threshold:
            if time.time() - self.last_failure_time < self.reset_timeout:
                return True  # Circuit is open — skip this provider
            self.failures = 0  # Reset timeout passed, try again
        return False
Enter fullscreen mode Exit fullscreen mode

If Claude returns three consecutive 5xx errors, the circuit breaker opens for 60 seconds. Requests skip directly to GPT-4o instead of waiting for timeouts. When the breaker resets, it tests the water with a single request before fully reopening.

The lesson: LLM APIs are external dependencies. Treat them like you'd treat a database connection or a third-party payment service. They will fail, and your users shouldn't notice.

The takeaway

Building with LLMs isn't hard because the models are complex — it's hard because the infrastructure around them needs to be resilient. Token management, structured parsing, and provider fallback are not optional extras after launch. They're the stuff you need on day one.

These three patterns have been running in production for months now. They're not particularly clever or novel. They're just battle-tested.

Got your own LLM API war stories? I'd love to hear them in the comments.

Top comments (0)