Don't Bolt LLM Calls Into Your App — Put Them Behind a Tiny API Instead

The problem: the five-line feature that metastasizes

Every team adds their first LLM feature the same way. Someone writes five lines inside the existing codebase:

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
summary = client.chat.completions.create(...)

It works. Then product asks for a second feature — chat. Then classification for the support inbox. Six months later, OpenAI calls are scattered across four modules, the API key lives in three config files, nobody knows which endpoint is burning tokens, and swapping models means a search-and-replace across the whole repo. The LLM stopped being a feature and became infrastructure — except nobody built it like infrastructure.

There's a boring pattern that prevents this: put LLM calls behind a tiny HTTP service of your own. Your app talks to one small FastAPI sidecar; the sidecar talks to the provider. Keys, prompts, and provider quirks live in exactly one place.

I built a free demo of this pattern — one endpoint, about 150 lines — so you can see the shape before committing to it.

The demo: one feature, done cleanly

The demo service has exactly two endpoints:

• GET /health — no auth, just liveness
• POST /features/summarize — takes {text, max_sentences}, returns a summary

The whole thing is a single app.py. Here's the interesting part — the auth dependency:

def require_api_key(x_api_key: str | None = Header(default=None)) -> None:
    expected = os.getenv("SERVICE_API_KEY")
    if not expected:
        raise HTTPException(status_code=503,
            detail="SERVICE_API_KEY is not set on the server.")
    if x_api_key != expected:
        raise HTTPException(status_code=401,
            detail="Invalid or missing X-API-Key.")

Your backend sends a shared X-API-Key header; browsers and mobile apps never see it, because they talk to your backend, not to this service. One key, one check, every feature endpoint protected by adding dependencies=[Depends(require_api_key)].

And here's the detail most demos skip — what happens when the OpenAI key is missing:

openai_key = os.getenv("OPENAI_API_KEY")
if not openai_key:
    return JSONResponse(status_code=503, content={
        "detail": "OPENAI_API_KEY is not set. Copy .env.example "
                  "to .env and add your OpenAI API key."
    })

Not a stack trace. Not a service that crashes on boot at 2 a.m. A clean 503 that tells the next developer exactly what to fix. Small thing, but it's the difference between a template and a toy.

The summarize endpoint itself is deliberately unremarkable:

@app.post("/features/summarize", dependencies=[Depends(require_api_key)])
def summarize(req: SummarizeRequest):
    # ... missing-key check above ...
    completion = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": f"Summarize in at most "
             f"{req.max_sentences} sentence(s). Keep key facts and numbers."},
            {"role": "user", "content": req.text},
        ],
    )
    return {"summary": completion.choices[0].message.content, ...}

Run it:

pip install -r requirements.txt
cp .env.example .env   # set SERVICE_API_KEY + OPENAI_API_KEY
uvicorn app:app --port 8000

curl -X POST localhost:8000/features/summarize \
  -H 'X-API-Key: YOUR_KEY' -H 'Content-Type: application/json' \
  -d '{"text": "Long ticket text...", "max_sentences": 2}'

That's the pattern. Your Rails, Django, Laravel, or Node backend doesn't care that Python and OpenAI are on the other side of the HTTP call — and that's the point.

Where the demo stops (on purpose)

The demo is honest about being a demo. The moment this serves real users, three production concerns show up:

1. Who spent what? A summarize call costs fractions of a cent — a runaway loop or one enthusiastic client costs real money. Production needs per-client token budgets, checked before the provider call: estimate the request, and if the client is over their daily cap, return 402 without spending anything. Settling with the provider's real usage numbers afterwards keeps the accounting honest.

2. What happens when the provider hiccups? OpenAI returns 429s and 5xx like everyone else. Without retries with backoff and a sane timeout, your users see raw failures for what was a two-second blip.

3. Can you prove what happened? When finance asks "why did the bill triple in March?", you need one JSON log line per call — endpoint, client, input/output tokens, latency — not a forensic expedition through application logs.

Add streaming chat (SSE) and a classify endpoint, and you've described the full version of this service — same request/response shape, with budgets, rate limits, retries, logging, and a provider abstraction so "switch to Claude" is a config change, not a refactor.

Try it

The demo is free and MIT-licensed — clone it, run the three smoke tests (/health → 200, no key → 401, no OpenAI key → clean 503), and wire summarize into your app this afternoon:

GitHub: https://github.com/mahuan2928/llm-feature-demo

If you want the production version — /features/chat with SSE streaming, /features/classify, per-client daily token budgets with HTTP 402 enforcement, rate limiting, retries with backoff, and JSON call logging — it's here:

Full version ($49): https://mahuan220.gumroad.com/l/gwwrrn

Start with the demo. You'll know within one integration whether the pattern fits — and if it does, the upgrade path is a drop-in, not a rewrite.

---

Built by Marty Ma — Senior AI & Data Engineer (Tokyo), 11+ years building production systems for enterprise clients.