How to hook an LLM into your app without shooting yourself in the foot

I spent the last few months building apitestgen.dev — a tool that generates API tests from an OpenAPI spec — and roughly 80% of the actual engineering time went into stuff that has nothing to do with prompting. Everyone talks about prompt engineering. Almost nobody talks about the boring plumbing that keeps an LLM integration from quietly destroying your app in production. Here's what actually bit me.

1. Streaming responses need a real state machine, not a string you keep appending to.

The naive version (buffer += chunk and re-render) works fine in a demo and falls apart the moment the model emits a tool call mid-stream, or the connection drops at token 400 of 800. You need to know: are we in plain text, inside a tool call, or recovering from a truncated response? Treat the stream as an explicit state, not a growing string.

2. Never trust the model's JSON.

Even with function calling or structured output modes, you will get malformed or half-truncated JSON often enough that "just JSON.parse it" is a production incident waiting to happen. Wrap every parse in a schema validator (I use Zod) and have a defined fallback path. Not a silent catch that returns {} and lets your UI render garbage three components downstream.

const result = ResponseSchema.safeParse(parsed);
if (!result.success) {
  // log it, retry once with a stricter re-prompt, or degrade gracefully
  // do NOT ship `{}` further down the pipeline
}

3. Cost control is not optional past day one.

The first time I let a retry loop run unbounded against a real API key, I found out the hard way that "retry on failure" and "infinite loop that bills you per token" look identical from the outside until you check the dashboard. Hard-cap retries, hard-cap max tokens per request, and put a per-user or per-session budget ceiling in front of the model call. Not after it.

4. Prompt injection from user-supplied content is a real vector, not a theoretical one.

If any part of your prompt includes content the user controls, assume someone will try to get the model to ignore your system prompt. For apitestgen, the OpenAPI spec itself is untrusted input. A description field is a perfectly good place to hide an instruction. Treat anything you didn't write yourself as data, not as instructions, and say so explicitly in the prompt structure. Separate user content from system instructions. Don't string-concat them into one blob.

5. Latency is a UX problem before it's an infra problem.

A 6-second wait with no feedback reads as broken. Stream partial output the moment you have it, even if it's just token-by-token text before the structured part arrives. Users forgive slow. They don't forgive silent.

None of this is exotic. It's the same discipline you'd apply to any external API you don't control: validate inputs, cap costs, handle partial failure, don't trust the wire format. The only thing that's different with LLMs is that the failure modes are quieter. A bad SQL query throws. A bad LLM response just looks plausible and ships.