DEV Community: zunairah

6 Sneaky LLM Bugs That Only Show Up After You Ship

zunairah — Sat, 12 Sep 2026 11:38:27 +0000

Okay, real talk. You know that feeling when your AI feature works perfectly in testing, you ship it on a Friday feeling like a genius, and then Monday morning something's on fire and you have no idea why?

Yeah. That feeling has a name, and it's usually one of these six things.

None of these are exotic. There's no galaxy-brain fix here. They're just small, easy-to-miss decisions that look totally fine in a demo and quietly turn into 2am pages once real users show up. Let's fix them now so you don't have to learn them the hard way.

Stop hiding "streaming or not" behind a flag

Raise your hand if you've written something like this:

python
def ask(prompt, stream=False):
if stream:
# return a generator
else:
# return a string

Feels efficient, right? One function, does everything. Except now every single place that calls ask() has to remember what stream=True does to the return type, and the day someone forgets, they're trying to call .upper() on a generator and wondering what they did wrong.

Just split it into two functions:

python
def ask(prompt: str) -> str:
"""Always returns a full string."""
...

def ask_streaming(prompt: str):
"""Always yields chunks. Always a generator."""
...

Boring? Sure. But now the function name tells you exactly what you're getting, and nobody has to hold extra state in their head to use it correctly. Future-you will say thanks.

Streaming calls need a context manager, not just a loop

Here's a fun one. You write a loop to print tokens as they stream in. It works great — until the loop throws an error halfway through (a rendering bug, a network blip, whatever). Does the connection actually close?

With a naive setup: often, no. It just... sits there. Leaking. Quietly. Until you're staring at your server's connection count going up and up with no idea why.

python
with client.messages.stream(...) as stream:
for text in stream.text_stream:
yield text

Wrapping it in a context manager means cleanup happens no matter what — even if things blow up mid-stream. It's a one-line change that you'll never notice... until the one day it saves you.

When trimming chat history, count in pairs, not messages

If you're building any kind of chatbot, you eventually need to cap how much history you send back to the model (context windows aren't infinite, and neither is your API bill). The natural instinct is "just keep the last N messages."

Here's the trap: if N lands you in the middle of a user/assistant back-and-forth, you get an orphaned assistant message with no user message before it — and a lot of APIs will just reject that outright.

The fix is almost embarrassingly simple once you see it:

python
recent = self.history[-(self.max_history_turns * 2):]

Multiply by 2, slice from the end — now you're always keeping whole conversational turns, never a half-finished one.

Forgetting to normalize vectors = wrong search results, zero errors

This is my favorite one because it's sneaky. No crash, no error message, nothing that tells you something's wrong. Just... search results that feel a little off.

If you're using FAISS with IndexFlatIP to do cosine similarity search, that trick only works if your vectors are normalized first. Skip that step, and you're silently doing plain inner-product search instead — which ranks things differently, and there's no red flag telling you why your "most similar" results feel not-quite-right.

python
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
normalized = vectors / np.clip(norms, 1e-10, None)

One line. Easy to forget. Impossible to notice until you go digging.

Not every error deserves a retry

Retrying failed API calls feels responsible — rate limits happen, timeouts happen, stuff breaks sometimes and trying again is the grown-up thing to do. But if you retry everything indiscriminately, you'll also retry your own bugs. Sent a malformed request? Cool, now you're going to fail the exact same way four times in a row, burning time and rate-limit budget for absolutely nothing.

Be picky about what you retry:

python
@retry(
retry=retry_if_exception_type((RateLimitError, APITimeoutError, APIError)),
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, min=1, max=20),
)
def call_model(...):
...

Transient stuff (rate limits, timeouts)? Worth a retry with backoff. Your own bug? Let it fail fast so you actually see it and fix it, instead of hiding it behind four identical failed attempts.

Have a plan B model, not just a retry loop

Retries are great for "oops, blip" moments. They don't help much if a model provider is having a genuinely bad day for an extended stretch. That's where a fallback model earns its keep — try the fast/cheap one first, and if it keeps failing, fall back to a stronger (or just different) model instead of just erroring out on your users.

python
try:
return call_model(primary_model, messages)
except RetryableErrors:
return call_model(fallback_model, messages)

It's the same trick most production AI gateways use behind the scenes. Costs you a few extra lines, saves you a very bad afternoon.

Honestly, none of these are hard once you know them — that's kind of the whole point. They're just the small stuff that's easy to skip when you're moving fast, and painful to debug when they finally bite. (If you want the full runnable versions of these patterns plus a few more — RAG, tool calling, memory — they're all written up in the AI & LLM Integration Cookbook, but the six above will already save you a rough night regardless.)

Now go add that context manager before you forget. 😄
Get the AI and LLM cookbook at 20% off
product link-https://payhip.com/b/0dUzx

6 LLM Integration Mistakes That Look Fine in a Demo and Break in Production

zunairah — Sat, 12 Sep 2026 11:21:34 +0000

Most LLM code you find online works great in a Jupyter notebook and falls apart the moment real traffic hits it. The bugs aren't exotic — they're small, structural decisions that don't show up until something goes wrong at the worst possible time. Here are six of them, and the fix for each.

Don't hide streaming behind a boolean flag

It's tempting to write one function with an if stream: branch. Don't. It means every caller has to know, out of band, which type they're going to get back — a plain string or a generator — and a wrong guess fails silently or crashes deep in your UI code.

Split it into two functions instead:

python
def ask(prompt: str) -> str:
"""Returns the full text response."""
...

def ask_streaming(prompt: str):
"""Yields text chunks as they arrive."""
...

Now the function signature is the documentation. ask() gives you something easy to log, cache, and unit test. ask_streaming() is unambiguously a generator, built for UIs. No flag, no ambiguity.

Wrap streaming calls in a context manager

If your loop over a streaming response throws an exception halfway through — network hiccup, a bug in your own rendering code, whatever — does the underlying HTTP connection actually close? With a naive implementation, often not. Connections leak quietly until you're wondering why your process is hoarding sockets.

The fix is boring but effective:

python
with client.messages.stream(...) as stream:
for text in stream.text_stream:
yield text

The context manager guarantees cleanup runs even on an error mid-stream. This is a one-line difference that only matters the day something actually goes wrong — which is exactly when you don't want to be debugging a connection leak too.

Trim chat history on turn pairs, not message count

A common way to cap conversation memory is to just keep "the last N messages." The bug: if N happens to land in the middle of a user/assistant pair, you end up with an orphaned assistant message with no matching user turn — and some APIs will outright reject that payload.

Slice on pairs instead:

python
def _trimmed_messages(self) -> list[dict]:
recent = self.history[-(self.max_history_turns * 2):]
return [{"role": "system", "content": self.system_prompt}] + recent

Multiplying by 2 and slicing keeps whole user→assistant exchanges intact, so you never truncate mid-conversation in a way the API can't parse.

Normalize your vectors before cosine similarity search

This one is sneaky because it fails silently. If you're using FAISS's IndexFlatIP (inner product) to approximate cosine similarity, that approximation is only valid if your vectors are unit-normalized first. Skip it, and you don't get an error — you get search results ranked in the wrong order, with no signal that anything's broken.

python
@staticmethod
def _normalize(vectors: np.ndarray) -> np.ndarray:
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
return vectors / np.clip(norms, 1e-10, None)

If your semantic search results look "close but weirdly off," this is one of the first things worth checking.

Only retry the errors that are actually transient

Blanket "retry on any exception" logic is a trap. If a request fails because you sent a malformed parameter, retrying it four times just guarantees the same failure four times — burning latency and rate-limit budget for nothing.

Scope retries to the errors that can plausibly resolve themselves:

python
@retry(
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, min=1, max=20),
retry=retry_if_exception_type((RateLimitError, APITimeoutError, APIError)),
reraise=True,
)
def _call_model(...):
...

Rate limits and timeouts are worth retrying with backoff. Your own bugs are not — those should fail fast so you actually see them.

Give yourself a fallback model, not just a retry loop

Retries handle transient failures. They don't help if a model is degraded or down for an extended window. Pairing retries with a fallback to a secondary model is the same pattern most production LLM gateways use under the hood:

python
try:
return _call_model(primary_model, messages)
except RETRYABLE_ERRORS:
return _call_model(fallback_model, messages)

Cheap/fast model first, stronger model as a safety net. Your app degrades gracefully instead of just erroring out.

Where these came from

I pulled these six lessons out of The AI & LLM Integration Cookbook — a set of 10 complete, copy-pasteable Python templates covering both the OpenAI and Anthropic APIs (quick-starts, tool calling, RAG over PDFs with LangChain and LlamaIndex, embeddings/vector search, and the production wrapper above). Each recipe includes a "why this pattern" note like the ones above, so you're not just getting code — you're getting the reasoning that usually only shows up after something's already broken in production once.

Worth a look if you're wiring up your first LLM feature or auditing an existing integration for exactly these kinds of gaps.

Get the AI and LLM cookbook at 20% off
product link- https://payhip.com/besttechbooks

Most "production-ready" LLM code I see in tutorials isn't production-ready. Here's what's usually missing.

zunairah — Sat, 12 Sep 2026 10:43:40 +0000

I spent the last few weeks pulling together 10 Python patterns I keep rebuilding on every LLM project, and three mistakes showed up over and over — including in my own early code.

Mixing sync and streaming behind a flag.
It's tempting to write one ask() function with an if stream: branch. Don't. Callers of the blocking version want a plain string they can log, cache, and unit test. Callers of the streaming version want a generator built for a UI. Collapsing both into one function with a boolean flag means every caller has to know which mode they're in — and testing gets messy fast. Two small functions beat one clever one.
Retrying every failure the same way.
Wrapping an API call in a retry loop feels like "production hardening" — until you realize you're retrying a malformed request four times with exponential backoff instead of failing fast. The fix is boring but important: only retry transient errors (rate limits, timeouts), and let everything else surface immediately. Pair that with a fallback model (cheap model first, stronger model if it keeps failing), and you've got the pattern most real LLM gateways actually use in production.
Re-embedding your entire document library on every restart.
This one's a silent cost killer. In RAG demos, it's common to load PDFs, chunk them, embed them, and query — all in one script, every single run. In production, you build the index once, persist it to disk, and load it on startup. Skipping this step is the single most common reason RAG demos rack up huge embedding bills and never make it past week one.

None of these are exotic. They're just the difference between "code that works when I run it" and "code that doesn't wake me up at 2am."

I ended up writing these patterns down properly — full runnable files, not fragments, covering both the OpenAI and Anthropic SDKs: streaming wrappers, tool/function calling, dynamic system prompts, RAG with LangChain and LlamaIndex, conversational memory with context trimming, vector search from scratch with FAISS, and the retry/fallback wrapper above.

Put them together into The AI & LLM Integration Cookbook — 10 copy-paste-adapt templates, each with a short "why this pattern" note so you're not just copying code, you understand the tradeoff behind it.

If you're tired of rebuilding the same LLM plumbing from scratch on every project, it might save you a weekend. Link in the comments — happy to answer questions about any of the patterns above in the meantime.