Okay, real talk. You know that feeling when your AI feature works perfectly in testing, you ship it on a Friday feeling like a genius, and then Monday morning something's on fire and you have no idea why?
Yeah. That feeling has a name, and it's usually one of these six things.
None of these are exotic. There's no galaxy-brain fix here. They're just small, easy-to-miss decisions that look totally fine in a demo and quietly turn into 2am pages once real users show up. Let's fix them now so you don't have to learn them the hard way.
- Stop hiding "streaming or not" behind a flag
Raise your hand if you've written something like this:
python
def ask(prompt, stream=False):
if stream:
# return a generator
else:
# return a string
Feels efficient, right? One function, does everything. Except now every single place that calls ask() has to remember what stream=True does to the return type, and the day someone forgets, they're trying to call .upper() on a generator and wondering what they did wrong.
Just split it into two functions:
python
def ask(prompt: str) -> str:
"""Always returns a full string."""
...
def ask_streaming(prompt: str):
"""Always yields chunks. Always a generator."""
...
Boring? Sure. But now the function name tells you exactly what you're getting, and nobody has to hold extra state in their head to use it correctly. Future-you will say thanks.
- Streaming calls need a context manager, not just a loop
Here's a fun one. You write a loop to print tokens as they stream in. It works great — until the loop throws an error halfway through (a rendering bug, a network blip, whatever). Does the connection actually close?
With a naive setup: often, no. It just... sits there. Leaking. Quietly. Until you're staring at your server's connection count going up and up with no idea why.
python
with client.messages.stream(...) as stream:
for text in stream.text_stream:
yield text
Wrapping it in a context manager means cleanup happens no matter what — even if things blow up mid-stream. It's a one-line change that you'll never notice... until the one day it saves you.
- When trimming chat history, count in pairs, not messages
If you're building any kind of chatbot, you eventually need to cap how much history you send back to the model (context windows aren't infinite, and neither is your API bill). The natural instinct is "just keep the last N messages."
Here's the trap: if N lands you in the middle of a user/assistant back-and-forth, you get an orphaned assistant message with no user message before it — and a lot of APIs will just reject that outright.
The fix is almost embarrassingly simple once you see it:
python
recent = self.history[-(self.max_history_turns * 2):]
Multiply by 2, slice from the end — now you're always keeping whole conversational turns, never a half-finished one.
- Forgetting to normalize vectors = wrong search results, zero errors
This is my favorite one because it's sneaky. No crash, no error message, nothing that tells you something's wrong. Just... search results that feel a little off.
If you're using FAISS with IndexFlatIP to do cosine similarity search, that trick only works if your vectors are normalized first. Skip that step, and you're silently doing plain inner-product search instead — which ranks things differently, and there's no red flag telling you why your "most similar" results feel not-quite-right.
python
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
normalized = vectors / np.clip(norms, 1e-10, None)
One line. Easy to forget. Impossible to notice until you go digging.
- Not every error deserves a retry
Retrying failed API calls feels responsible — rate limits happen, timeouts happen, stuff breaks sometimes and trying again is the grown-up thing to do. But if you retry everything indiscriminately, you'll also retry your own bugs. Sent a malformed request? Cool, now you're going to fail the exact same way four times in a row, burning time and rate-limit budget for absolutely nothing.
Be picky about what you retry:
python
@retry(
retry=retry_if_exception_type((RateLimitError, APITimeoutError, APIError)),
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, min=1, max=20),
)
def call_model(...):
...
Transient stuff (rate limits, timeouts)? Worth a retry with backoff. Your own bug? Let it fail fast so you actually see it and fix it, instead of hiding it behind four identical failed attempts.
- Have a plan B model, not just a retry loop
Retries are great for "oops, blip" moments. They don't help much if a model provider is having a genuinely bad day for an extended stretch. That's where a fallback model earns its keep — try the fast/cheap one first, and if it keeps failing, fall back to a stronger (or just different) model instead of just erroring out on your users.
python
try:
return call_model(primary_model, messages)
except RetryableErrors:
return call_model(fallback_model, messages)
It's the same trick most production AI gateways use behind the scenes. Costs you a few extra lines, saves you a very bad afternoon.
Honestly, none of these are hard once you know them — that's kind of the whole point. They're just the small stuff that's easy to skip when you're moving fast, and painful to debug when they finally bite. (If you want the full runnable versions of these patterns plus a few more — RAG, tool calling, memory — they're all written up in the AI & LLM Integration Cookbook, but the six above will already save you a rough night regardless.)
Now go add that context manager before you forget. 😄
Get the AI and LLM cookbook at 20% off
product link-https://payhip.com/b/0dUzx
Top comments (1)
The streaming-flag one is the sneakiest on the list because it corrupts the TYPE SYSTEM by convention: one function returning string-or-generator depending on a boolean means every caller is one forgotten branch away from treating a stream as a string, and the failure only shows up under real latency - which is to say, in production, at 2am, exactly as advertised. Splitting it into two functions with honest names is the same lesson as "don't return null, return an empty thing": make the wrong usage unrepresentable instead of documented. The meta-pattern across all six: every one is a decision that the demo can't punish. Demos run one request, one user, one happy path - the bugs live in cardinality, concurrency, and time, none of which exist in a demo. "Ship it Friday, fire on Monday" is just the demo environment billing you later.