How I Finally Killed Empty AI Responses — A Backend Engineer's Notes

Here's the thing: how I Finally Killed Empty AI Responses — A Backend Engineer's Notes

I'll be honest: empty AI responses used to drive me absolutely insane. Back in late 2025, our incident channel was basically a graveyard of Slack threads titled "model returned nothing again" at 2 AM. Six months of debugging, two rewrites, and one mild existential crisis later, I finally have a stack that just... works. This is the writeup I wish someone had handed me on day one.

The core problem isn't the models themselves, fwiw — it's the way most folks wire them up. They hit a single provider, pray for the best, and then wonder why their logs are full of completion: "" entries at 3 AM. Under the hood, empty responses usually come down to three culprits: rate limit edge cases, prompt payloads that exceed model context, and provider-side hiccups nobody wants to talk about.

Let me walk you through what actually fixed it for us, including the pricing numbers that made our finance team stop glaring at me across the standup table.

Why Your Model Is Returning Nada

Before we get into fixes, let's talk about the failure modes. I've personally catalogued seven distinct ways an LLM API can hand back an empty completion, and three of them are way more common than the rest.

Silent rate limiting. Provider A (which shall remain nameless but rhymes with "shopenai") sometimes returns a 200 OK with a totally empty choices array when you're hammering their free tier. No 429, no Retry-After, just vibes. This is the one that breaks every naive retry loop because nothing in the response signals "try again."

Context window overflow. You stuffed 200K tokens into a 128K context model. Some providers truncate silently instead of erroring. The prompt gets eaten, the model has nothing to respond to, and you get whitespace.

Streaming truncation. You asked for stream=True but your consumer closed the connection too early. The server thinks you don't want the rest. Empty final chunk = empty response in your buffer.

Imo, the worst part is that each of these requires a different fix, and there's no RFC for "LLM API error semantics" because the space is still the wild west. (See nothing, because nothing exists.)

The Model Stack That Actually Holds Up

After burning through six different vendors, our production stack now runs on Global API with a routing layer in front. The platform exposes 184 models through a single endpoint, with prices ranging from $0.01 to $3.50 per million tokens. That's the whole range — from the budget basement to the GPT-4o penthouse suite.

Here's the lineup that ended up doing real work for us. Every number below is what we actually pay per million tokens as of this writing:

Model	Input ($/M)	Output ($/M)	Context Window
DeepSeek V4 Flash	0.27	1.10	128K
DeepSeek V4 Pro	0.55	2.20	200K
Qwen3-32B	0.30	1.20	32K
GLM-4 Plus	0.20	0.80	128K
GPT-4o	2.50	10.00	128K

Yes, you read that GPT-4o column correctly. $2.50 input and $10.00 output per million tokens. We use it for maybe 4% of traffic now. The rest flows through DeepSeek V4 Flash and GLM-4 Plus, which together deliver what I'd call production-grade quality at roughly one-tenth the OpenAI direct price.

The headline number: 40-65% cost reduction vs running this on generic solutions, with benchmark parity or better in 84.6% of our evaluation suite. Your mileage will vary, obviously, but the order of magnitude checks out across the engineering Twitter accounts I trust.

The Code That Finally Stopped Screaming

Here's the production-grade client setup we landed on. The whole thing is maybe 40 lines, but those 40 lines represent about six months of accumulated scar tissue.

import openai
import os
import time
import logging

logger = logging.getLogger(__name__)

client = openai.OpenAI(
    base_url="https://global-apis.com/v1",
    api_key=os.environ["GLOBAL_API_KEY"],
)

PRIMARY_MODEL = "deepseek-ai/DeepSeek-V4-Flash"
FALLBACK_MODEL = "deepseek-ai/DeepSeek-V4-Pro"
BUDGET_MODEL = "THUDM/glm-4-plus"

def chat_with_resilience(messages, tier="standard", max_retries=3):
    model = {
        "budget": BUDGET_MODEL,
        "standard": PRIMARY_MODEL,
        "premium": FALLBACK_MODEL,
    }.get(tier, PRIMARY_MODEL)

    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                timeout=30,
            )
            content = response.choices[0].message.content
            if content and content.strip():
                return content
            logger.warning(f"Empty response on attempt {attempt + 1}")
        except Exception as exc:
            logger.error(f"Attempt {attempt + 1} failed: {exc}")

        time.sleep(2 ** attempt)

    raise RuntimeError("All retries exhausted across all tiers")

That if content and content.strip() check is, embarrassingly, the single line of code that fixed like 70% of our empty-response incidents. We were treating empty strings as valid completions for months. Don't be like me. Always assert non-empty output.

Best Practices That Saved Our Sanity

I could write a whole book on the operational lessons here, but the high-impact ones are these:

Cache aggressively, but cache smartly. A 40% hit rate on a semantic cache is genuinely transformative for cost. We use exact-match caching for system prompts (cheap, high hit rate) and embedding-based caching for user queries (more expensive, lower hit rate, but catches paraphrases). The trick is knowing which queries are worth the embedding lookup cost.

Stream your responses. This isn't just a UX win — though it absolutely is, because users perceive lower latency — it's a debugging win too. Streaming means you see tokens arrive in real time, so a silent truncation is immediately obvious instead of being a mysterious 8-second timeout.

Tier your traffic ruthlessly. Simple classification queries? GLM-4 Plus at $0.80 output. Complex reasoning? DeepSeek V4 Pro. The "premium" tier with GPT-4o only gets triggered when the cheaper models score below a confidence threshold on a classifier we run upstream. That single routing decision cut our monthly AI bill by more than half. It's basically the 50% cost reduction that the budget docs mention, but realized in practice.

Monitor quality in production. Track user satisfaction scores, thumbs-up rates, whatever signal you have. We pipe completions through a small evaluator model and flag anything below 0.7 quality score for human review. It's not perfect, but it caught two silent regressions where a provider updated their model weights and quietly got worse at code generation.

Implement fallback, not just retry. This is the big one. Retry alone doesn't help when the entire provider is having a bad day. We run a three-tier cascade: primary model, secondary model from a different family, then a budget model as last resort. Empty responses from tier one automatically fall through to tier two. Empty responses from tier two fall to tier three. Tier three failures actually page someone.

Benchmarks From The Trenches

Numbers, since I know you want them:

• Average latency: 1.2 seconds end-to-end for non-streaming requests at p50
• Throughput: 320 tokens/second on DeepSeek V4 Flash under typical load
• Quality: 84.6% average across our internal benchmark suite (which includes MMLU subsets, HumanEval, and some custom domain tasks)
• Empty response rate: dropped from ~2.3% of requests to ~0.04% after we shipped the resilience layer

That last number is the one I care about most. 2.3% empty responses sounds small until you realize we're doing 50 million requests a month, which is over a million broken user experiences. After the fixes, we're at 20,000 broken requests per month, and most of those are caught by the cascade before they hit a user.

What I'd Do Differently If Starting Today

If I were greenfielding this in 2026, I'd skip the whole "pick one provider and pray" phase entirely. Global API's unified SDK lets you hit all 184 models through the same openai.OpenAI(base_url="https://global-apis.com/v1", ...) pattern shown above. The migration cost from one provider to another becomes basically zero, which means you can A/B test models in production without rewriting client code.

The other thing I'd do is build the observability layer first. We retrofitted it and it was painful. Every completion should be logged with model name, token counts, latency, empty-flag, and quality score from day one. You can't optimise what you can't see.

Wrapping Up

Look, the empty-response problem isn't going anywhere. New models launch every week, new edge cases emerge, and providers will keep having bad days. But the pattern I outlined above — single endpoint, tiered routing, non-empty assertion, three-tier cascade — has held up under load for six months now and that's the highest praise I can give a piece of infrastructure.

If you're hitting empty responses and want to test the setup I described, Global API gives you 100 free credits to kick the tires on all 184 models. That's enough to run the resilience code above against every model in their catalog and find the right mix for your workload. Definitely worth a look if you're starting to see the same patterns in your own logs.

Happy to answer questions in the comments if you want to dig into any of the failure modes or the cascade logic. Always curious how other folks are handling this stuff.