There's a moment every engineering team hits when they move from prototyping with a hosted LLM API to running models in production. The demo works beautifully. The CEO is thrilled. Then the invoice arrives.
I've spent the last two years integrating generative AI into production systems — first building AI/ML tooling at Apple, then shipping LLM-powered features at SyenApp that serve over 100,000 users daily. Along the way, I've watched teams make the same expensive mistake: they treat inference as a commodity and ignore the architecture beneath it.
This post is the guide I wish I'd had when I first moved an LLM workflow from a Jupyter notebook to a system that needed to handle real traffic, real latency requirements, and a real budget.
The Hidden Tax of "Just Call the API"
When you're building a proof-of-concept, calling OpenAI's API or Anthropic's API is the fastest path to a working product. You send a prompt, you get a response, you move on. But at scale, this approach introduces three compounding costs that teams consistently underestimate.
Latency that kills conversion. At SyenApp, we built an AI-driven product search feature using OpenAI's API. It worked well in testing. In production, median response times hit 2.8 seconds under load. For an e-commerce search bar, that's a death sentence. Our data showed that every 500ms of added latency above one second correlated with a 7% drop in search-to-purchase conversion. We were literally paying more per query and losing revenue from the slowness.
Cost curves that don't flatten. Hosted API pricing is linear. You pay per token, and as usage grows, so does your bill — proportionally. There's no economy of scale. When our AI-powered customer support chatbot went from handling 200 conversations per day to 2,000, our monthly API costs jumped from $3,400 to $31,000. The unit economics stopped working.
Vendor lock-in disguised as convenience. Every model provider has slightly different prompt formatting, token counting, and response structures. The more deeply you integrate one provider's API, the harder it becomes to switch when a better or cheaper model emerges. And in this market, better models emerge every quarter.
The Case for Owning Your Inference Layer
The alternative isn't to build everything from scratch. It's to own the layer between your application and the model — the inference stack. This means deploying open source models on infrastructure you control (or infrastructure that's purpose-built for inference), with the flexibility to swap models, optimize for your specific workload, and scale independently of any single provider.
Here's what this looked like in practice when I rebuilt our search pipeline.
Step 1: Choosing the Right Model for the Job
Not every task needs GPT-4. When I audited our AI features at SyenApp, I found that 60% of our API calls were for relatively simple tasks: extracting product attributes from descriptions, generating search query expansions, and classifying user intent into one of twelve categories.
For these tasks, a fine-tuned Mistral 7B model matched the quality of GPT-4 at a fraction of the cost and latency. The key insight was decomposing our monolithic "send everything to GPT-4" approach into a routing layer that directed each task to the most efficient model.
# Simplified model routing logic
def route_request(task_type: str, complexity_score: float):
if task_type == "classification" and complexity_score < 0.6:
return "mistral-7b-finetuned"
elif task_type == "search_expansion":
return "mistral-7b-finetuned"
elif task_type == "product_recommendation":
return "llama-3-70b"
else:
return "gpt-4-fallback"
This routing pattern alone cut our inference costs by 45% in the first month.
Step 2: Optimizing Serving Infrastructure
Running your own models doesn't mean spinning up a GPU instance and running python server.py. Production inference requires attention to batching, quantization, and autoscaling — details that are easy to ignore in development but determine whether your system falls over at 2 AM.
Dynamic batching was the single biggest performance lever we pulled. Instead of processing requests one at a time, we batched incoming requests and processed them together on the GPU. This is counterintuitive — batching adds a small amount of latency per individual request — but the throughput gains are dramatic. We went from processing 12 requests per second on an A10G to 47 requests per second with a batch window of 50ms.
Quantization was the second lever. Running our Llama 3 70B model in FP16 required two A100 GPUs. After applying GPTQ 4-bit quantization, we ran the same model on a single A100 with negligible quality degradation on our evaluation benchmarks (less than 1.2% difference on our internal quality score). That's a 50% reduction in GPU costs for a model that our users couldn't distinguish from the unquantized version.
Step 3: Building for Model Portability
The AI model landscape moves fast. The model that's state-of-the-art today might be outperformed by something released next month. Your inference architecture needs to accommodate this without rewriting your application.
At Apple, I worked on internal ML tooling that tracked the lifecycle of dozens of models across teams. The most resilient systems I saw all shared one trait: they abstracted the model behind a consistent API contract. The application code never knew or cared which model was serving responses. It sent a request, received a response in a standardized format, and moved on.
# Abstract inference interface
class InferenceClient:
def __init__(self, endpoint: str, model_id: str):
self.endpoint = endpoint
self.model_id = model_id
async def generate(self, prompt: str, params: dict) -> InferenceResponse:
"""
Consistent interface regardless of underlying model.
Swap model_id and endpoint without touching application code.
"""
payload = {
"prompt": prompt,
"max_tokens": params.get("max_tokens", 512),
"temperature": params.get("temperature", 0.7),
}
response = await self._call_endpoint(payload)
return InferenceResponse(
text=response["output"],
tokens_used=response["usage"]["total_tokens"],
latency_ms=response["latency_ms"],
model=self.model_id,
)
When we migrated from Llama 2 to Llama 3 at SyenApp, this abstraction meant the swap was a configuration change, not a code change. The entire migration took 45 minutes, including running our evaluation suite.
What I Measure (And What You Should Too)
Inference optimization without measurement is just guessing. Here are the four metrics I track on every deployment:
Time to First Token (TTFT). For streaming applications — chatbots, search suggestions, real-time assistants — TTFT matters more than total generation time. Users perceive a system as responsive when they see output begin, even if the complete response takes a few seconds. Our target is under 200ms for TTFT on all interactive features.
Throughput at P95 latency. Average latency is misleading. I care about what happens at the 95th percentile under realistic load. If your P95 latency is three times your median, your system is going to feel broken for one in twenty users, and those users will be disproportionately the ones using your product the most.
Cost per successful inference. Not cost per request — cost per successful inference. Failed requests, retries, and timeouts all consume GPU time without delivering value. When I started tracking this metric separately, I discovered that 8% of our inference costs were going to requests that ultimately timed out or returned errors. Fixing the retry logic saved us $2,800 per month.
Model quality on production data. Lab benchmarks don't tell you how a model performs on your specific data distribution. I maintain a rolling evaluation dataset sampled from production traffic (with PII stripped) and run quality checks weekly. This is how we caught a 4% quality regression after a quantization change that looked fine on public benchmarks but underperformed on our domain-specific product descriptions.
The Infrastructure Decision That Changes Everything
The decision to own your inference stack versus renting it from an API provider isn't binary. The healthiest architectures I've built use a hybrid approach: self-hosted models for high-volume, latency-sensitive tasks where you've validated that an open source model meets your quality bar, and hosted APIs as a fallback for complex, low-volume tasks where the convenience premium is worth paying.
The infrastructure layer that connects these — the serving platform — is the most consequential technical decision you'll make in your AI stack. It determines how quickly you can adopt new models, how efficiently you use GPU resources, and how gracefully your system handles traffic spikes.
Get this layer right, and you can move as fast as the models improve. Get it wrong, and you'll spend your engineering time babysitting infrastructure instead of building product.
What's Next
In a follow-up post, I'll walk through a complete deployment of a fine-tuned Llama 3 model for production inference, including the evaluation framework I use to validate model quality before and after quantization, the autoscaling configuration that handles our traffic patterns, and the monitoring dashboard that keeps everything visible.
If you're currently running AI features in production and wrestling with cost, latency, or model migration challenges, I'd love to hear what's working for you and what isn't. The best practices in this space are being written in real time, and the more we share, the faster we all move.
Charles Walls is a Full Stack Engineer specializing in AI-integrated production systems. He currently works at SyenApp, where he builds LLM-powered features serving over 100,000 users. Previously, he developed AI/ML engineering tools at Apple. Find him on GitHub and LinkedIn.
Top comments (0)