The Problem With "Just Add AI"
Let's be honest: most of us are shipping LLM features without really understanding what's happening under the hood. Your PM wants AI chat. Your competitor just launched AI search. You've got an API key, a decent prompt, and a deadline. Ship it, right?
Not quite.
Here's the uncomfortable truth: every time you call an LLM endpoint, you're invoking a massive mathematical function with probabilistic outputs. And if you don't understand the maths — even at a basic level — you're essentially eval()-ing untrusted code in production.
Why This Actually Matters (Beyond the Hype)
I've seen three production incidents in the past six months that share a common pattern:
- A customer-facing chatbot that started hallucinating product specifications
- A document summarisation feature that occasionally inverted sentiment
- An AI-powered search that returned confident but completely wrong answers
In each case, the developer who built it treated the LLM as a deterministic API. They wrote integration tests that passed. They did manual QA that looked good. Then reality happened.
The core issue? LLMs are not databases. They're not even search engines. They're statistical models that predict token sequences based on learned probability distributions. When you don't grasp that distinction, you can't properly evaluate the hidden risk you're introducing.
What You Actually Need to Know
You don't need a PhD in machine learning. But you do need to understand a few fundamental concepts before shipping LLM features:
1. Temperature and Sampling
response = llm.generate(
prompt="Summarise this document",
temperature=0.9 # Wait, what does this actually do?
)
If you can't explain what that temperature parameter does mathematically, you can't reason about when your feature will behave unpredictably. (Hint: it's scaling the logits before applying softmax. If that sentence means nothing to you, that's the problem.)
2. Context Windows Aren't Magic
You've got 128k tokens of context. Brilliant. But do you understand how attention mechanisms degrade over distance? Why information at position 500 might be effectively invisible by position 100,000?
I've debugged code where developers assumed "it's in the context window, so the model will use it." That's not how attention works.
3. Embeddings Are Geometry
If you're building RAG (retrieval-augmented generation), you're doing vector similarity search. That means:
- Cosine similarity is a geometric operation in high-dimensional space
- Your retrieval quality depends on how well your embedding model captures semantic meaning
- "Close enough" in vector space doesn't always mean "semantically relevant"
When your RAG system returns irrelevant documents, it's not a bug in your code — it's a mathematical property of your embedding space.
The Organisational Gap
Here's where it gets tricky. Companies specialising in AI automation and software development have teams with this mathematical literacy. Your startup probably doesn't.
But the pressure to ship AI features is identical for both. Your competitors are launching. Your investors are asking. Your users expect it.
This creates a dangerous dynamic: the perceived cost of learning the fundamentals feels higher than the perceived risk of shipping without them.
Until production breaks. Then suddenly everyone wants to understand why "temperature" matters.
Practical Steps You Can Take Today
Read the model card for any LLM you're using. Actually read it. Understand the training data, known limitations, and evaluation metrics.
Implement proper monitoring that goes beyond error rates. Track output diversity, response lengths, and confidence scores (if available). Set up alerts for statistical anomalies.
Write property-based tests, not just example-based ones. If your LLM feature should never return PII, test that property across hundreds of random inputs.
Pair with someone who knows the maths when architecting LLM features. Even a few hours with someone who understands transformer architecture can save you weeks of debugging.
Document your assumptions about model behaviour. When they turn out to be wrong (they will), you'll know exactly what needs revisiting.
The Bottom Line
You wouldn't ship a database feature without understanding transactions and ACID properties. You wouldn't deploy distributed systems without understanding CAP theorem.
So why are we shipping LLM features without understanding the underlying mathematics?
The answer isn't to stop building with AI. It's to stop treating it as magic. These are mathematical systems with measurable properties and predictable failure modes — once you understand the fundamentals.
Your users deserve better than probabilistic bugs wrapped in a nice UI. And honestly, so do you.
Top comments (1)
the part that bites is the endpoint you tested against isnt frozen, they ship a new checkpoint whenever they want. your passing tests start lying and you dont notice. brutal. do you pin a version anywhere, or just rerun evals every push?