AI Twitter (and LinkedIn, and every tech newsletter) makes LLM engineering look like a solved problem: wire up an API call, add a system prompt, ship it. Anyone who's actually taken an LLM feature from prototype to production knows that's about 10% of the job. The other 90% is the unglamorous stuff - the failure modes nobody demos, the costs nobody mentions in the launch tweet, the "it worked great until real users touched it" moments.
Here are ten lessons that took me longer to learn than they should have, and that I wish someone had told me before I started.
- Your prompt is a piece of software, not a paragraph
Early on, prompts feel like writing - you tweak the wording until the output looks right, and move on. That mindset falls apart fast once a prompt has to survive contact with real, messy user input.
Treat prompts like code: version them, test them against a fixed set of inputs before every change, and track regressions. A prompt that works beautifully on your five test cases and silently breaks on the sixth isn't done - it's untested.
- Evaluation is the actual hard part
Building the feature is fast. Knowing whether it's good is slow. Most teams underinvest here because eval work doesn't produce a demo-able artifact - it produces confidence, and confidence doesn't screenshot well.
The fix isn't complicated, just unglamorous: build a small, real eval set from actual (or realistic) inputs early, and re-run it every time you touch a prompt, model, or pipeline step. Even 30-50 well-chosen examples beat vibes-based testing by a wide margin.
- Retrieval quality matters more than model choice
If you're building anything RAG-adjacent, it's tempting to assume a better model will fix bad answers. In practice, most "the AI is wrong" bugs I've debugged were retrieval bugs wearing a model-quality costume - the right chunk of context simply never made it into the prompt.
Before swapping models, check what your retrieval step is actually returning. Log it. Read it. It's usually the culprit.
- Structured output is worth the upfront pain
Free-form text output feels faster to build, until you're three regex patterns deep trying to parse a response that almost-but-not-quite follows the format you asked for. Investing in structured output (function calling, JSON schemas, whatever your provider supports) early saves you from a category of bugs that otherwise creeps in right before launch.
- Latency is a feature, not an afterthought
Users tolerate a slow first response far less than a slow search result, because they expect a "chat" to feel conversational. Streaming responses, showing intermediate steps for agentic workflows, and setting realistic timeouts aren't polish - they're core to whether the feature feels usable at all.
- Cost scales in ways that surprise you
A demo costing a few cents per call feels irrelevant, right up until it's running at production volume with retries, multi-step agent loops, and users who send far more requests than your test users ever did. Model cost, retry cost, and "the agent looped four times before finishing" cost are three different line items - track them separately, or the invoice will surprise you.
- Guardrails aren't optional, even for "internal tools"
It's tempting to skip input/output filtering on internal or low-stakes tools. Then someone pastes something unexpected into the input field, or an agent with tool access does something you didn't anticipate, and you're debugging an incident instead of shipping a feature. Basic guardrails scale down easily; retrofitting them after an incident doesn't.
- Agentic workflows fail differently than single-shot calls
A single bad response is annoying. A multi-step agent that makes a wrong decision early and confidently builds on it for five more steps is a much harder problem - the failure compounds instead of staying contained.
If you're building agents (and it feels like everyone is right now), invest early in step-level logging and the ability to see where a chain went wrong, not just that it did.
- MCP and tool ecosystems are changing the integration story fast
The Model Context Protocol and similar standards are quietly solving a problem that used to eat weeks of custom integration work per tool. If you haven't looked at how MCP-style tool servers work recently, it's worth revisiting — the ecosystem is moving fast enough that assumptions from six months ago are already outdated.
- The best LLM feature is often the smallest one
The instinct is to build the most impressive agentic pipeline possible. The features that actually stick with users tend to be narrower and more reliable: one job, done well, with predictable behavior - not a general-purpose agent that's occasionally magical and occasionally wrong in ways users can't predict.
None of this is meant to be discouraging - it's the opposite. LLM engineering is still young enough that most of these lessons come from hands-on scar tissue rather than established best practice. If you're early in this space, the gap between "the demo worked" and "it works reliably for real users" is exactly where the interesting engineering happens.
What's the lesson that took you the longest to learn? I'd genuinely like to know - drop it in the comments.
I'm a DevOps engineer with 10 years across AWS, Azure, Kubernetes, Terraform and Ansible. Previously: I built a self-healing pipeline with Prometheus and 50 lines of Python. I also maintain CronPort, a free cron expression converter for crontab, Kubernetes, GitHub Actions, AWS EventBridge and Terraform.
Top comments (0)