I've been building AI agents in production for the past two years. Not demos. Not weekend projects. Systems that real users talk to every day and get angry at when they break.
And the pattern I keep seeing? Engineers building elaborate machinery around the model. Custom orchestration layers. Hand-rolled retry logic. Massive tool routing systems. All to solve problems the LLM was already solving if you just let it.
Here's what I'd rip out if I could go back.
1. Custom Tool Selection Logic
You built a classifier that decides which tool the agent should use. Maybe a regex-based router. Maybe a whole separate model call just to pick the right function.
Stop.
Modern LLMs are shockingly good at tool selection when you give them well-named, well-described tools. The problem was never the model. It was your tool descriptions.
// Bad: vague tool name, model guesses wrong
{ name: "search", description: "Searches for things" }
// Good: specific name, clear scope, model nails it
{ name: "search_customer_orders", description: "Search customer order history by order ID, customer name, or date range. Returns order status, items, and tracking info." }
The fix isn't a smarter router. It's better tool design. Name your tools like you're writing an API for a junior dev who's never seen your codebase. Be embarrassingly specific.
Tool selection metrics can look great while the final answer is still garbage. I've seen this firsthand. The agent picks the right tool 95% of the time but still gives wrong answers because the tool descriptions don't explain what the returned data actually means.
2. Prompt Chains for Multi-Step Reasoning
I used to build 4-5 step prompt chains for anything complex. Break the problem down. Feed output A into prompt B. Parse the result. Feed it into prompt C.
Turns out a single well-structured system prompt with clear instructions handles most of this natively. The model already knows how to decompose problems. You just need to tell it what your constraints are and what good output looks like.
// Instead of chaining 3 prompts:
// 1. "Classify the user intent"
// 2. "Based on intent X, gather context"
// 3. "Now generate the answer"
// Just do this:
const systemPrompt = `You are a support agent for a logistics platform.
When a user asks a question:
1. Identify whether they need order status, account help, or technical support
2. Use the appropriate tool to get the data
3. Answer in plain English with the specific details they asked for
If you're unsure about intent, ask one clarifying question. Never guess.`
The chain approach also creates a hidden problem. Each step is a failure point. And debugging a 4-step chain when something breaks on step 3 is miserable. A single prompt with clear instructions is easier to observe, easier to eval, and fails more gracefully.
3. Retrieval Complexity Before Retrieval Quality
This one hurts because I've done it myself.
You spend two weeks building a hybrid retrieval pipeline. BM25 plus vector search plus re-ranking. Beautiful architecture. Looks great in a diagram.
Then you realize the actual problem is that your knowledge base documents are written in a way the model can't parse. Or your chunking strategy splits the answer across two chunks and neither one makes sense alone.
The retrieval pipeline doesn't matter if the underlying data is messy.
Before you optimize the search algorithm, ask yourself:
- If I showed this chunk to a human with no context, would they understand the answer?
- Are my documents written for the model or for the original author's brain?
- Am I chunking at logical boundaries or just every 500 tokens?
I've seen teams where retrieval "works" but answers are still wrong because the reference data itself contains outdated or incorrect information. That's not a retrieval problem. That's a data quality problem wearing a retrieval costume.
4. Custom Guardrails That Block Legitimate Use
You built a content filter. It catches bad inputs. Great.
Then users start complaining that normal questions get blocked. Someone asks about "terminating a contract" and the guardrail flags "terminating." Someone asks about shipping "explosive growth" and that trips another filter.
Rule-based guardrails at scale become a whack-a-mole game you can't win.
The LLM itself is already pretty good at understanding intent and context. Instead of building regex walls around the model, build guardrails INTO the model's instructions. Tell it what topics are off-limits. Tell it what information it should never reveal. Tell it to redirect gracefully instead of stonewalling.
// Instead of: regex filter that blocks "kill", "terminate", "destroy"
// Try this in your system prompt:
`If a user asks about topics outside your domain (logistics and order management),
politely redirect them. Never share internal system details, API keys,
or other customer data. You can decline requests, but always explain why
and suggest what you CAN help with.`
Guardrails and permissions are product design, not just safety theater. Treat them that way.
5. Agent Memory as a Separate System
You have your agent's database over here. Its memory system over there. A vector store somewhere else. And glue code holding all of it together with prayers and setTimeout.
The real question is simpler than the architecture you built: what does the agent actually need to remember between sessions?
Most agents don't need a sophisticated memory system. They need a well-structured context window. The conversation history plus a few key facts about the user. That's it. The model handles the rest.
When you DO need persistent memory, keep it close to your data. Don't build a separate memory service that has to sync with your database. Store memory where your data lives. Query it with the same tools.
The moment your agent's memory can't see its own database, you've created an integration problem disguised as a feature.
6. Sub-Agent Orchestration for Everything
Multi-agent architectures are seductive. One agent plans. One retrieves. One generates. One validates. They talk to each other through a message bus. It looks amazing on a whiteboard.
In production it's a nightmare to debug. When the answer is wrong, which agent broke? The planner? The retriever? The generator? You end up building observability tooling just to trace what happened across four agents when one would have been fine.
Start with one agent. Push it until it genuinely can't handle the complexity. Only THEN split into specialized sub-agents with clear, narrow responsibilities.
The rule I use: a sub-agent should exist only when the parent agent's context window literally can't hold the information it needs. Not because "separation of concerns" sounds good in a design doc.
Specialized agents make sense for high-context tasks where the prompt would blow up the token budget. General agents handle 80% of use cases with less operational overhead. Know which one you're building and why.
7. Evaluations That Test Happy Paths
This is the one that bites hardest.
You write 50 eval cases. The agent passes 48 of them. Ship it.
Then users find the 200 edge cases you didn't think of. The model hallucinates a tracking number. It confidently answers a question it should have said "I don't know" to. It uses data from one customer to answer another customer's question.
Good evals don't test whether the agent CAN answer correctly. They test whether it WILL answer correctly under pressure.
Build evals that target failure modes:
- What happens when the tool returns empty results?
- What happens when two tools return conflicting information?
- What happens when the user asks something slightly outside the agent's domain?
- What happens when the context is ambiguous?
The eval suite is the real moat. Not the model. Not the prompts. Not the architecture. The team that can systematically find and fix failure modes ships better agents than the team with the fancier framework.
The Uncomfortable Truth
Most of the complexity in your agent isn't making it smarter. It's making it harder to debug, harder to eval, and harder to change.
The best agent architectures I've built are embarrassingly simple. One model. Clear system prompt. Well-named tools. Good data. Ruthless evals.
Everything else is either premature optimization or an expensive lesson waiting to happen.
What's the most over-engineered thing you've built into an agent that turned out to be unnecessary?
Top comments (1)
Most teams overengineer context management in their AI agents, overlooking that LLMs are already proficient in this area. In our experience with enterprise teams, simplifying architecture by leveraging the LLM's innate capabilities can significantly reduce complexity and improve performance. Focus on integrating AI into real workflows rather than reinventing processes the model can already handle efficiently. - Ali Muwwakkil (ali-muwwakkil on LinkedIn)