There's a very specific kind of AI project that looks incredibly impressive in the architecture diagram and does almost nothing a simple version couldn't do better.
It has a vector database. It has a multi-agent orchestration graph. It has a fine-tuned model, a memory layer, custom tool wrappers, three retries with exponential backoff, and a couple of "future-proof" abstractions nobody's actually using yet. The agent at the center is simple. The scaffolding around it is a cathedral.
Here's the uncomfortable truth most teams learn the hard way: AI apps rarely fail because someone picked the wrong model or framework. They fail because layers got added before anyone could name the problem each layer was supposed to solve. The biggest mistake in building AI apps isn't starting too small — it's starting too big.
So here are 7 signs you've crossed into over-engineering, the simpler thing to do instead, and — at the end — a practical playbook for not falling into the trap in the first place. See how many feel a little too familiar.
1. You reached for a vector database before you needed one
"First, set up your vector database" became the default opening line of every AI tutorial — so teams spin up Pinecone or Chroma reflexively, before they've confirmed they even have a retrieval problem that requires embeddings.
The plot twist of the last year is how often that's overkill. Some of the most capable coding agents around quietly dropped vector search in favor of plain tool-driven search — grep, reading the file tree, asking for files by name. In one widely-cited case, ripping out the embedding pipeline and replacing it with grep reportedly outperformed the vector setup, by a lot.
That doesn't mean vector DBs are dead — they're still a strong fit for large, stable knowledge bases (product docs, FAQs, glossaries) with a good reranker. But if your data is small enough to fit in context, or searchable with keywords and filters, you may be maintaining an entire embedding-and-migration pipeline to solve a problem grep or a SQL WHERE clause already solves.
Instead: Start with the dumbest retrieval that works — keyword search, a filter, or just stuffing the relevant docs into the prompt. Add embeddings only when that measurably falls short.
2. You built an "agent" that's really a single prompt in a trenchcoat
Multi-agent systems are exciting. A planner agent, a researcher agent, a critic agent, a synthesizer agent, all passing messages around a graph — it feels like real engineering.
But a huge share of "agentic" apps are doing something a single well-structured prompt (or a short, linear sequence of two or three calls) would handle more reliably, more cheaply, and with far less to debug. Every extra agent multiplies your failure surface: more places to hallucinate, more handoffs to break, more latency, more cost, more nondeterminism.
If you can't clearly state what each agent does that a single call couldn't, you don't have a multi-agent system. You have one prompt wearing several hats and charging you for each.
Instead: Build the single-call version first. Only split into agents when you hit a concrete wall — a genuinely distinct sub-task, a real need for parallelism, or a step that must be independently verifiable.
3. You fine-tuned a model to teach it facts
Fine-tuning sounds like the serious, grown-up move — like you've graduated from "just prompting." So teams fine-tune a model on their company data expecting it to reliably know that information afterward.
This is one of the most common expensive mistakes in the space. Models memorize facts poorly and forget them unpredictably; fine-tuning on factual data is almost always the wrong tool. Facts belong in a retrieval layer you can update in seconds, not baked into weights you have to retrain to change. Fine-tuning is for shaping behavior — tone, format, task-alignment — not for storing knowledge.
Instead: If you want the model to know things, retrieve them (or just put them in the prompt). Reserve fine-tuning for when you need consistent style or structure that prompting can't reliably produce.
4. You added a memory system nobody asked for
"AI that remembers you" is a compelling pitch, so persistent memory layers, temporal knowledge graphs, and cross-session state get bolted on early — often to apps that are fundamentally single-shot.
Memory is a real and increasingly important layer for agents that genuinely span sessions and users. But it's also a whole system that has to decide what to keep, what to age out, and what to resurface — and if your app answers a question and moves on, that machinery is pure overhead. Worse, half-baked memory actively hurts: stale or wrongly-recalled context makes answers worse, not better.
Instead: Ask whether the task actually needs continuity across turns. If not, skip it. If it does, start with the simplest thing — a summary of the conversation passed forward — before reaching for a dedicated memory engine.
5. Your prompts have grown their own framework
It starts reasonably: a system prompt, a couple of examples. Then someone adds a templating engine, then conditional prompt-assembly logic, then a prompt "router," then a library of forty partials stitched together at runtime — and now understanding what the model actually receives requires running a debugger.
Complexity in the plumbing around the prompt is still complexity. When the assembled prompt becomes something no human can read in one sitting, you've traded a legibility problem you could see for one you can't.
Instead: Keep prompts as flat and readable as you can for as long as you can. When you do need dynamic assembly, log the final rendered prompt and read it regularly — if you can't follow it, the model's job is harder than it needs to be too.
6. You have no evals — but you have a lot of architecture
This is the tell that ties all the others together. Teams pour weeks into retrieval pipelines, agent graphs, and memory layers, and measure quality by vibes — clicking around, eyeballing outputs, shipping when it "feels good."
That's backwards. Skipping evaluation is one of the most common and dangerous mistakes in production AI. Without evals you can't answer the only questions that matter: Is it improving? Did that change break something? Does it behave consistently across inputs? Every layer you added was justified by an assumption — and without evals, not one of those assumptions has been tested. You could very likely delete half the architecture with zero quality loss and never know.
Instead: Build a small, high-quality labeled test set early — even 30-50 examples. Measure before and after every architectural change. Let evals, not aesthetics, tell you which layers earn their keep. (Bonus: evals usually reveal that your problem is retrieval quality or prompt clarity, not the thing you were about to build.)
7. You optimized for scale you don't have
Sharding, elaborate caching tiers, multi-region failover, a queue system, autoscaling GPU inference — for an app with a few dozen daily users and a roadmap that's mostly hypothetical.
Premature scaling is classic over-engineering wearing an AI costume. You're paying — in build time, complexity, and cognitive load — for traffic that may never arrive, and every one of those systems is now something you maintain and debug instead of improving the actual product. The irony is that scaling problems are good problems; they mean people are using the thing. Build the thing first.
Instead: Build for roughly 10x your current load, not 1000x. Make it easy to change. When real usage strains it, you'll know exactly which part to scale — and you'll scale the right one, because reality told you which it was.
How to Not Over-Engineer in the First Place
Recognizing the signs is half the battle. Avoiding them from day one is the other half. Here's the practical playbook — habits that keep an AI app lean without keeping it underpowered.
Start with the boring baseline and beat it. Before any architecture, build the crudest version that could possibly work: one prompt, the docs pasted in, no retrieval, no agents. Measure how good it actually is. That number is now your baseline — and every layer you consider has to beat it to justify existing. You'll be surprised how often the boring baseline is already good enough to ship.
Make "why" a required field. Adopt a simple rule for your team: no new layer goes in without a one-sentence answer to "what specific, observed problem does this solve?" "It might be useful later" and "the tutorial had one" don't count. If you can't name the failure it fixes, you don't have evidence you need it yet — you have a hunch. Hunches go in the backlog, not the codebase.
Write the eval before the feature. Flip the usual order. Before building the fancy retrieval pipeline, write the test that would prove it's better. If you can't define what "better" looks like measurably, you're not ready to build it — and if you can, you'll often discover a far simpler change moves the number just as much.
Add one layer at a time, and measure each. Never add three improvements at once. You'll have no idea which one helped, which did nothing, and which quietly made things worse. One change, one measurement, keep-or-revert. This alone prevents most accidental complexity, because layers that don't earn their keep get caught and removed immediately instead of calcifying.
Prefer boring, deletable tools. Given two options, pick the one that's easier to rip out. A plain function call is easier to delete than a framework. Keyword search is easier to delete than a vector store. Reversible decisions let you move fast because mistakes are cheap. Save the hard-to-undo commitments for the few places you're genuinely certain.
Optimize for the reader, not the résumé. A lot of over-engineering is really engineers building the impressive version for themselves. Ask instead: could a new teammate understand this system in an afternoon? Could you, six months from now, at 2 a.m., during an incident? Simple systems aren't less skilled — restraint is the harder skill. The senior move is usually the smaller one.
Delay the irreversible. Some choices are easy to change later (a prompt, a filter, a model swap). Some are painful (a database schema, an agent framework you've wired everything into, a fine-tuned model in your pipeline). Make the cheap, reversible choices freely and early. Delay the expensive, irreversible ones as long as you responsibly can — by then you'll actually know if you need them.
The One Rule Underneath All of It
If there's a single principle here, it's this:
Start with the smallest stack that solves the problem. Add a layer only when something specific and observable breaks.
Not "when a layer might be nice someday." Not "when the tutorial included one." When something breaks, in a way you can name and measure.
This isn't an argument for building sloppy or ignoring real complexity — plenty of AI apps genuinely need vector search, agents, memory, and scale. It's an argument for earning each layer. The best AI apps aren't the ones with the most impressive architecture diagram. They're the ones where every box on the diagram is there because something would break without it.
Complexity is easy to add and brutally hard to remove. Every layer you don't build is a layer you don't have to debug at 2 a.m.
Instead of asking "what could I add to make this more capable?" ask "what could I remove and still have it work?" Ship that version. Let reality tell you what to build next.
Which of these have you been guilty of? I'll admit to #3 and #7 — I've fine-tuned a model to "know" things it promptly forgot, and scaled an app for a stampede that was really about four people. Confess yours in the comments.
Top comments (6)
Spot on. The hardest part right now isn't building the architectural "cathedral"—it's having the discipline to know where to draw the line between a POC, an MVP, and a matured product.
Most teams jump straight to vector DBs and multi-agent graphs to solve problems a single prompt and simple retrieval could handle in an MVP. Architecture should be earned by failing simple evals, not anticipated by default. Great reminder to keep it lean until the limits actually force your hand!
"Architecture should be earned by failing simple evals, not anticipated by default" — that's better than anything in my post. Might have to steal it (with credit).
The POC/MVP/matured-product framing is exactly right too. So much complexity comes from building the matured-product version while you're still at POC, before you've earned a single one of those layers. Discipline really is the hard part — adding is easy and fun, restraint is the skill nobody tweets about. Thanks for the sharp addition.
4 is my category, so let me start by agreeing with it, with a number.
"Half-baked memory actively hurts" is exactly right. I counted my own store this week: 524 records, 257 of them (49%) have been overwritten at least once. Half of what I wrote down needed correcting. A memory layer that has no way to mark the old version as superseded isn't neutral overhead - it's a second, confident, wrong answer competing on relevance with the right one.
The test I'd offer for "does this app need memory at all" is narrower than the pitch: does a fact learned in session N come back in session N+1 without help? If the app answers a question and moves on, the answer is no and the whole layer is cost. That's most apps.
But the section I actually want to push on is #6, because it's the one that saved me and I think your playbook is one notch too generous.
You write: "Build a small, high-quality labeled test set early - even 30-50 examples."
I did that. Seventeen lessons, thirteen queries, hand-written. It said my ranker scored 92.3% precision@1 against 76.9% for a flat file. I nearly published that number.
Then I ran the same two versions against 498 real records, and the order flipped: the version the small set preferred by 23 points found the right answer half as often on real data - 15% against 30%. The entire advantage on the fixture was the damage on real data.
The reason isn't noise, and this is why I'd sharpen the advice rather than just add a caveat: seventeen examples is not a noisy version of five hundred, it's a different question. With sixteen competitors, a rare word is enough to win, so every retrieval mechanism clears the bar and they all look equally good. The differences aren't small - they're invisible.
So the version of your rule I'd write after that: a test set is only an eval if the task is hard enough on it that things can fail. If everything scores above 90%, you haven't built an eval, you've built a smoke test - and a smoke test will happily approve the layer you were hoping to justify.
One more, on "add one layer at a time, and measure each" - which I'd underline twice:
My benchmark used to report percentage points. It now counts how many questions moved. On 100 questions, "four points better" is four questions - but twenty-five better against five worse is a result, and fourteen better against thirty worse is the same four points and a disaster. That counter immediately killed my most promising change: on the corpus I'd been developing against it moved top-3 from 58% to 63%; on the store that actually runs in production the same code moved it from 55% to 51%. Same change, same metric, opposite sign. The second corpus was the real one.
Your "one change, one measurement, keep-or-revert" is right. I'd just add: make sure the measurement can count losses separately from wins, or an average will hide a change that helped a few people a lot and hurt many people a little.
Confessing mine: #4 and #6, in that order, and the second one is why I got away with the first.
This comment is better than the section it's correcting, and I mean that. You've taken two of my rules and shown exactly where they're too soft. Let me concede both, because you've earned it with numbers.
On #6 — you're right, and the sharper rule is yours: a test set is only an eval if things can fail on it. My "30–50 examples" advice quietly assumed the set could discriminate, and your 17-lesson case is the perfect counterexample: with sixteen competitors a rare word wins, so every method clears the bar and they all look equally good. "You haven't built an eval, you've built a smoke test — and a smoke test will happily approve the layer you were hoping to justify" is the line I wish I'd written. The 92.3% vs 76.9% fixture result flipping to 15% vs 30% on real data isn't a caveat to my point — it is the point, and it's more dangerous than having no eval at all, because a green number talks you out of skepticism. I'm going to amend that section and credit this.
On #7 (measure each layer) — the loss/win asymmetry is the part I genuinely underweighted. "Four points better" hiding twenty-five better against five worse versus fourteen better against thirty worse is exactly the kind of thing an average is designed to launder. Same delta, opposite meaning, and only one of them ships. Counting questions moved instead of reporting percentage points is a better default than what I wrote, and your 63%-on-dev / 51%-on-prod flip is the cleanest possible argument for it — same code, same metric, opposite sign, and the second corpus was the real one. The methodology bug wasn't the change; it was trusting the corpus you developed against.
On #4 — your memory test ("does a fact learned in session N come back in N+1 without help?") is narrower and better than my framing, and the 49%-overwritten number makes the "actively hurts" case concrete in a way my prose didn't. A store with no way to mark a version superseded isn't neutral overhead — it's a confident wrong answer competing on relevance with the right one. That's the whole failure mode in one sentence.
And your closing confession is the real lesson buried in all of this: #6 is why you got away with #4. A weak eval doesn't just fail to catch an unnecessary layer — it issues the layer a certificate. The bad memory store survived because the smoke test blessed it. That's the causal chain I under-drew in the post, and it's the most important thing in this thread.
Thanks for this. Genuinely the most useful comment I've gotten on anything I've written.
"A weak eval doesn't just fail to catch an unnecessary layer - it issues the layer a certificate" is better than anything I gave you, and I'm taking it.
One thing that makes it worse, and I'd rather hand it over than keep it: that certificate had a signature.
My 92.3% was not a weak measurement. It was a hardcoded string in the server's own metrics output, with a comment above it reading "CI-defended" and a unit test asserting the string was present. So the chain was: a number nothing computed → a test guarding the sentence that contained it → a green check → a layer that felt earned.
A number the server does not compute cannot fall. It isn't a bad eval, it's a painting of an eval, and it will hold up under exactly the kind of review that catches real regressions - because nothing about it is inconsistent. I only found it because someone in a comment thread asked for a harsher metric and I re-ran the command instead of quoting myself.
So the version I'd add under your certificate line: check whether your eval number is computed at request time or stored somewhere. If it can't move, it isn't measuring - and every layer it approved is unaudited.
Thanks for taking the two corrections in public. That's rarer than the post.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.