DEV Community

Cover image for 7 Signs You're Over-Engineering Your AI App (and How to Stop)
James Anderson
James Anderson

Posted on

7 Signs You're Over-Engineering Your AI App (and How to Stop)

Readers confess painful over-engineering realities

There's a very specific kind of AI project that looks incredibly impressive in the architecture diagram and does almost nothing a simple version couldn't do better.

It has a vector database. It has a multi-agent orchestration graph. It has a fine-tuned model, a memory layer, custom tool wrappers, three retries with exponential backoff, and a couple of "future-proof" abstractions nobody's actually using yet. The agent at the center is simple. The scaffolding around it is a cathedral.

Here's the uncomfortable truth most teams learn the hard way: AI apps rarely fail because someone picked the wrong model or framework. They fail because layers got added before anyone could name the problem each layer was supposed to solve. The biggest mistake in building AI apps isn't starting too small — it's starting too big.

So here are 7 signs you've crossed into over-engineering, the simpler thing to do instead, and — at the end — a practical playbook for not falling into the trap in the first place. See how many feel a little too familiar.


1. You reached for a vector database before you needed one

"First, set up your vector database" became the default opening line of every AI tutorial — so teams spin up Pinecone or Chroma reflexively, before they've confirmed they even have a retrieval problem that requires embeddings.

The plot twist of the last year is how often that's overkill. Some of the most capable coding agents around quietly dropped vector search in favor of plain tool-driven search — grep, reading the file tree, asking for files by name. In one widely-cited case, ripping out the embedding pipeline and replacing it with grep reportedly outperformed the vector setup, by a lot.

That doesn't mean vector DBs are dead — they're still a strong fit for large, stable knowledge bases (product docs, FAQs, glossaries) with a good reranker. But if your data is small enough to fit in context, or searchable with keywords and filters, you may be maintaining an entire embedding-and-migration pipeline to solve a problem grep or a SQL WHERE clause already solves.

Instead: Start with the dumbest retrieval that works — keyword search, a filter, or just stuffing the relevant docs into the prompt. Add embeddings only when that measurably falls short.


2. You built an "agent" that's really a single prompt in a trenchcoat

Multi-agent systems are exciting. A planner agent, a researcher agent, a critic agent, a synthesizer agent, all passing messages around a graph — it feels like real engineering.

But a huge share of "agentic" apps are doing something a single well-structured prompt (or a short, linear sequence of two or three calls) would handle more reliably, more cheaply, and with far less to debug. Every extra agent multiplies your failure surface: more places to hallucinate, more handoffs to break, more latency, more cost, more nondeterminism.

If you can't clearly state what each agent does that a single call couldn't, you don't have a multi-agent system. You have one prompt wearing several hats and charging you for each.

Instead: Build the single-call version first. Only split into agents when you hit a concrete wall — a genuinely distinct sub-task, a real need for parallelism, or a step that must be independently verifiable.


3. You fine-tuned a model to teach it facts

Fine-tuning sounds like the serious, grown-up move — like you've graduated from "just prompting." So teams fine-tune a model on their company data expecting it to reliably know that information afterward.

This is one of the most common expensive mistakes in the space. Models memorize facts poorly and forget them unpredictably; fine-tuning on factual data is almost always the wrong tool. Facts belong in a retrieval layer you can update in seconds, not baked into weights you have to retrain to change. Fine-tuning is for shaping behavior — tone, format, task-alignment — not for storing knowledge.

Instead: If you want the model to know things, retrieve them (or just put them in the prompt). Reserve fine-tuning for when you need consistent style or structure that prompting can't reliably produce.


4. You added a memory system nobody asked for

"AI that remembers you" is a compelling pitch, so persistent memory layers, temporal knowledge graphs, and cross-session state get bolted on early — often to apps that are fundamentally single-shot.

Memory is a real and increasingly important layer for agents that genuinely span sessions and users. But it's also a whole system that has to decide what to keep, what to age out, and what to resurface — and if your app answers a question and moves on, that machinery is pure overhead. Worse, half-baked memory actively hurts: stale or wrongly-recalled context makes answers worse, not better.

Instead: Ask whether the task actually needs continuity across turns. If not, skip it. If it does, start with the simplest thing — a summary of the conversation passed forward — before reaching for a dedicated memory engine.


5. Your prompts have grown their own framework

It starts reasonably: a system prompt, a couple of examples. Then someone adds a templating engine, then conditional prompt-assembly logic, then a prompt "router," then a library of forty partials stitched together at runtime — and now understanding what the model actually receives requires running a debugger.

Complexity in the plumbing around the prompt is still complexity. When the assembled prompt becomes something no human can read in one sitting, you've traded a legibility problem you could see for one you can't.

Instead: Keep prompts as flat and readable as you can for as long as you can. When you do need dynamic assembly, log the final rendered prompt and read it regularly — if you can't follow it, the model's job is harder than it needs to be too.


6. You have no evals — but you have a lot of architecture

This is the tell that ties all the others together. Teams pour weeks into retrieval pipelines, agent graphs, and memory layers, and measure quality by vibes — clicking around, eyeballing outputs, shipping when it "feels good."

That's backwards. Skipping evaluation is one of the most common and dangerous mistakes in production AI. Without evals you can't answer the only questions that matter: Is it improving? Did that change break something? Does it behave consistently across inputs? Every layer you added was justified by an assumption — and without evals, not one of those assumptions has been tested. You could very likely delete half the architecture with zero quality loss and never know.

Instead: Build a small, high-quality labeled test set early — even 30-50 examples. Measure before and after every architectural change. Let evals, not aesthetics, tell you which layers earn their keep. (Bonus: evals usually reveal that your problem is retrieval quality or prompt clarity, not the thing you were about to build.)


7. You optimized for scale you don't have

Sharding, elaborate caching tiers, multi-region failover, a queue system, autoscaling GPU inference — for an app with a few dozen daily users and a roadmap that's mostly hypothetical.

Premature scaling is classic over-engineering wearing an AI costume. You're paying — in build time, complexity, and cognitive load — for traffic that may never arrive, and every one of those systems is now something you maintain and debug instead of improving the actual product. The irony is that scaling problems are good problems; they mean people are using the thing. Build the thing first.

Instead: Build for roughly 10x your current load, not 1000x. Make it easy to change. When real usage strains it, you'll know exactly which part to scale — and you'll scale the right one, because reality told you which it was.


How to Not Over-Engineer in the First Place

Recognizing the signs is half the battle. Avoiding them from day one is the other half. Here's the practical playbook — habits that keep an AI app lean without keeping it underpowered.

Start with the boring baseline and beat it. Before any architecture, build the crudest version that could possibly work: one prompt, the docs pasted in, no retrieval, no agents. Measure how good it actually is. That number is now your baseline — and every layer you consider has to beat it to justify existing. You'll be surprised how often the boring baseline is already good enough to ship.

Make "why" a required field. Adopt a simple rule for your team: no new layer goes in without a one-sentence answer to "what specific, observed problem does this solve?" "It might be useful later" and "the tutorial had one" don't count. If you can't name the failure it fixes, you don't have evidence you need it yet — you have a hunch. Hunches go in the backlog, not the codebase.

Write the eval before the feature. Flip the usual order. Before building the fancy retrieval pipeline, write the test that would prove it's better. If you can't define what "better" looks like measurably, you're not ready to build it — and if you can, you'll often discover a far simpler change moves the number just as much.

Add one layer at a time, and measure each. Never add three improvements at once. You'll have no idea which one helped, which did nothing, and which quietly made things worse. One change, one measurement, keep-or-revert. This alone prevents most accidental complexity, because layers that don't earn their keep get caught and removed immediately instead of calcifying.

Prefer boring, deletable tools. Given two options, pick the one that's easier to rip out. A plain function call is easier to delete than a framework. Keyword search is easier to delete than a vector store. Reversible decisions let you move fast because mistakes are cheap. Save the hard-to-undo commitments for the few places you're genuinely certain.

Optimize for the reader, not the résumé. A lot of over-engineering is really engineers building the impressive version for themselves. Ask instead: could a new teammate understand this system in an afternoon? Could you, six months from now, at 2 a.m., during an incident? Simple systems aren't less skilled — restraint is the harder skill. The senior move is usually the smaller one.

Delay the irreversible. Some choices are easy to change later (a prompt, a filter, a model swap). Some are painful (a database schema, an agent framework you've wired everything into, a fine-tuned model in your pipeline). Make the cheap, reversible choices freely and early. Delay the expensive, irreversible ones as long as you responsibly can — by then you'll actually know if you need them.


The One Rule Underneath All of It

If there's a single principle here, it's this:

Start with the smallest stack that solves the problem. Add a layer only when something specific and observable breaks.

Not "when a layer might be nice someday." Not "when the tutorial included one." When something breaks, in a way you can name and measure.

This isn't an argument for building sloppy or ignoring real complexity — plenty of AI apps genuinely need vector search, agents, memory, and scale. It's an argument for earning each layer. The best AI apps aren't the ones with the most impressive architecture diagram. They're the ones where every box on the diagram is there because something would break without it.

Complexity is easy to add and brutally hard to remove. Every layer you don't build is a layer you don't have to debug at 2 a.m.

Instead of asking "what could I add to make this more capable?" ask "what could I remove and still have it work?" Ship that version. Let reality tell you what to build next.


Which of these have you been guilty of? I'll admit to #3 and #7 — I've fine-tuned a model to "know" things it promptly forgot, and scaled an app for a stampede that was really about four people. Confess yours in the comments.

Top comments (32)

Collapse
 
ofri-peretz profile image
Ofri Peretz

The point about coding agents dropping vector search for grep is the sharpest observation in here. In static analysis work I've run, I hit the exact parallel: teams would layer in embedding-based "semantic similarity" search to surface vulnerable code patterns, when a focused AST visitor already caught the same constructs — deterministically, with no pipeline to keep warm. Embeddings earn their spot when the query and the document don't share vocabulary; if they do, you're running an expensive approximation of a filter you already have. The retrieval problem often just dissolves once you write down, precisely, what question you're actually trying to answer.

Collapse
 
james_anderson_h profile image
James Anderson

"Running an expensive approximation of a filter you already have" — that's the cleanest statement of the whole failure mode, and your AST example is a sharper version of the grep one because static analysis makes the determinism gap so stark. An AST visitor doesn't approximate whether a construct is present; it knows, structurally, every time. Reaching for embedding similarity to surface a pattern the grammar already defines precisely is choosing a fuzzy, non-deterministic, pipeline-heavy answer to a question that had an exact one. You traded a guarantee for a vibe and paid extra for the privilege.

Your rule for when embeddings actually earn their spot is the keeper: when query and document don't share vocabulary. That's the real boundary. Semantic search is for bridging a vocabulary gap — the user says "login problems," the doc says "authentication failure." If the thing you're matching on is structural or lexical — an AST node type, a function name, an error code, a known construct — there's no gap to bridge, and the embedding is just a lossy filter with a GPU bill.

But the line I'll actually keep is your last one: the retrieval problem often dissolves once you write down precisely what question you're answering. That's the tell for half the over-engineering in the post. The vector DB frequently isn't solving a hard retrieval problem — it's substituting for the un-done work of specifying the question. Specify it precisely and you often discover it was a filter, a grammar, or a WHERE clause all along. The embedding was papering over a vague spec, not a hard search. Great addition.

Collapse
 
tokenlat profile image
TokenLat

Missing from the list, but the most expensive one I see: treating every LLM call as frontier-worthy. Most agent traffic is mechanical — summarize, classify, extract, format — and a cheap routed model handles ~80% of it cleanly. You only escalate to frontier for the genuinely hard 20%.

Teams that route by task instead of defaulting to one big model end up with a simpler harness AND a bill that doesn't make the CFO wince. "Start dumb, escalate on failure" applies to model choice exactly as much as to your retrieval layer.

Collapse
 
james_anderson_h profile image
James Anderson

This belongs on the list, and I think it's the highest-dollar item of all of them — over-engineering shows up in your model choice just as much as your architecture, and it hits the bill harder than anything else.

The framing that lands for me: defaulting every call to a frontier model is the same instinct as reaching for a vector DB before you've proven you need one. It's paying for capability you haven't confirmed the task requires. And you're right that most agent traffic is mechanical — summarize, classify, extract, format — where a small routed model is not just cheaper but often faster and more reliable, because you've narrowed the job. Frontier-for-everything is buying a supercomputer to run grep.

"Start dumb, escalate on failure" applied to routing is the perfect parallel to the rest of the post. The small model is the boring baseline; the frontier model is the layer you earn by watching the cheap one actually fail on the hard 20% — not by assuming it will. And the bonus you flagged is the part people miss: routing by task doesn't just cut the bill, it simplifies the harness, because most requests take the short path and only the genuine hard cases hit the complex one. Cheaper and simpler usually trade off; here they point the same way.

The one discipline I'd add: you need an eval to know where the 20% line actually is, or "escalate on failure" quietly becomes "escalate on vibes" and you drift back to frontier-by-default. But that's agreeing with you, not arguing. Great addition — genuinely one of the best comments on this post.

Collapse
 
tokenlat profile image
TokenLat

Agreed — and it's the most expensive item precisely because it's invisible. "Default to frontier" is the same reflex as reaching for a vector DB before proving you need one, except the bill shows up every month instead of once.

The win that actually moves the number: route per call, not per project. A 2-token intent check and a 4k-token legal analysis shouldn't get the same $3/M treatment. The cheap first cut is just classifying calls by whether the input has a clear shape and the output is checkable — if yes, it almost certainly doesn't need frontier. Most teams never measure which calls did, so they can't see the tax. Audit one week of traffic by call-type and the waste is obvious.

Collapse
 
eduzsh profile image
Edu Peralta

The coding agent point about dropping vector search for grep is the one that matches day to day work. When an agent can list the tree and read three files by name, an embedding pipeline mostly adds another place for stale chunks to win. The cathedral scaffolding you describe is what I see when teams add a planner, critic, and memory layer before they have a single failing task that proves any of those layers. Start with the dumb retrieval that fails in an obvious way. That failure is the only honest signal for what to build next.

Collapse
 
james_anderson_h profile image
James Anderson

"An embedding pipeline mostly adds another place for stale chunks to win" — that's the failure mode in one line, and it's sharper than how I put it. The agent that can list the tree and read three files by name is querying ground truth, live, every time. The embedding index is a cached, lossy snapshot of that same truth that now has to be kept warm, re-synced, and trusted — and the moment it drifts, it doesn't fail loudly, it just quietly serves a confident stale answer that beats the fresh one on similarity score. You added a whole pipeline and its most notable new capability is being wrong in a way grep never could be.

Your read on the cathedral is exactly right, and the sequencing is the whole point: planner, critic, and memory added before a single failing task that proves any of them is needed is architecture built on a hunch. Each of those layers is an answer to a question nobody has asked yet — and if you can't point to the failure it fixes, you don't have evidence, you have a guess wearing a diagram.

"Start with the dumb retrieval that fails in an obvious way. That failure is the only honest signal for what to build next." I might put that on the wall. It's the entire thesis distilled: the boring baseline isn't just cheaper, it's diagnostic — its failures tell you precisely which layer you've now earned, and nothing else does. Skip the baseline and you don't just over-build, you lose the only honest signal for what to build at all. Great comment.

Collapse
 
polterguy profile image
Thomas Hansen

You shouldn't even start at all - Creating AI agents today is a "commodity", you can easily pull off with a couple of sentences of natural language.

My project ==> hyperlambda.dev

Collapse
 
mudassirworks profile image
Mudassir Khan

sign #2 is the one i'd put first personally. the "planner + researcher + critic + synthesizer" graph is often just one good structured prompt with four sections, but with 4x the failure surface and 10x the debugging sessions.

we caught ourselves doing this on a RAG pipeline last year — 6 agents, 3 orchestration layers, weekly oncall rotations just to babysit state transitions. rewrote it as 2 calls (retrieval → generation) and an eval harness. latency dropped from 4s to 800ms, error rate went from 12% to under 2%.

the tell is whether you can explain the failure mode of each handoff. if you can't name the failure mode, the handoff is premature.

what's the threshold where you've found multi agent is actually warranted over a linear chain?

Collapse
 
james_anderson_h profile image
James Anderson

Those numbers are the whole post in one receipt — 6 agents + 3 orchestration layers + weekly oncall down to 2 calls and an eval harness, 4s → 800ms, 12% → under 2%. That's not a marginal cleanup, that's the complexity itself being the bug: most of the latency and most of the errors lived in the handoffs and state transitions, not the actual work. The moment you deleted the machinery, you deleted the failure surface it created. "Weekly oncall rotations just to babysit state transitions" should be printed on the wall of every team about to draw an agent graph.

Your tell is better than anything in the post: if you can't name the failure mode of a handoff, the handoff is premature. That's a cleaner test than my "what does each agent do that one call couldn't" — because it's not asking whether the split is conceptually justified, it's asking whether you understand it well enough to operate it. A handoff whose failure mode you can't name isn't a design decision, it's a place you've agreed to be surprised later.

On your question — where multi-agent actually earns it over a linear chain — my honest answer is it's rarely about the task being complex and almost always about one of three concrete things the linear version genuinely can't do:

True branching that needs isolated context. Not "four steps in sequence" (that's a structured prompt), but "depending on step 2, take a genuinely different path where the branches would pollute each other's context if they shared one window." If it's a straight line, it's a chain wearing a costume.
Independent parallelism with a real latency payoff. Five retrievals that don't depend on each other and fan out concurrently — where the parallelism buys you something measurable, not just a prettier diagram.
A step that must be independently verifiable / adversarial. A critic that has to not share the generator's context to do its job — which, per this whole thread, only counts if you've proven the critic can actually fail. A rubber-stamp critic is negative value.

If it's none of those three, a linear chain with a good structured prompt and an eval harness wins on latency, cost, and debuggability every time — as your rewrite demonstrated. And even when one of the three applies, I'd add: earn it on the failing task, not the anticipated one. The chain should fail first, in a way you can name, before the agent shows up to fix it. Great comment — the "name the failure mode of the handoff" line is going in the follow-up with credit.

Collapse
 
mnemehq profile image
Theo Valmis

Number of abstraction layers between the prompt and the actual API call is a good one to add here. Every layer someone adds for flexibility is another place a future debugging session has to search.

Collapse
 
james_anderson_h profile image
James Anderson

That's a great addition, and it's the plumbing version of the "single prompt in a trenchcoat" sign — complexity that hides not in what the system does but in how far the request travels to get there. Every abstraction layer between the prompt and the API call is sold as flexibility, and each one is a place a future debugging session has to stop and search. The cruel part is the asymmetry: the layer is added once, in a calm moment, to handle a hypothetical; it's then paid for on every incident, at 2 a.m., when you're tracing why the model received something you didn't expect.

And it compounds with the prompt-framework sign from the post, because the two failure modes stack. When "what did the model actually get?" requires stepping through six wrappers, a config resolver, and a template engine to answer, you haven't built flexibility — you've built distance between yourself and the one string that matters. The tell I'd offer: if you can't answer "what exact payload hit the API?" without a debugger, you have too many layers, no matter how clean each one looks in isolation.

The fix is the same discipline as everything else in the post — earn the layer. A layer that abstracts a variation you actually have is doing work; a layer that abstracts a variation you might have is just pre-paid debugging cost with no offsetting benefit yet. And most "for flexibility" layers are the second kind. Great catch — going on the list.

Collapse
 
promptshereai profile image
Fouad El Mourabit

Great read! Point #2 really hits home. I’ve seen so many ‘multi-agent’ systems that could have been solved with a single well-structured prompt and a few lines of logic. The complexity overhead is real, especially when it comes to debugging. Simplicity is definitely underrated in the current AI hype.

Collapse
 
james_anderson_h profile image
James Anderson

Thanks! And yeah — #2 is the one I see most in the wild. The debugging overhead is the part people underestimate: every agent handoff is another place for things to break silently, and a single prompt fails in ways you can actually read. Simplicity is underrated precisely because it doesn't demo as impressively as a graph of agents.

Collapse
 
promptshereai profile image
Fouad El Mourabit

Exactly, a single robust prompt is much easier to maintain than a fragile chain of agents. Glad we're on the same page!

Collapse
 
unitbuilds profile image
UnitBuilds

🙄 So... Making an entire IDE, with support for over 10 providers, with a custom VC system, a from-scratch Rust-based browser, remote desktop capabilities, cross-system droning, a custom rust-based MCP server, teams builder, orchestrator and cross-platform usage dashboard to track every API you use was overengineering? Oops

Collapse
 
james_anderson_h profile image
James Anderson

Ha — that's not over-engineering, that's a whole product suite wearing the trenchcoat of "I'll just build a little tool."

But here's the honest test the post would apply: over-engineering isn't about how much you built — it's about whether each piece was earned by a real problem or added on spec. So the actual question for your list is:

The 10-provider support — did users keep asking for providers you didn't have, or did you build all 10 before anyone requested #3?
The from-scratch Rust browser — did an existing engine actually fail you, or was "from scratch" the fun part? (This is the one I'd interrogate hardest. That's a cathedral inside the cathedral.)
The usage dashboard tracking every API — that one might genuinely be the least over-engineered thing on the list, given everything else you're running.

If every item traces back to a real observed need, it's not over-engineering — it's just a big product, and big products are allowed to be big. If half of them were "I'll need this eventually," then yeah… the post was written for exactly this. 😄

The tell is your own word: "oops." You don't say oops about the layers you're glad you built. You say it about the ones you built before anything forced your hand.

So — genuine question, not a dunk: how much of that stack has a user or a metric behind it, and how much was "while I'm in here"? Because the Rust browser is either the smartest or the most expensive decision in the whole thing, and I actually want to know which.

Collapse
 
unitbuilds profile image
UnitBuilds

Well currently I'm the only one using it, as I still want to pack in my durable workflows system, so no users testing it other than me. Practically everything was due to either limitations in existing infrastructure. The rust-based MCP using a custom format (no JSON) was to make it more understandable for LLMs, while stripping the serialization tax, essentially making tools execute in microseconds, instead of a few milliseconds, it also dropped ram overhead from needing node.js (50+mb) to 15mb. The providers was more a way to add capability, given I am a firm believer that each LLM has a specific niche, I mean compare Claude Fable vs Kimi K2.7 for GUI design, Kimi wipes the floor with Claude, so I give the option. The browser was for a few reasons, I started using my MCP-Lite to use the user's native browser, but chrome alone took up 10x more ram than the entire IDE (I built it with strict zero-alloc) and it still choked on any canvas page. So I built it from scratch, so I can strip the fluff and let the LLM see what's actually hiding behind the canvas, which hopefully will mean that there wont come a site the LLM cant interpret, without excessive screenshotting. The Drone system was so I can test features across devices (like the file transfer protocol and remote desktop), which in turn the Remote system was so I can in theory deploy the IDE to a VM and have a clean remote window into it, or have it do remote into another system for configuration (think AI tech support). the custom VC system was so when multiple agents run at once, they execute on the same codebase at once and resolve conflicts in realtime, not at merge and all choices and decisions are cleanly recorded, so if an app in production fails, it can tell you exactly what state the system was in at the time and how big the blast radius is and what fixes were tried before and failed, so it can make better edits. So nothing was wasteful per-say, but I probably should have shipped a MVP before overengineering it to be better than every competitor. In all honesty, it doesnt need a durable workflows system, but it would be a nice-to-have.

Thread Thread
 
unitbuilds profile image
UnitBuilds

Essentially the browser was a decision to build an AI browser. It doesnt even have a render output, because a LLM shouldnt need it, if it wasnt to inspect an element it can inspect the element, not take a full-page screenshot. Half the reason why we make them rely on screenshots, is because we force them to use browser designed for human usage. The rust browser is lighter, faster and tailored to LLM interpretation, you dont get a fancy page to look at, but it gets a clean BOM for everything, which in theory should improve it's capabilities to navigate and self-correct (eg. making sure it fits on a screen, nothing is off-screen, etc.) because it can use constraints to define the workspace and review it cleaner than what we can give it with CDP

Collapse
 
glenallen profile image
Glen Allen

One thing I’d add is that complexity should have an expiration condition, not just an entry condition. It’s reasonable to add a vector store, memory layer, or agent once a measurable problem justifies it, but teams should also define what evidence would tell them that the layer is no longer earning its place. Otherwise, temporary solutions quietly become permanent architecture.

Collapse
 
james_anderson_h profile image
James Anderson

Yes — an expiration condition, not just an entry condition. That's the piece I left out entirely, and it might be the more important half.

My whole post was about earning a layer's way in. You're pointing out that nobody ever schedules the review that lets it earn its way out. And the asymmetry is brutal: the entry condition gets scrutinized in a portfolio review with finance in the room, while the exit never gets checked at all — so every layer is effectively permanent the moment it lands.

"Temporary solutions quietly become permanent architecture" is exactly the failure. The fix is probably to write the kill condition at the same time you write the justification: we added this because X; we remove it when X is no longer true or a simpler thing covers it. Same discipline I argued for at the gate, just applied in both directions. Great addition.

Collapse
 
byteox2 profile image
Niuniu Ox

Sign #1 hit home — I watched a team spend three weeks standing up a vector DB + embedding pipeline for a docs assistant whose entire knowledge base was ~400 markdown files. Keyword search with a SQL WHERE clause beat it on relevance and p99 latency, and we deleted the whole embedding stack.

The pattern I've noticed: every extra layer is usually added to solve a problem nobody has measured yet. The "single prompt in a trenchcoat" test is a great filter — if you can't say what each agent does that one call couldn't, it's overhead.

Curious how you handle the pushback when a team wants the cathedral — e.g. "we'll need multi-agent eventually, so let's build it now." Do you have a go-to argument for deferring it that actually lands with stakeholders?

Collapse
 
james_anderson_h profile image
James Anderson

That ~400-files case is the whole post in one anecdote — three weeks of embedding pipeline losing to a WHERE clause on relevance and p99. And "every extra layer solves a problem nobody has measured yet" says it better than I did.

On the "we'll need it eventually, build it now" pushback, my go-to is the asymmetry argument: a single prompt is easy to split into agents later; a multi-agent graph is brutal to collapse back down. So deferring is cheap and reversible, while committing early is expensive and sticky. I don't argue against the cathedral on principle — that's a taste debate you can't win. I just make "eventually" concrete: "What's the observable signal that tells us it's time? Write it down now." Either they name a real trigger (clean deferral with a trip-wire) or they can't — which reveals "eventually" was a feeling, not a plan. Both outcomes win.

"Let's build it the moment we can name the problem it solves" is a much easier yes than "no."

Some comments may only be visible to logged-in visitors. Sign in to view all comments.