There's a very specific kind of AI project that looks incredibly impressive in the architecture diagram and does almost nothing a simple version co...
For further actions, you may consider blocking this person and/or reporting abuse
The point about coding agents dropping vector search for grep is the sharpest observation in here. In static analysis work I've run, I hit the exact parallel: teams would layer in embedding-based "semantic similarity" search to surface vulnerable code patterns, when a focused AST visitor already caught the same constructs — deterministically, with no pipeline to keep warm. Embeddings earn their spot when the query and the document don't share vocabulary; if they do, you're running an expensive approximation of a filter you already have. The retrieval problem often just dissolves once you write down, precisely, what question you're actually trying to answer.
"Running an expensive approximation of a filter you already have" — that's the cleanest statement of the whole failure mode, and your AST example is a sharper version of the grep one because static analysis makes the determinism gap so stark. An AST visitor doesn't approximate whether a construct is present; it knows, structurally, every time. Reaching for embedding similarity to surface a pattern the grammar already defines precisely is choosing a fuzzy, non-deterministic, pipeline-heavy answer to a question that had an exact one. You traded a guarantee for a vibe and paid extra for the privilege.
Your rule for when embeddings actually earn their spot is the keeper: when query and document don't share vocabulary. That's the real boundary. Semantic search is for bridging a vocabulary gap — the user says "login problems," the doc says "authentication failure." If the thing you're matching on is structural or lexical — an AST node type, a function name, an error code, a known construct — there's no gap to bridge, and the embedding is just a lossy filter with a GPU bill.
But the line I'll actually keep is your last one: the retrieval problem often dissolves once you write down precisely what question you're answering. That's the tell for half the over-engineering in the post. The vector DB frequently isn't solving a hard retrieval problem — it's substituting for the un-done work of specifying the question. Specify it precisely and you often discover it was a filter, a grammar, or a WHERE clause all along. The embedding was papering over a vague spec, not a hard search. Great addition.
Missing from the list, but the most expensive one I see: treating every LLM call as frontier-worthy. Most agent traffic is mechanical — summarize, classify, extract, format — and a cheap routed model handles ~80% of it cleanly. You only escalate to frontier for the genuinely hard 20%.
Teams that route by task instead of defaulting to one big model end up with a simpler harness AND a bill that doesn't make the CFO wince. "Start dumb, escalate on failure" applies to model choice exactly as much as to your retrieval layer.
This belongs on the list, and I think it's the highest-dollar item of all of them — over-engineering shows up in your model choice just as much as your architecture, and it hits the bill harder than anything else.
The framing that lands for me: defaulting every call to a frontier model is the same instinct as reaching for a vector DB before you've proven you need one. It's paying for capability you haven't confirmed the task requires. And you're right that most agent traffic is mechanical — summarize, classify, extract, format — where a small routed model is not just cheaper but often faster and more reliable, because you've narrowed the job. Frontier-for-everything is buying a supercomputer to run grep.
"Start dumb, escalate on failure" applied to routing is the perfect parallel to the rest of the post. The small model is the boring baseline; the frontier model is the layer you earn by watching the cheap one actually fail on the hard 20% — not by assuming it will. And the bonus you flagged is the part people miss: routing by task doesn't just cut the bill, it simplifies the harness, because most requests take the short path and only the genuine hard cases hit the complex one. Cheaper and simpler usually trade off; here they point the same way.
The one discipline I'd add: you need an eval to know where the 20% line actually is, or "escalate on failure" quietly becomes "escalate on vibes" and you drift back to frontier-by-default. But that's agreeing with you, not arguing. Great addition — genuinely one of the best comments on this post.
Agreed — and it's the most expensive item precisely because it's invisible. "Default to frontier" is the same reflex as reaching for a vector DB before proving you need one, except the bill shows up every month instead of once.
The win that actually moves the number: route per call, not per project. A 2-token intent check and a 4k-token legal analysis shouldn't get the same $3/M treatment. The cheap first cut is just classifying calls by whether the input has a clear shape and the output is checkable — if yes, it almost certainly doesn't need frontier. Most teams never measure which calls did, so they can't see the tax. Audit one week of traffic by call-type and the waste is obvious.
The coding agent point about dropping vector search for grep is the one that matches day to day work. When an agent can list the tree and read three files by name, an embedding pipeline mostly adds another place for stale chunks to win. The cathedral scaffolding you describe is what I see when teams add a planner, critic, and memory layer before they have a single failing task that proves any of those layers. Start with the dumb retrieval that fails in an obvious way. That failure is the only honest signal for what to build next.
"An embedding pipeline mostly adds another place for stale chunks to win" — that's the failure mode in one line, and it's sharper than how I put it. The agent that can list the tree and read three files by name is querying ground truth, live, every time. The embedding index is a cached, lossy snapshot of that same truth that now has to be kept warm, re-synced, and trusted — and the moment it drifts, it doesn't fail loudly, it just quietly serves a confident stale answer that beats the fresh one on similarity score. You added a whole pipeline and its most notable new capability is being wrong in a way grep never could be.
Your read on the cathedral is exactly right, and the sequencing is the whole point: planner, critic, and memory added before a single failing task that proves any of them is needed is architecture built on a hunch. Each of those layers is an answer to a question nobody has asked yet — and if you can't point to the failure it fixes, you don't have evidence, you have a guess wearing a diagram.
"Start with the dumb retrieval that fails in an obvious way. That failure is the only honest signal for what to build next." I might put that on the wall. It's the entire thesis distilled: the boring baseline isn't just cheaper, it's diagnostic — its failures tell you precisely which layer you've now earned, and nothing else does. Skip the baseline and you don't just over-build, you lose the only honest signal for what to build at all. Great comment.
You shouldn't even start at all - Creating AI agents today is a "commodity", you can easily pull off with a couple of sentences of natural language.
My project ==> hyperlambda.dev
sign #2 is the one i'd put first personally. the "planner + researcher + critic + synthesizer" graph is often just one good structured prompt with four sections, but with 4x the failure surface and 10x the debugging sessions.
we caught ourselves doing this on a RAG pipeline last year — 6 agents, 3 orchestration layers, weekly oncall rotations just to babysit state transitions. rewrote it as 2 calls (retrieval → generation) and an eval harness. latency dropped from 4s to 800ms, error rate went from 12% to under 2%.
the tell is whether you can explain the failure mode of each handoff. if you can't name the failure mode, the handoff is premature.
what's the threshold where you've found multi agent is actually warranted over a linear chain?
Those numbers are the whole post in one receipt — 6 agents + 3 orchestration layers + weekly oncall down to 2 calls and an eval harness, 4s → 800ms, 12% → under 2%. That's not a marginal cleanup, that's the complexity itself being the bug: most of the latency and most of the errors lived in the handoffs and state transitions, not the actual work. The moment you deleted the machinery, you deleted the failure surface it created. "Weekly oncall rotations just to babysit state transitions" should be printed on the wall of every team about to draw an agent graph.
Your tell is better than anything in the post: if you can't name the failure mode of a handoff, the handoff is premature. That's a cleaner test than my "what does each agent do that one call couldn't" — because it's not asking whether the split is conceptually justified, it's asking whether you understand it well enough to operate it. A handoff whose failure mode you can't name isn't a design decision, it's a place you've agreed to be surprised later.
On your question — where multi-agent actually earns it over a linear chain — my honest answer is it's rarely about the task being complex and almost always about one of three concrete things the linear version genuinely can't do:
True branching that needs isolated context. Not "four steps in sequence" (that's a structured prompt), but "depending on step 2, take a genuinely different path where the branches would pollute each other's context if they shared one window." If it's a straight line, it's a chain wearing a costume.
Independent parallelism with a real latency payoff. Five retrievals that don't depend on each other and fan out concurrently — where the parallelism buys you something measurable, not just a prettier diagram.
A step that must be independently verifiable / adversarial. A critic that has to not share the generator's context to do its job — which, per this whole thread, only counts if you've proven the critic can actually fail. A rubber-stamp critic is negative value.
If it's none of those three, a linear chain with a good structured prompt and an eval harness wins on latency, cost, and debuggability every time — as your rewrite demonstrated. And even when one of the three applies, I'd add: earn it on the failing task, not the anticipated one. The chain should fail first, in a way you can name, before the agent shows up to fix it. Great comment — the "name the failure mode of the handoff" line is going in the follow-up with credit.
Number of abstraction layers between the prompt and the actual API call is a good one to add here. Every layer someone adds for flexibility is another place a future debugging session has to search.
That's a great addition, and it's the plumbing version of the "single prompt in a trenchcoat" sign — complexity that hides not in what the system does but in how far the request travels to get there. Every abstraction layer between the prompt and the API call is sold as flexibility, and each one is a place a future debugging session has to stop and search. The cruel part is the asymmetry: the layer is added once, in a calm moment, to handle a hypothetical; it's then paid for on every incident, at 2 a.m., when you're tracing why the model received something you didn't expect.
And it compounds with the prompt-framework sign from the post, because the two failure modes stack. When "what did the model actually get?" requires stepping through six wrappers, a config resolver, and a template engine to answer, you haven't built flexibility — you've built distance between yourself and the one string that matters. The tell I'd offer: if you can't answer "what exact payload hit the API?" without a debugger, you have too many layers, no matter how clean each one looks in isolation.
The fix is the same discipline as everything else in the post — earn the layer. A layer that abstracts a variation you actually have is doing work; a layer that abstracts a variation you might have is just pre-paid debugging cost with no offsetting benefit yet. And most "for flexibility" layers are the second kind. Great catch — going on the list.
Great read! Point #2 really hits home. I’ve seen so many ‘multi-agent’ systems that could have been solved with a single well-structured prompt and a few lines of logic. The complexity overhead is real, especially when it comes to debugging. Simplicity is definitely underrated in the current AI hype.
Thanks! And yeah — #2 is the one I see most in the wild. The debugging overhead is the part people underestimate: every agent handoff is another place for things to break silently, and a single prompt fails in ways you can actually read. Simplicity is underrated precisely because it doesn't demo as impressively as a graph of agents.
Exactly, a single robust prompt is much easier to maintain than a fragile chain of agents. Glad we're on the same page!
🙄 So... Making an entire IDE, with support for over 10 providers, with a custom VC system, a from-scratch Rust-based browser, remote desktop capabilities, cross-system droning, a custom rust-based MCP server, teams builder, orchestrator and cross-platform usage dashboard to track every API you use was overengineering? Oops
Ha — that's not over-engineering, that's a whole product suite wearing the trenchcoat of "I'll just build a little tool."
But here's the honest test the post would apply: over-engineering isn't about how much you built — it's about whether each piece was earned by a real problem or added on spec. So the actual question for your list is:
The 10-provider support — did users keep asking for providers you didn't have, or did you build all 10 before anyone requested #3?
The from-scratch Rust browser — did an existing engine actually fail you, or was "from scratch" the fun part? (This is the one I'd interrogate hardest. That's a cathedral inside the cathedral.)
The usage dashboard tracking every API — that one might genuinely be the least over-engineered thing on the list, given everything else you're running.
If every item traces back to a real observed need, it's not over-engineering — it's just a big product, and big products are allowed to be big. If half of them were "I'll need this eventually," then yeah… the post was written for exactly this. 😄
The tell is your own word: "oops." You don't say oops about the layers you're glad you built. You say it about the ones you built before anything forced your hand.
So — genuine question, not a dunk: how much of that stack has a user or a metric behind it, and how much was "while I'm in here"? Because the Rust browser is either the smartest or the most expensive decision in the whole thing, and I actually want to know which.
Well currently I'm the only one using it, as I still want to pack in my durable workflows system, so no users testing it other than me. Practically everything was due to either limitations in existing infrastructure. The rust-based MCP using a custom format (no JSON) was to make it more understandable for LLMs, while stripping the serialization tax, essentially making tools execute in microseconds, instead of a few milliseconds, it also dropped ram overhead from needing node.js (50+mb) to 15mb. The providers was more a way to add capability, given I am a firm believer that each LLM has a specific niche, I mean compare Claude Fable vs Kimi K2.7 for GUI design, Kimi wipes the floor with Claude, so I give the option. The browser was for a few reasons, I started using my MCP-Lite to use the user's native browser, but chrome alone took up 10x more ram than the entire IDE (I built it with strict zero-alloc) and it still choked on any canvas page. So I built it from scratch, so I can strip the fluff and let the LLM see what's actually hiding behind the canvas, which hopefully will mean that there wont come a site the LLM cant interpret, without excessive screenshotting. The Drone system was so I can test features across devices (like the file transfer protocol and remote desktop), which in turn the Remote system was so I can in theory deploy the IDE to a VM and have a clean remote window into it, or have it do remote into another system for configuration (think AI tech support). the custom VC system was so when multiple agents run at once, they execute on the same codebase at once and resolve conflicts in realtime, not at merge and all choices and decisions are cleanly recorded, so if an app in production fails, it can tell you exactly what state the system was in at the time and how big the blast radius is and what fixes were tried before and failed, so it can make better edits. So nothing was wasteful per-say, but I probably should have shipped a MVP before overengineering it to be better than every competitor. In all honesty, it doesnt need a durable workflows system, but it would be a nice-to-have.
Essentially the browser was a decision to build an AI browser. It doesnt even have a render output, because a LLM shouldnt need it, if it wasnt to inspect an element it can inspect the element, not take a full-page screenshot. Half the reason why we make them rely on screenshots, is because we force them to use browser designed for human usage. The rust browser is lighter, faster and tailored to LLM interpretation, you dont get a fancy page to look at, but it gets a clean BOM for everything, which in theory should improve it's capabilities to navigate and self-correct (eg. making sure it fits on a screen, nothing is off-screen, etc.) because it can use constraints to define the workspace and review it cleaner than what we can give it with CDP
One thing I’d add is that complexity should have an expiration condition, not just an entry condition. It’s reasonable to add a vector store, memory layer, or agent once a measurable problem justifies it, but teams should also define what evidence would tell them that the layer is no longer earning its place. Otherwise, temporary solutions quietly become permanent architecture.
Yes — an expiration condition, not just an entry condition. That's the piece I left out entirely, and it might be the more important half.
My whole post was about earning a layer's way in. You're pointing out that nobody ever schedules the review that lets it earn its way out. And the asymmetry is brutal: the entry condition gets scrutinized in a portfolio review with finance in the room, while the exit never gets checked at all — so every layer is effectively permanent the moment it lands.
"Temporary solutions quietly become permanent architecture" is exactly the failure. The fix is probably to write the kill condition at the same time you write the justification: we added this because X; we remove it when X is no longer true or a simpler thing covers it. Same discipline I argued for at the gate, just applied in both directions. Great addition.
Sign #1 hit home — I watched a team spend three weeks standing up a vector DB + embedding pipeline for a docs assistant whose entire knowledge base was ~400 markdown files. Keyword search with a SQL WHERE clause beat it on relevance and p99 latency, and we deleted the whole embedding stack.
The pattern I've noticed: every extra layer is usually added to solve a problem nobody has measured yet. The "single prompt in a trenchcoat" test is a great filter — if you can't say what each agent does that one call couldn't, it's overhead.
Curious how you handle the pushback when a team wants the cathedral — e.g. "we'll need multi-agent eventually, so let's build it now." Do you have a go-to argument for deferring it that actually lands with stakeholders?
That ~400-files case is the whole post in one anecdote — three weeks of embedding pipeline losing to a WHERE clause on relevance and p99. And "every extra layer solves a problem nobody has measured yet" says it better than I did.
On the "we'll need it eventually, build it now" pushback, my go-to is the asymmetry argument: a single prompt is easy to split into agents later; a multi-agent graph is brutal to collapse back down. So deferring is cheap and reversible, while committing early is expensive and sticky. I don't argue against the cathedral on principle — that's a taste debate you can't win. I just make "eventually" concrete: "What's the observable signal that tells us it's time? Write it down now." Either they name a real trigger (clean deferral with a trip-wire) or they can't — which reveals "eventually" was a feeling, not a plan. Both outcomes win.
"Let's build it the moment we can name the problem it solves" is a much easier yes than "no."
love the article did not finish yet but what i read really interesting to me thanks you sharing
Thank you — that means a lot, and I'm glad it's landing so far! Would love to hear what you think once you reach the end.
i will absolutely give you a feedback
4 is my category, so let me start by agreeing with it, with a number.
"Half-baked memory actively hurts" is exactly right. I counted my own store this week: 524 records, 257 of them (49%) have been overwritten at least once. Half of what I wrote down needed correcting. A memory layer that has no way to mark the old version as superseded isn't neutral overhead - it's a second, confident, wrong answer competing on relevance with the right one.
The test I'd offer for "does this app need memory at all" is narrower than the pitch: does a fact learned in session N come back in session N+1 without help? If the app answers a question and moves on, the answer is no and the whole layer is cost. That's most apps.
But the section I actually want to push on is #6, because it's the one that saved me and I think your playbook is one notch too generous.
You write: "Build a small, high-quality labeled test set early - even 30-50 examples."
I did that. Seventeen lessons, thirteen queries, hand-written. It said my ranker scored 92.3% precision@1 against 76.9% for a flat file. I nearly published that number.
Then I ran the same two versions against 498 real records, and the order flipped: the version the small set preferred by 23 points found the right answer half as often on real data - 15% against 30%. The entire advantage on the fixture was the damage on real data.
The reason isn't noise, and this is why I'd sharpen the advice rather than just add a caveat: seventeen examples is not a noisy version of five hundred, it's a different question. With sixteen competitors, a rare word is enough to win, so every retrieval mechanism clears the bar and they all look equally good. The differences aren't small - they're invisible.
So the version of your rule I'd write after that: a test set is only an eval if the task is hard enough on it that things can fail. If everything scores above 90%, you haven't built an eval, you've built a smoke test - and a smoke test will happily approve the layer you were hoping to justify.
One more, on "add one layer at a time, and measure each" - which I'd underline twice:
My benchmark used to report percentage points. It now counts how many questions moved. On 100 questions, "four points better" is four questions - but twenty-five better against five worse is a result, and fourteen better against thirty worse is the same four points and a disaster. That counter immediately killed my most promising change: on the corpus I'd been developing against it moved top-3 from 58% to 63%; on the store that actually runs in production the same code moved it from 55% to 51%. Same change, same metric, opposite sign. The second corpus was the real one.
Your "one change, one measurement, keep-or-revert" is right. I'd just add: make sure the measurement can count losses separately from wins, or an average will hide a change that helped a few people a lot and hurt many people a little.
Confessing mine: #4 and #6, in that order, and the second one is why I got away with the first.
This comment is better than the section it's correcting, and I mean that. You've taken two of my rules and shown exactly where they're too soft. Let me concede both, because you've earned it with numbers.
On #6 — you're right, and the sharper rule is yours: a test set is only an eval if things can fail on it. My "30–50 examples" advice quietly assumed the set could discriminate, and your 17-lesson case is the perfect counterexample: with sixteen competitors a rare word wins, so every method clears the bar and they all look equally good. "You haven't built an eval, you've built a smoke test — and a smoke test will happily approve the layer you were hoping to justify" is the line I wish I'd written. The 92.3% vs 76.9% fixture result flipping to 15% vs 30% on real data isn't a caveat to my point — it is the point, and it's more dangerous than having no eval at all, because a green number talks you out of skepticism. I'm going to amend that section and credit this.
On #7 (measure each layer) — the loss/win asymmetry is the part I genuinely underweighted. "Four points better" hiding twenty-five better against five worse versus fourteen better against thirty worse is exactly the kind of thing an average is designed to launder. Same delta, opposite meaning, and only one of them ships. Counting questions moved instead of reporting percentage points is a better default than what I wrote, and your 63%-on-dev / 51%-on-prod flip is the cleanest possible argument for it — same code, same metric, opposite sign, and the second corpus was the real one. The methodology bug wasn't the change; it was trusting the corpus you developed against.
On #4 — your memory test ("does a fact learned in session N come back in N+1 without help?") is narrower and better than my framing, and the 49%-overwritten number makes the "actively hurts" case concrete in a way my prose didn't. A store with no way to mark a version superseded isn't neutral overhead — it's a confident wrong answer competing on relevance with the right one. That's the whole failure mode in one sentence.
And your closing confession is the real lesson buried in all of this: #6 is why you got away with #4. A weak eval doesn't just fail to catch an unnecessary layer — it issues the layer a certificate. The bad memory store survived because the smoke test blessed it. That's the causal chain I under-drew in the post, and it's the most important thing in this thread.
Thanks for this. Genuinely the most useful comment I've gotten on anything I've written.
"A weak eval doesn't just fail to catch an unnecessary layer - it issues the layer a certificate" is better than anything I gave you, and I'm taking it.
One thing that makes it worse, and I'd rather hand it over than keep it: that certificate had a signature.
My 92.3% was not a weak measurement. It was a hardcoded string in the server's own metrics output, with a comment above it reading "CI-defended" and a unit test asserting the string was present. So the chain was: a number nothing computed → a test guarding the sentence that contained it → a green check → a layer that felt earned.
A number the server does not compute cannot fall. It isn't a bad eval, it's a painting of an eval, and it will hold up under exactly the kind of review that catches real regressions - because nothing about it is inconsistent. I only found it because someone in a comment thread asked for a harsher metric and I re-ran the command instead of quoting myself.
So the version I'd add under your certificate line: check whether your eval number is computed at request time or stored somewhere. If it can't move, it isn't measuring - and every layer it approved is unaudited.
Thanks for taking the two corrections in public. That's rarer than the post.
Spot on. The hardest part right now isn't building the architectural "cathedral"—it's having the discipline to know where to draw the line between a POC, an MVP, and a matured product.
Most teams jump straight to vector DBs and multi-agent graphs to solve problems a single prompt and simple retrieval could handle in an MVP. Architecture should be earned by failing simple evals, not anticipated by default. Great reminder to keep it lean until the limits actually force your hand!
"Architecture should be earned by failing simple evals, not anticipated by default" — that's better than anything in my post. Might have to steal it (with credit).
The POC/MVP/matured-product framing is exactly right too. So much complexity comes from building the matured-product version while you're still at POC, before you've earned a single one of those layers. Discipline really is the hard part — adding is easy and fun, restraint is the skill nobody tweets about. Thanks for the sharp addition.