The Case Against Your Own Agent Stack: What Contrarians Get Right This Year

Scroll through anything published about AI agents this year and you’ll notice a pattern before you notice any individual argument. Almost everything is additive. Add a memory layer. Add a graph. Add an orchestration framework. Add a retrieval pipeline with three reranking stages. Add more agents to supervise the agents you already have. The implicit assumption underneath nearly all of it is that sophistication is progress, that if your system isn’t doing something more elaborate than it was six months ago, you’re falling behind.

I’ve built pieces of that stack myself, and I’ve defended some of those choices in earlier posts. So when I started running into a small cluster of pieces this year that argued the opposite, that the sophisticated thing you added probably didn’t earn its place, I paid closer attention than I normally would to takes with “you don’t need this” in the title. Most contrarian tech writing is just hype in reverse, a different flavor of confident and just as light on evidence. These three weren’t that. Each one is backed by something you can actually check: a benchmark result, a structural proof, or a plain description of how someone’s day actually goes. That’s the bar I wanted to hold this against, and it’s the bar I’m going to hold my own stack against for the rest of this piece.

Three cases. Each one is a place where the field reached for more structure when the actual problem was somewhere more boring.

Case one: your agent probably doesn’t need a vector database

I’ve shipped agent memory with a vector store more than once, and the decision usually gets made in about thirty seconds. Someone says “the agent needs to remember things across sessions,” and the reflex answer is: embed it, store it, retrieve by similarity. It’s the default because it’s what every tutorial does, not because anyone benchmarked it against the alternative first.

That’s the part a piece I read this year, Anubhav’s “Your AI Agent Doesn’t Need a Vector Database,” actually went and checked. The claim that stuck with me is the LoCoMo benchmark result: a baseline built from nothing more than a folder of text files and grep outscored a set of funded, purpose-built memory products on their own benchmark. Not a toy comparison rigged to make a point. The sophisticated systems, the ones with embeddings, similarity search, and in some cases graph-based memory structures on top, lost to something you could build in an afternoon.

Once I sat with that, the reasons stopped being surprising and started being obvious in hindsight. Vector similarity is a retrieval mechanism, not a reasoning mechanism. It’s good at “find me text that’s semantically close to this query.” It’s bad at the things memory actually needs, which are things like knowing that a fact from three weeks ago was superseded by a fact from yesterday, or that two stored memories directly contradict each other and one of them needs to win. A vector index has no concept of time and no concept of a correction. It just returns whatever’s closest in embedding space and leaves the model to sort out that two of the top five results disagree with each other.

There’s also a mismatch that’s easy to miss because it’s not about capability, it’s about fluency. Language models have spent enormous amounts of training on operating on files: reading them, searching them, editing them, listing directories, following import graphs. That’s not incidental, it’s the substrate most of their training data is made of. A grep-and-file-read loop plays directly to that fluency. A proprietary vector database's query API is a tool the model has to learn to use well within your specific context window, with no comparable depth of prior exposure to lean on. You're trading a skill the model already has for one it has to be taught on the fly, and then paying embedding and retrieval latency for the privilege.

Here’s roughly how I’d now lay out the decision, after actually thinking it through instead of defaulting to it:

WHEN A FILESYSTEM + GREP BASELINE IS ENOUGH WHEN YOU ACTUALLY NEED A VECTOR STORE
------------------------------------------------ ------------------------------------------------
Single-agent or small-team memory Retrieval across a corpus too large to
                                                    fit or scan in context at all
Facts that change over time and need Cross-document semantic search where
correction, not just accumulation keyword overlap is genuinely weak
Memory the model itself writes, Centralized memory shared by many agents
manages, and re-reads in its own loop that needs access control and auditing
Debuggable state, plain text you can Multi-hop or relational reasoning across
open, diff, and edit by hand thousands of entities where similarity
                                                     search is doing real narrowing work

The write/manage/read loop described in that piece is worth being concrete about, because “just use files” can sound hand-wavy until you see it as code. Here’s a version I’d actually run, no hosted service required:

import os
import subprocess
from datetime import datetime
MEMORY_DIR = "agent_memory"
def write_memory(topic: str, content: str) -> str:
    os.makedirs(MEMORY_DIR, exist_ok=True)
    path = os.path.join(MEMORY_DIR, f"{topic}.md")
    timestamp = datetime.utcnow().isoformat()
    with open(path, "a", encoding="utf-8") as f:
        f.write(f"\n## {timestamp}\n{content}\n")
    return path
def recall(query: str) -> str:
    # ripgrep if you have it, grep -r works fine too
    result = subprocess.run(
        ["rg", "-i", "-C", "2", query, MEMORY_DIR],
        capture_output=True, text=True
    )
    return result.stdout or "no matches"
def list_topics() -> list[str]:
    if not os.path.isdir(MEMORY_DIR):
        return []
    return [f[:-3] for f in os.listdir(MEMORY_DIR) if f.endswith(".md")]

Wire those three functions in as tools, let the model decide when to write, when to search, and when to just read a whole topic file into context because it’s short enough to fit, and you have a memory system with no embedding cost, no vector database to run or pay for, and state you can open in a text editor when something goes wrong. If you outgrow it, you’ll know exactly why, because you’ll have a specific failure: a corpus too big to grep through fast enough, or a multi-hop question the keyword search genuinely can’t answer. That’s a much better reason to add a vector store than “the tutorial did it this way.”

I don’t think this is an argument against vector databases existing. It’s an argument against reaching for one before you’ve benchmarked the boring baseline. Full-context and filesystem-plus-grep first, paid memory product only if it clears that bar by enough to justify the cost and the loss of debuggability. Most agent memory doesn’t need to clear that bar. Mine mostly hasn’t.

Case two: hypergraphs won’t save your RAG system either

If vector databases are last year’s default, graph-based RAG is this year’s, and hypergraphs are the version of it that shows up when a team decides the ordinary knowledge graph still isn’t expressive enough. The pitch is intuitive: a normal graph edge connects exactly two nodes, but a lot of real facts involve more than two participants. “Alice approved Bob’s expense report for the Q3 marketing budget on behalf of the finance team” has five participants tangled into one fact. Force that into pairwise edges and you either lose the fact that it’s one atomic event or you scatter it across a handful of binary edges that have to be reassembled at query time. A hyperedge, which can connect any number of nodes at once, seems like the structurally honest way to represent that. So teams build hypergraph RAG systems on the assumption that this honesty pays for itself in retrieval quality.

A piece I read this year by a writer going by Dustin, “Hypergraphs Won’t Make Your RAG System Better. Here’s What They Actually Change,” went and checked that assumption against the actual implementation of a hypergraph RAG paper rather than against its abstract. The finding was almost funny: HyperGraphRAG, a system explicitly built to argue for native hyperedges, stores its data in an ordinary graph database using binary edges under the hood. And the paper’s own authors show that this encoding, converting each hyperedge into a small cluster of binary edges around a reified node representing the event, loses nothing. No information disappears in the conversion. The “structurally honest” representation and the “boring” one are recoverable from each other exactly.

That’s not a minor implementation detail, it’s the whole argument. If a native hyperedge and a reified binary-edge cluster encode the identical incidence structure, and one is trivially reconstructable from the other, then the choice between them isn’t a modeling decision with consequences, it’s a storage format decision. A commenter on that piece, Felix Anderson, put the underlying math about as tightly as I’ve seen it stated: a hyperedge and a role-reified binary graph are the same incidence structure, and hypertree width only shifts by a constant under bounded arity. Hypertree width is the complexity measure that actually governs how expensive a query is to answer, not the number of hops you have to traverse and not how many participants got jammed into a single edge. If converting your representation only moves that number by a constant factor, and your facts have a bounded number of participants each (which almost all real-world facts do; five people in an expense approval, not five thousand), then you’ve spent real engineering effort buying a change that doesn’t touch the thing that actually determines query cost.

I want to be fair to why this trips people up, because I would have made the same mistake before reading this. Hop count is easy to reason about intuitively; more nodes between question and answer feels like it should mean a harder query. Hypertree width is not intuitive at all, it’s a measure from the theory of constraint satisfaction and query complexity, and it behaves differently from hop count in ways that aren’t visible unless you go looking for them. It’s entirely possible to add structural sophistication that reduces hop count for a specific example query while doing nothing, or even something mildly negative, to the complexity class the query actually belongs to. A hypergraph paper’s own worked examples can look great and still not tell you anything about the metric that governs the general case.

Here’s the comparison I’d actually want in front of me before choosing between a plain graph, a reified graph, and a native hypergraph store:

REPRESENTATION WHAT IT ADDS WHAT IT ACTUALLY CHANGES
--------------------- ------------------------------- --------------------------------
Plain binary graph Simplest to build and query Baseline; loses atomicity of
                         with standard graph tooling multi-participant facts
Reified binary graph Recovers atomicity via an Same incidence structure as a
(event node + roles) explicit "event" node hyperedge; hypertree width
                                                            shifts by a constant only
Native hypergraph Hyperedges as first-class No reduction in query
store objects, arguably cleaner complexity class over a
                         to write against reified graph; new storage
                                                             engine to run and maintain

None of this means graph structure is useless for RAG. Multi-hop relational retrieval genuinely benefits from graph structure over flat vector search, that part isn’t in dispute. What’s in dispute is the extra jump from ordinary graph to hypergraph, and the honest answer, once you look at the actual proof instead of the pitch, is that the jump buys you cleaner-looking data modeling and costs you a new kind of infrastructure to operate, without moving the number that determines whether your queries are fast or slow. If your retrieval is struggling, the fix that has evidence behind it is usually a better graph construction process or a better retrieval strategy over the graph you already have, not a fancier edge type.

Case three: AI can’t do the job because the job isn’t the code

The first two cases are about retrieval architecture, which is comfortably in my usual territory. The third one made me uncomfortable in a different way, because it’s not about a tool choice, it’s about what a senior engineering job actually consists of, and I recognized my own team’s shape in it more than I expected to.

Patrick Koss, writing as a tech lead running a five-person engineering team inside a company north of a thousand people, opens his piece “AI can’t do 95% of my job (and i’m a software engineer)” with a claim that sounds almost like a concession before it’s actually an argument: coding is the part of his job he spends the least time on, by a mile. His team operates on the “you build it, you run it” principle, so on-call for the systems his team owns sits with his own engineers, not a separate operations org that gets to treat production as someone else’s problem. His day starts at 8:30am with pull request review, and the code coming in from his engineers, most of it written with AI agents doing a large share of the typing, is genuinely higher quality than what he was reviewing two years ago. He’s not disputing that AI writes good code now. He’s granting it fully and then pointing out that it barely moves the needle on his actual job.

That’s the part worth sitting with, because it cuts against an assumption a lot of agent-stack thinking quietly makes, including a lot of my own thinking: that capability determines automation. If the model can write correct code, the reasoning goes, then writing code stops being work someone has to do, and the fraction of “the job” that gets automated tracks the fraction of the job that involved producing code. Koss’s argument is that this equation was already wrong before AI showed up, and AI just makes the error easier to see. A tech lead’s job was never mostly code production. It was always mostly coordination: deciding what gets built and in what order, negotiating scope with people who have competing priorities, reviewing and vouching for other people’s decisions, carrying the pager, mentoring engineers who are earlier in their careers than he is, translating between what a stakeholder asked for and what the system can actually support without falling over. None of that is a coding task with extra steps. It’s an organizational and interpersonal task that happens to produce code as one of its outputs, and a very automatable one at that, sitting inside a much larger set of tasks that aren’t automatable in the same way because they’re not fundamentally about producing artifacts, they’re about producing agreement, tradeoffs, and accountability among people.

I think this is the most underrated point in agent discourse this year, more than any specific benchmark result, because it explains why “the model got dramatically better at coding benchmarks” and “my job got dramatically easier” haven’t actually tracked each other for a lot of senior engineers, even ones using these tools heavily and getting real value from them. SWE-bench scores climbing from single digits to the seventies over a couple of years is a real, large capability jump. It just doesn’t automatically translate into “70% less of my job,” because the job was never 70% code production to begin with, especially not once you’re senior enough that your job includes owning the on-call rotation and the roadmap alongside the pull requests.

The honest caveat here is that this argument generalizes less cleanly than the first two. Vector databases and hypergraphs are technical claims you can, in principle, go verify against a benchmark or a proof, and I did. “What fraction of a senior engineer’s job is coordination versus code” is going to vary by company size, by team maturity, by how much of the organizational overhead is dysfunctional versus load-bearing, and by how senior the specific person is. A five-person team inside a thousand-person company with a “you build it, you run it” policy is a specific shape of job, not the shape of every engineering job. But the direction of the correction is one I think holds broadly: agent capability is a ceiling on how much of the code-production slice of a job could theoretically be automated, and it says close to nothing about how much of the coordination slice can be, because that slice was never bottlenecked on someone’s typing speed or code quality in the first place.

The throughline

Lay these three next to each other and the pattern isn’t really about vector databases, hypergraphs, or AI capability specifically. It’s about where each field assumed the bottleneck was versus where it actually was.

CASE WHERE COMPLEXITY WAS ADDED WHERE THE REAL BOTTLENECK WAS
------------------ ---------------------------------- --------------------------------
Agent memory Vector embeddings, similarity Whether the model can use a
                     search, sometimes a graph layer retrieval method it already
                     on top of that has deep fluency with
RAG structure Native hyperedges, a new The complexity class governing
                      storage engine, more query cost, which the fancier
                      elaborate graph modeling structure barely touches
"How much of the An assumption that model Whether the job was ever
job gets automated" capability alone predicts mostly about the thing the
                       the automatable fraction model is good at

In every case, the sophisticated option wasn’t wrong to exist. Vector databases have real use cases, hypergraphs might genuinely help somewhere I haven’t found yet, and AI agents are removing real toil from real engineering jobs, Koss says as much himself. What was wrong was skipping the step where you check whether the sophistication is addressing the actual bottleneck, versus addressing the bottleneck that’s easiest to build an elaborate solution for.

That’s a pattern I recognize from well outside agent engineering too. It’s easier to add a layer than to question whether the boring baseline was ever properly measured against it. A new abstraction is legible progress, something you can point to and say “we upgraded.” Checking whether grep already does the job, or whether your query complexity actually improved, or whether the thing eating your week was ever the thing you assumed it was, that's slower and less satisfying, and it sometimes tells you to stop building.

Here’s the short version of the checklist I’m actually going to run against my own stack before I add the next layer to it:

1. Have I benchmarked the boring baseline, not just assumed it loses?
   (full-context, grep, a plain graph, a human doing the coordination)
2. Does the new structure change the metric that actually governs cost
   or quality, or does it just look more sophisticated on a diagram?
3. Am I reaching for this because a benchmark or proof told me to,
   or because it's what the tutorials and the funded products default to?
4. If I strip this layer back out, what specifically breaks?
   If I can't name it precisely, I probably don't need the layer yet.
5. Am I solving the bottleneck I actually have, or the bottleneck
   that's most interesting to build a sophisticated solution for?

None of this is an argument for doing less engineering. It’s an argument for spending the engineering effort on confirming where the bottleneck actually lives before you build the elaborate thing that assumes you already know. The three pieces that stuck with me this year weren’t contrarian for the sake of it. They were contrarian because they did the unglamorous work of checking, and the checking didn’t agree with the default. That’s a much higher bar than “hot take with a strong headline,” and it’s the bar I’m trying to hold my own stack to from here.

Tags: ai-agents, rag, vector-database, software-engineering, llm, knowledge-graphs, ai-skepticism