DEV Community: Siddharth Bhalsod

Designers Are Shipping Code. Who Owns What Breaks?

Siddharth Bhalsod — Mon, 17 Aug 2026 04:48:20 +0000

In Figma Make, a designer can describe a checkout flow in plain English and watch working code appear next to the canvas. There is no handoff. No pull request waiting on an engineer who has never seen the layout. Just a prompt, a preview, and a publish button.

This is not hypothetical. Figma made Figma Make a core part of its platform in 2026, and its own research says most designers who lean into it believe it improves the quality of their work, not just the speed. The debate over whether designers should learn to code has run for a decade. That debate just became irrelevant. Not because anyone won it, but because the tooling made the question moot.

Somebody is now responsible for code that a designer typed one sentence to generate. Almost nobody has decided who that is.

The Old Debate Was About Syntax

The classic version of "should designers code" is a literacy argument: can a designer read a component tree, write a media query, contribute a small pull request. It stayed a literacy argument because the tools kept design and code in separate systems, with an engineer standing at the border between them.

Figma Make and the wave of tools chasing the same wedge, from v0 to Lovable to Rocket.new, didn't resolve that argument. They removed the border. A designer no longer needs to read code to produce it. The prompt is the interface. What ships isn't "a designer who learned React." It's functional code with no author who fully understands it, published by someone whose job was never system architecture.

That's a different problem than the one the old debate was having. It's the one the industry actually walked into, one release at a time, without anyone signing off on the tradeoff.

Closing the Distance Doesn't Close the Gap

The distance between having an idea and having running code has collapsed. The gap between code that runs and code that's sound hasn't moved at all.

LogRocket's independent review of Figma Make in 2026 put it plainly. The tool is genuinely useful for iteration and user testing, but the code it outputs is not production-ready, and it isn't accessible, semantic, or clean. Set that next to Figma's own survey finding that most designers who increased their AI usage believe it improved their output quality. Both can be true at once. A designer without an engineering background often can't tell finished-looking code from finished code, because the failure modes, a missing aria label, an unhandled edge case, a race condition in a state update, don't show up in the preview. The preview always looks done. That's the entire design goal of a prototyping tool.

When output that a tool markets as a prototyping aid gets promoted straight into production without anyone deciding to promote it, the organization has quietly redefined what counts as ready.

The Data on What Ships

This isn't a hypothetical risk. It's been measured, and the number isn't small.

A large-scale study of AI-authored commits in the wild tracked over 300,000 commits across more than 6,000 repositories. It found that roughly 15 percent of AI commits introduced at least one issue, and close to 23 percent of the catalogued issues were still unresolved at the latest snapshot of the repository. Separately, security vendor Apiiro reported a sharp jump in privilege escalation paths and architectural design flaws in AI-generated code. That figure is vendor research, not an independently replicated finding, and should be read that way until someone outside Apiiro reproduces it. Directionally, both point the same way.

That commit-level data came mostly from professional engineers using AI copilots inside teams that still had a review process. Tools like Figma Make are built for people publishing without that layer at all. If issue rates run this high with a reviewer in the loop, removing the reviewer doesn't make the code better. It just makes the defects someone else's problem to discover later, in production.

Where the Review Step Used to Be

The diagram above makes the shape of the problem visible. The traditional path from design to production had a built-in checkpoint: a second, technically trained person reviewing the code before it reached users. The AI-native path removes that step, not through a policy decision by anyone, but because the tool made the shortcut the default. Responsibility didn't disappear when the checkpoint did. It moved to whoever hit publish, and in a growing number of organizations, that person is a designer who was never asked to carry it.

The PM Who Signs Off Anyway

This isn't really a story about designers. It's a story about organizations approving a production pathway before deciding who owns its failures. A PM greenlights a Figma Make prototype for a real feature because it worked in the demo and the quarter needs velocity. A founder tells the design team to just build it because engineering is backlogged three sprints deep. Neither decision is unreasonable in isolation. Together, they quietly hand implementation risk to the person in the room who was never asked to carry it, without giving them the review process that used to catch what they'd miss.

The framing where designers who don't understand system architecture are "becoming obsolete" gets this backwards. The people actually at risk aren't designers shipping with AI tools. It's organizations that let a tool's shipping speed outrun their own process for deciding what's safe to ship. A nail gun lets anyone frame a wall fast. It doesn't make anyone a structural engineer, and the wall doesn't care who fired the nails.

"Should designers code" was never really the question worth asking. It was a proxy for a harder one most companies have avoided asking out loud: when a tool lets anyone generate functional software, who is accountable for what it does wrong. The market didn't answer that question. It just made it possible to avoid asking it for another quarter, until the checkout flow a designer shipped through a prompt corrupts an order, and someone has to explain why nobody reviewed it.

#mcp #security

Siddharth Bhalsod — Wed, 12 Aug 2026 06:42:12 +0000

Siddharth Bhalsod

Aug 12

MCP Has a Supply Chain Problem, and Patching One Package Won't Fix It

#mcp #supplychain #claude #security

5 min read

MCP Has a Supply Chain Problem, and Patching One Package Won't Fix It

Siddharth Bhalsod — Wed, 12 Aug 2026 06:41:48 +0000

In July 2025, a JFrog researcher found a way to get full command execution on a developer's machine by doing nothing more than connecting to the wrong server. The flaw lived inside mcp-remote, a small proxy tool that lets AI clients like Claude Desktop and Cursor talk to remote Model Context Protocol servers. It scored 9.6 out of 10 for severity. JFrog called it the first known case of full remote code execution achieved in the wild through an untrusted MCP connection.

The maintainer patched it within days. The npm package moved to version 0.1.16. Everyone moved on.

Nine months later, a different research team found the same category of flaw baked into the official MCP SDKs themselves, the code every other MCP server is built on top of. Not a rogue package this time. The reference implementation, in Python, TypeScript, Java, and Rust. Anthropic reviewed the finding and declined to change the architecture, calling the behavior expected.

That's the actual story. Not one vulnerable package that got fixed. A pattern that keeps resurfacing because of how the protocol treats trust.

What Most Teams Think an MCP Server Is

Most engineering teams treat an MCP server the way they treat a VS Code extension: something you add from a marketplace, that runs inside some kind of sandbox, that's safe by default because enough other people already installed it. That mental model is wrong, and the gap between what people assume and what actually happens is where the damage gets done.

An MCP server is a live process with the ability to execute tool calls on your behalf, often with the same file system, network, and shell access as the developer running it. It receives responses from wherever it's configured to look, and it treats parts of those responses as instructions to act on, not just data to read. This isn't a bug. It's the entire premise of the protocol: the model asks a question, the tool answers, and something downstream acts on that answer immediately.

The failure mode has a name. It's called command injection: when a program doesn't clearly separate the data it's processing from the instructions it executes, an attacker who controls the data can make the program run their command instead. That's precisely what happened in mcp-remote. A malicious server's authorization response contained a crafted URL. The client didn't just read it. It passed that value straight into a system command.

One Root Cause, Five Different Codebases

mcp-remote wasn't an outlier. Look at the disclosures from the past year and the same failure pattern shows up in unrelated projects, built by unrelated teams, shipped for unrelated purposes.

In April 2026, OX Security went further and found the same root cause sitting inside the official MCP SDKs, the libraries every one of those projects and thousands of others import directly. Their research, titled "The Mother of All AI Supply Chains," put a number on the exposure: over 150 million downloads, more than 7,000 publicly accessible servers, and an estimated 200,000 vulnerable instances in total. They identified four distinct ways to trigger it, including a zero-click path through prompt injection in IDEs like Windsurf and Cursor that requires no user interaction at all.

Anthropic's response was to leave the architecture as is. Not because the finding was wrong, but because fixing it at the protocol level would mean changing behavior that thousands of downstream projects now depend on. That's the part worth sitting with. This isn't a queue of unpatched CVEs waiting for attention. It's a design decision the ecosystem now has to build around, because the standard's steward isn't going to build the fix in for them.

MCP Servers Skip the Scrutiny npm Packages Get

Software supply chain security spent the last decade building tooling most engineers now take for granted. Lockfiles. Software composition analysis. Provenance attestation. A team adding a new npm dependency to a production service can, at minimum, run a scanner against it before merging.

MCP servers largely skip all of that, despite running with equal or greater privilege. Adding one to Claude Desktop or Cursor takes about as much scrutiny as installing a browser extension: paste a config block, restart, done. Few teams pin versions. Fewer read the source of a server before granting it filesystem or shell access.

BlueRock Security scanned more than 7,000 public MCP servers in early 2026 and found 36.7 percent potentially exposed to server-side request forgery, a flaw where an attacker tricks the server into making requests to internal systems it shouldn't reach. In one proof of concept against Microsoft's MarkItDown MCP server, researchers used exactly that path to pull AWS credentials straight from an EC2 instance's metadata endpoint. A misconfigured tool integration became a direct line into cloud infrastructure, and nothing about the exploit required a zero-day.

The Registry Doesn't Check Who's Publishing

Not every MCP compromise requires a technical vulnerability at all. In February 2026, Straiker's STAR Labs documented an operation that skipped code flaws entirely. A group calling itself SmartLoader spent three months building five fake GitHub developer personas, cross-forking each other's repositories to look like an active community, before submitting a trojanized version of a popular Oura Ring MCP server to a public registry.

The fork was functionally identical to the legitimate server. It passed as normal for months. Once installed, it quietly deployed an infostealer that pulled browser passwords, cloud session tokens, Discord credentials, SSH keys, and cryptocurrency wallet files, disguising its persistence mechanism as a Windows audio process. No CVE, no patch, no vendor to notify. Just patience and a registry with no moderation standing between a developer and a production credential store.

Why Teams Keep Getting Caught by This

The pattern behind all three problems is the same. MCP's entire pitch to engineering teams is speed: connect an agent to a new tool in an afternoon instead of waiting on a procurement cycle or a security review. That speed is real, and it's why adoption moved as fast as it did. It's also why the review step that used to catch this class of risk got skipped almost everywhere.

The GitHub Copilot remote code execution disclosed in August 2025 shows the same human factor from a different angle. An attacker didn't need to breach a network. They needed a config file, .vscode/settings.json, to get edited automatically by an agent nobody was watching closely enough to notice. Most teams don't have a mental category for "a JSON file is now an attack surface." They're building one in real time, usually after something has already gone wrong.

MCP made connecting an agent to a hundred tools take an afternoon instead of a quarter. That part of the pitch is true, and it's why teams adopted it as fast as they did. What the pitch leaves out is that revoking trust from a bad connection now takes exactly as little effort as granting it in the first place, which means most teams haven't built a process for doing either one deliberately. The protocol didn't create bad judgment. It just removed the friction that used to catch it before it shipped.

Vibe Engineering Won't Save You From Slowing Down

Siddharth Bhalsod — Mon, 10 Aug 2026 07:19:34 +0000

Sixteen senior open-source developers agreed to something most engineers would find uncomfortable: hand over their next few weeks of real work to a coin flip. Half their tasks would be done with AI tools. Half without. Every minute would be logged.

These weren't hobbyists. They averaged five years on the exact codebases they'd be working in, projects with over 22,000 GitHub stars and years of accumulated context in their heads. If anyone was positioned to use AI well, it was this group.

Before the study started, they predicted AI would cut their task time by 24 percent. After finishing, even having lived through the actual work, they still believed AI had saved them 20 percent. The stopwatch disagreed. They were 19 percent slower.

That's not a story about junior developers leaning on autocomplete they don't understand. It's a story about the exact profile the term "vibe engineering" was built to describe.

The Distinction Everyone Quotes, Few Apply Correctly

In late 2025, developer Simon Willison drew a line that spread fast through engineering circles. Vibe coding, he argued, is building software by prompting an AI and shipping whatever comes out, without reading it, without understanding it, without caring whether it's right. Vibe engineering is the opposite end of the same spectrum: experienced developers using tools like Cursor, Claude Code, or Devin to accelerate implementation, while staying fully accountable through testing, planning, code review, and disciplined source control.

The distinction is useful. The way most teams apply it isn't.

Somewhere between Willison's original post and a hundred LinkedIn recaps, "vibe engineering" quietly turned into shorthand for "an experienced engineer using AI." Seniority became the qualifying credential. If you'd been coding for a decade, the thinking goes, you were doing it responsibly by default. That's not what Willison described. He described a set of practices. Practices are something you do on a given task, not a status you carry into every one.

That gap between what the term means and how it gets used is where the METR study becomes relevant, and where most coverage of it gets the application wrong.

What the Study Actually Measured

The trial in question, run by AI research group METR between February and June 2025, was a randomized controlled trial, the same experimental design used to test drugs. Researchers recruited 16 experienced developers, gave them 246 real issues from their own repositories, and randomly assigned each issue to either allow or forbid AI tool use. When allowed, most developers reached for Cursor Pro paired with Claude 3.5 or 3.7 Sonnet, the frontier models at the time.

The gap between prediction, perception, and reality is the actual finding here, not the 19 percent on its own. These developers didn't just guess wrong going in. They kept believing the wrong thing after the work was already done. That's a harder problem than a bad estimate. It suggests experience doesn't automatically produce accurate self-assessment of AI's effect on your own output.

Access Is Not the Same as Practice

Here's where the study gets misapplied. METR's "AI-allowed" condition meant developers could use AI however they wanted. Nobody was required to write tests first, plan before prompting, or review every generated line the way Willison's vibe engineering describes. The trial measured what happens when experienced developers get access to AI tools. It did not measure what happens when they follow a specific disciplined practice around that access.

Access is not practice.

That distinction matters for anyone leading a team. A policy that amounts to "our engineers are senior, they'll use AI responsibly" is not vibe engineering. It's just access, with a job title attached as a substitute for process. Handing an experienced woodworker a table saw doesn't make the years of training show up automatically. The saw still requires the technique. Without it, an experienced hand can lose a finger just as easily as a beginner, sometimes more easily, because confidence outpaces caution.

The METR developers had the tenure. What the study suggests they didn't reliably have, in that six-month window with those specific tools, was a workflow that turned access into the disciplined outcome Willison's term promises.

The Study That Couldn't Be Repeated

There's a wrinkle here that most articles quoting the 19 percent figure leave out, and it surfaced almost a year later. In February 2026, METR published an update explaining why they were changing how they run this research. They'd started a follow-up experiment in August 2025 with a larger developer pool and newer tools, expecting to track how the effect changed over time. Instead, they found the new data unreliable. A growing number of developers were refusing to participate specifically because they didn't want to work without AI assistance, even temporarily, which skews any comparison between AI-allowed and AI-disallowed conditions.

Read that carefully, because the obstacle wasn't that AI got so good the slowdown vanished. It's that the population willing to go without it shrank enough to break the experiment's design. That's a different and arguably more interesting finding than the original 19 percent, and it's had a fraction of the coverage.

The practical takeaway: the original study is a snapshot of a specific six-month window, specific tools, and a specific group of 16 people. Treating it as a permanent verdict on experienced developers and AI is the same overreach as treating "vibe engineering" as a permanent status once you've hit five years of tenure. Both claims need re-proving under current conditions, not just cited from an older post.

What This Means for the Team You're Actually Running

If you manage engineers, or you're evaluating your own output, the practical question isn't whether your team is vibe coding or vibe engineering. It's whether anyone has actually checked. The developers in the METR study were confident, experienced, and wrong about their own speed, at the same time. That combination is the dangerous one, not inexperience on its own.

Teams that reward "ships fast with AI" in performance reviews without a pre-AI baseline to compare against are running the same experiment METR ran, just without the stopwatch. The senior engineer who says AI made them dramatically faster might be right. Or they might be describing the exact overconfidence sixteen experienced developers displayed under controlled measurement. Without a baseline, there's no way to tell which.

Vibe engineering isn't a rank you earn once and carry forward. It's a claim you have to keep proving, task by task, the same way a junior developer's output gets checked. The tenure doesn't remove that requirement. It just makes people less likely to ask.

Context Engineering Won the Naming War. Almost Nobody Has Built the System.

Siddharth Bhalsod — Mon, 10 Aug 2026 07:13:34 +0000

Ask ten AI teams what they do and most will say context engineering. Two years ago they said prompt engineering. The label changed fast, faster than infrastructure usually does.

Open the codebase behind that label and you'll typically find the same thing you'd have found in 2024: a system prompt, a retrieval call bolted on with string concatenation, and a truncation function that fires when the token count gets uncomfortable. The vocabulary evolved. The system underneath it mostly didn't.

That gap is the real story. It belongs to systems and infrastructure teams, not to whoever happens to write the prompts.

The Term Won. The System Didn't.

Here's the imposter version, the one half a dozen job postings currently list as a core skill: fetch a few chunks from a vector store, drop them above the user's question, ship it. That's retrieval-augmented prompting. It's useful, and it's also not a system, because nothing about it persists, nothing about it degrades gracefully, and nobody owns it once it's deployed.

The real version has three separate jobs running underneath one label, and they rarely live in the same part of the stack. State has to be stored somewhere between requests. Context has to be assembled from multiple sources within a latency budget, on every single call. And whatever gets loaded has to be pruned on a schedule, before stale content quietly degrades the model's output. Most teams that claim context engineering have built exactly zero of these three.

State Has to Live Somewhere

Persistent context state is a storage decision, not a prompt decision, and most teams haven't made it on purpose. They've defaulted into statelessness, then treated the resulting amnesia as a model limitation instead of an architecture choice nobody made deliberately.

Anthropic's memory tool, shipped for the Claude Developer Platform, takes a specific position here. Claude reads and writes files in a memory directory that the developer hosts. The model only issues the read and write instructions. Where those files actually live, local disk, object storage, a database, is entirely the developer's call.

That's a real tradeoff, not a solved problem. File-based storage is simple and auditable, but it doesn't give a team the query flexibility a vector store or relational schema would if they need to search across sessions instead of just replaying one. It also pushes versioning and conflict handling up into the application layer instead of a database engine. Teams that skip this decision by default end up with context that resets every session, then wonder why the agent forgets debugging context from three days earlier.

Assembly Is a Pipeline, Not a Copy-Paste

Getting the right context in front of the model, on every call, is an assembly step with its own latency budget and its own failure modes. It deserves to be treated like any other production service, not like glue code written once in a notebook.

This is what protocols like Model Context Protocol (MCP) are actually solving. MCP standardizes how a model requests data and tool access from external sources, instead of every team hand-rolling its own integration between the model and whatever systems hold the real information. The comparison worth making is an API gateway: one slow backend shouldn't take the whole request down with it, and one slow context source shouldn't either.

Most teams haven't built that resilience. Three upstream calls, a vector store, a ticketing system, an internal wiki, get made synchronously in the request path with no cache and no fallback. It works in the demo, because the demo never hits the slow dependency. It stops working the first time one does, and there's no owner for the timeout that should have caught it.

Pruning Needs to Be a Service, Not a Cron Job Someone Forgot

Deciding what gets cleared from context, and when, has to run continuously in production. It cannot be a decision made once in a design doc and left alone.

Logging went through the same evolution years ago. Print statements were fine for a single developer debugging locally. Production systems needed structured logging, retention policies, and alerting, because the volume and the stakes changed. Context pruning is sitting at the print-statement stage in most organizations right now.

Anthropic's own documentation makes the direction explicit. Context editing clears stale tool results once a conversation crosses a defined threshold. That capability was originally something developers configured client-side through an SDK. Client-side compaction is now deprecated in favor of server-side compaction, which Anthropic recommends because it handles token accounting and summarization automatically instead of asking every developer to rebuild that logic themselves. A truncation function someone wrote eight months ago doesn't know about a new content type or a changed usage pattern. It just quietly breaks.

Nobody Owns This Yet

None of these three jobs has a natural home in most org charts. The AI or ML team treats context as a prompting problem, because that's the layer they can see and measure. Platform engineering doesn't know it's in scope, because nobody told them a context pipeline needs the same on-call rigor as a payments API. Whoever owns data usually owns storage for everything except the one system actually feeding the model in production.

That gap doesn't show up in a demo. It shows up on turn forty of a long-running agent session, when the model references something that should have been pruned three exchanges ago, or forgets something it stored two days earlier because nobody built the retrieval path for old memory files. When it happens, the on-call engineer gets paged with "the agent is being weird," not a clear signal like "assembly exceeded latency budget," because nobody built a dashboard for a system nobody agreed was a system.

Winning the naming war was the easy part. A term spreads because it's useful shorthand, and context engineering is genuinely better shorthand than prompt engineering for what these systems now do. But shorthand raises expectations. Say the phrase out loud in 2026 and the person listening assumes a state store, an assembly pipeline, and a pruning policy exist behind it.

Most of the time, they don't.

It's a liability, sitting quietly in production, waiting for turn forty.

When RAG Is the Wrong Answer for Your LLM System

Siddharth Bhalsod — Mon, 20 Jul 2026 09:31:48 +0000

Ask ten engineering teams how they're grounding their LLM in company knowledge, and nine will answer retrieval before you finish the question. Not because they ran the comparison. Because retrieval is what the tutorials default to, what the frameworks assume, and what nobody has to defend in a design review.

That's not an architecture decision. That's a reflex.

Fine-tuning and long context aren't fallback options for when RAG gets weird in production. They answer a different question. Where does the information your model needs actually live. Get that question wrong, and no amount of chunking strategy or reranker tuning fixes it later.

The Question Everyone Skips

Every running program keeps the data it needs in one of three places. Baked into the binary at compile time. Loaded into memory before the program starts executing. Or fetched from disk the moment something asks for it.

LLM architecture makes the same three choices, just under different names. Fine-tuning bakes information into the model's weights. Long context loads it into working memory for a single request. Retrieval fetches it from an external index on demand. Seen this way, "RAG or fine-tuning" stops looking like a fair fight between two competitors. It's a decision tree missing its first branch, the one nobody asks out loud: does this even belong in the binary at all.

Fine-Tuning Is a Compile Step, Not a Database

Fine-tuning changes what a model does, not what it knows in any durable sense. That distinction gets lost constantly. Teams fine-tune a model on last quarter's product catalog expecting it to "know" the catalog, then wonder why it hallucinates SKUs from two versions ago.

FineTuneBench, a benchmark built specifically to test this, found fine-tuning succeeded at absorbing new factual updates only about 19% of the time. Not because the training ran badly. Because weights aren't a lookup table. They're a compressed, lossy compile step over whatever examples you fed in, and compression is a bad medium for facts that need to be exact and current.

What fine-tuning is genuinely good at is behavior: tone, output format, refusal patterns, domain-specific reasoning style. LoRA and QLoRA lowered the entry cost enough that the old "you need thousands of examples" rule doesn't hold anymore. For classification or format-locking tasks, 200 to 500 curated examples is often sufficient. Together AI offers this as managed infrastructure, tuning and inference in one stack, which suits teams who don't want to own GPU orchestration. Axolotl, the config-driven open-source framework, suits teams who want every knob exposed and version-controlled. Neither platform has a workflow for "add one new fact." That absence is the tell. Fine-tuning assumes you're compiling behavior, not writing to a database.

The strongest commercial case for fine-tuning right now isn't knowledge injection at all. It's distillation: taking a frontier model's behavior on a narrow task and compressing it into a smaller, cheaper open-weight model. That's a real, defensible use of the technique. Using it to keep a knowledge base current is not.

Long Context Is Preloading, Not a Free Lookup

The pitch for long context sounds like it eliminates the whole problem. Gemini 3.1 Pro and Gemini 3 Flash ship a million-token window. Claude's frontier models sit at the same mark. Just paste the whole knowledge base in and skip the architecture entirely.

Here's what that pitch leaves out: a million-token window doesn't mean a million tokens of equally reliable recall. Independent RULER-style evaluations in 2026 still show recall sagging well before models reach their stated ceiling, the same lost-in-the-middle effect that shows up at 50,000 tokens shows up again, just later, at whatever fraction of the window your corpus happens to occupy. And a study comparing retrieval against long context directly (LaRA, arXiv 2502.09977) found RAG still beating long-context approaches by 6 to 38% for smaller models, even at a relatively modest 128,000-token window. The advantage long context has is real, but it's concentrated in frontier-scale models with the budget to run them, not a universal upgrade.

This isn't a contradiction of the context rot argument from the token budgeting piece earlier in this series. It's the same mechanism, wearing a different hat. A window you fill because your corpus genuinely fits comfortably under where recall degrades is a legitimate design choice. A window you fill because the vendor advertises a million tokens and it's easier than building an index is the same context rot, just with a bigger invoice attached, since every one of those tokens gets reprocessed on every single call.

Retrieval's Real Advantage Isn't Quality. It's Decoupling.

Retrieval doesn't win because it's smarter than the other two. It wins because it's the only one of the three where the size of your knowledge base is decoupled from the cost of using the model. Add a thousand new documents to a RAG index and nothing retrains, nothing reprocesses. Add them to a fine-tuned model or a long-context prompt and you're paying, in training compute or in reprocessed tokens, for every addition.

That's why retrieval is the correct default for information that changes faster than monthly, which is the guidance most vendor documentation converges on: retrieval for grounding in facts that move, fine-tuning for behavior that should hold steady. It has nothing to do with retrieval producing better answers in the abstract. A well-tuned model with no retrieval will out-answer a badly-chunked RAG pipeline every time.

The flip side rarely gets said out loud: if your knowledge base is small and mostly static, the "fetch" step retrieval requires is overhead you're paying with no corresponding benefit. An index, a retriever, and a reranker are three new systems to operate and three new failure modes to debug, for a corpus that would have fit inside a single prompt.

Why Teams Default to Retrieval Anyway

The technical case above is not why most teams end up with RAG. The organizational case is simpler and less comfortable to say out loud. Retrieval lets a team ship without anyone on the call defending a tradeoff.

Fine-tuning requires someone to own a training pipeline, which means admitting the team needs ML infrastructure it may not have budgeted for. Long context requires someone to accept, explicitly, that recall degrades as the prompt grows, which means putting a number on an accuracy tradeoff in a meeting where nobody wants to be the one who signed off on it. Retrieval requires neither confession. The tradeoffs are still there, buried in chunking decisions and reranker configs, but they don't have to be said in the room where the architecture gets approved.

That's an incentive structure, not a technical argument, and it's exactly why the choice gets made once, in a sprint, by whoever set up the vector database first, and then never gets revisited even after the corpus and the query patterns have changed underneath it.

The three approaches aren't a ranked list with retrieval on top and the others as consolation prizes. They're three different places to keep the same information, and the only question worth asking is how often the underlying facts change, how large a model you can afford to run, and whether you can absorb reprocessing cost on every call. A default chosen once, in a sprint, and never revisited is still an architecture decision. It's just an accidental one, made by whoever happened to be in the room.

Retrieval as a First-Class Context Operation, Not a RAG Afterthought

Siddharth Bhalsod — Mon, 13 Jul 2026 16:02:47 +0000

A team ships retrieval-augmented generation in a single sprint. A vector database, an embedding call, a similarity search wired into the existing chat endpoint. The demo works. Support tickets get answered with citations attached, and everyone moves on to the next feature.

Six weeks later, the wrong document keeps winning. Not because the model got worse. Because nobody decided, on purpose, how the corpus should be split, searched, or ranked before the first real user typed a question. Those decisions got made anyway, by whatever the tutorial's defaults happened to be, and now they're load-bearing.

That's what "RAG as an afterthought" actually looks like in production. Not a missing feature. A set of irreversible decisions nobody remembers making.

The Feature That Isn't a Feature
Most teams reach for retrieval the way they'd reach for a caching layer: bolt it in front of the model, ship it, tune it later if something looks off. Early retrieval-augmented deployments really were built this way, added mainly to cut down on hallucinated answers rather than as a core design choice. That framing has aged badly. As retrieval-heavy systems move into workflows where a wrong answer costs money or triggers an audit, the technique stops behaving like a feature you can swap and starts behaving like a decision you already made, whether you meant to or not.

Article 3 argued you should start infrastructure with the data layer, not the model. This is a narrower, sharper version of that same instinct. Inside the data layer itself, retrieval isn't a call you make. It's a schema you commit to. Chunking, the process of splitting source documents into retrievable units, is the clearest example of what that means in practice. Decide it once, at design time, and every future query lives inside that decision, whether the team realizes it or not.

The Decisions That Are Actually a Schema
Retrieval quality problems get blamed on the model. Most of the time the real cause was set months earlier, in how the corpus got chunked and indexed. Fixed-size splitting, cutting every document into equal token counts, is the fastest way to get a RAG demo working and the slowest way to get it right for anything beyond a demo. Structure-aware splitting, breaking on headings or function boundaries, and hierarchical chunking, indexing a small child chunk for precise matching alongside a larger parent chunk for context, both consistently outperform naive fixed-size splitting in production benchmarks. The exact numbers vary enough across studies that no single ranking should be treated as settled, but the direction of the finding holds across most of them.

What the studies do agree on is the tension hierarchical chunking is built to solve. Small chunks retrieve precisely but arrive at the model missing surrounding context. Large chunks preserve context but dilute the similarity signal that made them findable in the first place. Frameworks like LlamaIndex ship hierarchical retrieval as a built-in pattern specifically because enough teams kept rebuilding a version of it by hand that it stopped making sense as custom code.

None of this is a setting anyone flips casually. Changing chunk strategy after launch means re-embedding and reindexing the entire corpus, not editing a config file. That's the schema comparison earning its keep. A database team wouldn't redesign a primary key structure as a Tuesday afternoon task, and a retrieval system's chunking strategy deserves the same caution, because it constrains every query the system will ever be asked to answer.

The Coupling Nobody Names
Chunk size, embedding model, and reranker don't operate independently, even though most architecture diagrams draw them as three separate boxes. Change the embedding model and the ideal chunk size shifts underneath it. Add a reranker and the number of candidates worth pulling in the first retrieval pass changes too. Production teams that treat these as three independent settings tend to discover the coupling the hard way. A routine embedding model upgrade quietly degrades answer quality, and nobody connects it back to a reranker still tuned against the old embeddings.

The fix isn't clever configuration. It's hybrid retrieval, combining dense vector search with a sparse keyword method like BM25 and merging the two ranked lists, paired with a dedicated reranker such as Cohere Rerank or an open cross-encoder like bge-reranker. Hybrid retrieval earns its cost because pure semantic search is weakest exactly where structured data is strongest: part numbers, error codes, proper nouns, the kind of tokens where an exact match beats a fuzzy one. Vector store choice mostly follows from scale rather than preference. pgvector inside an existing Postgres instance covers most teams under five to ten million vectors, and dedicated stores like Qdrant or Weaviate earn their added complexity once filtering and scale genuinely demand it.

This coupling is also why retrieval needs its own evaluation discipline, not one borrowed wholesale from general model evals. Article 4 covered eval systems as a sequencing problem across an entire pipeline. Retrieval needs a narrower version of that same layer: a check that scores whether the retrieved chunks were relevant at all, before anyone measures whether the final generated answer sounded right.

The Fork That Decides the Whole System's Shape
Underneath the tuning details sits a bigger, earlier choice. Naive single-pass retrieval, hybrid retrieval with reranking, or agentic retrieval, where the model itself decides whether to search again, aren't three tiers of the same architecture. They're three different systems, with different cost curves and different failure modes, and picking one after the others are already built means replacing the orchestration layer, not upgrading it.

The cost gap alone should settle most of the decision before a single line of code gets written. Naive retrieval runs at a fraction of a cent per query. Hybrid retrieval with reranking adds a modest per-query cost for a real jump in precision. Agentic retrieval, where the model plans, retrieves, evaluates, and sometimes retrieves again, can run ten to a hundred times the cost of the naive version, because a single user question might trigger several retrieval rounds instead of one.

One documented production overhaul shows the gap between architecture and afterthought directly. A support-knowledge-base deployment moved from naive single-pass retrieval to hybrid search with reranking and query rewriting, and its retrieval precision score moved from roughly half to over four-fifths in the process. That's one case, not a universal constant, but the direction is consistent with the broader research: architecture choice tends to dominate the outcome more than model choice does.

Choosing agentic retrieval also means choosing an orchestration story, not just a smarter loop. The common production pairing in 2026 is a retrieval-focused framework like LlamaIndex handling chunking, hybrid search, and reranking, wired into an orchestration layer like LangGraph managing the decision loop on top of it. Bolt that on after a naive pipeline already has real users, and the migration looks less like adding a feature and more like replacing the foundation while the building stays open.

What Retrieval Architecture Actually Costs Later

None of this stays technical for long. A founder who approves "agentic RAG" without knowing it means a per-query cost that scales with usage, not a flat line item, is going to be surprised by an invoice. A team that can't point to which chunk an answer actually came from is going to struggle the first time a regulator or an enterprise customer asks for an audit trail. A retrieval system nobody owns, tuned once during a hackathon and never revisited, degrades the same way an unmonitored database does. Quietly, until someone notices the answers have been subtly wrong for a while.

The teams that get this right treat retrieval the way they treat their production database: someone owns it, changes to it go through review, and its behavior gets measured on a schedule instead of when a customer complains. The teams that get it wrong treat it as a feature ticket, assigned to whoever happened to be free that sprint.

Nobody schedules a sprint to add a database after the product already has users depending on one shape of data. Retrieval deserves the same instinct. The chunking decision, the embedding model, the index structure: these are schema, not settings. Get them wrong on day one, and the fix six weeks later isn't a config change. It's a migration, with the downtime and risk that word implies for anyone who has run one at 2 a.m.

The team that ships a single similarity search call this sprint isn't ahead of the team that spent two weeks designing a retrieval architecture first. They're just borrowing time from a migration they haven't scheduled yet.

Context Compaction Patterns: When to Summarize, Truncate, or Retrieve

Siddharth Bhalsod — Wed, 08 Jul 2026 08:27:14 +0000

Context Compaction Patterns: When to Summarize, Truncate, or Retrieve

A context window fills up mid-session. Something has to leave. Most systems answer that problem with one move: summarize everything and hope the important parts survive. It's the same instinct that made "just add more context" the reflexive answer to the budgeting problem — a single technique, applied everywhere, regardless of what's actually being thrown out.

Compaction isn't one operation. It's three, and they destroy information in completely different ways. Summarizing keeps the gist and loses the precision. Truncating keeps the recent and drops the old outright. Retrieving loses nothing permanently, it just stops sitting in front of the model until something asks for it again. Pick the wrong one for a given piece of context, and the failure doesn't announce itself as a compaction bug. It shows up as the model forgetting a decision it already made, misreading a file it already read, or confidently inventing an API it saw an hour ago.

The Mistake Hiding Inside "Just Summarize It"

Ask an engineer how their system handles a full context window and the answer is usually one sentence: it summarizes. That's the imposter definition of compaction, and it's wrong in a specific way. Summarization is one tool in a set of three, best suited to exactly one kind of content: information where the gist matters more than the wording, and where there's no cheaper way to get it back later.

Truncation is not a worse version of summarization. It's a different bet entirely, that the old content isn't coming back and doesn't need a lossy stand-in either, because it's either genuinely stale or cheaply re-obtainable from somewhere else. Retrieval is not a fallback for when summarization would lose too much. It's the option that applies whenever the real thing still exists somewhere addressable, and fetching it again costs less than carrying a compressed copy around indefinitely.

Treat all three as interchangeable synonyms for "make it smaller," and a system will compact the wrong things the wrong way. A four-line grep result and a forty-turn architectural discussion do not fail the same way when compressed. Building one pipeline for both is the actual root cause of context management that quietly breaks agent behavior weeks into production, long after anyone remembers the shortcut was taken.

The One Question That Decides Which Technique to Use

There's a single question that sorts almost every piece of discardable context into the right bucket: if this disappeared right now, could you get it back, cheaply, from somewhere else?

If yes, retrieve. The content lives in a file, a database, a vector index, a URL. The model doesn't need to carry it, it needs to know it exists and how to ask for it again. Compressing it into a summary is strictly worse than dropping it and re-fetching the original when it's actually needed, because a summary of a file is a lossy copy of something you already have a lossless copy of.

If no, but the value sits in the narrative rather than the exact wording, summarize. A ninety-minute debugging conversation isn't reconstructable from any external source. Nobody logged the reasoning trail. But the decisions made along the way, the constraints established, the dead ends already ruled out, survive compression into a few paragraphs. The exact phrasing of turn forty doesn't matter. The fact that turn forty ruled out a database migration does.

If no, and the content has already been superseded by something more recent, truncate. An old grep result against a file that's since been edited isn't worth summarizing. It isn't worth a placeholder either. It's just wrong now, and the right move is to let it go rather than spend tokens compressing something that's no longer true.

The Trigger Matters As Much As the Technique

Even the right technique executed at the wrong moment causes damage. Wait until a conversation is nearly out of room before summarizing it, and the summary is built from an already-degraded view: the model is compressing a conversation it's already struggling to reason over cleanly. Run the same operation earlier, well before pressure builds, and the summary comes from a clean, complete picture instead of salvage material.

The same logic holds for the other two techniques. Truncate too early, before content is genuinely stale, and something still in use disappears with no summary to fall back on. Retrieve too late, after the model has already tried to answer from what it half-remembers instead of the real source, and the retrieval fixes an error that already happened instead of preventing it. Category tells you which technique to use. Timing tells you whether that technique does what it's supposed to.

Same System, Three Different Answers

The clearest proof this isn't theoretical is a single coding agent making three different calls inside one session. Claude Code doesn't run one compaction function against everything that gets long. It runs three, for three different problems. Old tool results that are no longer needed, a grep from many turns ago against a file that's since changed, get cleared outright: no summary, no reference kept, because the content is already stale and reconstructing it would mean re-running a command against a file that no longer matches. Large but still-relevant tool output, a long file read, a verbose test run, gets a different treatment: the full content moves to disk, and only a path reference stays in the model's view, pulled back in full if the work touches it again. Conversation history gets a third treatment entirely, because there's no disk copy of a debugging conversation sitting anywhere: when the session nears its limit, the turns themselves get compressed into a structured summary that preserves decisions and task state rather than exact wording.

Three techniques, three categories, inside one product. Cursor draws a similar line somewhere else. Ask it about a codebase and it doesn't truncate the repository or summarize old files to make room, it retrieves. The codebase gets indexed with embeddings ahead of time, and a request pulls back the handful of chunks that are actually relevant through semantic search, the same underlying bet as Claude Code's disk-backed tool outputs: the source is stable and cheaper to fetch again than to carry around compressed.

What Breaks When You Pick Wrong

The failure mode of picking the wrong technique rarely looks like a crash. It looks like a subtly worse answer that's hard to trace back to its cause. Summarize a tool output that needed to be reasoned over exactly, a JSON schema, a diff, an error stack, and the model starts working from its own paraphrase of that structure instead of the structure itself. It will still sound confident. It will just be wrong in ways that are expensive to catch, because the failure surfaces downstream of the compaction, not at the moment it happened.

Truncate something that hadn't actually been superseded yet, and the model loses a fact it still needed, with no summary to fall back on and no signal that anything is missing. This is the quieter failure. There's no error message, just a gap where a constraint used to be, and the next few turns build on an incomplete picture before anyone notices the output is already wrong.

Most teams don't choose the wrong technique through bad judgment. They choose it through convenience. Summarize-everything is the easiest thing to implement, one function call regardless of what's being compressed. Truncate-the-oldest is the second easiest, because it needs no model call at all. Retrieval is the one that gets skipped most often, not because it's technically harder, but because it requires deciding, in advance, that some piece of context deserves to live outside the conversation entirely, addressable rather than carried. That decision is architecture. It has to be made before the system is under pressure, not improvised the moment the window fills up.

Every compaction is a bet that the part being kept matters more than the part being thrown away. Most systems make that bet the same way for everything they discard. The ones that hold up under real use make it three different ways, on purpose, category by category, before they're ever forced to.

Building a Context Budget: A Practical Token Allocation Framework

Siddharth Bhalsod — Mon, 29 Jun 2026 10:10:47 +0000

Open the /context command in Claude Code and you'll see something most teams have never seen for their own product: a precise breakdown of where every token in the session went. System prompt, eight percent. Tools and skills, fourteen percent. Conversation history, sixty-one percent. Free space, seventeen percent. The command exists because someone decided a context window needed a statement, the same way a company needs a P&L.

Most teams building on top of these models have no equivalent. They know their token limit. They have almost no idea what's spending it.

That's not a tooling gap. It's a discipline gap. The teams getting hurt by context rot are rarely the ones with a context window that's too small. They're the ones who never asked where each token was going, or whether it had earned its place there.

This is the practice most teams skip, and the one worth building before you touch a single percentage.

What a Token Budget Actually Is

Most people use "token budget" to mean the ceiling: the advertised window size on the model's spec sheet. That's not a budget. It's a credit limit. A credit limit tells you the most you could spend. It says nothing about where the money is actually going.

A context window is capacity. The budget is the layer above it: the deliberate decision about what fills that capacity, in what proportion, reviewed by someone whose job it is to ask whether each slice still deserves the space. Capacity and governance are different problems, and conflating them is how teams end up with a 1-million-token window and the same quality complaints they had at 50,000.

The reason this matters operationally is that the categories competing for that capacity are in zero-sum competition with each other. Pull ten retrieved documents at 1,500 tokens apiece into a RAG pipeline and you've spent 15,000 tokens before the model has read the actual question. Every one of those tokens came out of the same fixed pool that conversation history, tool output, and your system instructions are also drawing from. Add more of one category and something else gets less. There's no way around that math, only choices about how to make it on purpose instead of by accident.

The Ceiling Is Not the Budget

A bigger window doesn't fix a budgeting problem. It postpones the moment the problem becomes visible, and it usually makes the eventual mess larger.

When Claude Code moved to a default 1-million-token context window, several practitioners who'd been getting strong results on the 200,000-token version reported the opposite of an upgrade. More room didn't mean better reasoning. It meant more space for stale tool output, abandoned approaches, and old file reads to accumulate without anyone noticing, because nothing forced a cleanup the way a tighter ceiling used to. One developer found his sessions actually improved after deliberately shrinking back down, treating the extra headroom the way you'd treat a hard drive with ten times the space: a place where clutter survives longer, not a reason to stop sorting it.

The practical version of this lesson: window size is RAM, not storage. Treat it like a constrained resource regardless of how large the number on the spec sheet is. A bigger number doesn't earn you the right to be lazier about what loads.

Nobody Owns the Four Categories

Every context window, regardless of product, splits into roughly four things competing for the same fixed space: system instructions, retrieved knowledge, conversation or session history, and tool output. In most teams, every one of those four grows on its own, because no single person is accountable for the total.

System instructions grow by accretion. Someone adds a clarifying line after a bad output, another engineer adds a guardrail after an incident, and six months later the system prompt is a record of every edge case anyone has ever hit, most of which will never recur. Retrieved knowledge grows because "pull a few extra chunks to be safe" feels free and almost never gets measured against whether the extra chunks changed the answer. Conversation history grows because trimming it feels like throwing away memory, even when most of what's being kept is no longer relevant to the current step. Tool output is the worst offender precisely because it's the least visible: a single page snapshot, a pulled list of records, or a raw API response can carry far more raw text than the model needs in that form, and unless something intercepts it, all of it sits in context anyway.

A recent review of agent memory architectures put the underlying issue plainly: the context window is the scarcest shared resource any agent system has, and how it gets allocated is a coordination problem that no single piece of the system can solve by itself. That's the part most teams miss. Budgeting context isn't an engineering task you assign to one function. It's a cross-cutting one, and cross-cutting problems are exactly the ones that go unowned by default.

Budgeting Is a Practice, Not a Percentage

The honest fix here isn't a universal split. Anyone offering "40 percent retrieval, 30 percent history, 20 percent tools, 10 percent system" as a template that fits every product is selling something that won't survive contact with a real codebase or a real support queue. A coding agent's ideal allocation looks nothing like a customer-support bot's, and a fixed-system-prompt template that worked for one will misallocate the other.

What actually transfers across products is the practice, not the numbers. Name the four categories explicitly. Give one person, not a committee, the job of asking whether each category still earns its share. Review it on a cadence, the same way a team reviews a cloud bill line by line rather than just checking they're under the annual cap. A cap tells you nothing about which service is burning the budget. A line-item review does.

Some teams are already doing pieces of this without naming it. Cursor's .cursorignore file is a budgeting decision made before the fact: entire categories of files are told they will never compete for context at all, rather than being added and then managed once they're already taking up space. When Cursor's agent needs to search broadly across a codebase, it can spawn a separate subagent with its own context window just for that search, so raw results never spend tokens out of the main conversation's budget. That's a team deciding, explicitly, that one category of work deserves its own ledger rather than sharing the main one.

Claude Code's /context breakdown is the other half of the same idea: a dashboard that exists specifically so someone can see the split before a session runs long enough to degrade. The dashboard isn't the discipline. Running it before every long session is.

Why This Doesn't Show Up on a Dashboard

The cost of skipping this practice doesn't throw an error. It shows up as a slow, undramatic decline. A support agent starts needing three exchanges where one used to do. A coding agent begins re-deciding things it already decided an hour earlier in the same session. Nobody gets paged, because nothing crashed. The decline gets blamed on the model, because the model is the part of the system anyone can name. The actual cause, an unaudited and unowned allocation of tokens, doesn't show up on any dashboard a team is currently watching.

This is the same root issue that's run through this series in different clothes. Wrong build order showed up as eval infrastructure nobody trusted. Here it shows up as context nobody owns. Both are organizational problems wearing technical costumes. The fix in both cases is the same shape: assign the question to a person, on a schedule, before the system grows large enough that no one can audit it by hand anymore.

A context window doesn't tell you when it's full of the wrong things. It just gets quietly worse and keeps answering anyway.

A token budget you never check isn't a budget. It's just a limit you haven't hit yet.

Context Window Economics: Why Your Token Budget Is a Product Decision

Siddharth Bhalsod — Mon, 22 Jun 2026 06:26:35 +0000

A model advertising a 200,000-token context window can start falling apart at 50,000 tokens. It won't throw an error. It won't flag a warning. It will just get worse, fluently and confidently, on a problem you can only catch by checking its work later.

This isn't a hypothetical. Chroma's 2025 research tested eighteen frontier models, including OpenAI GPT-4.1, Claude Code Opus 4, and Google Gemini 2.5, and found that every one of them degraded as input length grew, often well before the window was anywhere near full. Researchers gave this a name: context rot, a continuous decline in output quality that has nothing to do with hitting a limit.

Most teams don't know this exists. They treat the context window like disk space: bigger is better, and if something feels off, the fix is to feed the model more. A full ticket history instead of a summary. An entire file instead of the relevant function. Three months of chat logs instead of last week's. Each addition feels like it should help. Often it makes things worse, and nothing in the product tells you that it did.

The Capacity Myth
The assumption underneath most AI products is simple: more context means a smarter, more grounded system. Stack the tokens, the model wins.

The research says otherwise. In 2023, Stanford University researchers led by Nelson Liu tested how models handle multi-document question answering and found a U-shaped accuracy curve. Models retrieve information well when it sits at the start or end of the input. When the relevant fact lands in the middle, accuracy drops by more than 30 percent. They called it the lost-in-the-middle effect, and it has since replicated across six model families.

Chroma's broader study identified two more mechanisms compounding the problem. Attention dilution comes from how transformer attention scales: a 100,000-token input creates roughly 10 billion pairwise relationships competing for the model's focus. Distractor interference is sneakier. Content that's topically related but irrelevant doesn't just sit there harmlessly. It actively misleads the model toward the wrong answer.

Context window overflow is a hard stop: tokens get truncated, and you find out immediately. Context rot is the opposite. Performance erodes while everything still appears to be working, which is exactly why most teams don't think they have the problem.

None of this requires hitting the window's stated limit. A model with 200,000 tokens of capacity can show measurable degradation by 50,000. The decline is continuous, not a cliff, and that's exactly why it goes unnoticed until someone checks.

The real metric was never capacity. It's signal-to-noise: how much of what's sitting in the window actually deserves the model's attention right now.

The Budget Nobody Is Managing
Once signal-to-noise is the metric, the context window stops looking like storage and starts looking like a budget. Every product built on an LLM allocates a fixed, shared space across competing claims on that attention:

The system prompt and instructions
Retrieved context: documents, files, search results
Conversation history
Tool outputs: API responses, file reads, search results
Working memory the model uses mid-task

A team that decides to retrieve the top 20 documents instead of the top 5, or to keep full chat history instead of summarizing it, is deciding what the model pays attention to instead of something else. That's the same category of decision as choosing which features make a release. Most teams don't experience it that way. They experience it as a retrieval setting somebody configured once and never revisited.

This is also where the five-dimension framework from earlier in this series gets concrete. Context budget lives inside the interaction model dimension: not whether a product feels conversational, but whether what actually reaches the model at each turn is the right input for the job. A system can score well on every other dimension and still fail here, because nobody assigned ownership of what goes into the window.

The size of the bill is the easiest part of this to track, and the part most teams actually do track. API pricing runs largely per token, so a retrieval step that grabs 30 documents instead of 8 isn't more thorough, it's a standing tax on every call the product makes. At a million queries a month, the gap between a tight context and a bloated one shows up on the margin line, not the engineering backlog.

The Silent Failure, Again
We've made a version of this argument before in this series: eval failures are usually sequencing problems, not metric-selection problems. Skip the detection layer, and a system degrades invisibly until a user notices. Context bloat fails the same way, one layer over, and it needs the same kind of production monitoring this series has already argued for, just watching a different signal.

No error fires when context degrades. The response still looks plausible. It's fluent, it's confident, and it's often wrong in a way only the user catches, because the system has no idea its own attention has been diluted.

Picture a support agent fed a customer's full ticket history instead of a structured summary. Forty messages in, it recommends a fix it already tried, that the customer already rejected. The model didn't get a worse weight update overnight. Its context just got noisier, and nobody was watching for it.

How AI Native Products Actually Manage the Budget
Cursor and Claude Code make a useful contrast, because both face the worst version of this problem: real codebases generate far more text than any context window can comfortably hold.

Claude Code's answer is auto-compaction. As a session approaches its limit, it summarizes the older exchanges and replaces the raw history with a condensed record before forcing a hard reset. Practitioners tracking this closely have found that compacting earlier, around 60 to 75 percent of capacity rather than waiting for the automatic trigger near 95 percent, produces longer and higher-quality sessions. One developer monitoring usage independently found Claude Code reporting only 10 percent of capacity left while his own tracking showed 64 percent had actually been used, a 54-point gap traced back to deliberately conservative compaction thresholds. The unused space wasn't waste. It was headroom protecting signal-to-noise as the session went on.

Cursor takes a different angle on the same problem. Instead of expanding the window to fit a growing codebase, it indexes the codebase into a vector database and retrieves only the chunks semantically relevant to the current query. A natural-language question pulls back a handful of relevant files instead of the entire repository, assembled into context just for that request. The model never sees more than it needs, by design, not by accident.

Neither product asks how much it can fit. Both ask what deserves to be there. That's the actual skill underneath context window economics, and it has nothing to do with how large the advertised window is.

Why This Belongs on the Roadmap
There's a trust cost to context rot that's quieter than the API bill, and harder to track. Users rarely file a bug report that says the model's context got noisy around message forty. They just notice the product got worse over a long session, stop trusting it with anything that matters, and churn without explaining why.

The model didn't get dumber. Its context got noisier.

Ownership matters here too. If the context budget isn't anyone's explicit job, the way conversion rate or onboarding flow is somebody's job, it drifts by default toward "more," because more feels safer than a decision someone has to defend. The instinct to fix a struggling AI feature by feeding it more is the same instinct as fixing it by upgrading to a bigger model. Both substitute scale for design. In most cases, the actual fix is a smaller, better-curated context, and that's a decision that belongs with whoever owns the product, not buried in a retrieval config nobody revisits after launch.

Every additional token placed in front of a model is a decision about what it should attend to instead of something else. Treat it with the discipline of a feature cut or a pricing tier, not a default setting left over from launch week. The teams getting this right don't have the biggest context windows. They decided, on purpose, what doesn't get to be there.

Building a Production-Grade LLM Eval System From Scratch

Siddharth Bhalsod — Tue, 16 Jun 2026 10:08:13 +0000

Your LLM Eval Is Broken Before You Write the First Test

Most teams discover their eval system is broken the same way. They ship a prompt change that improves tone but silently tanks accuracy on edge cases. They upgrade their model version and something subtly changes — response length, citation patterns, how it handles ambiguity. Nobody catches it because the test suite was checking the wrong things. Or it wasn't running in CI. Or it existed on someone's laptop and that person has since left.

This is not a metrics problem. It is a sequencing problem.

The teams with working eval infrastructure — the ones where a prompt change doesn't become a post-mortem — built their system in a specific order. They defined what good looks like before they wrote a single test. They instrumented the system before they had enough data to justify it. They treated evaluation as architecture, not as a final validation step bolted on before launch.

In the AI Native series, Article 3 established that most teams build the wrong stack because they start with the model and work backward. The same mistake compounds inside the eval layer: most teams start with a framework and work backward. They install DeepEval or Braintrust, run a quick hallucination check, ship it, and call the eval layer done. The framework is not the system. The framework is one component inside a system that has to be deliberately designed.

This article is the design guide for that system. Not a framework tutorial — a sequencing blueprint.

The Wrong Starting Point

When a team decides to "add evals," the first thing they typically reach for is a library. pip install deepeval. Add AnswerRelevancyMetric. Run it against a few test cases. Green outputs feel like progress.

They are not progress. They are the illusion of instrumentation.

The problem is that answer relevancy is a generic metric. It tells you whether the model's response is topically related to the query — which is almost always true for any reasonably sized model and any reasonably coherent prompt. Passing this metric by default is like testing whether your e-commerce site can render a product page and calling the checkout flow validated.

The real question is not "does this output look relevant?" The real question is: what does quality actually mean for this specific system, in this specific product context, for this specific user?

That question is not a technical question. It is a product question. And it has to be answered before any eval framework is touched.

Layer One: Define Quality Before You Measure It

Consider two products that both use retrieval-augmented generation. The first is a legal research tool — lawyers use it to find case precedents before drafting filings. The second is a customer support assistant — customers use it to resolve billing disputes without calling in.

Both systems retrieve documents. Both generate responses. Both could fail on hallucination and answer relevancy. But the quality definitions are completely different.

For the legal tool, the most dangerous failure is a confident answer that cites a real case incorrectly — a paraphrase that changes the meaning of a ruling. For the support tool, the most dangerous failure is a refusal to resolve something the system should be able to handle — a hedge that sends the customer to a human unnecessarily.

Run the same generic metric set on both and you will get a score. That score will mean nothing to either product team.

This is why quality definition is Layer 1. Not Layer 4. Not "something we add later when we have real data."

The way to do it: write three to five failure statements before you write any test. Not metric names — failure statements. Things like: "The system confidently states a legal precedent that does not exist," or "The system routes a resolvable billing dispute to a human agent." These statements describe what broken looks like in terms your product team and your eval framework can both understand.

Then map each failure statement to a metric type. Some will map to built-in DeepEval metrics. Some will require a custom GEval criterion. Some will require a deterministic code-based check. The mapping is the architecture decision.

Layer Two: Instrument Without Waiting for Data

The second failure mode: teams wait until they have enough production data to build a "real" test suite. This feels responsible. It is actually how you end up with no eval coverage during the months when the system is most likely to change.

The practical answer is synthetic goldens.

DeepEval's Synthesizer can generate test cases from your knowledge base before a single real user has touched the system. If you are building a RAG pipeline, you feed it your document corpus and it generates realistic input/output pairs — questions a real user might ask, grounded in the content the system will retrieve. These are not perfect proxies for real traffic. They are good enough to establish a baseline and to catch the class of failures that break obviously.

GitHub's Copilot team runs comprehensive offline evaluations against every model before it reaches production — testing across metrics like latency, accuracy, and response consistency before any user interaction. They do not wait for user feedback to tell them the model regressed. The eval system surfaces regressions in the same pipeline that builds the release.

The minimum viable starting point is not fifty production examples. It is twenty-five synthetic goldens, two to three metrics that map to your failure statements, and a passing threshold. That is a real eval system. Run it before every prompt change, every model swap, every retrieval parameter update.

Layer Three: Structure the Test Suite Around Failure Modes, Not Features

This is the architectural distinction most teams miss.

The natural instinct is to organize test cases around features: here are the tests for the summarization flow, here are the tests for the question-answering flow, here are the tests for the refusal behavior. This organization feels logical. It mirrors how the product is structured.

The problem is that eval systems organized by feature tell you what broke but not why. When the summarization score drops three points, you know summaries got worse. You do not know whether the retrieval layer is returning worse context, whether the prompt changed something in formatting behavior, or whether a model update shifted the generation style.

Structure the test suite around failure modes instead. Each failure statement from Layer 1 becomes a test class. Each test class runs its specific metric. When a test class fails, the failure message is already diagnostic — it points to the component and the behavior, not just the feature.

In DeepEval, this looks like:

`from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

confident_hallucination = GEval(
name="ConfidentHallucination",
criteria="The output should never state a legal precedent with high confidence unless the retrieved context directly supports it.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.RETRIEVAL_CONTEXT],
threshold=0.8
)`

This is not the generic HallucinationMetric. It is a custom GEval criterion written in plain English, tied to the specific failure mode the legal research team identified. When it fires, it fires on a specific category of error — not on a score that requires interpretation.

DeepEval's recommendation, grounded in production experience, is to limit yourself to five metrics maximum: two to three generic system-specific metrics (contextual precision for a RAG pipeline, tool correctness for an agent) and one to two custom, use-case-specific metrics. The constraint is intentional. More metrics means noisier signals and harder-to-diagnose failures.

Layer Four: The Drift Problem No Test Suite Catches

There is a class of failure that well-designed test suites miss almost entirely. Call it version drift.

A model provider pushes a silent update. Not a new model version — an update to the weights behind the same model string. Your evals pass. Your prompts are unchanged. And quietly, over the following two weeks, something shifts. Users start submitting more corrections. Satisfaction scores drift down by a few points. The output that used to be crisp and structured gets slightly looser. Nobody changed anything. But the system got worse.

This is the failure mode that unit testing, however well structured, cannot catch. Offline evals run against a snapshot. They tell you whether the system performs acceptably on the dataset you created. They cannot tell you whether the live system is drifting from that baseline in production.

The answer is production monitoring — which is Layer 4 of the eval system and the layer most teams skip entirely.

Production monitoring means scoring a sample of real user interactions continuously using referenceless metrics. Referenceless because you will not have ground truth labels for live traffic. DeepEval provides these — metrics like AnswerRelevancyMetric, FaithfulnessMetric, and ConcisenessMetric that can run without a known correct answer.

The setup is straightforward: route ten to twenty percent of live traffic through your eval pipeline, aggregate scores on a rolling window, and alert when scores cross a threshold. Confident AI — the cloud platform built on top of DeepEval — handles the dashboard and monitoring infrastructure if you do not want to build it yourself. The point is not the tool. The point is that offline evals and production monitoring are two different systems solving two different problems, and you need both.

Teams that run only offline evals are flying blind during the longest part of a product's life: after launch.

The Build Order

The failure modes are not random. They follow directly from building the eval system in the wrong order.

Teams that instrument too late — after the system is in production — start with generic metrics and work backward to product meaning. They are always trying to retrofit quality definitions onto scores they do not fully trust.

Teams that organize by feature instead of failure mode always have a two-step debugging process: find the failing test, then figure out what the failing test actually means.

Teams that skip production monitoring ship a system that degrades invisibly until users tell them it has.

The right order is four layers, built in sequence:

Define quality as failure statements, before touching any framework.
Generate synthetic goldens and establish baselines, before waiting for real data.
Structure test classes around failure modes, not features.
Add production monitoring for drift, as a separate system from the offline test suite.

This is not how most teams build their eval layer. Most teams build Layer 2 first — the framework, the test cases, the CI run — and never get to Layers 1 and 4 at all.

The eval system that degrades invisibly is not a testing failure. It is a sequencing failure.

Most teams building AI products start with the model. That is the mistake. AI Native infrastructure has five layers, and none of them is the model - Read👇

Siddharth Bhalsod — Wed, 10 Jun 2026 08:41:36 +0000

Siddharth Bhalsod

Jun 10

What AI Native Infrastructure Looks Like in Practice

#ainative #aienhanced #aiautomation #aiops

8 min read