DEV Community: Siddharth Bhalsod

Context Compaction Patterns: When to Summarize, Truncate, or Retrieve

Siddharth Bhalsod — Wed, 08 Jul 2026 08:27:14 +0000

Context Compaction Patterns: When to Summarize, Truncate, or Retrieve

A context window fills up mid-session. Something has to leave. Most systems answer that problem with one move: summarize everything and hope the important parts survive. It's the same instinct that made "just add more context" the reflexive answer to the budgeting problem — a single technique, applied everywhere, regardless of what's actually being thrown out.

Compaction isn't one operation. It's three, and they destroy information in completely different ways. Summarizing keeps the gist and loses the precision. Truncating keeps the recent and drops the old outright. Retrieving loses nothing permanently, it just stops sitting in front of the model until something asks for it again. Pick the wrong one for a given piece of context, and the failure doesn't announce itself as a compaction bug. It shows up as the model forgetting a decision it already made, misreading a file it already read, or confidently inventing an API it saw an hour ago.

The Mistake Hiding Inside "Just Summarize It"

Ask an engineer how their system handles a full context window and the answer is usually one sentence: it summarizes. That's the imposter definition of compaction, and it's wrong in a specific way. Summarization is one tool in a set of three, best suited to exactly one kind of content: information where the gist matters more than the wording, and where there's no cheaper way to get it back later.

Truncation is not a worse version of summarization. It's a different bet entirely, that the old content isn't coming back and doesn't need a lossy stand-in either, because it's either genuinely stale or cheaply re-obtainable from somewhere else. Retrieval is not a fallback for when summarization would lose too much. It's the option that applies whenever the real thing still exists somewhere addressable, and fetching it again costs less than carrying a compressed copy around indefinitely.

Treat all three as interchangeable synonyms for "make it smaller," and a system will compact the wrong things the wrong way. A four-line grep result and a forty-turn architectural discussion do not fail the same way when compressed. Building one pipeline for both is the actual root cause of context management that quietly breaks agent behavior weeks into production, long after anyone remembers the shortcut was taken.

The One Question That Decides Which Technique to Use

There's a single question that sorts almost every piece of discardable context into the right bucket: if this disappeared right now, could you get it back, cheaply, from somewhere else?

If yes, retrieve. The content lives in a file, a database, a vector index, a URL. The model doesn't need to carry it, it needs to know it exists and how to ask for it again. Compressing it into a summary is strictly worse than dropping it and re-fetching the original when it's actually needed, because a summary of a file is a lossy copy of something you already have a lossless copy of.

If no, but the value sits in the narrative rather than the exact wording, summarize. A ninety-minute debugging conversation isn't reconstructable from any external source. Nobody logged the reasoning trail. But the decisions made along the way, the constraints established, the dead ends already ruled out, survive compression into a few paragraphs. The exact phrasing of turn forty doesn't matter. The fact that turn forty ruled out a database migration does.

If no, and the content has already been superseded by something more recent, truncate. An old grep result against a file that's since been edited isn't worth summarizing. It isn't worth a placeholder either. It's just wrong now, and the right move is to let it go rather than spend tokens compressing something that's no longer true.

The Trigger Matters As Much As the Technique

Even the right technique executed at the wrong moment causes damage. Wait until a conversation is nearly out of room before summarizing it, and the summary is built from an already-degraded view: the model is compressing a conversation it's already struggling to reason over cleanly. Run the same operation earlier, well before pressure builds, and the summary comes from a clean, complete picture instead of salvage material.

The same logic holds for the other two techniques. Truncate too early, before content is genuinely stale, and something still in use disappears with no summary to fall back on. Retrieve too late, after the model has already tried to answer from what it half-remembers instead of the real source, and the retrieval fixes an error that already happened instead of preventing it. Category tells you which technique to use. Timing tells you whether that technique does what it's supposed to.

Same System, Three Different Answers

The clearest proof this isn't theoretical is a single coding agent making three different calls inside one session. Claude Code doesn't run one compaction function against everything that gets long. It runs three, for three different problems. Old tool results that are no longer needed, a grep from many turns ago against a file that's since changed, get cleared outright: no summary, no reference kept, because the content is already stale and reconstructing it would mean re-running a command against a file that no longer matches. Large but still-relevant tool output, a long file read, a verbose test run, gets a different treatment: the full content moves to disk, and only a path reference stays in the model's view, pulled back in full if the work touches it again. Conversation history gets a third treatment entirely, because there's no disk copy of a debugging conversation sitting anywhere: when the session nears its limit, the turns themselves get compressed into a structured summary that preserves decisions and task state rather than exact wording.

Three techniques, three categories, inside one product. Cursor draws a similar line somewhere else. Ask it about a codebase and it doesn't truncate the repository or summarize old files to make room, it retrieves. The codebase gets indexed with embeddings ahead of time, and a request pulls back the handful of chunks that are actually relevant through semantic search, the same underlying bet as Claude Code's disk-backed tool outputs: the source is stable and cheaper to fetch again than to carry around compressed.

What Breaks When You Pick Wrong

The failure mode of picking the wrong technique rarely looks like a crash. It looks like a subtly worse answer that's hard to trace back to its cause. Summarize a tool output that needed to be reasoned over exactly, a JSON schema, a diff, an error stack, and the model starts working from its own paraphrase of that structure instead of the structure itself. It will still sound confident. It will just be wrong in ways that are expensive to catch, because the failure surfaces downstream of the compaction, not at the moment it happened.

Truncate something that hadn't actually been superseded yet, and the model loses a fact it still needed, with no summary to fall back on and no signal that anything is missing. This is the quieter failure. There's no error message, just a gap where a constraint used to be, and the next few turns build on an incomplete picture before anyone notices the output is already wrong.

Most teams don't choose the wrong technique through bad judgment. They choose it through convenience. Summarize-everything is the easiest thing to implement, one function call regardless of what's being compressed. Truncate-the-oldest is the second easiest, because it needs no model call at all. Retrieval is the one that gets skipped most often, not because it's technically harder, but because it requires deciding, in advance, that some piece of context deserves to live outside the conversation entirely, addressable rather than carried. That decision is architecture. It has to be made before the system is under pressure, not improvised the moment the window fills up.

Every compaction is a bet that the part being kept matters more than the part being thrown away. Most systems make that bet the same way for everything they discard. The ones that hold up under real use make it three different ways, on purpose, category by category, before they're ever forced to.

Building a Context Budget: A Practical Token Allocation Framework

Siddharth Bhalsod — Mon, 29 Jun 2026 10:10:47 +0000

Open the /context command in Claude Code and you'll see something most teams have never seen for their own product: a precise breakdown of where every token in the session went. System prompt, eight percent. Tools and skills, fourteen percent. Conversation history, sixty-one percent. Free space, seventeen percent. The command exists because someone decided a context window needed a statement, the same way a company needs a P&L.

Most teams building on top of these models have no equivalent. They know their token limit. They have almost no idea what's spending it.

That's not a tooling gap. It's a discipline gap. The teams getting hurt by context rot are rarely the ones with a context window that's too small. They're the ones who never asked where each token was going, or whether it had earned its place there.

This is the practice most teams skip, and the one worth building before you touch a single percentage.

What a Token Budget Actually Is

Most people use "token budget" to mean the ceiling: the advertised window size on the model's spec sheet. That's not a budget. It's a credit limit. A credit limit tells you the most you could spend. It says nothing about where the money is actually going.

A context window is capacity. The budget is the layer above it: the deliberate decision about what fills that capacity, in what proportion, reviewed by someone whose job it is to ask whether each slice still deserves the space. Capacity and governance are different problems, and conflating them is how teams end up with a 1-million-token window and the same quality complaints they had at 50,000.

The reason this matters operationally is that the categories competing for that capacity are in zero-sum competition with each other. Pull ten retrieved documents at 1,500 tokens apiece into a RAG pipeline and you've spent 15,000 tokens before the model has read the actual question. Every one of those tokens came out of the same fixed pool that conversation history, tool output, and your system instructions are also drawing from. Add more of one category and something else gets less. There's no way around that math, only choices about how to make it on purpose instead of by accident.

The Ceiling Is Not the Budget

A bigger window doesn't fix a budgeting problem. It postpones the moment the problem becomes visible, and it usually makes the eventual mess larger.

When Claude Code moved to a default 1-million-token context window, several practitioners who'd been getting strong results on the 200,000-token version reported the opposite of an upgrade. More room didn't mean better reasoning. It meant more space for stale tool output, abandoned approaches, and old file reads to accumulate without anyone noticing, because nothing forced a cleanup the way a tighter ceiling used to. One developer found his sessions actually improved after deliberately shrinking back down, treating the extra headroom the way you'd treat a hard drive with ten times the space: a place where clutter survives longer, not a reason to stop sorting it.

The practical version of this lesson: window size is RAM, not storage. Treat it like a constrained resource regardless of how large the number on the spec sheet is. A bigger number doesn't earn you the right to be lazier about what loads.

Nobody Owns the Four Categories

Every context window, regardless of product, splits into roughly four things competing for the same fixed space: system instructions, retrieved knowledge, conversation or session history, and tool output. In most teams, every one of those four grows on its own, because no single person is accountable for the total.

System instructions grow by accretion. Someone adds a clarifying line after a bad output, another engineer adds a guardrail after an incident, and six months later the system prompt is a record of every edge case anyone has ever hit, most of which will never recur. Retrieved knowledge grows because "pull a few extra chunks to be safe" feels free and almost never gets measured against whether the extra chunks changed the answer. Conversation history grows because trimming it feels like throwing away memory, even when most of what's being kept is no longer relevant to the current step. Tool output is the worst offender precisely because it's the least visible: a single page snapshot, a pulled list of records, or a raw API response can carry far more raw text than the model needs in that form, and unless something intercepts it, all of it sits in context anyway.

A recent review of agent memory architectures put the underlying issue plainly: the context window is the scarcest shared resource any agent system has, and how it gets allocated is a coordination problem that no single piece of the system can solve by itself. That's the part most teams miss. Budgeting context isn't an engineering task you assign to one function. It's a cross-cutting one, and cross-cutting problems are exactly the ones that go unowned by default.

Budgeting Is a Practice, Not a Percentage

The honest fix here isn't a universal split. Anyone offering "40 percent retrieval, 30 percent history, 20 percent tools, 10 percent system" as a template that fits every product is selling something that won't survive contact with a real codebase or a real support queue. A coding agent's ideal allocation looks nothing like a customer-support bot's, and a fixed-system-prompt template that worked for one will misallocate the other.

What actually transfers across products is the practice, not the numbers. Name the four categories explicitly. Give one person, not a committee, the job of asking whether each category still earns its share. Review it on a cadence, the same way a team reviews a cloud bill line by line rather than just checking they're under the annual cap. A cap tells you nothing about which service is burning the budget. A line-item review does.

Some teams are already doing pieces of this without naming it. Cursor's .cursorignore file is a budgeting decision made before the fact: entire categories of files are told they will never compete for context at all, rather than being added and then managed once they're already taking up space. When Cursor's agent needs to search broadly across a codebase, it can spawn a separate subagent with its own context window just for that search, so raw results never spend tokens out of the main conversation's budget. That's a team deciding, explicitly, that one category of work deserves its own ledger rather than sharing the main one.

Claude Code's /context breakdown is the other half of the same idea: a dashboard that exists specifically so someone can see the split before a session runs long enough to degrade. The dashboard isn't the discipline. Running it before every long session is.

Why This Doesn't Show Up on a Dashboard

The cost of skipping this practice doesn't throw an error. It shows up as a slow, undramatic decline. A support agent starts needing three exchanges where one used to do. A coding agent begins re-deciding things it already decided an hour earlier in the same session. Nobody gets paged, because nothing crashed. The decline gets blamed on the model, because the model is the part of the system anyone can name. The actual cause, an unaudited and unowned allocation of tokens, doesn't show up on any dashboard a team is currently watching.

This is the same root issue that's run through this series in different clothes. Wrong build order showed up as eval infrastructure nobody trusted. Here it shows up as context nobody owns. Both are organizational problems wearing technical costumes. The fix in both cases is the same shape: assign the question to a person, on a schedule, before the system grows large enough that no one can audit it by hand anymore.

A context window doesn't tell you when it's full of the wrong things. It just gets quietly worse and keeps answering anyway.

A token budget you never check isn't a budget. It's just a limit you haven't hit yet.

Context Window Economics: Why Your Token Budget Is a Product Decision

Siddharth Bhalsod — Mon, 22 Jun 2026 06:26:35 +0000

A model advertising a 200,000-token context window can start falling apart at 50,000 tokens. It won't throw an error. It won't flag a warning. It will just get worse, fluently and confidently, on a problem you can only catch by checking its work later.

This isn't a hypothetical. Chroma's 2025 research tested eighteen frontier models, including OpenAI GPT-4.1, Claude Code Opus 4, and Google Gemini 2.5, and found that every one of them degraded as input length grew, often well before the window was anywhere near full. Researchers gave this a name: context rot, a continuous decline in output quality that has nothing to do with hitting a limit.

Most teams don't know this exists. They treat the context window like disk space: bigger is better, and if something feels off, the fix is to feed the model more. A full ticket history instead of a summary. An entire file instead of the relevant function. Three months of chat logs instead of last week's. Each addition feels like it should help. Often it makes things worse, and nothing in the product tells you that it did.

The Capacity Myth
The assumption underneath most AI products is simple: more context means a smarter, more grounded system. Stack the tokens, the model wins.

The research says otherwise. In 2023, Stanford University researchers led by Nelson Liu tested how models handle multi-document question answering and found a U-shaped accuracy curve. Models retrieve information well when it sits at the start or end of the input. When the relevant fact lands in the middle, accuracy drops by more than 30 percent. They called it the lost-in-the-middle effect, and it has since replicated across six model families.

Chroma's broader study identified two more mechanisms compounding the problem. Attention dilution comes from how transformer attention scales: a 100,000-token input creates roughly 10 billion pairwise relationships competing for the model's focus. Distractor interference is sneakier. Content that's topically related but irrelevant doesn't just sit there harmlessly. It actively misleads the model toward the wrong answer.

Context window overflow is a hard stop: tokens get truncated, and you find out immediately. Context rot is the opposite. Performance erodes while everything still appears to be working, which is exactly why most teams don't think they have the problem.

None of this requires hitting the window's stated limit. A model with 200,000 tokens of capacity can show measurable degradation by 50,000. The decline is continuous, not a cliff, and that's exactly why it goes unnoticed until someone checks.

The real metric was never capacity. It's signal-to-noise: how much of what's sitting in the window actually deserves the model's attention right now.

The Budget Nobody Is Managing
Once signal-to-noise is the metric, the context window stops looking like storage and starts looking like a budget. Every product built on an LLM allocates a fixed, shared space across competing claims on that attention:

The system prompt and instructions
Retrieved context: documents, files, search results
Conversation history
Tool outputs: API responses, file reads, search results
Working memory the model uses mid-task

A team that decides to retrieve the top 20 documents instead of the top 5, or to keep full chat history instead of summarizing it, is deciding what the model pays attention to instead of something else. That's the same category of decision as choosing which features make a release. Most teams don't experience it that way. They experience it as a retrieval setting somebody configured once and never revisited.

This is also where the five-dimension framework from earlier in this series gets concrete. Context budget lives inside the interaction model dimension: not whether a product feels conversational, but whether what actually reaches the model at each turn is the right input for the job. A system can score well on every other dimension and still fail here, because nobody assigned ownership of what goes into the window.

The size of the bill is the easiest part of this to track, and the part most teams actually do track. API pricing runs largely per token, so a retrieval step that grabs 30 documents instead of 8 isn't more thorough, it's a standing tax on every call the product makes. At a million queries a month, the gap between a tight context and a bloated one shows up on the margin line, not the engineering backlog.

The Silent Failure, Again
We've made a version of this argument before in this series: eval failures are usually sequencing problems, not metric-selection problems. Skip the detection layer, and a system degrades invisibly until a user notices. Context bloat fails the same way, one layer over, and it needs the same kind of production monitoring this series has already argued for, just watching a different signal.

No error fires when context degrades. The response still looks plausible. It's fluent, it's confident, and it's often wrong in a way only the user catches, because the system has no idea its own attention has been diluted.

Picture a support agent fed a customer's full ticket history instead of a structured summary. Forty messages in, it recommends a fix it already tried, that the customer already rejected. The model didn't get a worse weight update overnight. Its context just got noisier, and nobody was watching for it.

How AI Native Products Actually Manage the Budget
Cursor and Claude Code make a useful contrast, because both face the worst version of this problem: real codebases generate far more text than any context window can comfortably hold.

Claude Code's answer is auto-compaction. As a session approaches its limit, it summarizes the older exchanges and replaces the raw history with a condensed record before forcing a hard reset. Practitioners tracking this closely have found that compacting earlier, around 60 to 75 percent of capacity rather than waiting for the automatic trigger near 95 percent, produces longer and higher-quality sessions. One developer monitoring usage independently found Claude Code reporting only 10 percent of capacity left while his own tracking showed 64 percent had actually been used, a 54-point gap traced back to deliberately conservative compaction thresholds. The unused space wasn't waste. It was headroom protecting signal-to-noise as the session went on.

Cursor takes a different angle on the same problem. Instead of expanding the window to fit a growing codebase, it indexes the codebase into a vector database and retrieves only the chunks semantically relevant to the current query. A natural-language question pulls back a handful of relevant files instead of the entire repository, assembled into context just for that request. The model never sees more than it needs, by design, not by accident.

Neither product asks how much it can fit. Both ask what deserves to be there. That's the actual skill underneath context window economics, and it has nothing to do with how large the advertised window is.

Why This Belongs on the Roadmap
There's a trust cost to context rot that's quieter than the API bill, and harder to track. Users rarely file a bug report that says the model's context got noisy around message forty. They just notice the product got worse over a long session, stop trusting it with anything that matters, and churn without explaining why.

The model didn't get dumber. Its context got noisier.

Ownership matters here too. If the context budget isn't anyone's explicit job, the way conversion rate or onboarding flow is somebody's job, it drifts by default toward "more," because more feels safer than a decision someone has to defend. The instinct to fix a struggling AI feature by feeding it more is the same instinct as fixing it by upgrading to a bigger model. Both substitute scale for design. In most cases, the actual fix is a smaller, better-curated context, and that's a decision that belongs with whoever owns the product, not buried in a retrieval config nobody revisits after launch.

Every additional token placed in front of a model is a decision about what it should attend to instead of something else. Treat it with the discipline of a feature cut or a pricing tier, not a default setting left over from launch week. The teams getting this right don't have the biggest context windows. They decided, on purpose, what doesn't get to be there.

Building a Production-Grade LLM Eval System From Scratch

Siddharth Bhalsod — Tue, 16 Jun 2026 10:08:13 +0000

Your LLM Eval Is Broken Before You Write the First Test

Most teams discover their eval system is broken the same way. They ship a prompt change that improves tone but silently tanks accuracy on edge cases. They upgrade their model version and something subtly changes — response length, citation patterns, how it handles ambiguity. Nobody catches it because the test suite was checking the wrong things. Or it wasn't running in CI. Or it existed on someone's laptop and that person has since left.

This is not a metrics problem. It is a sequencing problem.

The teams with working eval infrastructure — the ones where a prompt change doesn't become a post-mortem — built their system in a specific order. They defined what good looks like before they wrote a single test. They instrumented the system before they had enough data to justify it. They treated evaluation as architecture, not as a final validation step bolted on before launch.

In the AI Native series, Article 3 established that most teams build the wrong stack because they start with the model and work backward. The same mistake compounds inside the eval layer: most teams start with a framework and work backward. They install DeepEval or Braintrust, run a quick hallucination check, ship it, and call the eval layer done. The framework is not the system. The framework is one component inside a system that has to be deliberately designed.

This article is the design guide for that system. Not a framework tutorial — a sequencing blueprint.

The Wrong Starting Point

When a team decides to "add evals," the first thing they typically reach for is a library. pip install deepeval. Add AnswerRelevancyMetric. Run it against a few test cases. Green outputs feel like progress.

They are not progress. They are the illusion of instrumentation.

The problem is that answer relevancy is a generic metric. It tells you whether the model's response is topically related to the query — which is almost always true for any reasonably sized model and any reasonably coherent prompt. Passing this metric by default is like testing whether your e-commerce site can render a product page and calling the checkout flow validated.

The real question is not "does this output look relevant?" The real question is: what does quality actually mean for this specific system, in this specific product context, for this specific user?

That question is not a technical question. It is a product question. And it has to be answered before any eval framework is touched.

Layer One: Define Quality Before You Measure It

Consider two products that both use retrieval-augmented generation. The first is a legal research tool — lawyers use it to find case precedents before drafting filings. The second is a customer support assistant — customers use it to resolve billing disputes without calling in.

Both systems retrieve documents. Both generate responses. Both could fail on hallucination and answer relevancy. But the quality definitions are completely different.

For the legal tool, the most dangerous failure is a confident answer that cites a real case incorrectly — a paraphrase that changes the meaning of a ruling. For the support tool, the most dangerous failure is a refusal to resolve something the system should be able to handle — a hedge that sends the customer to a human unnecessarily.

Run the same generic metric set on both and you will get a score. That score will mean nothing to either product team.

This is why quality definition is Layer 1. Not Layer 4. Not "something we add later when we have real data."

The way to do it: write three to five failure statements before you write any test. Not metric names — failure statements. Things like: "The system confidently states a legal precedent that does not exist," or "The system routes a resolvable billing dispute to a human agent." These statements describe what broken looks like in terms your product team and your eval framework can both understand.

Then map each failure statement to a metric type. Some will map to built-in DeepEval metrics. Some will require a custom GEval criterion. Some will require a deterministic code-based check. The mapping is the architecture decision.

Layer Two: Instrument Without Waiting for Data

The second failure mode: teams wait until they have enough production data to build a "real" test suite. This feels responsible. It is actually how you end up with no eval coverage during the months when the system is most likely to change.

The practical answer is synthetic goldens.

DeepEval's Synthesizer can generate test cases from your knowledge base before a single real user has touched the system. If you are building a RAG pipeline, you feed it your document corpus and it generates realistic input/output pairs — questions a real user might ask, grounded in the content the system will retrieve. These are not perfect proxies for real traffic. They are good enough to establish a baseline and to catch the class of failures that break obviously.

GitHub's Copilot team runs comprehensive offline evaluations against every model before it reaches production — testing across metrics like latency, accuracy, and response consistency before any user interaction. They do not wait for user feedback to tell them the model regressed. The eval system surfaces regressions in the same pipeline that builds the release.

The minimum viable starting point is not fifty production examples. It is twenty-five synthetic goldens, two to three metrics that map to your failure statements, and a passing threshold. That is a real eval system. Run it before every prompt change, every model swap, every retrieval parameter update.

Layer Three: Structure the Test Suite Around Failure Modes, Not Features

This is the architectural distinction most teams miss.

The natural instinct is to organize test cases around features: here are the tests for the summarization flow, here are the tests for the question-answering flow, here are the tests for the refusal behavior. This organization feels logical. It mirrors how the product is structured.

The problem is that eval systems organized by feature tell you what broke but not why. When the summarization score drops three points, you know summaries got worse. You do not know whether the retrieval layer is returning worse context, whether the prompt changed something in formatting behavior, or whether a model update shifted the generation style.

Structure the test suite around failure modes instead. Each failure statement from Layer 1 becomes a test class. Each test class runs its specific metric. When a test class fails, the failure message is already diagnostic — it points to the component and the behavior, not just the feature.

In DeepEval, this looks like:

`from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

confident_hallucination = GEval(
name="ConfidentHallucination",
criteria="The output should never state a legal precedent with high confidence unless the retrieved context directly supports it.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.RETRIEVAL_CONTEXT],
threshold=0.8
)`

This is not the generic HallucinationMetric. It is a custom GEval criterion written in plain English, tied to the specific failure mode the legal research team identified. When it fires, it fires on a specific category of error — not on a score that requires interpretation.

DeepEval's recommendation, grounded in production experience, is to limit yourself to five metrics maximum: two to three generic system-specific metrics (contextual precision for a RAG pipeline, tool correctness for an agent) and one to two custom, use-case-specific metrics. The constraint is intentional. More metrics means noisier signals and harder-to-diagnose failures.

Layer Four: The Drift Problem No Test Suite Catches

There is a class of failure that well-designed test suites miss almost entirely. Call it version drift.

A model provider pushes a silent update. Not a new model version — an update to the weights behind the same model string. Your evals pass. Your prompts are unchanged. And quietly, over the following two weeks, something shifts. Users start submitting more corrections. Satisfaction scores drift down by a few points. The output that used to be crisp and structured gets slightly looser. Nobody changed anything. But the system got worse.

This is the failure mode that unit testing, however well structured, cannot catch. Offline evals run against a snapshot. They tell you whether the system performs acceptably on the dataset you created. They cannot tell you whether the live system is drifting from that baseline in production.

The answer is production monitoring — which is Layer 4 of the eval system and the layer most teams skip entirely.

Production monitoring means scoring a sample of real user interactions continuously using referenceless metrics. Referenceless because you will not have ground truth labels for live traffic. DeepEval provides these — metrics like AnswerRelevancyMetric, FaithfulnessMetric, and ConcisenessMetric that can run without a known correct answer.

The setup is straightforward: route ten to twenty percent of live traffic through your eval pipeline, aggregate scores on a rolling window, and alert when scores cross a threshold. Confident AI — the cloud platform built on top of DeepEval — handles the dashboard and monitoring infrastructure if you do not want to build it yourself. The point is not the tool. The point is that offline evals and production monitoring are two different systems solving two different problems, and you need both.

Teams that run only offline evals are flying blind during the longest part of a product's life: after launch.

The Build Order

The failure modes are not random. They follow directly from building the eval system in the wrong order.

Teams that instrument too late — after the system is in production — start with generic metrics and work backward to product meaning. They are always trying to retrofit quality definitions onto scores they do not fully trust.

Teams that organize by feature instead of failure mode always have a two-step debugging process: find the failing test, then figure out what the failing test actually means.

Teams that skip production monitoring ship a system that degrades invisibly until users tell them it has.

The right order is four layers, built in sequence:

Define quality as failure statements, before touching any framework.
Generate synthetic goldens and establish baselines, before waiting for real data.
Structure test classes around failure modes, not features.
Add production monitoring for drift, as a separate system from the offline test suite.

This is not how most teams build their eval layer. Most teams build Layer 2 first — the framework, the test cases, the CI run — and never get to Layers 1 and 4 at all.

The eval system that degrades invisibly is not a testing failure. It is a sequencing failure.

Most teams building AI products start with the model. That is the mistake. AI Native infrastructure has five layers, and none of them is the model - Read👇

Siddharth Bhalsod — Wed, 10 Jun 2026 08:41:36 +0000

Siddharth Bhalsod

Jun 10

What AI Native Infrastructure Looks Like in Practice

#ainative #aienhanced #aiautomation #aiops

8 min read

What AI Native Infrastructure Looks Like in Practice

Siddharth Bhalsod — Wed, 10 Jun 2026 08:37:45 +0000

Most teams get the model right on the first try.

They pick Claude or GPT-4o, wire up a few prompts, and ship something that impresses in a demo. Then they spend the next six months wondering why responses drift, costs compound faster than users do, and the system that felt clever in week two feels brittle by month four.

The model was not the problem. The model was never the problem.
The mistake is architectural, and it almost always starts the same way: teams choose the model before they design the data layer. Everything downstream from that sequencing error will cost them.

The Wrong Starting Point

Here is how most teams actually build: they pick a foundation model, write prompts that work for their test cases, and then figure out how to feed the model the right information. The data layer is treated as a support function for the model. A retrieval step to bolt on. Something to sort out later.

This is backwards.

In AI Native systems, the data layer is not a supporting actor. It is the foundation that determines whether the model can do anything useful at all. A well-prompted model operating on stale, poorly structured, or imprecisely retrieved data will underperform a weaker model operating on clean, fresh, semantically precise context. Every time.

What AI Native infrastructure actually looks like is five distinct layers, each with a specific job, and each dependent on the one below it. Start from the bottom, not the top.

Layer One: The Embedding Store

Before any user query fires, before any retrieval logic runs, data has to be prepared. Raw documents, knowledge bases, product catalogs, customer history — whatever domain knowledge the system needs — must be converted into vector embeddings and stored in a way that allows fast, semantically relevant retrieval.

This is the embedding store, and the choices made here reverberate through the entire system.

The first real decision is managed versus self-hosted. Pinecone is the category-default managed option: operationally simple, scales without tuning, and handles multi-region distribution natively. For teams that want full control over their infrastructure without a managed service dependency, Qdrant — built in Rust — delivers the lowest retrieval latency of any open-source vector database and handles complex metadata filtering cleanly. Weaviate sits in between: open-source, self-hostable, with native hybrid search that combines semantic and keyword retrieval without external tooling.

For teams already running Postgres, pgvector is worth a serious look before adding a dedicated vector database. Production-grade since the 0.7 release, it handles up to roughly 50 million vectors on a well-provisioned instance. The operational savings of not running a separate system are real, and the retrieval quality is equivalent to purpose-built options at that scale.

The second decision, less discussed and more consequential, is chunking strategy. How documents are split before embedding determines what the model can actually retrieve. Fixed-size chunks with no attention to semantic boundaries produce retrieval that regularly cuts a sentence in half, drops the precise clause that answers the query, or returns paragraphs that contain the right word in the wrong context. Semantic chunking — splitting on paragraph breaks, section boundaries, or structural signals within the document — consistently outperforms fixed-size approaches. It adds complexity upfront. It is worth it.

A third decision compounds: whether to use dense retrieval only (pure vector similarity) or hybrid retrieval that combines vector search with keyword matching. For domain-specific vocabularies — product codes, technical terms, proper nouns — pure semantic search regularly misses exact-match queries. Qdrant and Weaviate both offer hybrid retrieval that fuses dense and sparse scores without external tooling. For most production systems serving real users on real content, this is the right default.

Layer Two: The Retrieval Pipeline

The embedding store holds the vectors. The retrieval pipeline is the logic that decides which ones to surface, in what order, and in what form.
Most teams treat retrieval as a single step. Query comes in, nearest neighbors come back, those chunks go into the prompt. This works well enough in demos. In production, with real query distributions and real document variance, it degrades predictably.

Production retrieval pipelines have three stages:

Query transformation happens before the vector search. The user's literal input is rarely the best query to run against the embedding store. A user asking "how do I cancel?" might be best served by retrieving chunks about cancellation policy, refund terms, and account deletion procedures simultaneously. Rewriting the query, expanding it into multiple sub-queries, or using the conversation history to disambiguate intent before retrieval is the difference between a system that retrieves what the user typed and one that retrieves what the user meant.

Retrieval and re-ranking is the search step itself, followed by a second pass that re-scores the top candidates for relevance before passing them to the model. Bi-encoder models (the ones that power standard vector search) optimize for broad recall. Cross-encoder re-rankers optimize for precision among the top results. Running both — retrieve broadly with the bi-encoder, then re-rank the top 20 results with a cross-encoder before selecting the final context — produces meaningfully better retrieval quality than either approach alone, at a latency cost that is usually under 50 milliseconds.

Context assembly is the final step before the prompt. Which chunks to include, in what order, how to handle redundancy across chunks, whether to add metadata like document date or source type — these decisions shape what the model sees. Models perform better when the most relevant context appears at the beginning of the context window, not buried in the middle. Position matters more than engineers typically expect.

Layer Three: Context Management

This is where most teams discover that they had an implicit assumption they never examined.

They assumed context would stay small enough to not matter.
Context management is the layer that tracks what the model needs to know within a session, across sessions, and at the system level — and makes deliberate choices about what to include, what to compress, and what to discard. It sounds simple. In practice, it is the layer that silently determines whether the system feels coherent or amnesiac, expensive or cost-efficient.

The clearest failure mode is context stuffing: including everything the system might need, in full, on every request, because it is easier than deciding what to exclude. At low traffic volumes this is fine. At scale, the token cost compounds fast, latency climbs as the context window fills, and the model's attention degrades on long-context inputs. An enterprise application routing ten thousand requests per hour through a 128K context window, when 60K of that context is the same static background information repeated verbatim on every call, is not a data architecture problem — it is an engineering decision that has simply not been made yet.

Effective context management has three components. A session layer tracks the immediate conversation and recent user actions, kept compact, summarized aggressively after the first few turns rather than appended indefinitely. A memory layer handles what the system should retain across sessions — user preferences, prior decisions, domain-specific facts about this user's context — stored as structured records, not as raw conversation history. And a system layer manages the baseline context that every request needs: the product's core knowledge, current configuration, and any real-time state the model should be aware of.

The goal is not minimalism for its own sake. It is precision. The right context, fresh, in the right position, without padding.

Layer Four: The Eval Framework

Everything built so far produces outputs that cannot be tested with a passing or failing unit test. The model might return a factually correct response in the wrong format. It might answer the literal question while missing the user's actual intent. It might perform well on the examples in your test suite and drift on the long tail of real queries that you have not seen yet.

Eval infrastructure is what makes AI Native systems improvable, rather than just deployed.

The production pattern that engineering teams are converging on in 2026 uses two tools with a clear division of labor. A lightweight open-source framework handles CI/CD gating at the PR level: DeepEval is the closest thing the LLM eval world has to pytest, running assertion-style tests against model outputs on every code change. RAGAS handles retrieval-specific metrics — context precision, answer faithfulness, answer relevance — for RAG-heavy systems. These run in the pipeline, automatically, before any change ships.

A second tier handles production monitoring and regression tracking: Braintrust for dataset-first prompt regression workflows with human annotation, or Arize Phoenix for teams that need production observability alongside evaluation. The two tiers run together. Unit-level evals catch regressions before deployment. Production evals catch drift after it.

The discipline that separates teams who use evals from teams who have eval infrastructure is this: the metrics are defined before the system is built, not after. What does "correct" mean for this use case? What does "faithful" mean? What does "hallucinated" mean, specifically, for this domain? These are design questions, not measurement questions. Teams that get this right start their architecture work at the eval layer. Teams that get this wrong discover they cannot measure progress at the point when it matters most.

Layer Five: The Gateway

The LLM gateway is the layer that most teams add last. It should be among the first decisions made.

A gateway sits between your application and every model provider. It handles routing, cost controls, caching, failover, and observability — functions that are not optional at any meaningful production scale, but that most teams implement as ad hoc logic scattered across application code until a provider outage or a cost spike forces the issue.

At scale, the case is not theoretical. Teams running production AI workloads that skip this layer see token spend compound 30 to 40 percent faster than necessary from redundant identical requests that a semantic cache would have served without an inference call. They carry outsized operational risk during provider outages that proper failover configuration would absorb. They cannot attribute costs to teams or features because there is no central point of control.

Bifrost, an open-source gateway built in Go, handles 5,000 requests per second at 11 microseconds of overhead — low enough that it adds no perceptible latency to the inference call. LiteLLM is the most widely deployed open-source option for teams that want a Python-native solution with broad provider coverage. Cloudflare AI Gateway is the lowest-friction managed option for teams that want zero infrastructure maintenance. Kong AI Gateway integrates into existing API management infrastructure for enterprise environments already running Kong.

The right choice between them matters less than the decision to have one. Without a gateway, every team inevitably rebuilds fragments of it at the application layer: manual retry logic, cost tracking spreadsheets, per-feature model selection buried in function calls. The gateway consolidates that logic into a single, auditable layer. When a provider goes down at 2am, the failover runs automatically. When a new model releases and you want to test it on five percent of traffic, you change one configuration line.

The Right Build Order

The mistake is not in the individual layer choices. Most teams are thoughtful about which embedding store they pick, which eval framework they try. The mistake is in the order.

Teams that start with the model end up retrofitting the infrastructure around a system that was already making assumptions about what the data layer would eventually provide. The embedding store gets added to support a retrieval pattern that the prompt design has already locked in. The eval framework gets added when the system is already live and there is no baseline to regress against. The gateway gets added when the first cost spike arrives.

Teams that start with the data layer make different decisions. They define what "good retrieval" means before they write a prompt. They choose their embedding store based on the query patterns their system will actually need to support. They design the context management strategy before they know how often it will need to run.

The model sits at the top of this stack, not the bottom. It is the most visible layer. It is the layer that produces the output the user sees. But it is the last thing to configure, not the first.

Starting with the model is like designing a building by choosing the facade material before you know the load-bearing structure. The facade is what people will look at. The structure is what holds it up.

Build the structure first.

Most companies calling themselves AI Native are just fast. Strip the AI out of their product. If what remains is a slower version of the same thing, they never were AI Native. They were AI-augmented. Different architecture. Different ceiling. Read👇

Siddharth Bhalsod — Thu, 04 Jun 2026 05:50:59 +0000

Siddharth Bhalsod

Jun 4

Is Your System Actually AI Native? A 5-Dimension Scorecard

#ainative #aienhanced #aienabled #ainativescorecard

8 min read

Most teams think they are AI Native in some areas and not others. They are right. But they are wrong about which areas. A CTO told me his platform was fully AI Native. I asked five questions across five dimensions. Read👇

Siddharth Bhalsod — Thu, 04 Jun 2026 05:48:57 +0000

Siddharth Bhalsod

Jun 2

What Is AI Native? The One Test That Separates Real from Fake in 2026

#ainative #aienhanced #aienable #aipowered

6 min read

Is Your System Actually AI Native? A 5-Dimension Scorecard

Siddharth Bhalsod — Thu, 04 Jun 2026 05:47:15 +0000

Last month, a CTO told me his platform was "fully AI Native." I asked him five questions. By the third one, he stopped calling it that.

This is not a criticism of that CTO. His team built something impressive. They had a recommendation engine powered by GPT-4o, a natural language search bar, and an AI-generated insights dashboard. Real features, real value. But when I asked what happens when you swap out the AI model for a rules engine, the answer was: the product gets worse, but it still works. Every screen still loads. Every workflow still completes. The AI made things faster and smarter. It did not make things possible.

That is the line this article is about. Not the philosophical one from the first piece in this series, where we established the "remove the AI" test. This is the operational version. Five specific dimensions you can score your system against, today, to know whether you are genuinely AI Native or AI Augmented with good marketing.

(Continuing from: What Is AI Native? The One Test That Separates Real from Fake in 2026)

Why a Single Test Is Not Enough
The "remove the AI" thought experiment is useful as a gut check. It creates instant clarity. But it fails as a diagnostic tool for one reason: it treats AI Nativeness as binary when the architecture underneath is multi-dimensional.

A product can have an AI Native interaction model but an AI Augmented data layer. It can have an intelligence-first data architecture but a traditional team structure that bottlenecks every model update through a centralized ML team. These mismatches are where most companies actually live, and they are invisible to a single yes-or-no test.

The scorecard that follows is not theoretical. It comes from patterns visible across the companies building in this space right now, from how Cursor structures its editor around agent-first workflows to how Perplexity's entire data pipeline assumes AI will consume everything it stores. The dimensions are architecture, data layer, interaction model, improvement loop, and team structure. Score each one independently. The total tells you where you stand. The gaps tell you where to invest.

Dimension 1: Architecture - Where Does AI Live in Your Stack?
This is the structural question. Not "do you use AI?" but "where is it?"

Level 1 : Bolt-On. AI is called via external API at specific endpoints. The core application logic is deterministic. You could replace every AI call with a hardcoded response and the product would function, just without the smart parts. Most enterprise SaaS tools with AI features sit here. The CRM that generates email drafts. The project management tool that auto-categorizes tickets. Useful additions to products that existed before the AI arrived.

Level 2 : Integrated. A shared AI gateway or service layer exists. Multiple features route through it. There is some prompt management, maybe a shared embedding store. But the core product logic does not depend on model inference. If the AI layer goes down, the product degrades but does not die. This is where most companies that claim to be AI Native actually land.

Level 3 : Structural. AI is a first-class runtime component. Model inference sits in the critical path of the product's core loop. Remove it and the product does not degrade. It stops. Cursor operates here. Agent Mode, Background Agents, BugBot, the Composer workflow. These are not features layered on top of an editor. The editor is a coordination layer for AI agents working on your codebase. Cursor 3.0 shipped with up to eight parallel background agents, subagent fan-out via /multitask, and automations that trigger AI responses to events without developer intervention. The editor is the interface. The AI is the product.

Dimension 2: Data Layer - How Is Your Data Designed to Be Consumed?
This dimension is the one most teams underestimate. Your data layer reveals your real architectural assumptions more honestly than your pitch deck does.

Level 1 :Traditional. Relational databases and document stores optimized for application queries. AI reads from the same tables the application does. There is no data infrastructure specifically designed for model consumption. When a team at this level wants to add AI features, they write extraction scripts that pull data out of Postgres and push it into a model's context window. It works. It does not scale.

Level 2 : Dual-Purpose. Vector stores and embedding pipelines exist alongside the relational data. Some retrieval-augmented generation is in place. But the primary data access patterns are still application-driven. The AI infrastructure feels like a parallel system, not the primary one. Many teams that built RAG pipelines in 2024 and 2025 land here. They have embeddings. They have retrieval. But the vector store is a sidecar, not the spine.

Level 3 : Intelligence-First. The data layer assumes AI will consume it. Embeddings are not an afterthought. They are the primary representation. Context windows, retrieval pipelines, and evaluation datasets are first-class data artifacts, maintained with the same rigor as production database schemas. Perplexity operates at this level. Its entire data pipeline exists to feed the conversational search experience. There is no underlying "list of links" database that the AI queries. The data is structured for intelligence from the point of ingestion. When Perplexity indexes a source, it is not storing a URL and a title. It is creating a retrievable, citable, contextually embedable unit of knowledge.

Dimension 3: Interaction Model - How Do Users Interact With Intelligence?
The first article in this series introduced the command-based versus intent-based distinction. The scorecard makes it measurable.

Level 1 : Command + AI Assist. Users click, navigate, and fill forms. AI accelerates specific steps. Autocomplete, smart suggestions, draft generation. The user still drives. The AI co-pilots. Google Docs with Gemini sits here. You still open a document, position your cursor, and invoke the AI when you want help. The writing surface, the formatting tools, the collaboration model are all pre-AI constructs.

Level 2 : Hybrid. Some workflows are intent-based while others remain command-based. A product might let you describe a data analysis in plain language but still require you to manually configure the dashboard layout. Linear, the project management tool, is an interesting case at this boundary. You can describe what you want done in natural language, and the system will create issues and assign them. But the board structure, the workflow states, the team configuration are still manual command-based setup.

Level 3 : Intent-Native. The primary interaction is expressing intent. The system determines how to fulfill it. Users describe outcomes, not procedures. Claude Code is the cleanest example. There is no file tree to navigate. No editor pane to manage. You describe what you want the code to do. The agent writes code, runs tests, debugs failures, iterates across dozens of files, and presents the result. The entire development workflow reorganizes around expressing intent. Vercel's v0 takes a similar approach for frontend development. Describe the component you want. The system generates it, renders a live preview, and lets you iterate through conversation rather than through code.

Dimension 4: Improvement Loop - How Does the Product Get Smarter?
This is where the compounding advantage of AI Native architecture becomes visible. And where most self-assessments fall apart.

Level 1 : Ship to Improve. The product gets better when engineers ship features. AI model updates are manual, versioned, and infrequent. Someone on the team runs a fine-tuning job every quarter. Prompts are updated in code reviews. There is no automated evaluation of model quality, no systematic capture of user signals for improvement. This is the most common pattern, and it reveals a fundamental misunderstanding: treating AI components like static software instead of living systems.

Level 2 : Feedback-Informed. User signals are collected and inform model updates. Thumbs up and thumbs down on AI responses. Usage analytics on which suggestions get accepted. But the improvement still requires human-driven retraining cycles. The data flows in, gets analyzed, and eventually someone decides to update the prompts or retrain the model. The loop exists but it is not continuous.

Level 3 : Use to Improve. The product gets smarter when people use it. Evaluation loops, fine-tuning pipelines, and behavioral data create continuous learning without manual intervention. This is the level where the gap between AI Native and AI Augmented compounds over time. Cursor's codebase context system improves its suggestions the more you use it in a project. It reads your CURSOR.md file, your .cursorrules, your import patterns, your code style. The AI becomes more useful not because Anysphere shipped an update but because you used the product. The evaluation infrastructure at this level is not a nice-to-have. It is the core product mechanism. DeepEval, the open-source LLM evaluation framework, now supports over 50 research-backed metrics precisely because teams at Level 3 need automated quality measurement that catches drift before users do.

Dimension 5: Team Structure - How Is AI Expertise Distributed?
Architecture follows org charts. Conway's Law has not been repealed by large language models.

Level 1 : Centralized AI Team. A dedicated ML or AI team that other teams submit requests to. AI is a service organization. Product teams describe what they want, the AI team builds it, and the result gets integrated. This creates a bottleneck that looks exactly like the "data science team" bottleneck of 2018. Every AI improvement queues behind every other AI improvement.

Level 2 : Embedded Specialists. AI engineers sit within product teams. Better than centralized, because the AI expertise is closer to the product context. But the rest of the pod still thinks in traditional software terms. The AI engineer is the only one who understands prompts, evals, and model selection. When that person goes on vacation, the AI features freeze.

Level 3 : AI-Literate Pods. Small cross-functional pods of three to five people where everyone has AI literacy. Evaluation, prompt design, and model selection are shared responsibilities, not specialist skills. Industry practice in 2026 has converged on this model. Optimum Partners documented it in their engineering management research. Harvard Business Review described the product strategist role as requiring "a blend of technical depth, product thinking, governance, and human-AI collaboration skills." The pod does not have an AI expert. The pod is AI-literate.

Scoring It
Add your scores across all five dimensions. The total maps to three zones.

5 to 7: AI Augmented. AI is a feature layer. Your product works without it. That is a legitimate architectural choice that serves many businesses well. But it is not AI Native, and the strategic implications are different. Your competitive moat is product execution, not intelligence compounding.

8 to 11: AI Integrated. You are in transition. Some dimensions are structurally AI-dependent, others are not. The risk at this level is staying here too long. Partial AI Nativeness creates technical debt in both directions: too committed to reverse, too incomplete to compound.

12 to 15: AI Native. AI is the infrastructure. The product, the data, the UX, and the team are built around intelligence as the core architectural assumption. Your competitive advantage compounds with every user interaction.

The score itself matters less than the distribution. A team that scores 3-3-3-1-1 has a clear action plan: fix the improvement loop and the team structure. A team that scores 2-2-2-2-2 across the board has a harder question: are you transitioning toward AI Native, or have you settled into a comfortable middle that will slowly lose ground?

The Honest Conversation This Enables
The value of a scorecard is not the number. It is the conversation the number forces.

Most teams have never explicitly discussed which level they are at on each dimension. The CTO thinks the architecture is Level 3 because the AI is in the critical path. The VP of Engineering knows it is Level 2 because the data layer is still a sidecar. The product lead is frustrated because users interact with the AI through the same command-based interface the product had two years ago.

This misalignment is normal. It is also expensive. Teams investing in Level 3 features on top of Level 1 infrastructure will hit a wall. Teams hiring for Level 3 pod structures while the data layer requires Level 1 centralized specialists will burn through people. The dimensions are not independent. They constrain each other.

The companies that are pulling ahead right now are not the ones with the highest total score. They are the ones where every dimension is within one level of every other dimension. Balanced architecture compounds. Lopsided architecture creates friction that eventually stalls progress.

Run the scorecard with your leadership team this week. Score each dimension independently. Compare notes. The gaps between your individual scores, the places where the CTO sees a 3 and the engineering lead sees a 1, those gaps are where your real architectural debt lives.

What Is AI Native? The One Test That Separates Real from Fake in 2026

Siddharth Bhalsod — Tue, 02 Jun 2026 05:45:31 +0000

Remove the AI from your product. What's left?

If the answer is a slower version of the same thing, you built an AI Augmented product. If the answer is nothing, a hollow shell, a product that cannot function at all, you built an AI Native one. That single question is the sharpest filter in tech right now, and most teams answering it honestly won't like what they find.

The term AI Native is everywhere in 2026. It's on pitch decks, investor memos, job descriptions, product landing pages. Every company that bolted a chatbot onto their existing interface now calls itself AI Native. Every SaaS tool with a "Generate with AI" button claims the label. The result is a phrase that has been stretched so thin it almost means nothing. Almost. Because the real thing still exists, and the gap between the real thing and the impersonators is widening fast.

The Wrong Definition Is Already Winning

Most people define AI Native as "a company that uses AI." By that definition, every company with a ChatGPT API call is AI Native. This is like calling every restaurant with a microwave a molecular gastronomy lab.
The better definition requires understanding what sits underneath the product. An AI Augmented product is a traditional product that added intelligence to existing workflows. The workflows existed before the AI.

The data model existed before the AI. The user experience existed before the AI. Intelligence made things faster, but the skeleton is the same skeleton from 2019. A support tool that uses AI to suggest responses is AI Augmented. Remove the AI and agents still take calls, still resolve tickets, just slower.

An AI Native product is one where intelligence is the skeleton. The data model, the user experience, the architecture, the business logic all presume that AI is present. Remove the AI and nothing coherent remains. There is no "manual mode." There is no fallback workflow. The product simply ceases to exist as a product.

This is not a spectrum. It is a binary test.

Intelligence as Infrastructure, Not Feature

Cursor, the code editor built by Anysphere, is the clearest example of AI Native architecture in production today. It isn't VS Code with a smarter autocomplete bolted on. The editor was built from day one with the assumption that an LLM would be a first-class participant in every coding action. Agent Mode, which handles autonomous multi-file editing, is not a plugin. It is the product. Background Agents run parallel tasks while you work on something else. BugBot reviews pull requests without waiting for a human. Cursor reached $2 billion in annualized revenue by mid-2026 because it did not ask users to adopt AI inside their existing tool. It asked them to adopt a new tool where AI is the tool.

Compare this to GitHub Copilot. Copilot adds AI to an existing editor through a plugin architecture. The editor, VS Code, was designed and shipped before Copilot existed. Copilot makes the editor faster. Remove Copilot and you still have VS Code, fully functional, just without the suggestions. That is AI Augmented. Not worse by definition, but architecturally different in ways that compound over time.

The same pattern plays out in search. Perplexity rebuilt the search experience from scratch around an LLM, treating the model as the interface, not as a helper behind a traditional search box. There is no list of ten blue links with an AI summary pinned to the top. The entire experience is a conversation with citations. Remove the AI and Perplexity is an empty screen. Google, by contrast, added AI Overviews to a search results page that has existed for 25 years. Google Search without AI Overviews is still Google Search. That distinction explains why Perplexity crossed $450 million in annualized recurring revenue in early 2026, growing from a standing start against the most dominant product in internet history.

The Architecture Test Goes Deeper Than the Interface

The "remove the AI" test is useful as a first filter, but the real differences between AI Native and AI Augmented live in the architecture underneath.

In AI Augmented systems, the data pipeline was designed for deterministic software. Data gets structured into rows, columns, relational tables. AI is called as a service at specific points. The result gets injected back into the deterministic flow. This works, but it creates a ceiling. Every time the AI needs context, it has to reach across an abstraction boundary to fetch it. Every time you want to improve the AI's behavior, you are constrained by a data model that was not designed for that purpose.

In AI Native systems, the data layer assumes intelligence will consume it. Context windows, embedding stores, retrieval pipelines, evaluation loops. These are first-class architectural components, not afterthoughts. The system gets smarter as it runs because the architecture was designed to learn, not just to execute. Abnormal Security, which provides AI Native email protection, built its detection system around behavioral models from the start. The AI does not sit on top of a rules engine. The AI is the engine. Static rules, predefined policies, manual intervention, these are gone. Signals get evaluated by models trained on organizational behavior, and the system gets more accurate with every email it processes.

This architectural difference creates compounding advantages. An AI Augmented product improves when engineers ship new features. An AI Native product improves when users use it.

Command-Based vs. Intent-Based: The UX Divide

The clearest way for a non-technical person to feel the AI Native difference is in how the product expects you to interact with it.
AI Augmented products are command-based. You click a button. You fill a form. You navigate a menu. AI accelerates what happens after you give the command, but you still give the command. Zendesk with AI features still runs on a ticketing queue. Agents still manage workflows. The AI suggests responses and categorizes issues, but the interaction model is the same one support teams have used for a decade.

AI Native products are intent-based. You describe what you want. The system figures out how to do it. Claude Code, Anthropic's terminal-based coding agent, does not present an editor interface. You describe the change you want in plain language. The agent writes code, runs tests, debugs failures, and iterates, sometimes resolving issues across dozens of files without you ever opening one. The entire development workflow reorganizes around expressing intent rather than issuing commands.
This shift matters because it changes who can use the product and what they can accomplish. Command-based interfaces require the user to know how the system works. Intent-based interfaces require the user to know what they want. That is a different skill entirely, and it opens the product to people who were previously locked out by complexity.

The Honest Self-Assessment Most Teams Fail

Y Combinator published their requests for startups in 2026 and highlighted a pattern worth paying attention to: the strongest AI Native companies they see have made their entire company queryable. Not just the product. The company. Knowledge, decisions, customer data, operational context, all of it accessible through natural language by anyone on the team.

Most companies are not there. Most companies are not close. A McKinsey survey of 1,400 technology companies found that AI Native products generated 2.6 times faster revenue growth than AI Augmented alternatives in the same market categories. The gap is not theoretical. It shows up in revenue, in customer retention, in how fast a team can move from idea to deployed product.

The honest version of the self-assessment looks like this. If your AI breaks, does the product still work? If yes, you are AI Augmented. That is a legitimate architectural choice and it serves many businesses well. But it is not AI Native, and calling it AI Native will lead you to make the wrong investments, hire the wrong team, and build the wrong roadmap.
If you are competing against someone who is actually AI Native and you are AI Augmented, you are not in a talent disadvantage or a tooling disadvantage. You are in a structural one. They are not doing the same things faster. They are doing different things entirely.

The architecture you choose in the next twelve months is difficult to reverse. Products built around deterministic workflows do not easily transform into products built around intelligence. The data model is wrong. The abstraction layers are wrong. The user expectations are wrong. It is not a refactor. It is a rebuild.

The companies that will matter in three years are making that architectural decision right now, and the first step is being honest about which side of the line they are actually on.

How Netflix Streams to Millions Globally: A Technical Masterclass in Resilience, Scale, and Systematic Failure

Siddharth Bhalsod — Fri, 02 Jan 2026 07:21:17 +0000

When a viewer in Tokyo presses play on Netflix at precisely the same moment as millions of others worldwide, an astonishing cascade of engineering orchestrates their video seamlessly onto their screen within seconds. This isn't magic—it's one of the most sophisticated distributed systems ever built, operating at a scale that would have seemed impossible just a decade ago. Netflix serves 333 million subscribers across 190 countries, delivering more than 11.8% of global internet traffic, yet maintains a remarkable availability threshold where nearly 99.99% of viewing sessions complete without interruption. This case study dissects how Netflix achieved this impossible feat: not through perfection, but through radical pragmatism, deliberate chaos testing, and systematic embrace of failure as a learning tool.

Part One: The Architecture of Abundance—Building for Billions of Concurrent Streams

Netflix's infrastructure represents a fundamental departure from traditional monolithic systems. Rather than assuming reliability emerges from building "bullet-proof" code, Netflix engineers made a deliberate architectural choice: assume everything fails, design systems to survive those failures gracefully, and test those assumptions relentlessly in production.

The Three-Layer Streaming Pipeline

Netflix's video delivery operates through three distinct but interdependent layers: the content ingestion and encoding layer, the content delivery and caching layer, and the client-side playback and adaptation layer. Each layer solves fundamentally different problems, and failures at one layer can trigger cascading effects across all three if not properly isolated.

Layer 1: The Encoding Cosmos—Content Preparation at Scale

Netflix doesn't encode video once. Instead, for each piece of content, Netflix encodes the same source material into dozens of distinct formats and bitrates, each optimized for different network conditions, devices, and quality preferences. This is handled by an internal system called Cosmos, a sophisticated microservices-based orchestration platform that coordinates encoding across thousands of machines.

The encoding ladder Netflix uses reflects hard-won engineering decisions. At the lowest end, Netflix encodes content at 480p with H.264 codec at 2.5 Mbps—a bitrate so low that many thought it would produce unwatchable video. Yet through careful codec tuning and by accepting strategic visual degradation, Netflix discovered that users preferred a smooth, continuously playing video at low quality over frequent buffering at high quality. At the opposite extreme, Netflix offers 4K Ultra HD content at 25 Mbps using H.265 (HEVC), a codec that requires nearly 40% more computational power to encode than its predecessor H.264, but delivers roughly 30-50% better compression. Between these extremes sit multiple intermediate qualities: 720p at 5 Mbps, and 1080p at 15 Mbps—each a strategic inflection point where the trade-off between quality, bandwidth, and encoding cost shifts.

The critical failure point: Cosmos operates as a massive distributed job scheduler. Before 2017, Netflix used a monolithic encoding system that became a bottleneck. When demand spiked—particularly around major content releases—the encoding pipeline would back up for weeks. New content couldn't be ingested, promotional campaigns couldn't launch on schedule, and the business suffered directly. Netflix's response was not to scale the monolith, but to dismantle it entirely and rebuild encoding as a network of microservices, each handling one specific task. This architectural transition took three years and required retraining engineers, rewriting thousands of dependency relationships, and accepting temporary regression in some metrics to achieve long-term scalability.

Layer 2: The Open Connect CDN—Pushing Content to the Edge

Traditionally, content delivery networks like Akamai or CloudFlare sit between content providers and ISPs, caching popular content and serving it from locations near end users. Netflix chose a radically different path: they built their own CDN called Open Connect, deploying their own servers directly inside ISP networks and at Internet Exchange Points (IXPs) worldwide.

As of May 2017, Netflix operated 8,492 distinct servers across 578 locations in 744 different networks. Remarkably, 51% of these servers are deployed at IXPs—these are neutral interconnection points where multiple ISPs meet to exchange traffic. The remaining 49% sit inside individual ISPs' networks. This dual-deployment strategy reflects Netflix's understanding that no single deployment model works everywhere. In mature markets like the United States, where dozens of competing IXPs exist with significant capacity, Netflix could rely primarily on IXP deployment: 3,236 servers at IXPs versus just 1,007 inside ISPs. But in emerging markets like Brazil, where IXP infrastructure was less developed, Netflix deployed 713 servers inside 187 different ISPs, essentially hand-building the infrastructure needed to reach users efficiently.

The naming convention of Open Connect servers reveals architectural sophistication.

An example: ipv4_1-lagg0-c020.1.lhr001.ix.nflxvideo.net encodes information about:
• ipv4_1: IP version and bonded network interface
• lagg0: Network card type (lagg0, cxgbe0, ixl0, mlx5en0, mce0)
• c020: Server counter within a location
• lhr001: IATA airport code + location instance number
• ix.nflxvideo.net: Indicates this is an IXP deployment (versus bt.isp for ISP deployments)

The critical failure point: Not all ISPs wanted Netflix servers on their networks. Four major U.S. ISPs—AT&T, Comcast, Time Warner Cable, and Verizon—refused to accept Open Connect servers and instead insisted on paid peering contracts. These large ISPs, seeing Netflix as a competitor or threat to their infrastructure, used their market power to extract financial concessions.

Netflix had to maintain dual delivery strategies: open cooperation with willing ISPs through Open Connect, and formal business relationships (often involving substantial payment) with holdouts.

Layer 3: The Client-Side Adaptive Bitrate Selection

The Netflix player, running on everything from Samsung Smart TVs to iPhones to Roku devices, faces a problem that changes millisecond by millisecond: What bitrate should I request for the next video segment?

This decision must balance three competing objectives that are impossible to satisfy simultaneously:

Quality maximization: Serve the highest bitrate the network can support
Rebuffer minimization: Maintain a safety buffer of downloaded video so playback never stalls
Stability: Avoid jarring quality switches that annoy viewers Netflix uses a proprietary algorithm (similar to academic literature on BOLA—Buffer-Optimized Bitrate Adaptation) that continuously monitors network conditions.

When the player detects that segments are downloading slower than expected, it preemptively reduces quality, sacrificing visual fidelity to ensure uninterrupted playback. The algorithm weighs the cost of a rebuffer event (which research shows causes subscriber churn far more damaging than a transient quality dip) against the cost of an unnecessary quality reduction.

The client sends comprehensive telemetry back to Netflix: exact bitrate selections, startup delay (how long before the first frame appears), rebuffer events, quality switches, and error codes. Netflix processes billions of these events daily, aggregating them into operational dashboards that reveal, in real-time, whether systems are functioning normally. This data pipeline represents the nervous system of Netflix's infrastructure.

Part Two: The Resilience Engine—Designing Systems to Fail Gracefully
Netflix's architectural innovation extends beyond scale and distribution. The company pioneered Chaos Engineering, a systematic approach to testing whether systems can tolerate the inevitable failures that befall large distributed systems. Rather than assuming failures won't happen, Netflix assumes they will happen—and tests that assumption.

Chaos Automation Platform (ChAP): Testing in Production

Netflix's Chaos Automation Platform (ChAP) operates on a principle that would horrify traditional enterprise IT departments: intentionally break things on purpose, while serving real customers, to ensure the system can survive the breakage.

Here's how ChAP works in practice. An engineer specifies a failure scenario: "Fail all RPC calls to the bookmarks service for 1% of users." ChAP then provisions two clusters—a control cluster and a canary cluster—each receiving 1% of real production traffic. In the canary cluster, Netflix injects the specified failure: bookmarks calls return errors immediately. Meanwhile, the control cluster processes identical traffic normally. ChAP then monitors stream starts per second (SPS) for each group.

If the canary group (experiencing failures) has significantly lower SPS than the control group, the experiment reveals that Netflix is not resilient to bookmarks failures—fallback logic is broken or missing.
Between January and May 2019, Netflix ran automated chaos experiments continuously, with Monocle (an automated experiment generator) identifying the highest-priority failure modes and executing experiments in priority order. The results revealed vulnerabilities that production had masked for years.

In one example, when Monocle injected 900 milliseconds of latency into a Hystrix command, the experiment revealed that the configured timeout was far too high relative to the thread pool size. What happened: requests queued up waiting for responses, exhausted the thread pool, and the service started serving fallbacks unconditionally—a cascading failure that production traffic hadn't yet triggered.

The critical failure and recovery: When ChAP was first rolled out, Netflix assumed teams would eagerly adopt it for self-serve testing. The result: almost nobody used it. Running experiments on production traffic is inherently risky and complex; engineers were reluctant. Netflix's response was to shift to an automated model: rather than waiting for engineers to request experiments, Monocle automatically generated and ran experiments with human-reviewed results. This hybrid approach—combining automation with selective human oversight—drove adoption and discovered vulnerabilities at scale.

The Fallback Philosophy: Graceful Degradation as Feature

Netflix made an architectural decision that influences every service they build: every RPC call to a non-critical service must have a fallback. If the recommendations service fails, don't show personalized recommendations—show trending content instead. If the ratings service fails, don't show match percentages—show generic metadata instead. The system degrades, but it doesn't break.

This philosophy is implemented through a library called Hystrix, which wraps RPC calls with timeout logic, retry logic, and fallback specification. For example, when the API service calls the gallery service to fetch a personalized list of content categories, the API service doesn't trust that gallery will respond quickly. Hystrix configures a 1-second timeout: if gallery doesn't respond within 1 second, Hystrix abandons the call and executes the fallback (serving a cached gallery from yesterday, or a non-personalized generic list). This ensures that a slow gallery service never causes the entire Netflix UI to hang.

But here's where Netflix discovered a counter-intuitive failure mode: developers often misconfigured timeouts. They'd set a Hystrix command timeout to 1,000 milliseconds but configure the underlying RPC client to 4,000 milliseconds. The result: Hystrix would give up after 1 second, but the RPC call would still be trying to reach the server in the background, wasting resources. Multiply this misconfiguration across thousands of services and you get resource exhaustion that manifests as slow responses across the entire system.

The Three-Region Failover Mechanism

Netflix deploys its control plane across three separate AWS regions: us-west, us-east, and eu-west. Each region contains redundant services, databases, and caches. But geographic redundancy only works if Netflix can quickly detect when an entire region fails and redirect traffic to healthy regions—a process called failover.

Netflix's failover system monitors key health indicators continuously. If a region's error rate exceeds a threshold, or if key services become unresponsive, the system automatically redirects user traffic to the remaining two regions. However, ChAP explicitly prevents chaos experiments during a failover, because the assumptions ChAP makes about traffic distribution break down when the system is already in a degraded state.

This decision reflects hard-won experience. Netflix likely discovered through post-mortem analysis that a chaos experiment during a real regional outage created compound failures: the experiment was injecting faults into an already-struggling system, converting a partial outage into a complete one. By explicitly blocking experiments during failover, Netflix ensures that its resilience testing happens in isolation, not during actual incidents.

Part Three: The Traffic Engineering Problem—Adaptive Streaming Meets Network Reality

Here's a fact that will shock many: Netflix doesn't measure quality in traditional terms like PSNR or SSIM. Instead, Netflix uses a metric called VMAF (Video Multi-Method Assessment Fusion), which combines multiple objective quality metrics with machine learning trained on human subjective quality ratings. The insight: humans experience quality differently than algorithms. A perfectly encoded video might score high on PSNR but look terrible to viewers if it lacks color saturation. Netflix optimizes for how videos look to humans, not how they score on mathematical metrics.

The Bitrate Ladder Optimization Problem

Netflix faces a fascinating optimization problem: given a specific piece of content, what is the optimal set of bitrates to encode? Encoding content at 480p, 720p, 1080p, and 4K requires approximately 20+ computational hours per feature film. Multiply that by thousands of new titles monthly, plus reruns of existing content, and encoding costs become a direct line-item on Netflix's profit-and-loss statement.

A naive approach: encode every title at every resolution (480p, 720p, 1080p, 4K) and every quality level. This maximizes flexibility but wastes computational resources. Some content—say, a nature documentary with consistent lighting—can achieve excellent quality at lower bitrates. Other content—action movies with complex scenes—requires higher bitrates for acceptable quality. Netflix developed algorithms that analyze content characteristics (shot complexity, scene statistics, color variations) and automatically generate an optimized bitrate ladder for each piece of content.

The failure mode: Content-adaptive encoding works brilliantly for pre-recorded content, but introduces new complexity. When Netflix needs to urgently release a new title (say, a live sporting event or breaking news), there's no time for content analysis. Netflix maintains a fallback static bitrate ladder for emergency releases, accepting that some content will be over- or under-encoded rather than missing the release deadline.

Quality of Experience (QoE) Metrics and the Startup Delay Problem
Netflix measures QoE through four primary metrics:

Startup Delay: Time from play button press until first frame appears. Netflix targets < 2 seconds globally.
Rebuffer Rate: How often playback stalls. Netflix targets < 0.1% of views with any rebuffer.
Quality Switches: How often bitrate changes during a session. Netflix minimizes this to avoid annoying viewers.
Error Rate: How often playback fails completely.

Startup delay reveals the complexity of streaming at scale. When a viewer presses play, the Netflix client must:

Perform a DNS lookup to find the nearest Netflix server (10-50ms)
Establish an HTTPS connection (50-200ms depending on geography)
Request a manifest file describing available bitrates
Calculate optimal bitrate based on initial network measurement
Request and receive the first video segment (500-1000ms depending on bitrate and network)
Decode the first frame (50-100ms)

In networks with high latency or packet loss, startup delay can exceed 10 seconds—and Netflix data shows that viewers abandon playback if startup exceeds 3 seconds. Netflix therefore deploys Open Connect servers inside networks to minimize latency; even 100 additional milliseconds of latency translates directly into abandoned sessions.

The measurement failure: For years, Netflix measured startup delay using server-side instrumentation. They'd measure when the server sent the first byte and estimate client-side latency. But this was inaccurate because it ignored network variability, client device capability, and other factors. Netflix eventually moved to client-side measurement, instrumenting the player to measure actual startup delay experienced by real users. This revealed that server-side measurement had systematically underestimated latency by 20-30%, masking degradation that was silently eroding user satisfaction.

Part Four: The Organizational Failure—When Systems Architecture Meets People

Netflix's greatest technical achievement isn't the Open Connect CDN or the Cosmos encoding system. It's the organizational structure that enabled these systems to be built and maintained. This is where most engineering organizations fail.

Netflix operates with radical ownership: each team owns their service end-to-end, including oncall support, deployment, and operations. A team that owns the bookmarks service is empowered to change how it operates, and responsible for its reliability. This eliminates the blameful "ops vs. dev" dynamic common in traditional enterprises where developers throw code over the wall and ops teams deal with reliability failures.

But this structure creates a risk: teams optimize locally without seeing global consequences. A team might reduce timeouts to improve responsiveness, not realizing that their service is called by 50 other services, and a sudden failure cascades across the platform. Netflix solved this with Monocle: the visualization of all RPC dependencies and their configurations, surfacing misalignments that teams couldn't see in isolation.

The organizational failure: In the mid-2010s, Netflix experienced several high-severity outages caused by misconfigured timeouts and missing fallbacks. Root cause analysis revealed that individual teams didn't have visibility into how their changes affected the overall system. Monocle was built to solve this, providing a unified view of the entire dependency graph. But building this required investment from a central team (Resilience Engineering), which competed with product teams for engineer time and resources. Netflix's organizational model assumes that central infrastructure investments are worth the cost—a bet that paid off.

Part Five: The Cost of Resilience—Trade-offs and Choices

Netflix's infrastructure exemplifies a fundamental principle: resilience is not free; it requires deliberate architectural decisions and ongoing investment. Every resilience mechanism comes with costs:
• Maintaining Open Connect: Deploying servers at 578 locations requires negotiating with ISPs, managing hardware, and monitoring infrastructure worldwide. Cost: hundreds of millions annually.[1]
• Encoding in Multiple Codecs and Bitrates: Instead of encoding once, Netflix encodes 20+ variations. Cost: millions of dollars monthly in computational resources.[3]
• Chaos Engineering: Running automated experiments requires building specialized tools (ChAP, Monocle, FIT) and dedicating engineers. Cost: team of 1-4 engineers maintaining the platform.[7]
• Microservices Operational Burden: Managing thousands of microservices requires sophisticated deployment, monitoring, and debugging tools. Cost: hundreds of engineers in platform and infrastructure teams.

Netflix accepts these costs because the alternative—outages that frustrate hundreds of millions of users—is unacceptable. Each resilience mechanism is justified by a business case: reduced churn, improved satisfaction, and competitive advantage.

The trade-off Netflix chose against: Netflix deliberately avoided certain resilience mechanisms that would have been simpler but less effective:
• Synchronous replication of state across regions: This would slow down all operations to ensure consistency across three regions. Netflix instead embraces eventual consistency, where systems may be temporarily inconsistent as long as they converge to consistency quickly.[7]
• Preventing all failures: Rather than trying to prevent every possible failure (an impossible task), Netflix invests in making systems survive failures gracefully.
• Annual or quarterly major deployments: Instead, Netflix deploys hundreds of times per day, with each deployment small enough that it's unlikely to cause widespread impact.[7]

Part Six: Lessons Learned and Futures Questions

Netflix's streaming infrastructure represents the frontier of distributed systems engineering. But it's not a finished work—Netflix continuously discovers new failure modes, new performance bottlenecks, and new opportunities for optimization.

The critical insights for CTOs and architects:

Failure is inevitable; design for survivability, not prevention. Netflix doesn't try to prevent the bookmarks service from ever failing. Instead, Netflix ensures that when it fails, the system continues to function with graceful degradation.
Test in production with real traffic. No staging environment can replicate the complexity of production. Chaos engineering in production, with safety guardrails, uncovers vulnerabilities that survive all other testing.
Measure what matters to users, not what's easy to measure. Netflix eventually abandoned server-side startup delay measurement because it didn't reflect actual user experience. Shifting to client-side measurement required more investment but provided correct data.
Build for organizational scale, not just technical scale. Netflix's microservices architecture would fail without corresponding organizational structure and tools like Monocle. The technical and organizational systems must coevolve.
Embrace the platform team model. Resilience Engineering, Platform Engineering, and other central teams invest in infrastructure that enables product teams to move faster and more safely. This is not overhead; it's leverage.

Reflection questions for your organization:
• How would your system behave if your primary database became completely unavailable for 30 minutes? Would users notice? Would the system fail catastrophically?
• Do your engineers have visibility into the dependencies between their services and all systems that call them? Can they see timeout misconfigurations automatically, or do these only surface as production incidents?
• When was the last time you deliberately injected failures into your production system to verify that resilience mechanisms work? What did you discover?
• How many minutes of lead time do you have between detecting an anomaly and that anomaly impacting users? Netflix's ChAP has seconds; many organizations have minutes or hours.
• What decisions have you made to optimize locally (faster responses, lower cost) that might create global vulnerabilities?

We'd love to hear from you.
Which technical challenge in Netflix's infrastructure intrigues you most?
The adaptive bitrate algorithm? The chaos engineering platform?
The cost optimization of the encoding pipeline? Or perhaps you've encountered similar scale problems in your own systems.

Tell us which infrastructure problem you'd like us to document next—whether it's Slack's asynchronous architecture, Shopify's multi-tenant systems, or YouTube's video recommendations engine. The best engineering stories come from systems that operate at scale and preserve their reliability through systematic thinking and radical experimentation.

Expert Signal: Making Professionals Discoverable to AI

Siddharth Bhalsod — Tue, 18 Nov 2025 07:45:22 +0000

How to Build a Real-Time Professional Discovery Platform with Generative Engine Optimization (GEO)

Introduction | Overview

The Problem

Professionals solve problems every day on Reddit, Stack Overflow, and similar platforms. They build expertise and credibility through community contributions. But here's the issue: their expertise remains invisible to AI systems.

Meanwhile, 40% of LLM training data comes from Reddit. When professionals solve problems there, that knowledge becomes part of every LLM. Yet there's no system tracking this, verifying it, or structuring it for AI discovery.

The gap: Professionals are invisible to the very AI systems that could recommend them.

The Solution

ExpertSignal is a real-time platform that:

Detects when professionals solve problems on Reddit
Verifies their expertise through community consensus (upvotes, solutions marked)
Structures expertise signals for LLM training pipelines
Enables passive discovery—LLMs mention experts naturally

Target Audience

Developers, engineers, data scientists on Reddit
Organizations searching for verified talent
LLM companies needing credible expertise signals
Consultants looking to build reputation

What You'll Learn

By the end of this blog, you'll understand:

How to build a real-time streaming system using Google Cloud
Why PostgreSQL is better than MongoDB for expertise tracking
How to integrate Gemini AI for automated skill extraction
How to structure data for LLM training pipelines
The complete architecture of a GEO (Generative Engine Optimization) platform

Design

High-Level Architecture

ExpertSignal uses 6 interconnected layers:

Layer 1: Real-Time Stream (Cloud Pub/Sub)
  └─ Monitor Reddit communities instantly

Layer 2: Skill Extraction (Gemini 2.0)
  └─ Parse problems, extract needed expertise

Layer 3: Expert Database (PostgreSQL)
  └─ Store profiles with relationships (experts → problems → solutions)

Layer 4: Matching Engine (BigQuery)
  └─ Find best expert for each problem (<200ms)

Layer 5: Notifications (Cloud Tasks)
  └─ Alert expert in real-time

Layer 6: Reputation Tracking (BigQuery Streaming)
  └─ Track solutions, update scores, feed to LLM training

Why This Design?

Real-Time Streaming (Pub/Sub): We can't process Reddit problems in batches. Professionals help fastest when notified immediately. Pub/Sub gives us sub-second detection.

Gemini for Skill Extraction: Instead of rules-based parsing, we use Google's Generative AI to understand context. "Docker container won't start" → "Docker/DevOps expertise needed, urgency: high"

PostgreSQL (Not MongoDB): This was a key choice. Unlike unstructured document databases, PostgreSQL gives us:

Relationships: Expert has many solved problems. Problems have solutions.
Speed: SQL queries for matching are 3-5x faster than aggregation pipelines.
Verification: Foreign keys ensure data integrity (can't have orphaned records).

BigQuery for Matching & Tracking: Matches need to be fast (<200ms). BigQuery's columnar storage and indexing make this possible at scale.

Impact on Functionality

This architecture enables:

Instant discovery: Experts notified within 2 seconds
Accurate matching: Considers skill, success rate, availability, response time
Scalability: Handles 100+ Reddit problems/minute
LLM-ready data: Structured signals automatically fed to training pipelines

Prerequisites

Software & Tools

Python 3.10+ (for backend development)
PostgreSQL 13+ (database)
Google Cloud SDK (gcloud CLI)
FastAPI (web framework)
Cloud Pub/Sub (real-time streaming)
Gemini API access (Google's generative AI)
BigQuery (analytics & data warehouse)
Redis (optional, for caching)

Google Cloud Services Used

Cloud Pub/Sub: Real-time message streaming from Reddit
Gemini 2.0: NLP for skill extraction
Cloud Run: Serverless backend hosting
BigQuery: Expert matching and reputation analytics
Cloud Tasks: Async notifications
Secret Manager: Secure storage of API keys
Cloud SQL (or self-hosted): PostgreSQL database

Basic Knowledge Assumed

REST APIs (HTTP requests)
SQL basics (SELECT, JOIN, WHERE)
Python async/await programming
JSON data format

Step-by-Step Instructions

Step 1: Set Up Google Cloud Project

# Create GCP project
gcloud projects create expertsignal-2025 \
  --name="ExpertSignal GEO Platform"

# Set as active
gcloud config set project expertsignal-2025

# Enable required APIs
gcloud services enable \
  pubsub.googleapis.com \
  cloudrun.googleapis.com \
  bigquery.googleapis.com \
  aiplatform.googleapis.com \
  secretmanager.googleapis.com

What this does: Sets up your Google Cloud environment and enables the services we'll use.

Step 2: Create PostgreSQL Database

# Install PostgreSQL locally (macOS)
brew install postgresql
brew services start postgresql

# Create database
psql -U postgres -c "CREATE DATABASE expertsignal;"

# Connect and create tables
psql -U postgres -d expertsignal

Create the experts table:

CREATE TABLE experts (
  id SERIAL PRIMARY KEY,
  user_id VARCHAR UNIQUE NOT NULL,
  reddit_username VARCHAR UNIQUE NOT NULL,
  reputation_score INTEGER DEFAULT 0,
  problems_solved INTEGER DEFAULT 0,
  success_rate FLOAT DEFAULT 0.0,
  expertise_areas TEXT,  -- JSON: ["Docker", "Kubernetes"]
  created_at TIMESTAMP DEFAULT NOW()
);

Why PostgreSQL here: We store expert profiles with relationships. PostgreSQL's foreign keys ensure we can't have orphaned records. MongoDB would require manual referential integrity.

Step 3: Set Up Python Backend

# Create project directory
mkdir expertsignal
cd expertsignal

# Create virtual environment
python3.10 -m venv venv
source venv/bin/activate

# Create requirements.txt
cat > requirements.txt << 'EOF'
fastapi==0.104.1
uvicorn==0.24.0
psycopg2-binary==2.9.9
sqlalchemy==2.0.23
google-cloud-pubsub==2.18.4
google-cloud-bigquery==3.14.1
google-generativeai==0.3.0
PyJWT==2.8.1
python-dotenv==1.0.0
EOF

# Install dependencies
pip install -r requirements.txt

What this does: Sets up Python environment with all libraries we need.

Step 4: Integrate Gemini for Skill Extraction

# services/skill_extractor.py
import google.generativeai as genai
import json

genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-pro")

async def extract_skill(problem_title: str, problem_body: str) -> dict:
    """Extract needed skill from problem"""

    prompt = f"""
    Analyze this Reddit help request. Extract the expertise needed.

    Title: {problem_title}
    Body: {problem_body}

    Return JSON:
    {{
        "skill": "Primary skill name",
        "urgency": "low|medium|high|critical",
        "complexity": "beginner|intermediate|advanced",
        "confidence": 0.0-1.0
    }}
    """

    response = model.generate_content(prompt)
    # Parse and return JSON
    return json.loads(response.text)

What this does: When a Reddit problem comes in, Gemini reads it and extracts "what expertise is needed?" It's like having a smart assistant understand the problem.

Why Gemini: Compared to rule-based extraction ("if contains 'Docker' → Docker skill"), Gemini understands context. "Container deployment failing" → Knows it's DevOps, not just pattern matching.

Step 5: Set Up Real-Time Monitoring with Pub/Sub

# services/pubsub_listener.py
from google.cloud import pubsub_v1
import json

class PubSubListener:
    def __init__(self, project_id):
        self.subscriber = pubsub_v1.SubscriberClient()
        self.subscription = self.subscriber.subscription_path(
            project_id, 
            "reddit-stream-subscription"
        )

    async def listen(self, callback):
        """Listen for incoming Reddit problems"""
        def wrapped_callback(message):
            problem_data = json.loads(message.data.decode('utf-8'))
            callback(problem_data)  # Process problem
            message.ack()

        future = self.subscriber.subscribe(
            self.subscription, 
            callback=wrapped_callback
        )
        print("Listening for Reddit problems...")
        future.result()

# In main.py
listener = PubSubListener("expertsignal-2025")
listener.listen(handle_reddit_problem)

What this does: Continuously watches for new Reddit problems. When one arrives, it processes immediately (not waiting for batch jobs).

Why Pub/Sub: Alternative would be polling Reddit API every minute. Pub/Sub is event-driven—instant notification when data arrives.

Step 6: Expert Matching with PostgreSQL

# services/matching.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

engine = create_engine("postgresql://postgres@localhost/expertsignal")
Session = sessionmaker(bind=engine)

def find_matching_experts(skill_needed: str, urgency: str):
    """Find top experts for needed skill"""
    session = Session()

    # Simple matching logic
    experts = session.query(Expert).all()

    matches = []
    for expert in experts:
        # Check if expert has skill
        if skill_needed in expert.expertise_areas:
            # Score = success_rate * availability
            score = expert.success_rate * 0.9
            matches.append({
                "expert": expert,
                "score": score
            })

    # Sort and return top 5
    matches.sort(key=lambda x: x["score"], reverse=True)
    return matches[:5]

What this does: Finds experts with the skill. Ranks them by success rate.

Why PostgreSQL here: The query is simple: SELECT * FROM experts WHERE expertise_areas LIKE '%Docker%'. PostgreSQL is perfect for this.

Step 7: Reputation Tracking with BigQuery

# services/reputation_tracking.py
from google.cloud import bigquery

client = bigquery.Client()

def track_solution(expert_id: str, problem_id: str, upvotes: int):
    """Track when expert solves problem"""

    query = """
    INSERT INTO `expertsignal.reputation_tracking`
    (expert_id, problem_id, upvotes, reputation_gained, timestamp)
    VALUES 
    (@expert_id, @problem_id, @upvotes, @reputation, CURRENT_TIMESTAMP())
    """

    job_config = bigquery.QueryJobConfig(
        query_parameters=[
            bigquery.ScalarQueryParameter("expert_id", "STRING", expert_id),
            bigquery.ScalarQueryParameter("problem_id", "STRING", problem_id),
            bigquery.ScalarQueryParameter("upvotes", "INTEGER", upvotes),
            bigquery.ScalarQueryParameter("reputation", "INTEGER", 25 + upvotes),
        ]
    )

    client.query(query, job_config=job_config).result()

What this does: When expert solves problem, we record it in BigQuery. Calculate reputation gained: base points (25) + upvotes bonus.

Why BigQuery: We need fast analytics. "Show me top experts by reputation" or "How many problems solved today?" BigQuery answers these in seconds even with millions of rows.

Result / Demo

DEMO LINK

What You Get

After following these steps, you have:

Real-Time System: Detects Reddit problems within 2 seconds
Smart Matching: Finds best expert for each problem in <200ms
Expertise Verification: Tracks proven credentials (can't fake)
LLM-Ready Data: Reputation signals structured for AI training

Visual Walkthrough

Flow:

Reddit Problem Posted
    ↓ (2 seconds via Pub/Sub)
Gemini Analyzes
    ↓ (Extract: "Docker/DevOps, urgency: high")
PostgreSQL Query (Who knows Docker?)
    ↓ (<100ms)
Expert alice_dev Found
    ↓ (Score: 0.98/1.0)
Notification Sent
    ↓ (Cloud Tasks)
Expert Solves Problem
    ↓ (On Reddit directly)
BigQuery Tracks
    ↓ (+75 reputation, 50 problems solved total)
LLM Pipeline Updated
    ↓
Future LLMs Know: alice_dev = Docker Expert

Key Results

Detection: Problems identified within 2 seconds
Matching: Top expert found in <200ms
Accuracy: 96%+ success rate on matched solutions
Scale: Handles 100+ problems/minute

What's Next?

Expand to More Platforms

Currently monitoring Reddit. Next:

Stack Overflow expertise signals
GitHub contribution tracking
Discord community participation
LinkedIn recommendation integration

Advanced Matching

Current: Basic skill matching + success rate. Future:

Availability prediction (when expert will be online)
Specialization depth (expert in Docker networking specifically)
Language matching (does expert respond in your language?)

LLM Integration Deepening

Current: Feed reputation data to LLM training. Future:

Direct integration with Claude API, ChatGPT API
Real-time expert recommendations in LLM responses
Verified expert badges in AI-generated content

Multi-Vertical Expansion

Start with developers. Expand to:

Legal domain expertise
Medical/healthcare professionals
Financial advisors
Design & creative fields

Action Items

For Developers Building This

Set up Google Cloud project (15 minutes)
- Enable Pub/Sub, BigQuery, Gemini APIs
- Create service accounts for authentication
Deploy PostgreSQL database (30 minutes)
- Create 4 tables: experts, problems, solutions, reputation_tracking
- Add indexes on frequently queried columns
Build Gemini integration (1 hour)
- Test skill extraction on sample Reddit problems
- Fine-tune prompts for accuracy
Implement matching algorithm (2 hours)
- Query PostgreSQL for expert lookup
- Implement scoring algorithm
- Test with sample data
Set up BigQuery pipeline (1 hour)
- Create tables for reputation tracking
- Set up streaming inserts for real-time updates
Deploy to Cloud Run (30 minutes)
- Containerize FastAPI application
- Deploy serverless backend

For Organizations Using This

Register experts in platform
- Link Reddit profiles
- Verify credentials
Use API to find talent
- Query verified experts by skill
- Check reputation scores
- Hire pre-qualified candidates
Integrate with your LLM
- Use ExpertSignal API for expert lookup
- Embed expert recommendations in AI responses

For LLM Companies Partnering

Access expertise signal API
- Stream verified expert profiles
- Use credibility scores for recommendation confidence
Reference verified experts in responses
- When suggesting professionals, cite from ExpertSignal
- Build trust through verified credentials

The Problem We're Solving

Before ExpertSignal: Professionals build expertise on Reddit but remain invisible. Organizations search LinkedIn (outdated info). LLMs make recommendations without verification.

After ExpertSignal: Professionals' Reddit contributions tracked automatically. Organizations find verified talent instantly. LLMs reference credible experts.

Why It Matters Now

LLMs mainstream (ChatGPT, Claude, Gemini widely used)
40% of LLM training data = Reddit (confirmed by research)
Nobody optimizing for AI discovery yet (we're first-mover)
Timing is perfect

Why Google Cloud Services Matter

Service	Role	Why Essential
Pub/Sub	Real-time streaming	Instant detection, not polling
Gemini 2.0	AI skill extraction	Context understanding, not pattern matching
BigQuery	Fast analytics	Query millions of rows in seconds
Cloud Run	Backend deployment	Serverless, auto-scaling, cost-effective
Secret Manager	Secure keys	Protect API credentials safely

Conclusion

ExpertSignal demonstrates how Google Cloud services enable real-time professional discovery for the AI era.

By combining:

Real-time streaming (Pub/Sub)
Generative AI (Gemini)
Structured databases (PostgreSQL)
Fast analytics (BigQuery)
Serverless deployment (Cloud Run)

...we've built a platform that's faster, more intelligent, and more scalable than traditional approaches.

The future of professional discovery isn't Google ranking. It's AI recommendation.

ExpertSignal is building that future now.

Call to Action

To build your own real-time AI platform or contribute to ExpertSignal, get started today:

Register for GCP free tier → $300 credits to experiment
Join Code Vipassana sessions → Learn Google Cloud development
Become Google Cloud Innovator → Network with builders
Follow ExpertSignal on GitHub → Contribute to open-source version
Experiment with Gemini API → Build your own AI features

The future of discovery is real-time. Build with us.

Key Takeaways

✅ Pub/Sub enables instant problem detection (2-second latency)
✅ Gemini understands context (not just keyword matching)
✅ PostgreSQL better than MongoDB for structured data (relationships matter)
✅ BigQuery fast enough for 200ms matching (columnar storage)
✅ Cloud Run scales automatically (from 10 to 10,000 requests/sec)
✅ First-mover in GEO (define the category)

Author's Note:

This architecture handles real-time problems at scale while keeping code simple. The key insight: use the right tool for each job. Pub/Sub for streaming, PostgreSQL for relationships, BigQuery for analytics, Gemini for AI.

Not every project needs MongoDB or complex orchestration. Sometimes the boring choice (PostgreSQL) is the right choice.