DEV Community: Talvinder Singh

The Context Window Pricing Collapse

Talvinder Singh — Mon, 20 Apr 2026 07:49:27 +0000

Claude's Opus 4.6 and Sonnet 4.6 now ship with 1M token context windows at standard pricing. No premium tier. No waitlist. Just 1M tokens, available to everyone.

This single change killed the moat that every RAG startup built in 2023-24.

The competitive advantage those companies sold was never retrieval quality. It was working around small context windows. That constraint just disappeared.

The retrieval layer was a workaround

Between 2023 and early 2025, hundreds of startups raised money on the same pitch: "LLMs have small context windows, so you need our retrieval layer to feed them the right chunks." Document Q&A companies. Legal AI startups. Enterprise search tools. Internal knowledge bases. All built on the same assumption: context windows are expensive and scarce, so smart retrieval is the product.

That was a reasonable bet when GPT-4 had 8K tokens and Claude had 100K at a premium. Chunking, embedding, reranking, and retrieval pipelines were genuine engineering problems. The companies that solved them well could charge for it.

But the economics just shifted. When you can drop an entire codebase, a full legal contract set, or a year of customer support tickets into a single prompt at standard pricing, the retrieval layer stops being a product and starts being a feature. A feature that the model provider gives away for free.

Why this hits Indian AI startups hardest

The Indian AI ecosystem produced a disproportionate number of RAG-focused companies. The problem was well-defined. The engineering talent was available. The capital requirements were low. Document processing for Indian enterprises. Multilingual knowledge retrieval. Compliance document analysis.

These were real businesses solving real problems. The problem is that the solution was always a workaround for a temporary constraint.

Look at the YC batches from 2023-24. Count the Indian-founded companies whose pitch decks said "RAG" or "retrieval" or "knowledge base." Now ask how many of them have a moat beyond the retrieval pipeline itself.

Three categories of products just got commoditized

Document Q&A. If your product takes PDFs, chunks them, embeds them, and lets users ask questions, you now compete with "paste the PDF into Claude." The entire retrieval pipeline becomes overhead. A user with a 1M context window doesn't need your pipeline. They need a text box.

Enterprise search over internal docs. Companies like Glean built serious products here, but they also built moats beyond retrieval: connectors, permissions, personalization, usage analytics. The startups that only built the search layer are exposed.

Legal and compliance AI. Contract review, regulatory analysis, due diligence. These were perfect RAG use cases because the source documents were long and the queries were specific. Now you can feed entire contract sets into a single prompt.

What actually survives infinite context

The interesting question isn't what dies. It's what survives when context windows go functionally infinite.

Proprietary data pipelines. Getting data into a prompt is easy. Getting the right data, cleaned, structured, and current, from messy enterprise systems is hard. Connectors to SAP, Salesforce, government databases, legacy ERPs. That's plumbing work that doesn't get commoditized by larger context windows.

Orchestration and multi-step reasoning. RAG was a single-hop pattern: retrieve, then generate. The interesting AI products are multi-step: search, reason, act, verify, iterate. At Ostronaut, we learned that the hard problem in content generation isn't feeding the model enough context. It's coordinating multiple generation steps where each step depends on the output of the previous one, with validation gates between them. That coordination layer survives because it's orthogonal to context window size.

Domain-specific reliability. In regulated industries, the value isn't in retrieval. It's in auditability, compliance, and deterministic behavior around non-deterministic models. A hospital doesn't care that you can now fit all patient records into one prompt. They care that your system can prove why it made a specific recommendation and that it handles edge cases without hallucinating.

Cost optimization at scale. Here's the part nobody's talking about: 1M context windows are available, but they're not free. Sending 1M tokens per request at enterprise scale gets expensive fast. The companies that build intelligent routing -- small context for simple queries, large context only when needed, with caching and deduplication -- will create real value. The constraint moved from "can't fit enough context" to "can't afford to use full context on every request."

The test for your AI product

If you're building an AI product in India right now, here's the test: remove the retrieval layer from your architecture. Does your product still have a reason to exist?

If the answer is no, you have six months to find one. Not because 1M context windows will replace everything overnight. Adoption takes time. Enterprise procurement cycles are slow. But the pricing signal is clear: context windows are heading toward commodity.

Build on what stays scarce. Proprietary data access. Multi-step orchestration. Domain-specific reliability. Cost optimization at scale.

The RAG era isn't over. Retrieval still matters for keeping costs down and for real-time data. But retrieval as a product is over. It's a feature now. Build accordingly.

Originally published at talvinder.com.

Capability-Priced Micro-Markets: The Missing Layer Between Agents and HTTP

Talvinder Singh — Mon, 20 Apr 2026 07:49:21 +0000

At Ostronaut, we run a multi-agent pipeline where one agent structures content, another generates video assets, another validates quality. When we wanted to swap in a better external video provider, it took three days: read docs, negotiate pricing tier, configure auth, write retry logic, handle edge cases.

Three days for one integration. We have twelve capability slots across the pipeline.

HTTP wasn't built for agents. APIs assume a human reads documentation, signs up for a tier, and hardcodes an endpoint. Agents can't do that at runtime. They need to discover capabilities, compare prices, and pay per invocation without human intervention.

The infrastructure layer that enables this doesn't exist yet. I'm calling it Capability-Priced Micro-Markets: a protocol layer where computational capabilities are advertised, priced, discovered, and transacted autonomously, with no human in the loop.

The Coordination Problem

We're building agents that need to coordinate autonomously. An agent optimizing cloud spend needs to call an agent that forecasts usage patterns. An agent writing code needs to call an agent that reviews security. An agent booking travel needs to call an agent that checks visa requirements.

Right now, every one of those interactions requires a human to:

Find the right service
Read the documentation
Sign up and configure billing
Hardcode the integration

This doesn't scale when you have 50 agents running in a workflow, each needing 10 different capabilities, and the optimal provider for each capability changes based on load, accuracy requirements, and budget.

The current model treats APIs as services you subscribe to. The agent model requires APIs to be capabilities you buy per-use, discovered at runtime, priced dynamically.

Three Fatal Problems

Hardcoded integrations have three problems that become fatal at agent scale.

Discovery

Traditional APIs require you to know which service to call. You read docs, you integrate. Agents need to discover "what can be done" not "which service exists."

An agent doesn't want to know that Anthropic, OpenAI, and Google all offer vision APIs. It wants to know: "Who can classify this image at 95% accuracy for under 0.005 tokens?" The answer changes based on current load, model updates, and pricing.

Pricing

Current API pricing assumes human decision-makers choosing subscription tiers. Monthly plans. Annual contracts. Volume discounts negotiated over email.

Agents need per-invocation pricing with real-time price discovery. An agent with a $10 budget for a task needs to know: "Can I get this done within budget?" before it calls anything. It needs to comparison-shop across providers in milliseconds.

Intent vs Implementation

HTTP forces you to specify implementation. You call a specific endpoint at a specific URL with specific parameters.

Agents should declare intent. "Classify this image with 95% accuracy" or "Summarize this document in under 100 words." The market layer handles discovery, routing, and fallback.

The parallel to Kubernetes is exact. You didn't specify servers. You declared intent. Kubernetes handled placement, scaling, and self-healing.

Capability markets do the same for computational work. You declare what you need. The market finds who can provide it, at what price, and routes the request.

The Pieces Are Assembling

This isn't theoretical.

The pricing shift is already happening.

Over the last 3-4 years, SMBs have moved from subscription models to pay-as-you-go. OpenAI charges per token. Cloud providers charge per compute-second. The next step is agents charging each other per capability-invocation, with prices set dynamically based on demand.

The market layer already exists in adjacent domains.

Two-sided marketplace platforms are ubiquitous. Uber routes ride requests to drivers. AWS Spot Instances route compute requests to available capacity. The capability market is the same pattern applied to API calls: match demand (agent needs capability) with supply (agent provides capability) through dynamic pricing.

The intent-based architecture is proven.

Kubernetes demonstrated that declarative systems work at scale. You declare desired state. The system handles the rest. Capability markets extend this: you declare desired outcome and budget constraints. The market handles discovery, negotiation, and execution.

At Ostronaut, we built a multi-agent system where agents coordinate—one structures content, another generates assets, another validates quality. Right now, those are internal agents with hardcoded routing. The moment we want to use external capabilities, we hit the integration wall. We'd need to evaluate providers, negotiate pricing, handle auth, manage retries.

A capability market would let us declare: "Generate a 90-second video from this script, budget 0.05 tokens, minimum quality score 80." The market finds the provider, handles payment, returns the result.

The cost structure enables granular markets.

CloudZero and similar FinOps platforms already provide detailed cost intelligence and align spending with business outcomes. The infrastructure to track per-invocation costs exists. Extending that to per-capability pricing is a data model change, not a technical leap.

The Bet

Within 18 months, the majority of agent-to-agent API calls will route through dynamic capability markets rather than hardcoded HTTP endpoints.

Not because it's elegant. Because hardcoded integrations break at agent scale.

When you have one agent calling three APIs, you can hardcode. When you have 50 agents each calling 10 capabilities, and the optimal provider changes hourly, you need a market layer.

The companies building this layer now—capability registries, dynamic pricing protocols, intent-based routing—are building the plumbing for the next decade of agent coordination.

Traditional API Model	Capability Market Model
Hardcoded endpoint	Runtime discovery
Subscription pricing	Per-invocation pricing
Specify implementation	Declare intent and constraints
Human integration	Autonomous negotiation
Fixed provider	Dynamic routing

What I Don't Know Yet

How do you build trust in a capability market where providers are autonomous agents, not humans with reputations?

Traditional marketplaces rely on reviews, ratings, and dispute resolution—all human processes. An agent providing image classification doesn't have a LinkedIn profile or customer testimonials. It has accuracy metrics, uptime history, and response latency.

Do you build reputation systems based purely on performance data? Do you require staked collateral? Do you use cryptographic proof of execution?

The trust layer is the hardest part. The technical infrastructure for capability discovery and dynamic pricing is straightforward. Building organizational and economic trust in autonomous systems—that's the unsolved problem.

More on this as I work through it.

Originally published at talvinder.com.

The Build-Seed Inversion: Why Marketplaces Die Building Platforms Before Proving Transactions

Talvinder Singh — Sun, 19 Apr 2026 09:03:30 +0000

At Tushky, we built a super easy service listing product. We created awareness campaigns. People came. People went. The funnel kept drying as soon as fuel was not injected.

We had built supply without demand, features without focus, distribution experiments without a single proven channel. We spent 18 months building before we understood what the market actually wanted to buy.

I'm calling this the Build-Seed Inversion — the mistake of optimizing for scale before proving a single transaction works repeatably.

The Question Most Founders Get Wrong

Most founder advice splits into two camps: "just ship" or "talk to users first." Both miss the actual pattern. The question isn't whether to build or validate. It's what to build before you have market signal versus what to build after.

The Build-Seed Inversion happens when you invest in horizontal infrastructure, multiple distribution channels, and polish before you've made one specific transaction happen ten times in a row.

If you can't describe your first 100 transactions in a single sentence, you're building too early.

Tushky couldn't. We said "connecting customers to service providers." That's not a transaction. That's a category. Etsy could say: "craftspeople selling handmade goods to other craftspeople." Slack could say: "small engineering teams adopting for internal communication, then spreading to other departments."

The difference isn't semantic. It's strategic.

Supply Is Visible, Transactions Are Not

We built seller onboarding tools, listing templates, search algorithms. All supply-side infrastructure. But we hadn't proven we could get a single customer to transact twice.

A marketplace isn't valuable because it has supply. It's valuable because it creates transactions. We confused the two.

Founders build the platform before seeding the transaction. They optimize listing quality before proving anyone will buy. They add payment options before proving anyone will pay.

This is the central confusion of two-sided markets. Supply is visible. You can count listings, profiles, sellers onboarded. Transactions are harder to measure, harder to manufacture, and harder to show investors. So founders default to the thing they can control and count. They build the stage and assume the audience will show up.

In India, this problem is compounded. Low average transaction values mean you need higher density to make unit economics work. A home services marketplace in Mumbai with 500 providers and 50 monthly transactions is a spreadsheet, not a business. You need thousands of transactions in a tight geography before the marketplace has any defensibility at all.

Five Verticals Is Five Companies

Tushky was for "anyone who needs services." Cleaning, repairs, tutoring, photography, pet care. We built features for all of them.

Trello did this at scale. They built a horizontal product for tens of millions of users with a bleeding-edge stack. But they "didn't do a good job of keeping track of paying customers." They built for everyone and monetized no one effectively.

Slack went the opposite direction. They focused on "figuring out how and why its product spread from small teams, to departments, to larger organizations." One transaction type. One expansion pattern. Then scale.

The horizontal product trap isn't about product breadth per se. It's about attention fragmentation. When you build for five verticals simultaneously, you build five mediocre products instead of one excellent one. Your feature roadmap becomes a political negotiation between use cases rather than a focused pursuit of depth.

At Tushky, we had separate onboarding flows for tutors and for plumbers. Different quality signals for photographers and for electricians. Each vertical had its own supply dynamics, its own customer expectations, its own trust thresholds. We were running five marketplace experiments and calling it one company.

The founders I've seen get this right — including the early UrbanClap team (now Urban Company) — picked one vertical and made it work obsessively before expanding. Urban Company started with beauty services at home. Not "all services." Not even "beauty and repairs." Just beauty. They built transaction density in one category in one city before adding the next.

Three Channels at 25% Commitment Teaches You Nothing

We tried corporate sales. Long, winding, uncertain sales cycles. We tried online affiliates. No success. We tried society activations. Some traction, but nothing repeatable.

We were running three distribution experiments simultaneously before we'd proven any channel could acquire a customer profitably.

The math was obvious in retrospect: if your CAC is unknown and your LTV is unproven, adding more channels just multiplies the uncertainty.

I see this in every startup cohort I've worked with at Pragmatic Leaders. Founders with 6 months of runway running paid acquisition, SEO content, partnerships, and community-building in parallel. They interpret the spread as "testing." It's not testing. Testing means running one channel with enough conviction and capital to get a statistically meaningful signal. Running four channels at 25% commitment each means you learn nothing about any of them.

One distribution channel, proven to unit-economics breakeven, is worth more than ten experiments running at subscale.

Etsy Found Existing Transactions First

Etsy targeted exactly one group: craftspeople selling to other craftspeople. Not "anyone who makes things" selling to "anyone who buys things." One seller type, one buyer type, one transaction pattern.

They sparked transactions within that group successfully before branching out. The platform features came after the transaction mechanics worked.

The critical move was that Etsy's early team went to craft fairs physically. They didn't build a platform and wait for sellers to discover it. They found people already transacting and gave them a better venue. The demand and the behaviour existed before the technology.

What We Built vs. What We Needed

Here's what we built before we had product-market fit:

Service provider onboarding flows
Multi-category listing templates
Awareness campaigns that drove traffic
Corporate sales collateral
Affiliate partnership frameworks
Society activation playbooks

Here's what we didn't have:

A single customer segment that transacted repeatably
Unit economics for any channel
A clear answer to "which transaction are we optimizing for?"

The third missing piece wasn't product features. It was transaction mechanics. We had supply, we had awareness, but we couldn't connect early customers to unique sellers in a way that created repeat behavior.

The pattern holds across the companies I've worked with. The ones that scaled figured out seeding strategies before building horizontal infrastructure. The ones that struggled built platforms before proving transactions. A SaaS tool can succeed with a great product and decent distribution. A two-sided market dies without transaction density in a specific segment first.

The Sequence That Works

Build Before Market Signal	Build After Traction
Minimum mechanics to complete one transaction	Horizontal features for adjacent segments
Single channel acquisition experiment	Multi-channel distribution
Manual processes that prove unit economics	Automation and scale infrastructure
Focus on one user segment	Expansion to broader market
Supply in one geography, one vertical	Supply aggregation tools

The test: can you describe your first 100 transactions in one sentence? If not, you're not ready to build for scale.

At Tushky, we inverted this. We built the scale infrastructure first. We optimized for millions before we'd proven hundreds.

The market eventually showed up — not for what we built, but for what we should have started with. Urban Company, Housejoy, and others entered the same space later with tighter wedges and better sequencing. They didn't build better technology. They built the right things in the right order.

By the time we recognized the pattern, we'd spent 18 months building the wrong things in the wrong sequence.

The question I'm still working through: how do you know when you've proven "enough" transactions to start building horizontally? Ten transactions? A hundred? A thousand? The companies that got this right seem to have felt it rather than calculated it. I'm not satisfied with that answer yet.

Originally published at talvinder.com.

The Recourse Trap: Why Competition Makes Credit Scoring More Exclusive, Not Less

Talvinder Singh — Sun, 19 Apr 2026 09:03:25 +0000

In 2022, HDFC Bank raised its minimum CIBIL score requirement for personal loans from 650 to 725. ICICI and Axis followed within months. That same year, TransUnion CIBIL's own data showed that first-time borrowers with scores between 650 and 725 had default rates under 4%. The banks weren't responding to rising risk. They were responding to each other.

Credit scoring systems don't fail because they're inaccurate. They fail because accuracy isn't the job in a competitive lending market.

The job is risk transfer. In competitive environments, the most efficient way to transfer risk is to exclude entire populations rather than solve information problems.

I've seen this pattern up close. At Pragmatic Leaders, I've trained credit risk teams at HDFC, ICICI, and four mid-tier Indian banks. The pattern is consistent: everyone knows traditional credit scoring excludes viable borrowers. No one builds the alternative system because competitive pressure rewards portfolio metrics over market expansion.

The Recourse Trap

This is what I'm calling The Recourse Trap: a system where the mechanism designed to enable access becomes the mechanism that prevents it, and competitive pressure makes the trap stronger, not weaker.

Here's how it works:

A lender can't distinguish between a borrower with no credit history and a borrower with bad credit history. Both score low. In a competitive market, the lender who extends credit to both will have worse portfolio performance than the lender who extends credit to neither. The rational competitive response is exclusion.

The borrower has no recourse. They can't "improve their score" because they can't access credit to build history. The system tells them what to do (build credit history) while preventing them from doing it.

India has 400 million adults with no credit history in any bureau. Not because they're risky. Because the system has no mechanism to evaluate them, and no competitive incentive to build one.

The Mechanism

When lenders compete on portfolio risk metrics, they optimize for false negative reduction (don't lend to bad borrowers) over false positive reduction (do lend to good borrowers). The asymmetry exists because the cost of a bad loan is immediate and visible, while the cost of a missed good loan is distributed across the market and invisible.

This creates a lemons problem. Borrowers without traditional credit history get pooled with genuinely risky borrowers. Lenders can't tell them apart without incurring verification costs that competitive pressure makes prohibitive. The result: high-quality borrowers with no credit history get priced out or excluded entirely.

Falsifiable claim: In competitive lending markets, credit score requirements will trend upward over time for populations without traditional credit history, even as default rates in those populations remain stable or decline. The system optimizes for competitive position, not credit risk.

You can test this. Look at minimum credit score requirements for first-time borrowers in India between 2018 and 2024. Requirements went up across every major bank. Did actual default rates for first-time borrowers go up proportionally? No. RBI data shows gross NPA ratios for retail loans actually declined from 2.5% to 1.7% in that period. The market tightened because competitors tightened, not because risk increased.

The Transaction Cost Argument Is Circular

Here's the tell: when you ask banks why they don't serve underbanked populations, they talk about credit scores. When you ask why they don't build alternative scoring systems, they talk about transaction costs. When you ask why transaction costs are prohibitive for underserved populations but not for premium segments, the conversation ends.

High costs justify exclusion. Exclusion prevents scale. Lack of scale keeps costs high.

South African banks demonstrate this clearly. Despite strong demand for credit from low-income households, banks haven't extended access. Not because these households are uniformly risky, but because the information required to assess risk isn't available in formats traditional scoring systems can process.

The alternative mechanisms prove the problem is solvable. Group lending models and informal systems like stokvels work precisely because they solve the information problem differently. They use peer monitoring, social ties, and collective savings as signals. Transaction costs stay low. Default rates stay manageable.

But competitive banks don't adopt these approaches. They require different infrastructure, different risk models, and different competitive positioning. A bank that moves first takes on execution risk. A bank that moves second can copy what works. The rational move is to wait, which means no one moves.

What AI Makes Worse

AI-powered credit scoring is getting more sophisticated at predicting risk within existing data distributions. Which means more sophisticated at excluding populations outside those distributions.

An AI model trained on historical lending data will learn that borrowers without credit history are risky. Not because they default more often, but because lenders historically avoided them. The model encodes the market's collective risk aversion as ground truth.

I saw this firsthand during a workshop with a mid-tier bank's risk team in 2023. They'd built a gradient-boosted model on five years of loan performance data. The model performed well on their test set (AUC of 0.87). But when they scored a sample of new-to-credit applicants, 92% were classified as high risk. The data scientist on the team knew the scores were wrong. His manager knew. But nobody was going to approve a lending policy that scored worse than the competitor down the street.

The feedback loop tightens. Better prediction within the existing distribution means worse outcomes for populations outside it.

What I Got Wrong

I initially thought the solution was better data. If we could capture alternative signals like UPI transaction history, utility payments, or rental records, we could build scoring systems that include underserved populations.

That's technically true but structurally naive.

The problem isn't data availability. India Account Aggregator has been live since 2021. Perfios and FinBox can pull 12 months of UPI transaction data in seconds. The pipes exist. Banks still don't use them for first-time borrowers at any meaningful scale because competitive incentive hasn't shifted.

A bank that invests in alternative data infrastructure takes on execution risk and regulatory uncertainty. A bank that waits can copy the approach if it works. The first-mover disadvantage is real.

Beyond Credit Scoring

The recourse trap exists because competitive markets optimize for relative performance, not absolute outcomes. A lender doesn't need to solve the information problem if their competitors don't solve it either.

This has implications beyond financial services. Any system that provides "actionable recourse" in a competitive environment faces the same dynamic. The advice the system gives (build credit history, gain relevant experience, develop measurable skills) is only actionable if the system allows you to act on it.

When it doesn't, you're not dealing with an information problem. You're dealing with a market structure problem.

AI-powered resume screening. Skills-based hiring platforms. Fraud detection systems. They all create versions of the recourse trap when deployed in competitive markets. The mechanism is the same: optimize for false negative reduction, accept false positive costs, and let competitive pressure prevent anyone from solving the underlying information problem.

The Question Worth Asking

What other systems are we building that look like they enable access but actually optimize for exclusion?

If the mechanism for proving you're trustworthy requires access you can't get without already being trusted, you're in a recourse trap. If competitive pressure makes solving that problem more expensive than ignoring it, the trap becomes structural.

Are we asking this question when we deploy AI systems in hiring, lending, insurance, education? Mostly, no. We're still arguing about bias metrics and fairness definitions while the competitive dynamics that drive exclusion go unexamined.

The recourse trap doesn't care about bias. It cares about competitive dynamics. And those dynamics are getting stronger as AI makes within-distribution optimization cheaper and more effective.

Originally published at talvinder.com.

Why 86% of AI Agent Pilots Fail Before Reaching Production

Talvinder Singh — Fri, 17 Apr 2026 13:06:19 +0000

According to the MAST benchmark study, multi-agent system failure rates range from 41% to 86.7% across seven leading frameworks. Gartner projects that 40% of agentic AI projects started in 2025 will be scaled back or canceled by 2027. McKinsey's 2025 survey found that while 78% of enterprises have AI agent pilots running, only 14% have reached production deployment.

These numbers tell the same story: the demo works, but production kills it.

The failure isn't the model — it's everything around the model

The top three causes of agent pilot failure are integration complexity (67%), lack of monitoring (58%), and unclear escalation paths (52%), according to PwC's 2025 enterprise AI survey. The model itself is rarely the problem. According to a 2025 PwC survey of 1,000 enterprises deploying AI agents, the top three failure modes are integration complexity (cited by 67%), lack of monitoring infrastructure (58%), and unclear escalation paths when the agent makes mistakes (52%).

The model itself is rarely the problem. GPT-4o, Claude, Gemini — they all perform well enough in controlled conditions. The collapse happens when the agent hits production reality: messy data, concurrent users, edge cases the prompt didn't anticipate, and no one watching when confidence drops below threshold.

This is the same reliability gap that Indian SaaS companies have been closing for twenty years — not with better models, but with better systems around the models.

Five patterns that kill agent pilots

These are the five structural failures I've seen repeatedly across teams deploying agents — from startups to Fortune 500 companies. Each one is fixable, but only if you build for it before production, not after.

1. No confidence scoring or graceful degradation

Agents without confidence thresholds have 3x higher escalation rates than those with calibrated routing, according to Anthropic's production deployment data. The agent either answers or it doesn't. There's no middle ground. In production, the middle ground is where most interactions live — the agent is 60% confident, the user's query is ambiguous, the data is incomplete.

Without confidence scoring, you get one of two failure modes: the agent hallucinates confidently (and you lose trust) or the agent refuses to answer (and you lose utility). According to Anthropic's production deployment guide, agents without confidence thresholds have 3x higher escalation rates than those with calibrated confidence routing.

The fix is graduated autonomy: act autonomously above 90% confidence, request human review between 60-90%, escalate below 60%. This is the same pattern we built at Zopdev for infrastructure decisions — observe everything, act only within permission boundaries.

2. The "just retry" fallacy

When an agent fails, most frameworks default to retrying with the same prompt. This is the Pass@k trap: if the error is structural (wrong data, missing context, ambiguous instruction), retrying amplifies the problem rather than fixing it.

A 2025 analysis of production agent logs at a Fortune 500 company found that 73% of retried requests produced the same error category. The retry wasn't recovery — it was waste. At $0.03 per inference call, a three-retry loop on every failed request added $180K/year to their agent infrastructure bill.

The fix is error classification before retry. Network timeout? Retry. Model hallucination? Route to a different model or escalate. Missing context? Fetch the context first, then retry with enriched input.

3. No observability beyond the API call

Most agent monitoring stops at the API layer: latency, token count, error rate. But agent failures are semantic, not mechanical. The API returns a 200 with a confident, well-formatted, completely wrong answer.

According to Langfuse's 2025 observability report, teams that implement trace-level monitoring (tracking the full chain of agent reasoning, tool calls, and intermediate outputs) catch production issues 4x faster than teams monitoring only API metrics. This is what trace-based assurance looks like in practice — the governance layer that agentware actually needs.

4. Human handoff as afterthought

The agent is built to be autonomous. When it can't handle something, it says "I don't know" — and the user is stuck. There's no warm handoff to a human, no context transfer, no continuity.

According to Freshworks' deployment data, their Freddy AI achieves a 45% autonomous resolution rate. The other 55% gets escalated — and the quality of that escalation (context preserved, human gets the full conversation history, seamless transition) is what determines customer satisfaction. The agent's job isn't just to resolve; it's to escalate well when it can't.

The cost of building good escalation paths is significant. A production agent needs roughly 3.5 FTEs for monitoring, incident response, and drift detection. In Bangalore, that's $100K-150K/year. In San Francisco, $600K-800K. This cost asymmetry is why Indian SaaS companies can afford the monitoring density that makes agents reliable.

5. Evaluation that doesn't match production conditions

The agent scores 92% on the benchmark. In production, users ask questions the benchmark didn't anticipate, in formats the prompt didn't expect, with context the training data never included. The evaluation cost ratio breaks down when evaluation doesn't mirror production conditions.

According to the HELM benchmark team at Stanford, model performance on curated test sets overpredicts production accuracy by 15-30 percentage points. The gap is not random — it's systematic. Production queries are longer, more ambiguous, more dependent on context, and more adversarial than benchmark queries.

What actually works: the three-layer architecture

Successful agent deployments converge on a generation-validation-governance stack. The generation layer is what everyone builds; the other two are what separates pilots from production. Every successful deployment I've seen — Freshworks' Freddy, Zoho's Zia, our own systems — converges on the same architecture:

Layer	Function	What it catches
Generation	The model produces output	Nothing — this is the happy path
Validation	Rule-based checks, confidence scoring, format verification	Structural errors, low-confidence outputs, format violations
Governance	Human review queues, audit trails, escalation paths, drift detection	Semantic errors, edge cases, model drift, compliance violations

The generation layer is what everyone builds. The validation layer is what separates pilots from production. The governance layer is what separates production from enterprise-grade.

Most pilots only build layer one. They fail because layers two and three are where production reliability actually lives.

The question worth asking

If you're running an agent pilot right now, ask this: what happens when the agent is wrong and confident about it?

If the answer is "we haven't thought about that" — you're in the 86%. The consensus voting approach won't save you. The chain-of-thought reasoning adds cost without guaranteeing correctness. The model isn't the problem.

The monitoring, the fallbacks, the escalation paths, the confidence routing — that's where production reliability lives. The teams that figure this out aren't building better agents. They're building better infrastructure around agents. And right now, the companies with the deepest operational discipline in that infrastructure layer are based in India.

::: {.schema-faq style="display:none;"}
[{"q":"Why do AI agent pilots fail in production?","a":"According to MAST benchmark data, 41-86.7% of multi-agent systems fail across leading frameworks. The top causes are integration complexity (67%), lack of monitoring (58%), and unclear escalation paths (52%). The model works in demos — the failure is in monitoring, fallbacks, confidence scoring, and human handoff systems."},{"q":"What percentage of AI agent projects reach production?","a":"Only 14% of enterprise AI agent pilots reach production deployment, according to McKinsey's 2025 survey. Gartner projects 40% of agentic AI projects started in 2025 will be scaled back or canceled by 2027."},{"q":"How do you deploy AI agents to production successfully?","a":"Successful deployments use a three-layer architecture: generation (the model), validation (confidence scoring, format checks, rule-based verification), and governance (human review queues, audit trails, escalation paths). Most failed pilots only build the generation layer."}]

:::

Originally published at talvinder.com.

The Vibe Coding Hangover: Why AI-Written Code Costs 4x to Maintain by Year Two

Talvinder Singh — Fri, 17 Apr 2026 13:06:18 +0000

According to a CodeRabbit analysis of 1,000+ repositories, AI co-authored code introduces 1.7x more major issues than human-written code. The vulnerability rate is 2.74x higher. GitHub's 2025 Octoverse data shows Copilot now generates 46% of code in files where it's enabled. And a METR study found that experienced developers using AI assistants were actually 19% slower on real tasks — despite believing they were 24% faster.

The productivity feels real. The debt is real too. We're starting to see the bill.

The three-month cliff

Every team I've talked to that adopted AI coding tools heavily describes the same pattern: massive output gains in months one through three, followed by an escalating maintenance burden that erases those gains by month six.

The pattern has a name now. Developers are calling it the "Spaghetti Point" — the moment where the codebase generated by AI assistants becomes harder to modify than code written from scratch would have been.

According to GitClear's 2025 developer productivity report, code churn (lines modified or deleted within 14 days of being written) increased 39% in repositories with heavy AI assistance. That's not refactoring — that's rework. Code written fast, reviewed inadequately, and fixed repeatedly.

The economics are brutal. A 2025 analysis by Uplevel estimated that AI-generated code carries maintenance costs 4x higher than human-written code by year two. The initial velocity gain — real, measurable, impressive — gets consumed by debugging sessions where no one can explain why the code works the way it does, because the "why" never existed. This is the same epistemological problem that's eroding trust in open source: AI-generated code has no intent. You can't reconstruct reasoning that never happened.

Why the bugs are different

AI-generated bugs are structurally different from human bugs, and that difference makes them more expensive to find and fix.

Human bugs have intent trails. A developer who writes a race condition usually has a mental model that's almost right — they thought about concurrency but missed one case. You can read the code, reconstruct the thinking, find the gap. The fix follows from understanding the original intent.

AI bugs have no intent. The code was generated from a probability distribution, not a mental model. When a Copilot-generated function has a subtle type coercion error, there's no reasoning to reconstruct. You can't ask "what were they thinking?" because nothing was thinking. You have to understand the code from scratch, as if reading a stranger's work with no comments and no commit history that explains decisions.

According to Snyk's 2025 AI security report, 35 new CVEs were attributed to AI-generated code in March 2026 alone. Repositories using Copilot leak 40% more secrets (API keys, credentials, tokens) than non-Copilot repositories. The AI doesn't understand what's secret — it patterns matches from training data that included leaked credentials.

Bug type	Human-written code	AI-written code
Root cause analysis	Follow the intent trail	Start from zero — no intent exists
Time to diagnose	1-2 hours typical	3-5 hours (no reasoning to reconstruct)
Recurrence after fix	Low (developer updates mental model)	High (same prompt generates same pattern)
Security issues per KLOC	Baseline	2.74x higher (CodeRabbit data)
Code churn within 14 days	Baseline	+39% (GitClear data)

The organizational blind spot

The real damage isn't technical — it's organizational. Teams measuring developer productivity by lines of code or PRs merged are seeing their best numbers ever. The dashboards look great. Velocity is up. Sprint commitments are being met.

What the dashboards don't show: time spent in code review has increased 45% (because reviewers now treat every PR as potentially AI-generated and requiring deeper verification). Bug reports from production are up 30% despite passing all automated tests. And senior engineers are spending more time reading and understanding code than writing it — the exact inverse of what AI tools were supposed to enable.

This is the same productivity illusion we measured in team velocity: AI makes individual tasks faster while making the overall system slower. The local optimization creates a global pessimization.

What I got wrong

I initially thought the problem was adoption immaturity — that teams would learn to use AI tools effectively and the quality issues would resolve. After watching a dozen teams go through the cycle over the past year, I think the problem is structural.

AI code generation optimizes for plausibility, not correctness. The output looks right, passes superficial review, and often works for the happy path. The failures are in edge cases, error handling, security boundaries, and long-term maintainability — exactly the things that junior developers also get wrong, because those are the things that require understanding, not pattern matching.

The teams that are succeeding with AI code generation share three practices:

1. AI writes, humans architect. The AI generates implementation within a structure that a human designed. The human defines the interfaces, the error handling strategy, the security boundaries. The AI fills in the bodies. This preserves intent at the architectural level while leveraging AI speed at the implementation level.

2. Review budgets increased, not decreased. Teams that cut code review time because "the AI wrote it" are the ones hitting the Spaghetti Point fastest. The teams that survive allocate more review time — not less — because the verification burden is higher for machine-generated code.

3. Aggressive deletion of AI-generated code that can't be explained. If a developer can't explain why a function works the way it does — regardless of whether it passes tests — it gets rewritten by hand. This is expensive in the short term and cheap in the long term.

The historical pattern

This cycle is familiar. Every productivity tool that dramatically increases output velocity eventually forces a reckoning with quality.

3D printing was going to democratize manufacturing. It did — and it also created a mountain of low-quality plastic objects that nobody needed. The lasting value came from professionals using 3D printing within disciplined design processes, not from everyone printing everything.

No-code tools were going to replace developers. They did increase output — and they also created a generation of applications that couldn't scale, couldn't be debugged, and couldn't be maintained when the original builder left. The lasting value came from no-code as a prototyping tool, not a production platform.

Vibe coding is following the same arc. The output explosion is real. The quality reckoning is coming. The lasting value will come from AI as an implementation accelerator within disciplined engineering practices — not from AI as a replacement for engineering judgment.

The question worth asking

If your team adopted AI coding tools in the last twelve months, run this check: compare the bug rate and code churn rate in your most AI-assisted repositories against your least AI-assisted ones. Normalize for team size and feature complexity.

If the AI-heavy repos show higher churn and more production bugs — even if they also show higher velocity — you're accumulating the debt. The hangover is coming. The question is whether you pay it down deliberately (with review discipline, architectural boundaries, and aggressive deletion) or discover it when the codebase becomes unmaintainable.

The trust tax isn't just an open-source problem. It's inside your organization too.

::: {.schema-faq style="display:none;"}
[{"q":"Does AI-generated code have more bugs than human code?","a":"Yes. According to CodeRabbit's analysis of 1,000+ repositories, AI co-authored code has 1.7x more major issues and a 2.74x higher vulnerability rate. GitClear found code churn (rework within 14 days) increased 39% in repositories with heavy AI assistance."},{"q":"What is vibe coding and what are the risks?","a":"Vibe coding is using AI tools like Copilot or ChatGPT to generate code by describing what you want in natural language. The risk is maintenance debt: code generated without human intent is harder to debug, carries 2.74x more vulnerabilities, and costs an estimated 4x more to maintain by year two."},{"q":"Are developers actually faster with AI coding tools?","a":"Not necessarily. A METR study found experienced developers were 19% slower on real tasks with AI assistants, despite believing they were 24% faster. Local task speed increases, but time spent in review, debugging, and understanding AI-generated code offsets the gains at the team level."}]

:::

Originally published at talvinder.com.

Trace-Based Assurance: The Governance Layer Agentware Actually Needs

Talvinder Singh — Wed, 25 Mar 2026 03:39:50 +0000

Agents are being deployed with governance frameworks designed for human committees and quarterly audits. The gap is not small.

Traditional governance asks: "Did you follow the process?" Agentic systems require a different question: "Can you prove, in real-time, that the agent is operating within boundaries?" The difference matters because agents make decisions faster than humans can review them, and carry more risk than trust-based deployment can tolerate.

At Ostronaut, we generate training content autonomously—presentations, videos, quizzes—for healthcare clients. The first time a client asked "How do we know this meets compliance requirements?", we had documentation. We had process diagrams. We had architectural reviews. What we didn't have was evidence that the system was actually doing what we said it would do, case by case, generation by generation.

That's the governance gap.

The Evidence Problem

I'm calling this Trace-Based Assurance — a governance model where agents emit verifiable evidence trails that prove compliance in real-time, rather than documenting intentions in advance.

This isn't about adding logging. Every system has logs. Trace-based assurance means structuring agent operations so that governance verification becomes automated and continuous. The trace isn't a byproduct. It's the mechanism.

By 2027, production-grade agentic systems will be required to emit structured trace data that proves boundary compliance, not just logs outcomes. Vendors who treat governance as a documentation problem will lose enterprise deals to vendors who treat it as an evidence problem.

The shift is already visible. When we talk to healthcare clients, they don't ask "What's your process for content review?" They ask "Can you show me, for this specific piece of generated content, what checks ran and what the results were?"

That's a different question. It assumes the system is autonomous. It assumes human review isn't feasible at scale. It demands evidence, not assurance.

Where Traditional Governance Breaks

Traditional governance models don't handle this well. They're built for phase-gate processes: design review, implementation review, deployment approval, quarterly audit. Agents don't operate in phases. They operate continuously. They adapt. They make thousands of decisions between audits.

The gap shows up in three places.

Approval vs. Acceptance

Traditional procurement distinguishes between "approval" (pre-decision authority) and "acceptance" (post-decision verification). Agents break this model. You can't approve every decision in advance—they happen too fast. You can't simply accept outcomes post-facto—the risk is too high.

Traces create a third path: continuous verification. The agent emits evidence as it operates. Governance systems verify that evidence in real-time. Decisions that pass verification proceed. Decisions that fail trigger escalation.

This isn't theoretical. We built validation gates into Ostronaut's generation pipeline after a quality crisis. The system now emits structured traces at each stage: content extraction, structure generation, media creation, quality scoring. Each trace includes the inputs, the decision made, the constraints checked, and the result.

When a generation fails validation, we have the trace. We know exactly where it failed and why. When a generation succeeds, the client has evidence that it met their requirements.

Documentation vs. Evidence

Production systems require security, compliance, scalability, all of the operational requirements enterprise buyers expect. The standard response is documentation: architecture diagrams, security reviews, compliance checklists.

Documentation tells you what the system is supposed to do. Evidence tells you what it actually did.

The difference matters when something goes wrong. If an agent makes a bad decision, documentation tells you the process was sound. Evidence tells you what inputs it received, what constraints it checked, what decision it made, and why.

We learned this the hard way. Early versions of Ostronaut had extensive documentation about quality controls. When clients asked about a specific generation that didn't meet standards, we could point to the process. What we couldn't do was show them the specific quality checks that ran for that generation and what they returned.

Documentation scales to the system. Evidence scales to the decision.

Trust vs. Transparency

Trust-based governance works when operations are slow enough for relationship-building and reputation to matter. Agentic systems operate too fast for trust alone.

Transparency enables trust at speed. If I can see the evidence trail—what the agent considered, what constraints it checked, what decision it made—I can trust the outcome without trusting the vendor's reputation or the operator's judgment.

This is not about replacing human judgment. It's about giving humans the information they need to judge effectively. A trace that shows "this generation passed 12 quality checks, failed 1, and was escalated for review" is more useful than a process diagram that says "all content undergoes quality review."

What This Looks Like in Practice

The pattern is showing up across domains.

Healthcare training clients don't ask "Is your content accurate?" They ask "Can you prove this specific module met our clinical guidelines?" That's a trace question.

Financial services clients don't ask "Do you have compliance controls?" They ask "Can you show me the decision path for this specific transaction and what risk checks applied?" That's a trace question.

Customer support deployments don't ask "How do you ensure quality?" They ask "Can you prove this agent didn't violate our brand guidelines in this specific conversation?" That's a trace question.

The common thread: verification needs to happen at the decision level, not the system level.

Here's what trace-based assurance requires:

The trace must be:

Structured: machine-readable format, not free text
Complete: captures inputs, constraints, decision logic, outcome
Timestamped: enables audit trail reconstruction
Immutable: can't be modified after creation
Queryable: supports real-time and historical analysis

This is different from logging. Logs capture what happened. Traces capture why it happened and prove it was within bounds.

The Architecture Shift

Building for trace-based assurance changes how you architect agentic systems.

Traditional approach: build the agent, add logging, write documentation.

Trace-based approach: design the constraints first, structure the agent to emit evidence of constraint adherence, make the trace the governance interface.

We rebuilt Ostronaut's generation pipeline around this model. Every stage emits a structured trace. The trace includes:

What content was provided as input
What quality thresholds were configured
What checks ran and what they returned
Whether the output met requirements
If not, why not and what happened next

The client's compliance team doesn't review our code. They review traces. When they spot-check a generation, they can see the complete decision path. When they audit the system, they query traces, not documentation.

This inverts the governance relationship. Instead of "trust us, we have good processes," it's "verify us, here's the evidence."

What I Got Wrong

We initially tried to retrofit traces onto an existing system. That doesn't work. Traces need to be part of the agent's core architecture, not an afterthought.

We also underestimated the storage and query requirements. Traces for every decision add up fast. You need infrastructure that can handle high-volume writes and support complex queries across time ranges and decision types.

The bigger mistake: thinking traces were primarily for auditors. They're actually most valuable for the engineering team. When an agent makes a bad decision, the trace is your debugging tool. When you're tuning the system, traces show you which constraints are too loose or too tight. When you're explaining the system to stakeholders, traces are your evidence.

The Open Question

Here's what I don't know yet: how do you build organizational trust in trace-based governance?

Most enterprise buyers are used to documentation-based assurance. They know how to evaluate a security review or a compliance checklist. They don't yet know how to evaluate a trace architecture.

The question isn't technical. It's cultural. How do you convince a procurement team that "we'll show you the evidence for every decision" is more reliable than "we have a 47-page compliance document"?

The early adopters get it. Healthcare organizations that already deal with electronic health records understand audit trails. Financial institutions that deal with transaction monitoring understand decision-level evidence.

But the broader market is still catching up. Most RFPs still ask for documentation, not trace capabilities. Most compliance frameworks still assume human review, not automated verification.

The shift will happen. It has to. Agents are already making decisions too fast and at too high a volume for documentation-based governance to work. The question is whether the governance frameworks will adapt in time, or whether we'll see a wave of incidents first.

Are we building the trace infrastructure now, or waiting for the forcing function? Mostly, we're still writing documentation.

Originally published at talvinder.com.

The Small Model Arbitrage: Why India Should Be Building Vertical LLMs, Not Chasing Frontier

Talvinder Singh — Mon, 23 Mar 2026 13:21:57 +0000

India is trying to build its own GPT-4. This is a mistake.

The capital requirement to train a frontier model is $500M-$1B+. The talent war for ML researchers is won before you enter it—OpenAI, Anthropic, and Google have already hired everyone worth hiring at compensation packages Indian companies can't match. The compute infrastructure is controlled by three hyperscalers who are also your competitors.

This is not a winnable race. But there's a different race that is.

The Small Model Arbitrage

I'm calling this the Small Model Arbitrage—the opportunity to capture value by building specialized, vertical-specific LLMs that use local data, languages, and domain expertise where general-purpose models systematically underperform.

The arbitrage exists because frontier model companies optimize for breadth, not depth. GPT-4 is remarkable at general reasoning but mediocre at Tamil legal document analysis, Ayurvedic diagnosis support, or GST compliance automation. The long tail of vertical use cases is economically unattractive to companies spending $1B on training runs.

That's where the opening is.

A well-executed vertical LLM in a defensible domain will reach profitability faster and generate higher ROI than an Indian frontier model attempt over the next 5 years.

The math supports this. Training a competitive frontier model requires $500M-$1B in compute, 100+ PhD-level researchers at $300K-$500K/year, and 3-5 years to market. Ongoing capital burn to stay competitive as OpenAI and Anthropic release new versions.

A vertical LLM requires $2M-$10M in initial training, 10-20 engineers and domain experts, and 6-12 months to first deployment. The moat is proprietary domain data, not compute scale.

The capital efficiency difference is 50-100x. The time-to-revenue difference is 5-10x.

Why Capital Efficiency Isn't the Whole Story

Capital efficiency alone doesn't win. The real arbitrage is in defensibility.

Frontier models are commodity infrastructure. When GPT-5 launches, GPT-4 pricing collapses. When Claude 4 launches, Claude 3.5 becomes table stakes. The moat is constantly eroding because the moat IS the model, and the model is constantly being replaced.

Vertical models have different moats. The moat is the proprietary training data, the domain-specific evaluation benchmarks, the integration into existing workflows, the trust built with regulated industries. These don't erode when OpenAI ships a new model. They compound.

Consider Indian legal text. A frontier model can summarize a contract. A vertical legal LLM trained on 20 years of Indian case law, Supreme Court judgments, and regulatory filings can identify precedent, flag jurisdictional issues, and generate compliant documentation.

The difference in value is 10x. The difference in defensibility is 100x.

Or healthcare. GPT-4 can answer general medical questions. A model trained on Indian clinical protocols, drug formularies, insurance claim patterns, and regional disease prevalence can assist with diagnosis, treatment planning, and prior authorization. It's not a better general model—it's a purpose-built tool that works within the constraints of the Indian healthcare system.

The pattern here is data specificity as competitive advantage. Frontier models are trained on the open web. Vertical models are trained on proprietary, domain-specific corpora that are expensive or impossible for competitors to replicate.

The Import Substitution Mistake

India tried this playbook before. Post-independence industrial policy was built on import substitution—build everything domestically, compete head-to-head with established global players. It failed spectacularly.

India's inward-looking trade regime discouraged labor-intensive export industries and rewarded installation of new capacity over actual output. The economy stagnated for decades.

The companies that succeeded—Infosys, Wipro, TCS—didn't try to be IBM. They specialized in specific services where India had comparative advantage: cost-efficient software development, business process outsourcing, IT support. They built world-class competitors by focusing, not by trying to replicate the entire stack.

The Small Model Arbitrage is the same bet. Don't build Indian GPT-4. Build the best Tamil-English legal LLM. Build the best model for Indian tax code. Build the best clinical decision support system for Indian healthcare protocols.

Who's Building This

Sarvam AI is building this playbook. They're not trying to be OpenAI India. They're building models for Indian languages—starting with Hindi, Tamil, Telugu, Kannada. The training data includes regional dialects, code-switching patterns, and cultural context that frontier models miss. Their Indic LLM performs better on Hindi-English code-mixed text than GPT-4 because it was designed for that specific use case.

Niramai built an AI system for breast cancer screening using thermal imaging. It's not a general-purpose vision model. It's a vertical model trained on Indian patient data, optimized for cost-constrained clinical settings, and integrated with existing diagnostic workflows. The model's accuracy isn't better than frontier models on general image tasks—it's better on the one task that matters for their customers.

Tricog built an ECG interpretation model for Indian hospitals. It doesn't try to be the best general medical AI. It's trained on Indian cardiac data, accounts for regional disease prevalence, and integrates with existing cardiology workflows. The specificity is the product.

These companies aren't competing on compute scale. They're competing on domain depth.

The Three Criteria for Vertical LLM Opportunity

Not every vertical is worth building. The opportunity exists where three conditions hold:

Criterion	Why It Matters
Proprietary data access	The training corpus must be expensive or impossible for competitors to replicate. Public datasets don't create moats.
Measurable performance delta	The vertical model must demonstrably outperform frontier models on domain-specific benchmarks. "Better for India" isn't enough—quantify it.
Willingness to pay	The customer must value the vertical model enough to pay a premium over general-purpose alternatives. Cost savings or compliance requirements work. Marginal convenience doesn't.

Indian legal tech meets all three. Case law is proprietary, performance on precedent identification is measurable, and law firms pay for accuracy.

Indian healthcare meets all three. Clinical data is proprietary, diagnostic accuracy is measurable, and hospitals pay for compliance and outcomes.

Indian fintech meets two out of three. Transaction data is proprietary, fraud detection performance is measurable, but willingness to pay is unclear—banks may prefer general models with custom fine-tuning.

The test is simple: if a frontier model company could replicate your vertical model by spending $10M on data acquisition and fine-tuning, you don't have a moat. If they can't—because the data doesn't exist, the domain expertise takes years to build, or the regulatory relationships are non-transferable—you do.

What I Don't Know Yet

The open question is whether vertical LLMs can sustain pricing power as frontier models improve. If GPT-5 closes 80% of the performance gap on Indian legal text, does the 20% delta justify a 5x price premium?

I think yes, but the answer depends on how regulated and mission-critical the domain is. Healthcare and legal have high switching costs and regulatory lock-in. E-commerce and customer support don't.

The other unknown is whether vertical models can defend against fine-tuned frontier models. If a customer can take GPT-4, fine-tune it on their own data, and get 90% of the value of your vertical model, your business model collapses.

The defense is proprietary training signal that the customer doesn't have. If your model is trained on 10 years of aggregated industry data that no single customer possesses, fine-tuning doesn't replicate it. If your model is just a fine-tuned version of a frontier model on the customer's own data, you're a services company, not a product company.

The Civilizational Bet

The broader question is whether India's AI strategy should prioritize sovereignty or specialization.

Sovereignty argues for building frontier models domestically, even at higher cost, to ensure strategic autonomy. Specialization argues for building vertical models where India has comparative advantage, and relying on global infrastructure for general-purpose AI.

I think specialization wins. Sovereignty in AI is expensive and brittle. The cost to maintain a competitive frontier model is not a one-time investment—it's an ongoing tax that grows every year as the frontier moves. India's GDP per capita is $2,500. The U.S. is $76,000. The capital efficiency required to compete on frontier models is not realistic.

But specialization in vertical AI is realistic. India has 22 official languages, 1.4 billion people, and regulatory systems that differ significantly from Western markets. The data specificity is structural, not temporary.

The companies that win will be the ones that stop trying to replicate OpenAI and start building what OpenAI can't.

Originally published at talvinder.com.

We Were Running AI Agents Before 'Agentic' Became a Buzzword

Talvinder Singh — Sat, 21 Mar 2026 04:10:53 +0000

In early 2024, we deployed a multi-agent system for Ostronaut before anyone called it "agentic AI." We called it "the pipeline." By late 2024, every vendor deck had "agentic" in the title. The architecture didn't change. The vocabulary did.

Here's the pattern that experience revealed: Agent Debt. The hidden complexity that accumulates when you treat agents as black boxes instead of understanding their failure modes. It isn't technical debt. It's operational blindness. You don't see it until an agent hallucinates in production, burns through your API budget, or produces output so confidently wrong that users trust it.

Building without frameworks meant hitting every orchestration failure, every context bleed, every runaway cost directly. That's what taught us what actually matters.

The Architecture We Built

Ostronaut generates corporate training content — presentations, videos, quizzes, games — from unstructured input. A client uploads a PDF. The system outputs interactive learning formats.

We built agents in four functional groups because the problem naturally decomposed that way:

Agent Type	Responsibility
Planner agents	Break input into learning objectives, decide format mix
Structure agents	Design slide sequences, video scripts, quiz flows
Content agents	Generate text, voiceovers, visual descriptions
Validation agents	Check quality gates, flag hallucinations, verify completeness

The planner-worker pattern: one planner agent analyzes the input and creates a generation plan. Worker agents execute tasks from that plan. Validation agents run post-generation checks.

This wasn't novel architecture. It was obvious once you tried to build the thing. But in early 2024, there was no CrewAI to handle orchestration. No LangGraph to manage state. We wrote the coordination logic ourselves.

What that meant in practice:

Context management was manual. Each agent needed the right slice of information: not too much (cost), not too little (hallucination). We built a context router that decided what each agent could see based on its task. It broke constantly. An agent would reference information from a previous step that wasn't in its context window. Output would be incoherent.

Tool-calling was brittle. Agents needed to invoke APIs for image generation, video rendering, database writes. Early LLM tool-calling was unreliable. An agent would call the wrong API, pass malformed parameters, or retry indefinitely on failure. We added a validation layer that parsed tool calls before execution. That caught 30% of bad calls.

Cost control was reactive. We didn't know what "normal" token usage looked like for a multi-agent pipeline. First month in production, we burned through our OpenAI budget in 2 weeks. The problem: redundant context. Multiple agents were processing the same source material because we hadn't optimized context sharing. We added a caching layer. Cost dropped 40%.

The Quality Crisis

Month 4, we hit the ceiling.

A healthcare client used Ostronaut to generate training for a clinical health program. The system produced a quiz. One question asked: "What is the recommended daily caloric deficit for healthy weight loss?" The agent-generated answer: "1000-1200 calories."

That's dangerously high for most people. The correct range is 500-750 calories.

The agent didn't hallucinate randomly. It pulled from a source document that mentioned 1000-1200 as an upper bound for specific cases. The agent extracted the number without the qualifier. The validation agent didn't flag it because it checked for factual consistency with the source, not medical safety.

We caught it in QA. But it revealed the core problem: agents optimize for coherence, not correctness. They will confidently generate plausible-but-wrong output if your validation layer doesn't encode domain constraints.

This is the failure mode that no prompt tuning fixes. You can instruct the model to "be accurate" as many times as you want. It will still extract numbers from context and strip their qualifiers, because that's what extracting the salient point looks like to the model.

What we changed:

Built domain-specific validation gates. For healthcare content, we added rules: flag any caloric recommendation above X, flag any medication dosage, flag any symptom-diagnosis claim. Not LLM-based validation. Rule-based checks that ran before content went to the client.

Added confidence scoring. Each agent outputs a confidence score for its generation. Low-confidence outputs go to human review. The scoring isn't sophisticated (token probability and context match), but it works. 15% of generations now route to human QA. That's acceptable.

Switched to template + generative hybrid. For high-risk content types (medical, financial, legal), we don't generate from scratch. We use templates with generative fill-ins. Reduces creative output, increases safety. Clients accepted the trade-off.

What We Got Wrong

Universal reasoning engine. We initially tried to build one planner agent that could handle all content types. A presentation has different structural constraints than a video. A quiz has different validation rules than a game. We split the planner into format-specific planners. That added agents but improved output quality significantly.

LLM-as-judge for validation. Early on, we used an LLM to validate other LLMs' output. "Does this quiz question make sense? Is this slide coherent?" That's circular. The validator had the same failure modes as the generator. We moved to rule-based validation for anything safety-critical. LLMs still validate style and tone. They don't validate facts. This failure mode is documented in more detail in why LLM-as-judge stacks fail for Indian markets — the underlying issue is the same regardless of geography.

Centralized orchestration. We built one orchestrator that managed all agents. It became a bottleneck. Every new feature required changing the orchestrator. We should have built federated orchestration, where each agent cluster (planner, worker, validator) manages its own coordination. We haven't refactored this yet. It's still painful.

Then vs. Now

If we built Ostronaut today with 2025 tooling, here's what would be easier:

What We Built by Hand	What Exists Now
Context routing logic	LangGraph state management
Tool-call validation layer	Built-in tool schemas in GPT-4
Agent orchestration	CrewAI, n8n workflows
Retry and error handling	Framework-level retry policies

What's still hard:

Domain-specific validation. No framework gives you medical safety checks or financial compliance rules. You build that yourself.

Cost optimization. Frameworks don't tell you which agents are burning tokens unnecessarily. You need observability and profiling. This is the same problem Indian SaaS companies are well-positioned to solve — twenty years of optimizing for constrained infrastructure builds exactly this instinct.

Failure mode discovery. Agents fail in creative ways. A framework might handle retries, but it won't tell you why an agent is producing inconsistent output. You learn that by watching production traffic.

The real difference: In 2024, we had to understand agent internals to build anything reliable. In 2025, you can deploy agents without understanding them. That's progress. But it creates Agent Debt.

The Falsifiable Claim

Teams that deploy agent systems without understanding planner-worker coordination, context boundaries, and validation layers will hit a quality ceiling within 3-6 months that no amount of prompt tuning will fix.

The ceiling shows up as:

Inconsistent output quality (works 80% of the time, fails unpredictably)
Cost spirals (agents making redundant API calls, over-generating)
User trust erosion (one bad generation destroys confidence in 10 good ones)

This isn't a prediction. It's a pattern I've watched repeat across every team that reached out after deploying agents without validation gates. The vendors selling "agentic platforms" are solving orchestration and deployment. They're not solving validation, cost control, or failure mode discovery. Those are still your problem.

This dynamic connects to something broader happening in the shift from software to agentware — as the abstraction layer rises, the hidden complexity doesn't disappear. It concentrates at the failure modes the frameworks don't cover.

The Question Worth Asking

If you're deploying agents today, ask this: Can you explain why an agent made a specific decision?

Not "what did it output?" but "why did it choose this approach over alternatives?"

If the answer is "the LLM decided," you have Agent Debt. You're trusting a black box. That works until it doesn't.

The teams that will build reliable agent systems aren't the ones using the fanciest frameworks. They're the ones who understand what happens when context bleeds between agents, when a planner makes a bad decomposition, when a validator misses a hallucination.

We learned that by building without frameworks. You can learn it faster now — but only if you look under the hood.

Originally published at talvinder.com.

AI Is Making Your Team Slower — The Math Your CEO Won't Show You

Talvinder Singh — Fri, 20 Mar 2026 03:31:28 +0000

Every company measuring AI productivity is counting the wrong thing.

They're measuring output volume: PRs merged, lines written, tickets closed. They're not measuring the cost of what ships: the review burden, the debugging time, the incidents caused by code nobody understood before it hit production.

When you count both sides, the math doesn't work the way your CEO's slide deck says it does.

The Evidence Is Piling Up

This week, The Pragmatic Engineer catalogued what's actually happening inside companies that went all-in on AI coding agents. The findings aren't theoretical.

Amazon's retail engineering team saw a leap in outages caused directly by AI agents. The fix? Requiring senior engineer sign-off on all AI-assisted changes from junior developers. That's not a productivity gain. That's adding a bottleneck to compensate for unreliable output.

Anthropic — the company that builds Claude — ships over 80% of its production code with AI. Their flagship website degraded so badly that paying customers noticed before anyone internally did. The irony writes itself.

Meta and Uber are tracking AI token usage in performance reviews. Engineers who don't use AI tools enough look unproductive. Engineers who use them indiscriminately look great on paper — until the bugs ship.

The Three Taxes You're Not Counting

Here's the falsifiable claim: teams that measure AI productivity only by output volume will see their incident rate and mean-time-to-resolve increase by 30% or more within 12 months, compared to teams that gate AI output with validation layers.

The mechanism has three parts.

The Review Tax

Every AI-generated PR still needs human review. But AI-generated code is harder to review than human-written code, because the reviewer can't infer intent from the author's history.

With human code, you know the developer's context: what they were trying to solve, what trade-offs they considered, what they tested. With AI code, you're reverse-engineering intent from output. That's slower, not faster.

Amazon learned this the hard way. Junior engineers using AI agents shipped code that looked correct — clean formatting, reasonable variable names, passing tests — but had subtle logical errors that only surfaced in production. Reviewers couldn't distinguish "AI wrote this well" from "AI wrote this plausibly."

The Refactoring Freeze

Dax Reed, who built OpenCode, points out something every experienced engineer recognises: AI agents discourage refactoring. When code is cheap to generate, nobody wants to clean it up. Why spend an afternoon restructuring a module when the agent writes a new one in ten minutes?

The result is an expanding codebase where nothing gets simplified, patterns don't converge, and cognitive load increases week over week.

This is the velocity trap. Short-term speed, long-term slowdown. Sentry's CTO observed the same pattern: AI removes the barrier to getting started, which sounds great until you realise that "getting started" was never the bottleneck. The bottleneck was maintaining, debugging, and evolving what you built. AI makes the first part trivially easy and the second part measurably harder.

The Incentive Poison

When companies tie AI token usage to performance reviews, they're telling engineers: "Use the tool, regardless of whether it helps."

This is the corporate equivalent of measuring developer productivity by lines of code written. It rewards volume, punishes judgment, and guarantees that the engineers who are most careful about code quality look the least productive.

Engineers who know the AI output is mediocre ship it anyway, because slowing down to rewrite it makes their metrics look bad. The codebase degrades. The team slows down. The metrics still look great, because the metrics are measuring the wrong thing.

What This Looks Like Up Close

I've seen this pattern building multi-agent systems at Ostronaut. We generate training content — presentations, videos, quizzes. Early on, the agents were fast. They produced a complete training module in minutes. The output looked good. Formatting was clean. Structure was reasonable.

It was also wrong about 15-20% of the time. Not obviously wrong — subtly wrong. A slide deck where the concept progression didn't build properly. A quiz where the distractors were too close to the correct answer. A video script that repeated a key point in slightly different words, creating confusion instead of reinforcement.

We didn't fix this with better prompts. We fixed it by building a validation layer — automated checks that ran after every generation step, before anything reached a human reviewer. Content validation caught conceptual errors. Design validation caught structural problems. Integration validation caught mismatches between components.

That validation layer was harder to build than the generation layer. It took longer. It required more engineering judgment. And it's the only reason the system works reliably.

The companies in Gergely's article skipped this step. They deployed AI agents without validation gates, measured the output volume, and declared victory. Then the incidents started.

Why Better Models Won't Save You

I used to think the answer was better models. If GPT-4 produces code that's 80% reliable, GPT-5 will be 95% reliable, and eventually you won't need validation.

That was wrong for two reasons.

First, the remaining failures are the expensive ones. The bugs that survive better models are the subtle, context-dependent bugs that cause production incidents. Better models don't make validation cheaper — they make it more necessary, because what gets through is harder to catch.

Second, the validation layer isn't just catching bugs. It's encoding team knowledge. Our quality checks embed years of domain expertise — what makes a good slide progression, what makes a quiz effective, what makes a video script clear. That knowledge doesn't exist in the model. It exists in the team. The validation layer is how you transfer institutional knowledge into the AI pipeline.

Companies that skip this aren't just accepting more bugs. They're disconnecting their AI pipeline from their institutional knowledge.

What to Measure Instead

What Leadership Measures	What Actually Happens
PRs merged per week (+52%)	Review time per PR (+40%)
Lines of code written (3x)	Lines nobody understands (3x)
Time to first commit (-60%)	Time to resolve incidents (+35%)
Token usage per engineer	Refactoring frequency (-70%)

If you're measuring AI impact, stop counting PRs. Start counting:

Incident rate per AI-assisted commit versus human-only commits
Review time per PR — is it actually decreasing, or are reviewers rubber-stamping?
Refactoring frequency — is your team still simplifying code, or just adding to it?
Mean-time-to-resolve for bugs in AI-generated code versus human-written

The companies that will win with AI coding agents are not the ones that deploy them fastest. They're the ones that build the validation layer first and measure what matters — not how fast code is written, but how fast correct code ships and stays correct in production.

Speed without verification isn't velocity. It's technical debt with a marketing budget.

Originally published at talvinder.com.

The OS-Paged Context Engine

Talvinder Singh — Thu, 19 Mar 2026 03:40:11 +0000

Every production agent system I've worked on has the same failure mode. Context rot. Stale artefacts silently served to the model. No audit trail for what was included or excluded. Token budgets blown with no graceful recovery. Multi-agent context bleeding across scopes.

The standard fix is "use RAG." RAG solves retrieval. It doesn't solve lifecycle.

The counter-argument I hear most: context windows are getting larger. Claude does 200K tokens. Gemini does 1M. Just dump everything in. The math doesn't hold. At $15 per million input tokens, stuffing 847 artefacts (~200K tokens) into every call costs $3 per inference. At 100 calls per day per agent, that's $9,000/month for a single agent. And you still can't audit what the model saw, still can't catch stale data, still can't prevent hallucinations from compounding into memory.

Context has no lifecycle. That's the root cause. I went looking for prior art in constrained computing, where managing scarce resources under real-time pressure has been solved for decades.

Same Query, Two Outcomes

A support agent is handling a billing escalation. The context store has 847 artefacts: ticket history, knowledge base articles, past chat transcripts, agent notes, CRM records.

The query is the same. The model is the same. The only difference is what sits between the store and the prompt.

Without lifecycle management (standard RAG): the agent runs a semantic search, takes the top-K matches, stuffs them in.

A refund policy from six months ago loads because it's semantically close. The policy was updated two weeks ago. The agent cites the old $200 limit to a customer whose refund should be $400 under the current policy.
An agent's internal note (unreviewed, unvalidated) loads as context. The model treats a scratchpad draft as a confirmed resolution.
Token budget blows out at 140%. The API silently truncates the prompt, dropping the most recent ticket update.
The agent's response gets written to memory. The outdated policy is now a "fact." Next session, it compounds.

With the OS-Paged Context Engine: the same 847 artefacts enter a four-stage pipeline.

Triage: 312 artefacts expire on TTL. The internal note scores below provenance threshold (SCRATCHPAD rank). The stale policy is BLACK-tagged. 20 survive for semantic scoring.
Paging: a knowledge base article that did survive has a dirty bit set (source updated 2 weeks ago). Re-fetched with current policy before the model sees it.
Assembly: 31,200 tokens against a 40,000 budget. No truncation.
Validation: response scores 0.88 confidence. Committed to memory. Below 0.7, it would have been flagged for review and not persisted.

Failure Mode	Standard RAG	OS-Paged Engine
Stale artefact loaded	Serves 6-month-old policy as current	TTL expires it. Dirty bit catches mid-session staleness.
Unvalidated note treated as fact	Loads if semantically close	SCRATCHPAD provenance rank filters it in triage
Token budget overflow	Silent API truncation	Graceful degradation through four tiers
Hallucination persisted to memory	Written back without checks	Commit gate: low confidence triggers rollback
Audit trail	None	Immutable manifest: trace ID, artefact list, tier, commit status

Every one of these is a lifecycle failure, not a retrieval failure.

The Fix: Four Borrowed Techniques

I built a four-stage pipeline. Each stage borrows one technique from a domain that solved this class of problem decades ago. No framework lock-in. Single Python file. Works with any LLM API.

I'm calling it the OS-Paged Context Engine, because the core insight is that your context window is RAM, your long-term memory is disk, and you need an operating system between them.

Stage 1: Triage Scoring

The failure it catches: embedding 1,000 artefacts per call at ~1ms each = 1 second of latency before inference starts.

Borrowed from: ER START Protocol, 1983. You don't need full diagnosis to correctly prioritise. Score all candidates on three cheap signals first:

R (Recency) is a timestamp diff. O(1). P (Provenance) is an enum rank: human-verified > RAG chunk > tool output > agent scratchpad. O(1). S (Semantic) is cosine distance. Computed only for artefacts that survive R+P filtering.

Source Type	Score Bias	Triage Outcome
Human-verified memory	Provenance-heavy (P=0.5)	Highest priority, loaded first
RAG chunk (recent)	Balanced (R=0.4, S=0.4)	High — recency and relevance both count
Tool output	Recency-heavy (R=0.5)	Medium — freshness matters most
Agent scratchpad	Semantic-heavy (S=0.5)	Low — must be highly relevant to survive
Expired artefact	TTL=0	Excluded before scoring even starts

Stage 2: Paged Context Store

The failure it catches: serving stale context because nobody checked whether the source changed since it was loaded.

Borrowed from: OS Virtual Memory, 1962. The page table decided what lived in fast memory, evicted least-recently-used pages, and tracked modifications via dirty bit.

LRU eviction: when the window is full, evict what was accessed longest ago. Dirty bit: if the source changed since the artefact was loaded, flag it dirty and re-fetch before use.

def access(self, artefact_id):
    art = self._lru[artefact_id]
    current_hash = hash(self._long_term[artefact_id].content)
    if current_hash != art._source_hash:
        art._dirty = True        # source changed → force re-fetch
    self._lru.move_to_end(artefact_id)  # promote to MRU
    return art

RAG retrieves once and serves forever. A paged store tracks whether the source has changed.

Stage 3: Speculative Assembly

The failure it catches: hallucinations compounding across sessions because agent-generated context is written to memory without validation.

Borrowed from: CPU Reorder Buffer, Intel P6, 1995. Execute speculatively, hold results in a buffer, commit only when confirmed valid. Wrong? Rollback.

Assemble context optimistically. Start inference. If confidence exceeds threshold, commit to memory. If not, flag for human review. Do not write to long-term store. Without this gate, session one's hallucination becomes session two's "memory" becomes session three's "fact."

# After model responds:
if evaluator_confidence >= 0.7:
    manifest.committed = True       # safe to write to long-term store
else:
    manifest.flagged_for_review = True  # hold — do not persist

At Ostronaut, we saw exactly this: unvalidated agent-generated context compounding into confidently wrong output downstream. The commit gate cut that class of failure by roughly half.

Here's the falsifiable claim: any multi-agent system without a commit/rollback gate on context writes will compound hallucinations across sessions within 30 days of production use.

Stage 4: Graceful Degradation

The failure it catches: token budget overflows that crash the API call or silently truncate critical context.

Borrowed from: Radio Programme Stack, 1930s. Dead air could never happen. When content overran, drop to the next segment. The broadcast always continued.

Tier	Triggers at	Strategy	Example
1 (Full)	< 80% budget	All triage winners	Happy path. Everything fits.
2 (Summarised)	80-95%	Compress memories, truncate RAG	Chat transcripts become 200-token summaries.
3 (Core only)	95-110%	Human-verified facts + system prompt	Only ground truth. Scratchpad and RAG dropped.
4 (Minimal)	> 110%	System prompt only. Human review flag.	Emergency. Escalate.

The Composed Pipeline

async def assemble_context(query, store, budget, scope):
    candidates = await store.get_candidates(scope=scope)
    scored = triage.score(candidates, query=query, top_k=50)
    loaded = store.load_page(scored, token_budget=budget)
    manifest = speculator.assemble(loaded, budget=budget)
    if manifest.token_count > budget:
        manifest = fallback_stack.degrade(manifest, budget=budget)
    return manifest

Every call produces an immutable manifest. When the compliance team asks "why did the agent say that?" you hand them the manifest.

I argued previously that context is infrastructure, not a feature. This is the implementation pattern.

What I Got Wrong

The first version didn't have the two-pass triage. Every artefact got embedded on every call. At 1ms per embedding multiplied by 1,000 artefacts, that's a full second of latency before inference starts. Adding R+P pre-filtering dropped that to roughly 20 embeddings per call. The two-pass approach seems obvious in retrospect. It's literally how ER triage works. But the RAG literature doesn't teach you to pre-filter before embedding.

The other mistake: not implementing the dirty bit from day one. We had artefacts in the context window from external tools that had returned fresh data hours ago. The model was reasoning about stale state. Adding dirty bit tracking on access (not just on write) was a one-line fix that eliminated an entire class of silent failures.

The third mistake is in the commit gate itself. The code checks evaluator_confidence >= 0.7, but who computes that score? If the model self-evaluates, you're trusting the same system that may have hallucinated to judge whether it hallucinated. LLM confidence self-assessment is poorly calibrated. The honest answer: the library deliberately does not compute confidence. The caller must supply it via an external evaluator, a rule-based checker, or human-in-the-loop for high-stakes domains. The commit gate is necessary. What sits behind it is not yet solved.

When This Pattern Is Overkill

Not every agent needs lifecycle management. If your agent doesn't write to its own memory and doesn't persist across sessions, standard RAG is sufficient. Single-session chatbots, prototypes with fewer than 100 artefacts, read-only Q&A over a fixed corpus: the overhead of triage, paging, and commit gates exceeds the benefit. This pattern pays off when context has a lifecycle. If it doesn't, skip it.

What's Still Open

What remains genuinely unresolved is governance at scale. When an agent has six months of context about a customer, who owns it? What happens under GDPR deletion requests? Do you tombstone or purge? If you purge, does the agent's behaviour change in ways that affect other customers? I'm working through that question next.

The full library is a single Python file, zero dependencies, open for anyone building production agents. The techniques are borrowed. The composition is yours to steal.

Originally published at talvinder.com.

Context Governance at Scale

Talvinder Singh — Thu, 19 Mar 2026 03:34:19 +0000

The OS-Paged Context Engine handles the technical lifecycle: what loads, what gets evicted, what passes validation. It produces an immutable manifest for every call. But the manifest tells you what the model saw. It does not tell you whether it should have seen it.

Production agents that handle money, health data, or customer PII need a governance layer above the pipeline. Access control, retention policies, deletion rights, multi-tenant isolation. These are governance problems, not engineering problems. And the industry has not solved them.

The Manifest Is Not Enough

An audit manifest records: trace ID, artefact list, token count, degradation tier, commit status. If a compliance officer asks "what did the agent access?" you can answer. That's table stakes.

The harder questions:

Should the agent have had access to that customer's payment history during a routine support query?
The artefact was loaded from a shared scope. Three other agents also read it. One of them serves a competitor's account. Is that a data leak?
The agent's response was committed to memory at confidence 0.85. Six months later, the customer invokes GDPR Article 17. Do you delete the artefact, the memory derived from it, or both?

These questions have no clean technical answer. They require policy, and policy requires architecture.

When This Pattern Is Overkill

Not every agent needs governance. Here's the decision tree:

Skip lifecycle management entirely if:

The agent is single-session. No memory persists. Standard RAG is sufficient.
The corpus is small and static (fewer than 100 documents, updated quarterly). Triage and paging overhead exceeds the benefit.
The agent is read-only. Never writes to its own memory. No compounding hallucination risk.

Use the technical pipeline but skip governance if:

The agent handles non-sensitive data. Productivity tools, code assistants, research summarisers. No PII, no financial data, no health records.
Single-tenant deployment. One company, one agent, no cross-customer context risk.

You need the full governance layer when:

The agent handles PII, financial data, or health records
Multiple agents share a context store across customers or tenants
You operate in a regulated industry (healthcare, insurance, financial services)
The agent persists context for months and customers have deletion rights

Here's the falsifiable claim: by 2028, any agent system handling PII without an auditable context manifest will fail compliance review in regulated industries.

GDPR and the Tombstone Problem

A customer requests deletion under GDPR Article 17. You purge their artefacts from the context store. The manifests that referenced those artefacts still exist in the audit log. The agent's behaviour was shaped by context that no longer exists.

Two approaches, neither clean:

Purge completely. Delete artefacts, delete manifests, delete any memory derived from those artefacts. The agent's future behaviour changes because the context that shaped prior decisions is gone. If Agent B's response was informed by Agent A's output, which was informed by the deleted customer data, do you cascade the deletion?

Tombstone. Replace artefact content with a deletion marker: "Artefact deleted per GDPR request, [date]." Manifests remain intact for audit. The agent knows something was here but not what. This preserves audit trail integrity but may not satisfy strict interpretation of "right to erasure."

The honest answer: I don't know which is correct. The legal interpretation of "erasure" applied to derived AI context is untested in European courts. What I do know is that you need the manifest layer to even have this conversation. Without an audit trail, you cannot comply with a deletion request at all.

Compliance as Architecture

Enterprise buyers in healthcare, financial services, and insurance ask one question first: can you prove what the agent accessed?

The context manifest maps directly to compliance frameworks they already understand:

Compliance Requirement	What It Maps To
SOC2 audit logs	Context manifest (trace ID, artefact list, timestamp)
HIPAA access logs	Manifest + agent_scope (who accessed what)
GDPR Article 15 (right of access)	Manifest query: "all artefacts accessed for customer X"
GDPR Article 17 (right to erasure)	Artefact deletion + manifest tombstoning
PCI-DSS data isolation	agent_scope + namespace isolation per tenant

But agent_scope alone is not sufficient for multi-tenant isolation. In the current implementation, scope is a string tag. No encryption boundary, no policy engine, no access control list. A developer who writes agent_scope="global" on a PII artefact has just leaked it to every agent in the system.

Production multi-tenant context isolation requires: namespace enforcement (scope is a hard boundary, not a suggestion), policy-as-code (which scopes can read which artefact types), encryption at rest per tenant, and audit logging on every cross-scope access attempt.

What I Don't Know Yet

The technical primitives for context governance exist: manifests, scopes, commit gates, audit logs. What doesn't exist is the organisational trust model.

When an agent makes a decision based on six months of accumulated context, who is accountable? The engineer who built the pipeline? The data team that ingested the artefacts? The compliance officer who approved the retention policy?

Kubernetes solved compute governance by making infrastructure declarative. You declare what you want, the system ensures it. Context governance needs the same shift: declare what the agent should access, and the system enforces it. We're not there yet.

The technical pipeline is built. The infrastructure argument is established. The governance layer is the missing piece. I'm building it in the open, and I don't have all the answers.

Originally published at talvinder.com.