DEV Community: Sonal Jain

Your AI Demo Works. That's the Problem: Crossing the PoC-to-Production Gap

Sonal Jain — Mon, 29 Jun 2026 07:00:27 +0000

The demo always works. That is precisely why it should worry you.

You have sat in the meeting. Someone shares their screen, types a question, and an AI assistant answers it perfectly. It pulls the right number, drafts the right email, summarises the right document. The room nods. A budget gets approved. And then, somewhere between that flawless thirty-second demo and a system your customers actually rely on, the project goes quiet — slips, balloons in cost, or never ships at all. This is the most expensive pattern in enterprise software right now, and the demo is what sets the trap.

Here is the uncomfortable truth almost no vendor will say out loud during a pitch: a working demo proves the easiest twenty percent of the problem. The hard eighty percent — the part that decides whether you get value or a write-off — is everything the demo carefully avoided. This article is about that gap: why it exists, why it kills projects, and exactly what the minority of teams who cross it do differently.

80%

of AI projects fail to deliver on their goals, by RAND's analysis

~84%

of those failures are organizational and operational, not the technology

20 / 80

the demo is the first 20%; production is the other 80%

Sources: RAND Corporation analysis of why AI projects fail; the 20/80 split is the production gap pattern Shanti sees across delivery.

Why a Demo That Works Tells You Almost Nothing

A demo is a performance, and like any performance, it is staged. None of that is dishonest — it is simply what a demo is for. The problem is the conclusion you draw from it. When you watch a flawless demo, your brain quietly fills in: "if it can do that, it can do the real thing." It usually cannot, not yet, because the demo removed every condition that makes the real thing hard.

Think about what a demo silently controls for. The input is hand-picked — the question the presenter typed is one they know the system handles well. The data is clean and curated, not the decade of inconsistent, half-empty, contradictory records sitting in your actual systems. There is a human in the loop steering it, ready to rephrase a prompt if the first attempt wobbles. There is no real load, no concurrent users, no malicious input, no compliance requirement, no integration with the seven other tools your business depends on, and no question of what it costs when ten thousand people use it instead of one. Strip those conditions away and you have a magic trick. Add them back and you have engineering.

The demo measures capability. Production measures reliability.

Those are different questions. "Can it answer this well, once, when I set it up?" is not "Will it answer reliably, safely, and affordably, every day, when reality pushes back?" Most AI buying decisions are made on the first question and then judged on the second.

The 80% Nobody Puts in the Demo

If the demo is the visible tip, here is the mass under the water — the work that turns a prototype into something a business can actually depend on. None of it is glamorous, and that is exactly why it gets skipped in the sales cycle and discovered in the budget overrun.

In the demo	In production (the real work)
One clean, hand-picked input	Messy real-world inputs, missing fields, contradictory records, and inputs nobody anticipated
It answered correctly	Defined behaviour for when it is wrong: how it flags uncertainty, escalates, and fails safely
A person steering the prompt	Unattended operation with guardrails, so a non-expert user cannot break it
Runs on a laptop	Scales to real concurrency without latency collapse or runaway cost
Standalone screen	Integrated with your CRM, billing, auth, and the tools your team already lives in
No one is watching the data	Security, access control, a data-processing agreement, and audit trails for compliance
It worked today	Monitoring, evaluation, and maintenance so it still works in three months as data and models drift

Look down that right-hand column and notice something: almost none of it is about the AI model. It is data engineering, error handling, integration, security, observability, and operations. The model — the bit everyone obsesses over in the demo — is one component in a system that is mostly plumbing. That is not a criticism of the technology; it is a description of what software has always been. AI did not change it. AI just made the demo so convincing that people forgot the plumbing exists.

A useful gut check before you fund anything:

ask the team to demo the system being

wrong

. Feed it a malformed request, an out-of-scope question, a date that does not exist. A production-minded partner will show you graceful failure on cue. A demo-ware vendor will look uncomfortable, because they only built the happy path.

The Failures Are Organizational, Not Technical

The instinct, when an AI project stalls, is to blame the model — it is not smart enough, the prompt needs work, maybe a newer model will fix it. That is almost never the real reason. RAND's analysis of why AI initiatives fail puts more than eighty percent of failures down to organizational and operational causes, not the underlying technology. In plain language: the model was usually capable enough. The project failed for human and structural reasons.

Those reasons rhyme across almost every stalled project we have been called in to rescue. The problem was framed around a technology rather than a measurable business outcome, so nobody could say what "done" or "working" actually meant. The data the system needed was not ready — scattered, dirty, or locked in systems no one could integrate with. The team mistook the prototype for the product and tried to ship the demo, then watched it crumble on contact with real users. And there was no plan for the long tail: monitoring, retraining, and maintenance, the unglamorous work that keeps an AI system alive after launch. The technology was the easy part. The discipline around it was missing.

This is genuinely good news, even though it does not sound like it. It means success is not gated behind some breakthrough you cannot control. It is gated behind decisions and engineering practices you absolutely can — if you and your partner take the eighty percent as seriously as the twenty.

Set the Production Bar Before You Build the Demo

The single highest-leverage move you can make is also the cheapest: define what "production-ready" means before anyone builds a prototype. Most teams do this backwards. They build the demo, get excited, and only then discover what real use demands. By then the demo's shortcuts are baked into expectations and the budget. Flip the order. Decide the bar first, then build a prototype that tests the hardest part of clearing it — not the easiest part of looking impressive.

Writing the bar down is not bureaucracy; it is the contract that protects you. It turns "the demo worked" into something you can actually hold a vendor to. Here is the minimum it should pin down:

The outcome, in business terms. Not "use AI for support" but "resolve 40% of tier-1 tickets end to end without a human, measured weekly." If you cannot measure it, you cannot tell success from theatre.
The accuracy and failure bar. What level of correctness is acceptable, and exactly what must happen when the system is unsure or wrong — escalate, flag, refuse? Wrong-answer behaviour is a product requirement, not an afterthought.
Latency and cost at real scale. How fast must it respond, and what is the cost per request when usage is ten or a hundred times the demo? Get a written estimate for "what happens when this doubles."
The integration surface. Which existing systems it must talk to, and who owns that work. Integration is where timelines quietly go to die.
Security, data, and compliance. Where data lives, who can see it, whether it trains anyone else's model, and what regulation applies. In regulated industries this is the wall most pilots hit.
The operating plan. Who monitors it, how it is evaluated over time, and what maintenance is budgeted. Launch is the start of the work, not the end.

A prototype built against a real production bar is worth ten polished demos. It tells you whether the

hard part

is solvable. The demo only ever told you the easy part already was.

How to Be in the Winning Minority

If most AI projects fail to deliver and most of those failures are self-inflicted, then crossing the gap is not luck — it is a method. The teams who land in the minority that actually ships value tend to do the same handful of things, and none of them require a bigger model.

They treat the proof of concept as a test of the riskiest assumption, not a sales artefact — they point the prototype straight at the messiest data and the nastiest edge case, because that is what they are actually unsure about. They pilot against real users and real inputs early, in a contained way, so reality corrects the plan while it is still cheap to change. They scope to a measurable outcome and write it into the contract, so "working" has a definition both sides agreed to. They budget for the eighty percent — the integration, the security, the monitoring, the maintenance — from day one, instead of discovering it as a series of unwelcome surprises. And they pick a partner who shows them production systems used by real people, not a reel of demos, and who is comfortable demonstrating failure as confidently as success.

That last one is the quiet differentiator. Anyone can show you a demo that works. Far fewer can point to something live, handling real load, that has survived contact with real users for a year. When you are evaluating a partner, ask to see the boring parts: the monitoring dashboard, the error-handling, the thing that broke once and how they caught it. The willingness to show you the unglamorous eighty percent is the single best signal that they know it exists — and that they will build it for you instead of handing you a demo and an invoice.

We Build for the 80% the Demo Skips

Shanti Infosoft is a CMMI Level 5 software engineering firm. We take AI from a working prototype to a hardened production system — with the integration, security review, human QA, and monitoring that decide whether you get value or a write-off. You get a named senior team, written fixed-scope estimates, and full IP and source ownership.

Book a Free 20-Min Call →

Explore AI Development Services

Frequently Asked Questions

Why do so many AI projects fail after a successful demo?

Because the demo only proves the easiest part. It runs on clean, hand-picked data, in a controlled setting, with a human steering it. Production brings messy inputs, edge cases, security and compliance requirements, integration with existing systems, monitoring, and cost at scale. RAND's analysis finds that the majority of AI project failures are organizational and operational, not down to the model's raw capability.

What is the difference between an AI proof of concept and a production system?

A proof of concept answers "can this work at all?" on a narrow, favourable example. A production system answers "will this work reliably, safely, and affordably for real users, every day, when things go wrong?" The PoC is roughly the first 20% of the effort; the remaining 80% is data pipelines, error handling, evaluation, security, integration, monitoring, and maintenance.

How long does it take to move an AI prototype into production?

For a well-scoped use case, expect roughly 2 to 5 months from a working prototype to a hardened production system, depending on data quality, integration complexity, and compliance needs. Be sceptical of anyone promising "production-ready" in a couple of weeks — that usually means shipping the demo as if it were the product.

How do I avoid building an AI demo that never reaches production?

Define the production bar before you build the demo: the accuracy, latency, cost, and failure behaviour real use requires, plus how integration, security, and monitoring will be handled. Then build a prototype that tests the hardest part of clearing that bar, pilot it against real data and real users early, and treat the demo as a probe of the riskiest assumption rather than a finished product.

Is the AI model usually the reason a project fails?

Rarely. In most stalled projects the model was capable enough; the project failed because the outcome was never defined in measurable terms, the data was not ready, the prototype was mistaken for the product, or no one owned the ongoing operation. Those are fixable, controllable problems — which is why the gap is crossable with the right discipline and partner.

Written by

Rishabh Jain

AI Consultant & Founder, Shanti Infosoft LLP

Shanti Infosoft is a CMMI Level 5 software engineering firm. We deliver every project with written, fixed-scope estimates, full IP and source-code ownership for the client, and a named team of senior engineers. We specialise in taking AI from prototype to production: 700+ projects delivered across web and mobile development, AI integration, and offshore engineering.

700+ Projects Delivered | CMMI Level 5 | 4.9★ on Clutch | 38,000+ hrs on Upwork

Telling Clients the Truth About AI Timelines

Sonal Jain — Mon, 29 Jun 2026 05:30:18 +0000

The most valuable thing a project manager does on an AI build isn't planning the work. It's telling the client the truth about it early, especially when the truth is "this will take longer than the demo suggested." Clients don't leave over honest bad news delivered on time. They leave over surprises delivered too late to act on.

I've delivered enough AI projects to know the technical risks are manageable. The relationship risk is the one that sinks engagements, and it's almost always a communication failure, not a code failure. Here's how I handle it.

Why AI makes expectation-setting harder

AI raises expectations before the first meeting. The client has seen a viral demo, maybe built something impressive in an afternoon with a chatbot, and walks in believing the hard version is nearly as fast. It isn't. A polished pilot can take weeks while the production system behind it takes months. If I let that gap sit unspoken, every honest update later sounds like an excuse.

So I set the frame at the start, in plain language: the demo proves the idea can work; getting it to work reliably, on your real data, inside your systems, every time, is the actual project. We put the same point sharply in The AI demo works. That's the problem. Saying it on day one isn't lowering ambition. It's the difference between a client who trusts my updates and one who reads them as backpedaling.

Deliver bad news on a schedule, not when cornered

The instinct under pressure is to wait, to hope the slip recovers before anyone asks. It almost never does, and waiting converts a manageable conversation into a crisis. My rule is simple: the client hears about a risk while there's still time to do something about it.

What that looks like in practice:

Flag risk early and specifically. Not "things are a bit tight." Instead: "The CRM integration is messier than scoped. That puts the date at risk by roughly a week. Here are two options."
Always bring options, not just problems. Cut scope, move the date, add a person, ship a smaller first version. A problem with choices attached is a decision. A problem alone is just anxiety I've handed the client.
Separate facts from forecasts. "This is what we know" and "this is what we expect" are different statements. Blurring them is how confidence quietly erodes.
Make the trade-offs theirs. It's their budget and their business. My job is to lay out the real choices clearly, not to make them in the dark and present a result.

This is also why I'm careful with accuracy and reliability targets. "More accurate" sounds like a quick promise and can be an open-ended one. I commit to a number against a test set, and I explain why I won't promise perfection: these systems are probabilistic, and a confident "100%" is a lie I'd have to walk back.

The honesty compounds

Here's the part that took me years to fully trust. Every time I deliver hard news early and well, the relationship gets stronger, not weaker. The client learns that when I say the date holds, it holds, because I'm the same person who told them when it didn't. That credibility is the most valuable asset on a project, and it's built entirely out of small, uncomfortable, on-time truths.

The opposite compounds too. One buried problem that surfaces at the deadline can cost you a client who would have happily accepted the same news three weeks earlier. The information was the same. The timing was everything.

Key takeaways

The biggest risk on an AI project is usually the relationship, and it fails through poor communication, not poor code.
Set the demo-versus-production frame on day one so honest updates later don't read as excuses.
Deliver bad news while the client can still act on it. Late truth becomes a crisis; early truth stays a decision.
Always bring options, separate facts from forecasts, and let the client own the trade-offs.
Commit to accuracy as a number against a test set, and refuse to promise perfection from a probabilistic system.

FAQ

When should I tell a client a project is slipping? The moment you're confident the risk is real and there's still time to respond. Early enough to choose a path beats waiting for certainty that arrives too late to matter.

How do I deliver bad news without losing the client? Pair it with options and a clear recommendation. Clients handle hard news well when you hand them choices and the information to decide. They handle it badly when it's a surprise with no way out.

Should I promise a specific accuracy figure for an AI feature? Promise a number tied to an agreed test set, never perfection. Explaining why 100% isn't realistic builds more trust than a confident figure you'll later have to retract.

If you've been burned by an AI project where the bad news always arrived too late, that's a delivery and communication problem worth fixing before the next one. The team at Shanti Infosoft builds software on honest timelines, and we're glad to talk through how we'd de-risk yours.

The 5 Things AI Projects That Don't Get Cancelled Do Differently

Sonal Jain — Sun, 28 Jun 2026 07:00:08 +0000

Gartner expects more than 40% of agentic AI projects to be dead by the end of 2027. Here's the part nobody puts on the scary slide: the other ~60% are doing five specific, unglamorous things — and none of them is "had a better model."

We've delivered AI projects that shipped and stayed shipped, and we've been called in to rescue ones that didn't. The difference between the two groups is remarkably consistent, and remarkably learnable. It has almost nothing to do with how advanced the technology is and almost everything to do with how the project is framed, scoped, and run. If you're about to start an AI initiative — or you're worried about one already underway — this is the checklist that keeps you on the right side of that line.

40%+

Of agentic AI projects Gartner expects cancelled by 2027

~84%

Of AI failures are organizational, not technical (RAND)

5 habits

Separate the projects that ship from the ones that get pulled

1. They Start Absurdly Narrow

The single biggest predictor of an AI project surviving is how small it started. Cancelled projects tend to begin with sweeping ambition — "transform customer service with AI," "build an autonomous operations agent." Surviving projects begin with something almost embarrassingly specific: "automatically categorise and route the 400 support tickets we get every day," or "draft first-pass responses to the five most common refund requests."

Narrow scope does three things at once. It makes success measurable, it makes the build achievable in weeks rather than quarters, and — crucially — it produces a win the organisation can see and believe in before the budget patience runs out. RAND's finding that around 84% of AI failures are organizational rather than technical is really a finding about this: projects collapse under their own ambition and the organisation's lost faith, not under a technical limit. A small thing that visibly works buys you the credibility to do the next, slightly bigger thing.

The rule:

if you can't describe your first AI deliverable as a single, countable task with a number attached, it's too big. Cut it down until you can. The teams that ship would rather automate one workflow completely than ten workflows partially.

2. They Define the Number That Means "It Worked"

Surviving projects decide, before they build, exactly how they'll know it worked — and they write it down. Not "improve efficiency." Something a spreadsheet can settle: "cut average ticket-handling time from 12 minutes to 4," "reduce manual data entry by 30 hours a week," "answer 80% of tier-1 queries without a human."

This matters for a reason that's almost political. A project with a clear target can declare victory and earn its next round of investment. A project with a vague goal can never definitively succeed — which means it's permanently vulnerable to the budget review that asks "what did we actually get for this?" Vague goals don't protect a project; they leave it defenceless. The teams that don't get cancelled make their success undeniable on purpose.

3. They Keep a Human in the Loop — and Say So

Counterintuitively, the projects that survive are usually less autonomous than the ones that fail. The cancelled projects often over-reach for full automation, hit the inevitable wrong answers, lose trust, and get switched off. The surviving projects design the human in from the start: the AI does the heavy lifting and a person handles the edge cases, the high-stakes calls, and the quality check.

This isn't a lack of ambition — it's how you build trust and how you stay safe. It also reflects what the data keeps showing about AI-generated work: in CloudBees' 2026 research, the same teams that trusted AI output overwhelmingly also reported more incidents from it. The lesson generalises beyond code. AI is a powerful drafter and a poor final authority. The projects that last treat it that way — and they're transparent with stakeholders that a human still owns the outcome, which is precisely what makes leadership comfortable enough to keep funding it.

4. They Ship Fast, Then Improve in Public

Surviving projects get something real in front of real users quickly — in weeks — and then improve it based on what actually happens. Cancelled projects tend to disappear into a long build, perfecting in private, until the day they emerge to discover the requirements changed, the stakeholders cooled, or the thing they built solved a problem nobody has anymore.

A fast, visible first version does something a perfect invisible one can't: it keeps the organisation engaged and the feedback flowing. Each improvement is witnessed. Momentum compounds. The project becomes a living thing people are invested in, rather than a line item they're waiting to see justified. Speed-to-something-real is a survival trait.

Habit	Projects that survive	Projects that get cancelled
Scope	One narrow, countable task	"Transform the business with AI"
Success metric	A specific number, agreed up front	"Improve efficiency" — unmeasurable
Autonomy	Human in the loop on edge cases	Full autonomy, then lost trust
Delivery	Ship in weeks, improve in public	Long private build, stale at launch
Ownership	One named, accountable owner	"The committee" / nobody

5. They Have One Named, Accountable Owner

Every AI project that survives has a single person whose job it is to make it succeed — someone with the authority to make decisions and the accountability when it goes sideways. Cancelled projects almost always have the opposite: ownership smeared across a committee, where everyone is consulted and no one is responsible, and the project drifts until the budget review ends it.

This is the human counterpart to the technical "named accountability" that good AI governance demands. A real owner makes the unglamorous calls — cutting scope, killing a feature, pushing back on a stakeholder — that keep a project alive. Without one, every hard decision becomes a meeting, every meeting becomes a delay, and delay is how AI projects quietly die. When we deliver for clients, we insist on a counterpart owner on their side for exactly this reason: shared accountability is what gets things shipped.

From our delivery experience:

when a client brings us a project that's already struggling, the fix is almost never a better model. It's narrowing the scope, agreeing a real success metric, designing the human back into the loop, getting a working version live fast, and naming one accountable owner. Five unglamorous moves — and they rescue more projects than any technical upgrade ever has.

The Survival Checklist

Before you greenlight an AI project — or to diagnose one that's wobbling — run it against these. Every box you can honestly tick moves you toward the surviving 60%.

The first deliverable is a single, countable task you can describe in one sentence
There's a specific number that defines success, agreed in writing before the build
A human is designed into the loop for edge cases and high-stakes decisions
A real, usable version will be live in weeks — not perfected in private for months
One named person owns the outcome and has authority to make the hard calls
You can already picture the demo where you declare the first win

The 60% Isn't Lucky. It's Disciplined.

It's comforting to imagine the AI projects that succeed had some technical edge — a better model, more data, smarter engineers. The reality is more useful than that, because it's something you can choose. The projects that don't get cancelled win on discipline: narrow scope, a hard metric, a human in the loop, fast visible delivery, and clear ownership. Every one of those is a decision available to you on day one, regardless of your budget or your tech stack.

That's genuinely good news. It means surviving the 40% cull isn't about being the most advanced — it's about being the most deliberate. If you want a partner who runs AI projects this way by default — scoped tight, measured honestly, governed properly, shipped fast — that's exactly how we work. Tell us about your project, or see how we structure AI development so it lands in the 60%.

What the First 30 Days Look Like

The five habits sound simple in the abstract, so here's how they show up in the part of a project that actually decides its fate: the first month. Cancelled projects almost always lose the plot in these early weeks — they spend them gathering requirements forever, or jumping straight into a model with no agreed target. Surviving projects use the first 30 days to lock in the conditions for success.

In week one, the goal isn't to build anything — it's to choose the one narrow task and write down the number that will define success. This is the most important meeting of the entire project, and it's the one most teams skip. If you leave week one without a single countable deliverable and a metric attached to it, you're already drifting.

In weeks two and three, the work is to get a thin, end-to-end version of that one task working — not polished, but real, touching real data, with the human checkpoint already in place. A rough version that runs the whole loop teaches you more than a beautiful version of half the loop. It surfaces the integration problems, the data issues, and the edge cases while they're still cheap to fix.

By the end of week four, the surviving project has something it can show: the narrow task working on real inputs, the metric being measured, and a clear list of what to improve next. That demo is what renews the organisation's faith and unlocks the next phase. The cancelled project, by contrast, is usually still in a requirements document at week four — and the clock on leadership's patience is already running.

The 30-day test:

if, one month in, you can't put a working version of one real task in front of a stakeholder with a number next to it, the project isn't behind — it's miscalibrated. Stop, narrow the scope, and reset. That correction is far cheaper now than the cancellation that's otherwise coming.

Frequently Asked Questions

CTA Box

Want Your AI Project in the 60% That Survives?

We run AI projects scoped tight, measured honestly, governed properly, and shipped fast — the five habits that keep them alive. If you're starting an initiative or rescuing one that's wobbling, tell us where it stands. Named team, written estimates, CMMI Level 5.

→ Book a Free 20-Min Call

AI Development Services

Generative AI

Machine Learning

View Portfolio

Author Card

Written by

Rishabh Jain

AI Consultant & Founder,

Shanti Infosoft LLP

700+ Projects Delivered

Google Cloud AI Certified

AWS ML Certified

4.9★ on Clutch

38,000+ hrs on Upwork

CMMI Level 5

Contact

The AI Bill You Didn't Budget For (Tokens and Upkeep)

Sonal Jain — Sun, 28 Jun 2026 05:30:09 +0000

The true cost of an AI project is dominated by what comes after the model works: token usage at real volume, the eval and monitoring you have to keep running, integration upkeep, and maintenance. Most organizations misjudge AI costs, and a meaningful share underestimate badly. The model fee is often a small fraction of the real total.

I sit in the budget conversations, so this is the part I push hardest on. Here's where the money actually goes and how I plan for it without scaring the client off a project that's genuinely worth doing.

The model is the cheap part

The number everyone fixates on, the per-call API price, is falling. The bill is still rising, because volume is climbing faster than price drops. Token consumption across the industry has grown many times over in barely a year, driven by agents and long context windows. An agent that loops, retries, and carries a fat context can quietly cost many times a single clean call.

The cautionary tales are real: a single team handing an AI coding tool to thousands of engineers and burning its entire annual AI budget in a few months; usage that ran far past what anyone modeled because nobody priced the agent doing more than the happy path. These aren't exotic failures. They're what happens when nobody models usage at production volume.

So when I budget, I don't price the demo's token cost. I estimate cost per request at realistic complexity, multiply by realistic volume, and add headroom for the agent doing more work than the happy path suggests. Then I make that a line the client can see, because a token bill that arrives as a surprise is a trust problem, not just a finance one.

The costs that show up after launch

Total cost of ownership on AI is shaped less by the build and more by the operational lifecycle that follows. The items that catch teams out:

Evals you keep running. The harness that defines "done" isn't a one-time build. Models change, data drifts, and you re-run it to catch regressions.
Monitoring and observability. You need to see what the agent is doing in production, what it's costing, and where it's going wrong. That tooling and the attention to watch it are real budget.
Integration upkeep. Every system you connected to will change its API, its data, or its auth. Prompt and connector drift is a maintenance line, not a one-off.
Maintenance reserve. A sensible rule of thumb is to hold back 15–25% of build cost per year for upkeep, and to add a meaningful buffer to vendor quotes for the integration surprises that always surface.

None of this is visible in a prototype. All of it lands once the thing is live, which is exactly the pattern we describe when we argue you have to engineer for ownership, not just cheap creation, in The build-vs-buy line just moved.

How I present it to a client

I split the budget into two clear buckets and never blur them: build (get it working and proven against the evals) and run (tokens, monitoring, eval upkeep, maintenance, per month). Most cost surprises come from showing a client only the build number and letting them assume that's the whole cost.

A simple framing that lands:

"Here's what it costs to build and prove it works."
"Here's the monthly run cost at your expected volume, including a buffer for the agent doing more than the happy path."
"Here's the annual reserve for keeping it healthy as systems around it change."

Three honest numbers beat one optimistic one. A client can plan around the truth. They can't plan around a build figure that silently excludes everything that happens after launch.

Key takeaways

Most organizations misjudge AI cost. The model fee is a fraction of true total cost of ownership.
Token prices are falling but bills are rising, because volume and agent looping outpace the price drops. Budget at production volume, not demo volume.
The big costs arrive after launch: eval upkeep, monitoring, integration drift, and maintenance.
Hold a yearly maintenance reserve (15–25% of build) and buffer vendor quotes for integration surprises.
Present build and run as two separate, honest numbers. One optimistic figure is how trust breaks.

FAQ

Why is my AI bill rising if token prices are dropping? Because you're using far more tokens. Agents retry and loop, context windows grow, and usage scales with adoption. Volume beats the per-token discount.

What's a reasonable maintenance budget for an AI system? A common guideline is 15–25% of build cost annually, plus a buffer on integration work. The exact figure depends on how many systems it touches and how fast they change.

How do I avoid a runaway token bill? Model usage at real volume before launch, set cost monitoring and alerts from day one, and cap agent retries and context where you can. Visibility early is what prevents the surprise.

If you're budgeting an AI project and the only number on the page is the build cost, that's the gap that hurts six months in. The team at Shanti Infosoft can help you build an honest build-plus-run budget before you commit.

Agent Washing: Only ~130 of the Thousands of 'AI Agent' Vendors Are Real.

Sonal Jain — Sat, 27 Jun 2026 07:00:02 +0000

Gartner estimates that out of the thousands of vendors marketing "AI agents," only around 130 are the real thing. The rest are something else wearing a new label - a chatbot with a fresh coat of paint, a rules engine rebranded, last year's automation tool with "agentic" stapled to the datasheet. The industry has a name for this now: agent washing. And if you are shopping for an AI agent in 2026, it means the odds are overwhelmingly that the thing being pitched to you is not what the pitch says it is.

That sounds grim, but it is actually empowering - because the gap between a real agent and a re-skinned one is not subtle once you know what to look for. You do not need to be technical to catch it. You need five questions and the nerve to insist on straight answers. This article hands you both.

The stakes are simple. Buy a washed agent and you pay agent prices for chatbot capability, then spend months discovering it cannot actually do the job. Catch the washing in the sales call and you save the budget, the time, and the credibility you would have burned defending the choice internally.

What "Agent Washing" Actually Is

Agent washing is the practice of relabelling existing technology as an "AI agent" to ride the hype, without the capabilities the term implies. It is the 2026 version of every bandwagon rebrand that came before it - and it works because "agent" has no agreed, enforced definition in the market, so anyone can claim it.

To catch it, you need a working definition of the real thing. A genuine AI agent does three things a chatbot or a fixed automation does not:

It pursues a goal over multiple steps - it plans and sequences actions toward an outcome, rather than answering one prompt at a time or following a single hard-coded script.
It takes real action in your systems - it reads from and writes to your tools to actually complete the work, not just suggest what you should do.
It adapts - when a step fails or the situation shifts, it adjusts its approach instead of breaking, within the guardrails you set.

Most washed "agents" fail at least one of these. A re-skinned chatbot answers but does not act. A rebranded automation acts but cannot adapt - it shatters the moment reality deviates from its script. A "co-pilot" relabelled as an agent only ever suggests and hands the work back to you. None are bad tools. They are just not agents, and they should not carry agent pricing or agent expectations.

The core test in one line:

a real agent

finishes a multi-step job by acting in your systems and adapting when things change.

If the thing being sold only answers, only follows a fixed script, or only suggests - it is washed.

The 5-Question Test: Real Agent or Re-Skin?

Take these into your next vendor call. Each targets a specific way washing hides. Vague, deflecting, or demo-only answers are the tell - a real agent vendor answers all five concretely and without flinching.

Question 1: "Show me it completing a multi-step task end to end, on real data."

Not a scripted demo on a clean sandbox - a genuine task with messy, realistic inputs. Watch whether it actually finishes the job or just produces a suggestion you would still have to execute. Washing tell: the demo only ever shows the happy path, or the "agent" stops at recommending and a human does the doing.

Question 2: "What exactly can it do in my systems - read, write, or just talk?"

A real agent integrates and acts: it logs into tools, updates records, sends, triggers workflows. Push for the specific actions and the specific systems. Washing tell: it "integrates" but on inspection only reads or displays information, or "takes action" only by drafting something for you to send yourself. Answering is not acting.

Question 3: "What happens when something goes wrong or unexpected mid-task?"

This is the question that exposes rebranded rules engines fastest. A real agent adapts, retries, or escalates intelligently. Washing tell: the honest answer is some version of "it follows the configured flow" - meaning it is a fixed script that breaks on anything off-path. Ask for a concrete example of it recovering from a surprise.

Question 4: "Where does a human stay in the loop, and how do I control what it does automatically?"

Counterintuitively, a good answer here describes guardrails, approval gates, and an audit log - because real agents take real actions and serious vendors design the controls in. Washing tell: either there are no controls to speak of (it cannot actually act, so none are needed), or the vendor cannot explain how you would gate, pause, or audit it. Both are red flags pointing in opposite directions.

Question 5: "How is this priced, and what is the cost per task at my volume?"

Real agents call AI models on every step, so their cost scales with usage and a credible vendor can model your cost per completed task. Washing tell: flat per-seat pricing identical to ordinary SaaS often signals there is no real model-driven agent underneath - just software with an agent label. Ask what happens to the bill when volume doubles; the answer reveals the architecture.

The pattern across all five:

real agents

act, adapt, and need governance

and a serious vendor speaks fluently about all three. Washed agents answer, follow scripts, and suggest - and their vendors deflect to demos, vague "AI-powered" language, and roadmap promises when you press on specifics.

The Quick Buyer's Checklist

Before you sign anything that calls itself an AI agent, confirm you can answer yes to these. Every "no" is a place agent washing hides.

☐ You have seen it complete a real, multi-step task end to end on messy data - not a scripted happy-path demo.
☐ You can name the specific actions it takes in your specific systems - and confirmed they include writing, not just reading or suggesting.
☐ The vendor gave a concrete example of it adapting to a failure or surprise mid-task.
☐ There are real controls - approval gates, guardrails, an audit log, and a kill switch.
☐ You understand the cost per completed task at your volume, and what happens when usage scales.
☐ You spoke to a reference customer running it in production for a use case like yours - and asked them what broke.
☐ You own your data, prompts, and any resulting configuration - with no lock-in that makes leaving impossible.

Real Agent, Wrong Fit Is Still a No

One nuance worth holding: passing the test means the technology is genuinely an agent - not that it is right for you. A real agent pointed at the wrong workflow, or one that needs human judgment it cannot supply, will still disappoint. The five questions filter out washing; your own use case has to filter out mismatch. The best vendors help with both - they will tell you when an agent is overkill for your problem and a simpler automation would serve you better. That honesty is itself a strong signal you are dealing with the real 130, not the washed thousands.

What This Means For You

"Only ~130 are real" is not a reason to avoid AI agents - it is a reason to shop like a professional. The capability is genuine and valuable when you find it. The market is just crowded with imitations because the label is free and the hype is loud. Your defence costs nothing: five pointed questions, a checklist, and the discipline to treat a demo as marketing rather than proof.

Ask the questions. Insist on real tasks, real systems, real failure-handling, real controls, and honest pricing. The vendors who answer cleanly are the ones worth your budget. The ones who deflect just told you everything you needed to know - and saved you from finding out the expensive way.

Not Sure If What You're Being Sold Is a Real Agent?

Bring us the vendor pitch, the demo, or the workflow you want to automate. We will run it through the test above, tell you honestly whether it is a real agent or a re-skin, and whether an agent is even the right tool for your problem - with a fixed written estimate if you want us to build the real thing instead.

Get an Honest Second Opinion

See How We Build Real Agents

About Shanti Infosoft

Shanti Infosoft LLP is a CMMI Level 5 software engineering company that builds genuine AI agents and integrations - software that pursues a goal, acts in your systems, and adapts, with the guardrails and audit trails to govern it. We will also tell you honestly when an agent is the wrong tool and a simpler custom build serves you better. You get a named senior team, fixed-scope written estimates, and full source-code and IP ownership - no washing, no lock-in.

Frequently Asked Questions

What is "agent washing"?

Agent washing is relabelling existing technology - a chatbot, a rules-based automation, a suggestion-only co-pilot - as an "AI agent" to ride the hype, without the capabilities the term implies. Gartner estimates only around 130 of the thousands of vendors marketing AI agents are offering the real thing, which is why a buyer's odds of being pitched a re-skin are high.

What makes something a real AI agent versus a chatbot?

A real agent pursues a goal over multiple steps, takes real action in your systems (reading and writing, not just answering or suggesting), and adapts when a step fails or the situation changes. A chatbot answers one prompt at a time, a rebranded automation follows a fixed script and breaks off-path, and a co-pilot only suggests. Genuine agents act, adapt, and need governance.

How do I tell if a vendor is agent washing?

Ask five things: show it completing a real multi-step task on messy data; name the exact actions it takes in your systems; explain how it handles something unexpected mid-task; describe the human controls, approval gates, and audit log; and give the cost per completed task at your volume. Concrete answers indicate a real agent; deflection to demos, vague "AI-powered" language, or flat SaaS pricing indicates washing.

Why does pricing reveal whether an agent is real?

Real agents call AI models on every step, so their cost scales with usage and a credible vendor can model your cost per completed task and explain what happens when volume doubles. Flat per-seat pricing identical to ordinary software often signals there is no model-driven agent underneath - just relabelled SaaS.

If an agent passes the test, is it right for my business?

Not necessarily. Passing means it is genuinely an agent, not that it fits your workflow. A real agent aimed at a task that needs human judgment it cannot supply will still disappoint. Filter out washing with the five questions, then filter out mismatch with your own use case - and favour vendors honest enough to tell you when a simpler automation would serve you better.

Scope Creep on AI Projects: How a PM Holds the Line

Sonal Jain — Sat, 27 Jun 2026 05:30:03 +0000

Scope creep is the slow expansion of a project's work beyond what was agreed, and on AI builds it's harder to spot because "can it also just handle this?" sounds tiny and is rarely tiny. The way you hold the line is not by saying no, but by making the cost of each addition visible at the moment it's requested, so the client chooses with eyes open.

Scope creep lands on more than half of all projects. On AI work I'd put it higher, and the reason is specific to how these systems behave. Let me explain why, and what I actually do about it.

Why AI invites scope creep

A traditional feature has visible edges. "Add a report" is clearly more work than the dashboard you scoped. AI blurs those edges. The agent already produces fluent, plausible output, so every new request feels like a small nudge rather than new scope. "It already answers billing questions, can it also handle refunds?" Refunds means a new action, new permissions, new failure modes, and a whole new set of eval cases. But it doesn't feel that way, because the surface looks the same.

There's a second trap. Because AI is probabilistic, "make it a bit more accurate" can be an unbounded request. Going from 80% to 90% task completion might be an afternoon or might be a month, and the client genuinely can't tell. If I don't surface that, I've accepted unbounded work disguised as a tweak.

Hold the line at the moment of the ask

The mistake is letting small additions through and reconciling at the end. By then the schedule is gone and the conversation is adversarial. I handle every request the same way, the moment it lands:

Name what it actually touches. Refunds isn't "one more intent." It's an action, a permission scope, a new failure mode, and new eval cases. I say that out loud.
Price it in time, not just yes/no. "That's roughly three days including the evals to prove it's safe." A number reframes the request from a favor into a decision.
Make the trade explicit. "We can add refunds, or we hold the date. Adding it moves launch by about a week. Your call." Most clients make sensible trades when they can see the trade.
Write it down as a change. A one-line change note with the time impact. Not bureaucracy, just a shared record so nobody relitigates it later.

Notice I never said no. I made the cost visible and handed the decision back. That's the whole job. Clients don't resent paying for value they chose. They resent surprises, and surprises are what unmanaged scope creep manufactures.

A few habits that prevent it earlier

Define the input distribution up front. A lot of "creep" is really scope that was vague to begin with. If we agreed exactly which cases the agent handles, the new case is visibly outside the line.
Tie scope to the eval set. When "done" is a fixed set of test cases, a new request is obviously a new case, which makes the extra work concrete instead of arguable.
Keep a parking lot. Good ideas that aren't in scope go on a visible list for the next phase. People let go of a request far more easily when it's captured, not killed.
Watch the accuracy asks specifically. When someone wants "better," I make us agree the target number and the test set first. Otherwise "better" never ends.

The build-vs-buy decision has shifted partly because AI made custom work cheaper to start, which we cover in The build-vs-buy line just moved. Cheaper to start is exactly why scope discipline matters more now, not less. When building feels easy, the temptation to keep adding is constant.

Key takeaways

Scope creep hits over half of all projects and is harder to see on AI work, where new requests look deceptively small.
"Make it more accurate" can be unbounded. Pin it to a target number and a test set or it never ends.
Hold the line at the moment of the ask: name what it touches, price it in days, make the trade explicit, write it down.
You don't say no. You make the cost visible and hand the decision back.
Tie scope to the eval set and keep a parking lot so good-but-out-of-scope ideas aren't lost.

FAQ

Isn't being strict about scope bad for the client relationship? The opposite. Clients trust a PM who tells them the real cost before they commit. They lose trust when work quietly expands and the date slips with no explanation.

How do I price a change I'm unsure about? Give a range and say it's a range, then timebox an investigation if needed. "Two to five days; let me spend an hour confirming" is honest and still actionable.

What if the client insists everything is in scope? Then point back to the agreed input distribution and eval set. Scope is whatever you wrote down together. If it isn't on that list, it's a change, and that's a fact, not an opinion.

If your AI project keeps growing a little every week and the date keeps moving, the problem is usually a missing change process, not a difficult client. The team at Shanti Infosoft can help you set scope you can actually defend.

OpenAI's Co-Founder Called Today's AI Agents 'Slop.' He's Not Wrong - Here's the Nuance.

Sonal Jain — Fri, 26 Jun 2026 07:00:49 +0000

Andrej Karpathy - a founding member of OpenAI and one of the most credible voices in the field - looked at the current crop of fully autonomous AI agents and called them, bluntly, "slop." Coming from a true believer, that lands harder than any skeptic's hot take. But here is the part the headlines cut off: in the same breath, Karpathy said we are entering the "decade of agents," not the year of agents. He is not bearish on AI agents. He is bearish on the timeline the hype is selling you.

That distinction is the whole game for anyone making a decision about AI agents right now. Karpathy is not saying agents do not work. He is drawing a hard line between two things the marketing deliberately blurs: the loop-running coding agents that genuinely deliver value today, and the fully autonomous "set it and forget it" agents that are still years from being reliable. Confuse the two and you either dismiss a real tool or buy a fantasy. Get the line right and you can act with confidence.

This is the most useful kind of contrarian take: not "AI agents are overhyped, ignore them," but "here is exactly which part is real now and which part is 2030." Let's draw that line clearly.

What Karpathy Actually Said - and Why It Carries Weight

Speaking on the Dwarkesh Patel podcast, Karpathy pushed back on the idea that we are months away from autonomous agents that can be handed a goal and trusted to run a complex job end to end with no human watching. His "slop" critique was aimed precisely there - at agents marketed as fully autonomous that, in practice, lose the thread, take wrong actions, and need constant correction. His point was not that the technology is fake. It was that the gap between an impressive demo and a system you can actually trust unsupervised is large, and closing it is the work of a decade, not a launch cycle.

What makes this worth more than a random skeptic's opinion is the source. Karpathy is long on AI by any measure - he has spent his career building it and believes deeply in where it is going. When someone who wants agents to succeed tells you the autonomous version is not ready, that is signal, not noise. He is doing the buyer a favour: separating the genuine, usable progress from the marketing that overstates its maturity.

The honest summary:

"Decade of agents, not year" is not a put-down. It is a calendar. The capability is real and compounding; the fully autonomous, no-human-needed version is just further out than the hype implies. Plan to the calendar, not the demo.

What Is Real Right Now: Agents-in-the-Loop

Strip away the autonomy fantasy and there is a category of AI agent delivering serious value today - and Karpathy is bullish on it. The pattern is the human-in-the-loop coding and workflow agent: software that runs a multi-step loop toward a goal, but with a person steering, reviewing, and approving the consequential moves. It works because the human supplies exactly what the autonomous version lacks - judgment at the moments that matter.

Where this category already earns its keep:

Coding assistance that runs a loop. An agent that reads a task, writes code, runs the tests, sees the failures, and iterates - with the developer reviewing and approving. This is real, in production, and saving experienced teams meaningful time daily.
Research and synthesis. Agents that pull from many sources, cross-check, and assemble a draft a human then verifies and refines. The agent does the legwork; the person owns the conclusion.
Bounded operational workflows. Lead research and first-touch drafting, ticket triage and routing, data pulls into a weekly brief - high-volume, rule-bounded work where the agent drafts and acts on low-stakes steps and a human gates anything that writes, sends, or spends.

The common thread is not that these agents are dumber. It is that they operate inside a frame a human controls. They are trusted with the steps where a mistake is cheap and reviewed on the steps where it is not. That is not a limitation to apologise for - it is the design that makes them safe to deploy now.

A concrete example of in-loop value

Consider a sales team drowning in inbound leads. A human-in-the-loop agent watches the form submissions, researches each company, drafts a tailored first reply, and queues it - then a rep skims the queue and approves with one click, or edits the rare one that needs a human touch. The agent has not replaced the salesperson; it has eliminated the two hours a day they spent on research and typing, and it never lets an after-hours lead go cold. Crucially, the consequential moment - actually sending - still passes through a person. That single design choice is the difference between a tool a team trusts and an autonomous system that emails the wrong segment at 2 AM. The value is real, it is available today, and it is safe precisely because the human stayed in the loop.

This is why the in-loop pattern is not a temporary crutch. Even as models improve, the highest-leverage design for consequential work will keep a human at the decision points that carry real risk, while the agent absorbs the volume around them. Karpathy's optimism about the decade is, in large part, optimism about how far this collaborative pattern can be pushed - not a promise that the human disappears.

What Is Still 2030: Fully Autonomous Everything

The version Karpathy calls slop - and the version most likely to waste your budget - is the agent sold as needing no human at all: hand it a complex, open-ended objective, walk away, and trust the outcome. Several hard problems stand between today and that promise, and none of them are close to solved.

Reliability over long horizons. An agent that is 97% reliable on any single step is wildly unreliable over a fifty-step task - small errors compound into a wrong final result. Long, unsupervised chains are exactly where today's agents lose the plot.
Knowing when it is wrong. Autonomy requires an agent to recognise its own uncertainty and stop. Current systems are confidently wrong as readily as confidently right, which is fatal when no human is watching.
Recovering from the unexpected. Real environments throw curveballs. Humans improvise; agents tend to barrel ahead on a broken plan. Graceful recovery from surprise is still largely missing.
Trustworthy action on live systems. An autonomous agent with write access to your real tools is one bad inference away from real damage. The permissions, guardrails, and audit trails that make that safe are still being figured out.

The buyer's tell:

any vendor promising a fully autonomous agent that needs no human oversight for a consequential workflow is selling you the 2030 version on a 2026 invoice. The honest pitch describes where the human stays in the loop - and why that is a feature, not a shortcoming.

How to Act on This: Buy the Real Thing, Skip the Fantasy

The decade-not-year framing is genuinely freeing, because it tells you exactly how to behave. You do not have to wait, and you do not have to overcommit. You meet the technology where it actually is.

Adopt human-in-the-loop agents now. For coding, research, and bounded operational workflows, the value is here today. Deploy it, measure it, and capture the gains while your competitors argue about autonomy.
Design for the human, not around them. Build the review gates, approvals, and audit logs in from the start. This is not a stopgap until autonomy arrives - it is good design that also happens to be the only safe design right now.
Treat "fully autonomous" claims as a red flag. When a pitch leans on no-human-needed autonomy for anything that matters, ask precisely where a human gates the consequential actions. A non-answer tells you what you need to know.
Climb the ladder as trust earns it. Start at assist, move to supervised, and loosen the leash only when your own logs prove the agent is reliable on a given workflow. Let evidence, not marketing, set the pace.
Position for the decade. The capability is compounding. The teams that build the data, the guardrails, and the in-loop muscle now are the ones ready to safely widen autonomy as it genuinely arrives - rather than starting from zero in 2030.

What This Means For You

Karpathy gave the market a rare gift: a credible insider telling the truth about timeline. The right response is not cynicism about AI agents - it is precision. There is a real, valuable, human-in-the-loop category you should be using today, and there is a fully autonomous fantasy you should refuse to pay 2026 money for. The skill is telling them apart, and now you can.

Build for the decade by capturing the value that is already here, designed honestly around the human in the loop. That is how you get the upside of AI agents without becoming a cautionary tale - and how you are ready to extend autonomy the moment it is genuinely earned.

Build the Agent That Works Today - Not the One That Ships in 2030

We build human-in-the-loop AI agents for the workflows where they already deliver - with the review gates, guardrails, and audit logs that make them safe to run now. Bring us a workflow and we will tell you honestly what an agent can take over today, where a human stays in the loop, and what it costs - fixed written estimate, no hype.

Book a Free 20-Min Call

See How We Build Agents

About Shanti Infosoft

Shanti Infosoft LLP is a CMMI Level 5 software engineering company building production-grade AI agents and integrations - not demo-ware. We build the category that works today: human-in-the-loop coding, research, and operational agents, with review gates, guardrails, and full audit trails designed in. We will tell you plainly where AI already wins, where a human still needs to gate the call, and what it will realistically cost - with a named senior team, fixed written estimates, and full IP ownership handed to you.

Frequently Asked Questions

Did Karpathy say AI agents do not work?

No. He called today's fully autonomous agents "slop," but in the same conversation said we are in the "decade of agents." His critique targets the timeline and the no-human-needed autonomy being marketed - not the technology itself. He is bullish on AI agents long-term and on the human-in-the-loop versions that work today.

What is the difference between an agent that works now and one that does not?

The reliable ones keep a human in the loop: they run a multi-step loop toward a goal but a person reviews and approves consequential actions. The unreliable ones are sold as fully autonomous - hand them an open-ended goal and walk away. Over long, unsupervised chains today's agents compound small errors and act confidently when wrong, which is why the in-loop design is what works.

What does "decade of agents, not year" mean for my business?

It means the capability is real and compounding, but the fully autonomous version is years out, so you should adopt the human-in-the-loop agents that deliver today rather than wait for or overpay for autonomy. Build the data, guardrails, and review muscle now, and widen autonomy as evidence - not marketing - earns it.

How do I spot a vendor overselling autonomy?

Ask exactly where a human gates the consequential actions - anything that writes, sends, or spends. A vendor selling genuine value will describe the in-loop design as a feature; one overselling the 2030 fantasy will dodge the question or promise no human is needed for a workflow that clearly carries real risk.

Should I wait for fully autonomous agents before investing?

No. Waiting forfeits the value that is already here and leaves you starting from zero when autonomy matures. Adopt human-in-the-loop agents now for coding, research, and bounded workflows, design honestly around the human, and climb the autonomy ladder as your own logs prove reliability. That captures today's upside and positions you for the decade.

What 'Done' Means for an AI Feature (Write It Down)

Sonal Jain — Fri, 26 Jun 2026 05:30:48 +0000

For an AI feature, "done" means it passes a written set of acceptance tests on real inputs at an agreed success rate, not that it produced a good answer once in a demo. If you can't state the metric and the threshold, the feature isn't defined, and you'll argue about it at sign-off instead of agreeing on it up front.

This is the conversation I have at the start of every AI engagement, because it's the one that prevents the worst conversation at the end of one. Here's how I run it.

Traditional "done" breaks on AI

Normal software has a clean contract: input X returns output Y. You write a test, it passes or fails, everyone agrees on the state. Generative AI breaks that. The same input can return different output across runs. So "it worked when I tried it" is no longer evidence the feature is finished. It's evidence it can work, on that input, that time.

That's why I don't accept "the demo passed" as a definition of done. A demo is one sample from a probabilistic system. I need to know how the feature behaves across the real input distribution, and I need that agreed before we build, because acceptance criteria are how the client and the team agree on what "done" actually means.

Define done as a measurable target, not a vibe

For an AI feature, I write acceptance criteria as numbers tied to a test set:

Task completion rate on a fixed set of realistic cases. "The agent resolves the request correctly in at least N% of the golden test cases."
Tool/argument correctness for agents that take actions. Did it call the right function with the right inputs, not just produce nice text?
Groundedness for anything retrieval-based. Is the answer supported by the source, or a confident fabrication?
Refusal quality. When it shouldn't answer, does it decline cleanly instead of guessing?
Latency and cost per request, because a correct answer that takes nine seconds or burns a fortune in tokens may still fail the real requirement.

The exact metrics depend on the feature. The discipline doesn't: every criterion is a number against a named test set, with a threshold the client signed off on.

The eval harness is a deliverable, not overhead

Here's the part teams skip and then regret. You can't measure any of that without an eval harness, a repeatable way to run the feature against your golden cases and grade the output. So I scope it as a real deliverable with its own hours: pick a handful of key tasks, define success, build golden examples, and grade them, starting with simple deterministic checks before anything fancier.

A practical sequence I use:

Weeks 1–2: agree the key tasks and what success means for each. Collect real examples into a golden set.
Weeks 3–4: build the harness, add basic metrics (success rate, steps, latency, cost), and establish a baseline.
From there: every change runs against the harness, so "did this improvement break something else?" is a number, not an opinion.

This is the same instinct as defining production requirements before you prototype, which we make the case for in The AI demo works. That's the problem. The eval set is how you make "production-ready" measurable instead of a feeling.

Why this protects everyone

When "done" is a written, measured target, sign-off stops being a negotiation. The client knows exactly what they're accepting. The team knows exactly when they're finished. And nobody is stuck in the bad meeting where a stakeholder says "but it gave me a wrong answer yesterday" and the team says "it works on our machine." Both can be true for a non-deterministic system. The eval is what turns that standoff into a number you agreed on weeks earlier.

Key takeaways

"It worked in the demo" is not a definition of done for AI. One sample from a probabilistic system proves capability, not completion.
Write acceptance criteria as metrics against a fixed test set: completion rate, tool correctness, groundedness, refusal quality, latency, cost.
The eval harness is a real deliverable. Scope it with hours, not as an afterthought.
Agree the thresholds before building, so sign-off is confirmation, not a fight.
Start grading with simple deterministic checks before reaching for complex graders.

FAQ

How do you test something that gives different answers each time? You stop testing single outputs and start measuring rates across a fixed set of real cases. Done becomes "passes the agreed threshold on the golden set," not "gave a good answer once."

What's a golden set? A curated collection of real, representative inputs with known good outcomes. You run every version of the feature against it so you can see, in numbers, whether you're getting better or worse.

Isn't building evals expensive? It costs hours up front and saves far more in disputed sign-offs, regressions, and rework. It's the cheapest insurance on an AI project.

If your team is about to build an AI feature and "done" still means "the demo looked good," that's worth fixing before a line of code ships. The team at Shanti Infosoft can help you define measurable acceptance criteria and the eval harness to back them.

Writing Code Was Never the Bottleneck. Governing It Is.

Sonal Jain — Thu, 25 Jun 2026 07:00:40 +0000

Here is a paradox worth sitting with. In a 2026 survey of software leaders by CloudBees, 92% said they trust the code AI produces - and in the same breath, 81% reported that AI-generated code has caused more security or operational incidents, not fewer. Trust went up. Incidents went up too. Both numbers, from the same room of people.

That is not a contradiction to explain away. It is the single most important shift in software delivery right now, stated in two figures. Writing code - the thing we spent decades optimising, hiring for, and treating as the constraint - is no longer the bottleneck. The bottleneck moved. It is now governing the code: reviewing it, verifying it, and deciding whether the confident-looking thing the AI just produced is actually safe to ship.

If you run a product, a team, or a vendor relationship, this reframe changes where you should be spending money and attention. Most organisations are still optimising the part that is now cheap and under-investing in the part that is now expensive. This article is about flipping that.

The New Bottleneck: Code Got Cheap, Review Got Expensive

For most of software's history, the cost of a feature was dominated by the cost of writing it. Hiring was about who could produce more good code. Process was about removing friction from writing. Review existed, but it was a checkpoint on a relatively small, human-paced flow of changes.

AI broke that flow open. Code is now abundant and fast to produce. A developer with a good assistant can generate in a morning what used to take days. But every one of those changes still has to be understood, verified, and trusted before it touches production - and a human reviewing has not gotten faster. So the queue backed up somewhere new.

The 81% figure is what that backed-up queue produces. When generation outpaces review, things slip through: an authorization check that was never added, a dependency with a known flaw, a subtle logic error wrapped in confident, tidy syntax. The incidents are not because AI writes uniquely bad code. They are because more code is reaching production with less human understanding behind it than ever before.

The one-line version:

AI made writing code 10x cheaper and left reviewing it exactly as expensive. The constraint did not disappear - it moved downstream, to the one place most teams have not reinforced.

Why "We Trust It" and "It Causes Incidents" Are Both True

The trust-and-incidents paradox is not irrational. It is the predictable result of how AI-generated code feels versus how it behaves.

Trust is earned by fluency, not correctness

AI-generated code is articulate. It is well-structured, consistently formatted, and reads like something a competent engineer wrote on a good day. Humans equate fluency with competence - we always have - so the code earns trust by looking right. The problem is that insecure code and secure code are equally fluent. The confidence the output projects is not evidence about its correctness; it is a property of the writing style.

Incidents are caused by the gaps fluency hides

Meanwhile, the actual failures live in what is absent or subtly wrong - the missing permission check, the edge case not handled, the assumption that holds in the demo and breaks under real data. These are exactly the things a quick, trusting read glides over, precisely because the surrounding code looks so reassuring. High trust lowers scrutiny; lowered scrutiny lets the gaps through; the gaps become incidents. Both numbers are true because they describe two different things: how the code looks and what it does.

The trap for leaders:

the more impressive your AI tooling, the more your team will trust its output, and the

deliberate your governance has to be to compensate. Capability and required oversight rise together. Treating better tools as a reason to review less is exactly backwards.

What "Governing Code" Actually Means

"Governance" sounds like bureaucracy. It is not. In a world of abundant AI-generated code, governance is simply the set of cheap, mostly-automated gates that let you ship fast and safely - the discipline that converts a flood of code into trustworthy releases. It has five practical layers.

Human review that scales. Not reading every line - that no longer fits the volume - but focused human attention on the parts that carry real risk: anything touching authorization, money, data exposure, or irreversible actions. The skill shifts from writing code to judging it.
Automated guardrails on every change. Static analysis, security scanning, dependency checks, and policy rules that run on every commit and fail the build when something dangerous appears - so the machine catches the mechanical mistakes before a human ever has to.
Tests as the contract. A test suite that encodes what the code must and must not do, so AI-generated changes are checked against intent automatically rather than trusted on sight. Tests become the thing you trust; the code is just an implementation that has to pass them.
Provenance and traceability. Knowing what was AI-generated, what was reviewed, by whom, and why - so when an incident happens you can trace it, learn from it, and tighten the gate, instead of guessing.
Clear ownership. Every change has a human who is accountable for it shipping. "The AI wrote it" is not an owner. Accountability is what keeps the other four layers from quietly eroding.

Notice that four of these five are largely automated and run in seconds. Governance done well is not slow. It is the thing that lets you go fast, because it turns "we hope this is fine" into "the gates passed, so we know the obvious failure modes are covered."

The Investment Has to Move

If code is cheap and review is the constraint, your spending should reflect it - and in most organisations it does not yet. The dollars and the headcount are still aimed at producing more code, when the leverage has moved to verifying it.

Hire and train for judgment, not just output. The valuable engineer is increasingly the one who can quickly assess whether a change is correct and safe, architect the guardrails, and own the call - not only the one who can produce the most lines.
Fund the pipeline, not just the people. Automated review, security scanning, and a real test suite are now core infrastructure, not nice-to-haves. They are where you get caught failures cheaply.
Measure incidents and review throughput, not velocity alone. "How much code did we ship" is a vanity metric in an age of abundant code. "What reached production with real human understanding behind it, and what did it cost us in incidents" is the number that matters.

A 6-Point Governance Self-Audit

Run your team - or a vendor you are evaluating - through these. The 81% who report more incidents are overwhelmingly the ones answering "no" to most of them.

☐ Does every change get focused human review on the high-risk parts (auth, money, data, irreversible actions)?
☐ Do automated security and quality gates run on every commit and block the build on failure?
☐ Is there a real test suite that encodes intent, so AI-generated changes are checked against it automatically?
☐ Can you tell what was AI-generated and trace any change back to a reviewer?
☐ Does every change have a named human owner accountable for it shipping?
☐ Are you measuring incidents and review capacity - not just how fast you ship?

What This Means For You

The CloudBees numbers are a gift, because they name the shift before it costs you a major incident. Trust in AI code is not the problem - earned trust, backed by gates, is exactly the goal. Blind trust, backed by nothing, is what turns 92% confidence into 81% more incidents.

If you are buying software, ask your vendor not "do you use AI?" but "how do you govern what the AI produces?" If you run a team, stop optimising the part that is now cheap and start reinforcing the part that is now the constraint. Code abundance is here to stay. The winners will be the ones who treated governance scarcity as the real problem - and built the cheap, automated, human-anchored gates that let them ship fast without shipping incidents.

Govern Your AI-Generated Code Before It Governs Your Incident Log

We build with AI and govern it on purpose - focused human review on what matters, automated security and quality gates on every change, and tests that encode intent. Show us your delivery pipeline and we will tell you where the governance gaps are and give you a fixed written estimate to close them.

Talk to Our Engineering Team

How We Build & Review

About Shanti Infosoft

Shanti Infosoft LLP is a CMMI Level 5 software engineering company that builds custom web and app products, AI integration, and custom software with governance built in. We use AI to make writing code fast, and we put real review where it now matters: focused human QA on high-risk changes, automated security and quality gates on every commit, and a test suite that encodes intent. You get a named senior team, fixed-scope written estimates, and full source-code and IP ownership - code you can trust because it was governed, not just generated.

Frequently Asked Questions

How can 92% trust AI code while 81% report more incidents from it?

Because trust and incidents measure different things. AI-generated code is fluent and well-structured, which earns trust by looking right - but the failures live in what is missing or subtly wrong, which a confident, trusting read glides over. High trust lowers scrutiny, lowered scrutiny lets gaps through, and the gaps become incidents. Both figures, from CloudBees' 2026 survey, are true at once.

What does it mean that "governing code, not writing it" is the new bottleneck?

AI made producing code fast and abundant, but every change still has to be understood, verified, and trusted before it ships - and human review did not get faster. So the constraint moved from writing code to reviewing and governing it. That is where teams now lose time and where incidents originate.

Is governance just bureaucracy that slows us down?

No. Done well, governance is mostly automated - security scans, quality gates, and tests that run in seconds on every commit - plus focused human review only on high-risk changes. It is what lets you ship fast safely, because it converts "we hope this is fine" into "the gates passed." It speeds you up by catching failures cheaply instead of in production.

What should I ask a software vendor about their AI use?

Not "do you use AI?" but "how do you govern what it produces?" Ask whether high-risk changes get human review, whether automated security and quality gates run on every change, whether they have a real test suite, whether they can trace changes to a reviewer, and who is accountable for each release. The answers separate disciplined delivery from "the AI wrote it."

Where should we move our engineering investment?

Toward judgment and verification. Hire and train engineers who can quickly assess whether a change is correct and safe and own the call, fund your review pipeline (automated scanning and a real test suite) as core infrastructure, and measure incidents and review capacity rather than raw shipping velocity. The leverage has moved from producing code to verifying it.

Why AI Projects Slip: The Demo-to-Production Gap

Sonal Jain — Thu, 25 Jun 2026 05:31:05 +0000

Most AI projects miss their date for one reason: the demo proved the easy 20% of the problem, and everyone planned the schedule as if that was the whole job. The fix is to scope the boring 80% (data, integration, evals, edge cases) before you commit a date, and to tell the client what "done" actually requires.

I run delivery for a living. I've watched a flawless Friday demo turn into a three-month grind, and it almost always traces back to the same planning mistake. So let me walk through where the time actually goes, and how I scope around it now.

The demo is a measurement of capability, not reliability

A demo runs on inputs someone hand-picked. The data is clean, the user cooperates, the happy path is the only path. That's not cheating. It's what a demo is for. But it removes every condition that makes the real system hard, which is exactly why a working prototype tells you so little about the delivery date.

The pattern holds across the industry: a polished pilot runs six to twelve weeks, and the production deployment behind it runs six to twelve months. Most AI failures come from organizational and operational issues, not the model itself. When I read a plan that assumes the demo timeline carries through to launch, I know where the slip is coming from before the project even starts.

Where the time actually goes

When I break down a real AI build, the model work is a slice, not the cake. The hours hide in:

Integration with systems nobody documented. Legacy interfaces, auth that was never designed for machine access, a CRM with twelve years of inconsistent data. Teams routinely spend most of their build time here, on connectors, not on the agent.
Data readiness. Cleaning, structuring, and governing the data is real engineering, and it lands on the schedule whether you planned for it or not. A lot of projects stall here.
Evals. You cannot say a feature is done until you can measure that it's done. Building that measurement is a deliverable in its own right.
The unglamorous edge cases. The 20% of inputs that aren't clean are where the agent compounds small errors into a wrong answer a customer sees.

None of this shows up in a demo. All of it shows up in the burndown.

How I scope it now

I stopped estimating from the demo. I estimate from production requirements, and I define those before anyone writes a prototype. That mirrors what we argue at Shanti Infosoft in The AI demo works. That's the problem. define what production needs first, then build toward it.

My scoping checklist before I commit a date:

What's the real input distribution? Not the demo's three examples. The messy thousand.
What systems does this touch, and who owns the integration on their side? Undocumented interfaces are the single most common reason my estimates need a buffer.
How will we know it works? If we can't write the acceptance test, the feature isn't scoped, it's wished for.
What happens when the model is wrong? A human in the loop, a fallback, a confidence threshold. This is design work, and it takes time.
Who operates this after launch? Monitoring and ownership are part of "done," not a phase-two afterthought.

Then I add a buffer where the risk is highest, usually integration and data, and I make that buffer visible to the client rather than burying it. A schedule with no buffer isn't optimistic. It's wrong, and everyone finds out at the worst possible time.

Key takeaways

A demo proves capability under ideal conditions. It does not predict your delivery date.
Plan for the 80% the demo skips: integration, data readiness, evals, edge-case handling, and operations.
Pilots are weeks; production is months. Estimate from production requirements, not the prototype.
If you can't write the acceptance test, the feature isn't scoped yet.
Put the buffer where the risk lives, and show the client why it's there.

FAQ

How long does a real AI project take? It depends on integration and data complexity, but the honest framing is that a polished pilot in six weeks can still need months of production hardening. Scope the hardening explicitly instead of discovering it.

Why do AI estimates miss so often? Because they're anchored to the demo. The demo removed the hard conditions on purpose, so estimating from it skips the work that actually consumes the schedule.

What's the single best way to de-risk an AI timeline? Define what "done" means in production before you prototype, write the acceptance tests, and buffer the integration and data work.

If you're staring at a great AI demo and trying to figure out the real timeline behind it, that's a conversation I have every week. The team at Shanti Infosoft is happy to walk through your scope and give you an honest read on what production will actually take.

The demo always works. That's the problem.

Sonal Jain — Tue, 23 Jun 2026 13:51:43 +0000

Every AI project I've run has a moment that feels like a finish line and isn't. Someone in the room sees the demo work, the answer comes back clean, and the mood shifts to "great, when can we ship it." I've learned to brace for that moment, because the gap between a demo that works once and a feature people can depend on every day is where most of the real project actually lives.

The numbers back up the feeling. Depending on whose study you read, somewhere between 80% and 90% of AI pilots never reach production. The reason is rarely the model. The model is usually the part that works. What stalls projects is everything around it: the data that's messier in production than in the curated demo set, the integrations into tools nobody scoped, the question of who's accountable when an answer drifts six weeks after launch. (More on why so many stall: the things AI projects that don't get cancelled do differently.)

So when a client expects magic, my job isn't to deflate them. It's to redraw the finish line somewhere honest.

Scoping when the expectation is "it just knows"

The trickiest conversations happen early, before any code, when the client describes what they saw a competitor do or what ChatGPT did for them over the weekend. That's a real reference point and I don't dismiss it. But a weekend prompt and a production feature are different animals, and the difference is almost all the work.

What I do instead of promising the magic: I get specific about the unglamorous list. Where does the data come from, and who keeps it clean? What happens when the model is confident and wrong? Who reviews the edge cases, and how often? None of that shows up in a demo. All of it shows up in week three of production. Putting it on the table during scoping isn't pessimism. It's the only way the estimate means anything.

LLM accuracy is a range, not a number

This is the expectation I reset most often. Traditional software is deterministic. You give it the same input, you get the same output, and "done" is a thing you can point at. LLMs aren't like that. The same question can return a slightly different answer, and "correct" depends on context. Hallucination rates in real, live conditions vary wildly by domain. A general assistant might be fine. Anything touching legal or medical content carries much higher risk and needs heavy guardrails before it goes anywhere near a user.

I've stopped saying "the AI will be accurate." I say: here's the accuracy band we're targeting, here's how we'll measure it, and here's what we do with the cases that fall outside it. A human review step, a confidence threshold, a fallback. Clients are far more comfortable with a known failure mode than with a promise that has no floor. The teams that get burned are the ones who sold a number they couldn't hold.

Where the budget actually goes

The other expectation gap is cost, and it's brutal because it's invisible early. The proof of concept is cheap. It's a sliver of the budget and it makes everyone optimistic. Then production shows up with the bill nobody itemized: monitoring, a rollback plan, retraining when behavior drifts, incident response when it breaks at an awkward hour. I've seen the real first-year cost of running a thing land many times above what the POC suggested, and the surprise is what damages trust, not the number itself.

So I price the boring parts out loud, in the first estimate, even when it makes the proposal less exciting. A POC that omits the cost of keeping the thing alive isn't a smaller version of the project. It's a different project.

What "done" means now

For a normal feature, done is when it passes QA and ships. For an AI feature, I treat done as the point where it runs reliably without me standing next to it. That means it's monitored, someone owns it, there's a clear path when an answer goes sideways, and the people using it understand what it can and can't do. The demo proved the idea was possible. Done proves we can live with it.

This is the part of delivery I care most about at Shanti Infosoft, because it's the part clients feel every day after the excitement of launch wears off. The demo earns the meeting. The reliable version earns the renewal. My whole job is making sure we don't confuse the two, and that the client doesn't either.

The demo always works. Treat that as the start of the hard part, not the end of it, and the rest of the project gets a lot more honest.