Production-grade AI agents are not what most people are building. They're building demos — very good ones, in enormous numbers — and then discovering that the distance between a demo that works and an agent that ships is most of the actual engineering. Google and Kaggle just published a number that puts a size on that distance, and it deserves more attention than the marketing framing it arrived in.
Their 5-day AI Agents Intensive recap, published August 3, reports 353,000 registered participants. It also reports just over 6,000 submitted capstone projects, from 12,000+ active capstone participants. Google frames the course as taking developers "from vibe to live."
Do the arithmetic on their own funnel and the story inverts: roughly one registrant in fifty-nine finished with a project. That is not a knock on the course — free five-day cohorts always leak, and 6,000 working agent projects is a genuinely large number. It's a knock on how we talk about agent development, because the drop happens at a very specific place, and it is the same place teams stall inside companies.
TL;DR
- Production-grade means measured, not working. A demo proves your agent can succeed once. Production requires knowing the success rate, the cost, and the failure shape.
- Google's own funnel: 353,000 registered → 12,000+ active capstone participants → 6,000+ submissions. The leak isn't motivation, it's the six things a demo never has.
- Those six: an eval suite, durable state, tool contracts that fail loudly, a hard cost/step budget, per-run traces, and a security boundary that treats tool output as hostile.
- Build the eval suite first, before any refactor. Without a score, every later change to your agent is a guess dressed as progress.
- Vibe coding is the right tool for "can this work?" and the wrong tool for "how often does this work?" Switch deliberately.
What are production-grade AI agents?
Production-grade AI agents are agents whose behavior is measured rather than observed. You know the success rate across a scored set of real tasks, the cost and latency per run, and the shape of the failures — not merely that the thing worked the last time you ran it.
That definition is worth being precise about, because "production" usually gets read as a deployment question — is it on a server, does it have a URL. It isn't. You can deploy a prototype in an afternoon; Cloudflare Workers will host an agent endpoint before your coffee's cold. Deployment is the easy half.
If you can't state your agent's success rate as a number, you don't have a production agent — you have a demo that hasn't failed in front of you yet.
The hard half is epistemic. In normal software, you know what the function does because you wrote the branches. In an agent, the control flow is produced at runtime by a model, so you don't know what it does — you know what it did, on the traces you happened to look at. Production-grade is the state of having replaced that anecdote with a measurement.
Google's recap actually gestures at the right scope. Their claim is that the courses "navigated the entire lifecycle of designing, securing, and deploying production-grade AI agents in the cloud." Note that securing sits in the middle of that sentence, between design and deploy, which is exactly where it belongs and almost never is.
Why do so few agent prototypes reach production?
Here is the funnel as Google reported it, with the conversion math made explicit:
| Stage | Reported figure | Share of registrations |
|---|---|---|
| Registered for the 5-day course | 353,000 | 100% |
| Active capstone participants | 12,000+ | ~3.4% |
| Capstone projects submitted | 6,000+ | ~1.7% |
For context on the series rather than this cohort: Google says the 5 Day Intensive program has reached over 2 million learners since 2024, and reports 392,000 active participants in the Kaggle Discord — a community figure spanning the program, not a subset of this cohort's registrations, so it doesn't belong in the funnel above.
Two capstones they name — Palimpsest, a historical manuscript transcription tool, and Project ARIES, a space-weather research system — are exactly the shape of thing that works: narrow domain, verifiable output, a human who can tell right from wrong at a glance. That's not a coincidence, and it's the single most transferable lesson in the whole recap.
💡 Key insight: The agents that get finished are the ones where correctness is cheap to check. If you can't grade your agent's output quickly, you won't iterate on it, and you won't ship it.
What does a production agent have that a demo doesn't?
Six things. Every one of them is absent from a working prototype by definition, and each is where a team stalls.
1. An eval suite. Your demo is a single successful trace on a happy path you chose, possibly after several attempts you didn't count. Production needs a scored set: fifteen to thirty real task instances, each with a programmatic pass criterion, run on every change. This is the highest-leverage artifact in agent engineering and the most commonly skipped — you can't tell an improvement from a regression without it, so every subsequent change becomes a coin flip. If you're weighing how heavy a harness to build, the small-and-honest end of the eval spectrum beats the elaborate framework you abandon.
2. Durable state. Prototypes hold conversation and memory in a process. Production agents get restarted mid-task, run for hours, and resume. That means task state is a persisted record with a schema and a version, not a Python list — and it means you need to decide what happens to a half-finished task when the process dies. Most teams discover this requirement during their first deploy, which is the expensive time to discover it.
3. Tool contracts that fail loudly. Demo tools are happy-path functions that return a string. Production tools need timeouts, bounded retries with backoff, schema validation on the way out, and idempotency keys on anything that writes. Agents retry. An agent that retries a non-idempotent create_invoice call is a billing incident. Put these guarantees in the tool layer where the model can't route around them — this is a large part of the argument for wrapping your capabilities in a real MCP server instead of scattering ad-hoc functions through the prompt.
4. A hard cost and step budget. Agents are loops, and loops with retries inside them multiply cost geometrically rather than linearly. One confused run can cost more than a thousand normal ones. Budget three dimensions and fail closed on each:
budgets:
max_steps: 25 # tool calls per task, hard cap
max_tokens: 150000 # per run, across all model calls
deadline_seconds: 300 # wall clock, enforced by the runner
on_exceed: return_partial # a handled outcome, not a crash
The on_exceed line is the part people miss. Hitting a budget must be a normal outcome that returns partial work and an explanation, not an exception that loses the run.
5. Per-run traces. You cannot debug an agent from log lines, because the interesting failure is never one call — it's the shape of a sequence. You need every step captured: prompt, tool inputs and outputs, token counts, latency, and the decision. When someone reports "the agent did something weird on Tuesday," a trace is the difference between a fix and a shrug.
6. A security boundary that assumes tool output is hostile. Any text your agent retrieves can contain instructions aimed at your agent, and an agent holding broad credentials will follow them. This is not theoretical — agents have already been turned against the repositories they were working in. The fix is scope rather than detection: narrowest credential per tool, explicit confirmation on irreversible actions, and never a path from retrieved content straight into a privileged call.
The order to build them in
Sequence matters more than the list, because the wrong order wastes weeks:
Freeze the prototype and write the evals. Do not refactor first. You need a baseline number on the thing that currently exists, however ugly, or you'll never know whether your beautiful rewrite made it worse.
Write the task spec the agent is actually working from. Ambiguity in the task shows up as variance in the output, and most "the model is dumb" complaints are underspecified goals. This is the whole case for spec-driven development with agents — the spec is the thing evals grade against.
Harden the tools. Timeouts, idempotency, validation. Cheapest reliability win available, and it's ordinary backend work you already know how to do.
Add budgets and traces together. They're the same instrumentation pass, and each is nearly useless without the other.
Then deploy. By this point deployment is genuinely the boring step — an agent on Cloudflare Workers is an afternoon once the agent is worth deploying.
Common mistakes
Treating eval as a phase instead of a fixture. "We'll add evals before launch" means you'll ship on vibes and add evals never. The suite exists from day two or it doesn't exist.
Grading with an LLM before grading with code. LLM-as-judge is a real technique, and it's the second thing you reach for, not the first. Anything checkable programmatically — did the JSON parse, did the file get written, does the number match — must be checked programmatically. Judges are for the residue.
Confusing a longer prompt with more reliability. Past a point, added instructions trade one failure mode for another and the aggregate barely moves. The eval suite is what tells you where that point is. Without it, prompt length only grows.
Optimizing the model choice first. Model swaps are the most visible lever and usually the smallest. A weaker model with a tight spec, clean tools, and real budgets beats a stronger model wired into a loose harness — and it's cheaper on every run.
Letting the demo set the scope. The demo succeeded on a narrow, well-formed input. Production traffic includes the empty case, the enormous case, the malformed case, and the adversarial case. Sample real inputs before you promise a success rate.
FAQ
What makes an AI agent production-grade?
A production-grade AI agent is one whose behavior is measured rather than observed — it has a scored evaluation suite, durable state that survives a restart, tool calls with explicit timeouts and idempotency, a hard cost and step budget, per-run traces you can debug from, and a security boundary that assumes tool output is hostile. A demo proves the agent can succeed once. Production requires knowing how often it fails and what it costs when it does.
Why do so few AI agent prototypes make it to production?
Because the work that makes an agent shippable is almost entirely different from the work that makes it demo. A demo is a single successful trace on a happy path you chose. Production is the distribution of every path, including the ones where a tool times out, the model loops, or a document contains instructions aimed at your agent. Google and Kaggle's 5-day agents course is a clean illustration: 353,000 people registered and just over 6,000 submitted a capstone project, roughly one in fifty-nine.
What is the first thing to build after an agent prototype works?
An evaluation suite, before any refactor and before any new feature. Collect fifteen to thirty real task instances, write a programmatic pass criterion for each, and run the whole set on every change. Without it you cannot tell an improvement from a regression, which means every subsequent change to the agent is a guess. Everything else — memory, tools, deployment — is easier to get right once you can score it.
How do you stop an AI agent from running up unbounded cost?
Budget it in three dimensions at once and fail closed on each: a maximum step or tool-call count per task, a maximum token spend per run, and a wall-clock deadline. Agents are loops, and a loop with a retry inside it multiplies cost geometrically rather than linearly, so a single confused run can cost more than a thousand normal ones. Treat exceeding a budget as a normal, handled outcome that returns partial work, not as a crash.
Is vibe coding useful for building AI agents?
Yes, for the first version — natural-language prototyping is the fastest way to find out whether an agent idea is worth pursuing at all. It stops being useful the moment the question changes from "can this work?" to "how often does this work?", because that question is answered by evals and traces, not by another prompt. Use vibe coding to find the shape, then engineer the thing you found.
What is the most common security mistake in agent deployments?
Trusting tool output as data when the model reads it as instructions. Any text your agent retrieves — a web page, a PDF, an issue comment, a code file — can contain directions aimed at the agent, and an agent holding broad credentials will follow them. The fix is scope, not detection: give each tool the narrowest credential that works, require confirmation for irreversible actions, and never let retrieved content reach a privileged tool call unmediated.
The take
353,000 people wanted to build agents badly enough to register for a five-day course. That's the healthiest signal in this industry right now, and the course materials are still up on Kaggle Learn as self-paced content, so the funnel isn't closed.
But "from vibe to live" is doing a lot of quiet work as a phrase. Vibe gets you the prototype, fast, and that part is genuinely solved — it's why 353,000 people could plausibly sign up. Live is evals, durable state, tool contracts, budgets, traces, and a security model. None of that is glamorous and all of it is ordinary engineering discipline applied to a non-deterministic component.
If you have a prototype that works and you're wondering what's next: it's not a better model and it's not a longer prompt. Write thirty test cases and score them. The number you get back is the first true thing you'll know about your agent.
Sources
- Anant Nawalgaria and Brenda Flynn, Inside our 353,000-person vibe coding course — Google blog, August 3, 2026. Source of all participation figures quoted here: 353,000 registrations, 392,000 Discord participants, 12,000+ active capstone participants, 6,000+ capstone submissions, 2M+ learners since 2024, and the "designing, securing, and deploying production-grade AI agents in the cloud" and "from vibe to live" quotes.
- 5-Day AI Agents Intensive Course with Google — Kaggle Learn. The course materials, still available as self-paced content.
The conversion percentages in the funnel table are my own arithmetic on Google's reported figures, not numbers Google published.
Written for umesh-malik.com — no-fluff technical writing on AI, Web Dev, and Engineering.
Originally published at umesh-malik.com
Keep reading on umesh-malik.com:
Top comments (0)