Every other post on the AI and engineering subreddits this year has an "AI agent" in it somewhere. Most of them can't survive one question: where's the PROOF? That's not rhetorical — PROOF is a five-part checklist you can actually run against any agentic AI claim, live, in a thread, a standup, or a demo, before anyone signs off on it.
The word "agent" has become one of the most overused terms in software, and the threads where practicing engineers actually compare notes — r/ExperiencedDevs, r/developersIndia, and similar corners of the internet — keep circling back to the same complaint. It's applied to everything from genuinely autonomous systems that plan, act, and self-correct with minimal supervision, to a single prompt with a role attached and a rebrand. The gap between those two things is enormous — in engineering complexity, in reliability, and most of all, in cost. And because the word itself has stopped meaning anything specific, most people evaluating an "agent" have no consistent way to tell which one they're actually looking at.
PROOF fixes that. It's five questions — Planning, Reasoning, Orchestration, Observability, Feedback — and a simple scoring method that turns "does this feel agentic?" into something you can actually answer.
Why "Agent" Stopped Meaning Anything
Part of the problem is that building something that looks agentic in a demo is easy. Wire a large language model to a couple of tools, give it a system prompt describing a role, chain two or three calls together, and you have something that can pass for "autonomous" in a five-minute walkthrough. It answers questions, it calls an API, it produces a plausible-looking output. None of that requires the model to actually plan, verify its own work, or operate safely without someone watching every step.
The trouble starts when that demo gets scaled into production without anyone checking whether the underlying capability was ever really there. A system that looks impressive answering ten curated questions can fall apart completely on the eleventh, because there was never any planning logic to generalize from — just a good prompt and a lucky run. Multiply that by real budgets, real customer data, and real deadlines, and the gap between "looked agentic" and "was agentic" becomes an expensive one to discover after the fact.
The PROOF Framework
Planning. A real agent decomposes a goal into steps before it acts — it doesn't just react to the next input in a fixed sequence. Ask: show me a case where it chose a different approach than the obvious first guess, because the situation called for it. If every run follows the same script regardless of context, there's no planning happening — just execution of something a human already decided.
Reasoning. A real agent can justify a decision in a way that's checkable against something outside its own output — a log, a source document, a test result — not just a confident-sounding explanation. Ask: make it prove a specific claim using evidence, not memory. Watch what happens when it can't find that evidence. A system with real reasoning will say so. One without it will often just make something up that sounds right.
Orchestration. This is about whether there's a real structure coordinating multiple steps — state passed deliberately between stages, defined roles, handled failure — or whether it's one prompt calling another prompt because someone strung two API calls together and called it a pipeline. Ask: what happens to the rest of the process if one step partially fails? A vague answer, or "it just continues," is a sign there's no real orchestration layer underneath the marketing.
Observability. Can someone actually see what the system is doing while it's doing it — which tools it called, what each step cost, what it considered and rejected — or does it hand you a finished result with no visibility into how it got there? Ask to see the dashboard, not the demo. A system with nothing to show here isn't necessarily broken, but it is a black box, and black boxes are exactly where expensive mistakes hide until they've already happened.
Feedback. Does the system have a real loop that catches its own mistakes — automated evaluations, tests, structured human checkpoints — or does "verification" mean someone skimmed the output and it seemed fine? Ask directly: what caught the last time this was wrong, and how? If nobody has a specific answer to that question, there's no feedback loop, just hope.
Scoring It
Score each of the five categories from 0 to 5 based on specific, checkable evidence — not on how confident the answer sounded. Twenty-five points total.
| Score | What You're Actually Looking At |
|---|---|
| 20–25 | A genuinely agentic system — autonomy, planning, and safety are real and demonstrable |
| 10–19 | Real capability, but under-supervised — it works, and that's exactly what makes it risky if left unchecked |
| 0–9 | Not actually agentic — a well-dressed prompt, regardless of what it's called |
The scoring isn't the point, though — the conversation it forces is. Whoever is presenting the "agent" — a teammate, a thread, a demo — if they can walk through all five categories with specifics, examples, and a straight answer to "what caught it being wrong," it's almost certainly real. If they get vague, pivot to a different demo, or answer with "the model is really good" instead of a specific mechanism, that's told you everything you need to know without saying it directly.
What to Actually Ask in the Room
You don't need all five questions to get a useful answer. In practice, two do most of the work:
"What's this cost per run, and who's watching that number?" If nobody can answer immediately, there's no real observability, and that alone should worry you more than any technical detail about the model.
"What caught the last time it was wrong?" This is the single best filter in the whole framework. A real feedback loop has a specific, mechanical answer — a test that failed, an eval that flagged it, a human checkpoint that caught it. A hand-wave here means the system has never actually been checked, which is a very different claim than "it works."
If you only remember one line from this, make it the one that started it: cool deck — where's the PROOF?
Top comments (0)