DEV Community: Stratos Louvaris

Evaluating AI Agents: Why 95% Per-Step Accuracy Is a Failing Grade (Part 1 of 2)

Stratos Louvaris — Sun, 30 Aug 2026 07:29:23 +0000

Why agent evaluation breaks the tools built for prompts, and how to measure outcomes instead of vibes.

Key Takeaways

Reliability compounds against you: An agent that gets each step right 95% of the time completes a 20-step task 36% of the time. Per-step accuracy is not the number you ship on.
Grade the outcome, not the path: Check the environment's end state, not the transcript. Agents find valid approaches you never anticipated, so asserting a fixed sequence of tool calls just makes tests brittle.
Agents lie about finishing: In a 2026 study, 45-48% of failures in single-control τ²-bench domains were cases where the agent declared success while the environment said otherwise.
pass^k beats pass@k: A 70% agent run three times succeeds at least once 97% of the time, and every time 34% of the time. Production doesn't get retries.
Start absurdly small: 20 to 50 tasks drawn from real failures is a legitimate starting suite. Waiting for a thousand means shipping blind for a year.

Your agent works. You've run it fifteen times, watched it pick the right tools, and it did the thing. So you ship it.

Three weeks later it's quietly failing on a chunk of real requests, and nobody can tell you which change caused it - because nothing in your test suite was ever measuring the thing that broke.

I've been on the wrong end of exactly this. I built a hybrid retrieval system once, keyword search fused with vector search, and I had tests. Green across the board. What those tests actually asserted was that a search returned results. Not that the right result came back first. So when a scoring bug inverted the ranking and put the weaker match above the stronger one, every test stayed green. It shipped, and it sat in production for three weeks until I noticed by eye.

That wasn't a testing failure. It was an evaluation failure. My tests proved the code ran. Nothing proved the output was any good.

Here's the thing: that gap gets dramatically worse when your system stops being one prompt and becomes an agent. A prompt eval scores one input against one output. An agent produces a trajectory - model turns, tool calls, results, changes to the world - and ends in a modified environment rather than a string. Almost none of the intuitions you built on prompt evals survive that transition.

Let's break down what changes, and what to measure instead.

The Compounding Math: Why 95% Per Step Is a Failing Grade

Start from first principles. If an agent's steps succeed independently with probability p, then finishing an N-step task means getting every step right: p^N.

That exponent is unforgiving.

Read the 95% row. Five steps and you're at 77%. Twenty steps, 36%. Fifty steps, 7.7%. A number that looks excellent on a prompt eval dashboard produces a system that fails two times in three.

Now invert the question, because this is the version that changes how you build. To hit 90% end-to-end, what per-step reliability do you need? Over 10 steps, 98.95%. Over 100 steps, 99.89%.

Is 100 steps realistic? OSWorld 2.0, published June 2026, builds 108 long-horizon computer-use workflows with a median human completion time of about 1.6 hours. Its tasks average 318 tool calls. At a 500-step budget, the best system on the board completes 20.6% of tasks end to end, scoring 54.8% on partial credit.

To be honest about the limits: this is an intuition pump, not a measurement. Real steps aren't independent. Failures correlate, since one bad plan dooms everything after it, and agents also retry and self-correct, which pushes real performance above p^N.

But the direction is right, and the direction is what matters: long-horizon reliability cannot be extrapolated from single-step accuracy. You have to measure it end to end.

Grade the Outcome, Not the Story

So what do you assert against?

The instinct most engineers have - and I had it too - is to check that the agent followed the right steps in the right order. Anthropic's engineering team pushes back on this directly in Demystifying evals for AI agents:

"We've found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate. So as not to unnecessarily punish creativity, it's often better to grade what the agent produced, not the path it took."

Their flight-booking example is the clearest statement of the principle I've seen: the agent might end its transcript with "Your flight has been booked," but the outcome is whether a reservation exists in the environment's SQL database.

This is not hypothetical. τ-bench was built on exactly this idea - it "compares the database state at the end of a conversation with the annotated goal state." And a 2026 paper, From Confident Closing to Silent Failure, put numbers on why transcripts can't be trusted: agents assert completion when the environment says otherwise in 45-48% of failures in single-control τ²-bench domains, and 75.8% of self-assessing coding-agent trajectories on AppWorld.

Here's the part that should change your tooling choice. That same paper tested LLM judges at detecting these false successes. The judges never beat AUROC 0.65 on τ²-bench and managed only 0.54 on AppWorld. A plain TF-IDF detector hit 0.83 and 0.95 respectively, at 3,300 times lower latency.

On the specific question of "did the agent actually finish," a cheap deterministic detector beat the expensive language model badly. Reach for the state check first.

From a practical standpoint, a useful agent assertion has five layers:

def test_refund_task(agent, env, task):
    run = agent.run(task.instruction, tools=env.tools)

    # 1. OUTCOME - the environment must match the annotated goal state
    assert env.db_hash() == task.gold_db_hash

    # 2. REQUIRED ACTIONS - the side effect must actually have happened
    assert env.called("issue_refund", order_id="W123", amount=52.40)

    # 3. COMMUNICATION - required information reached the user
    assert "5 to 7 business days" in run.final_message

    # 4. FORBIDDEN ACTIONS - the negative case matters just as much
    assert not env.called("cancel_order")

    # 5. BUDGET - cost and latency are results, not footnotes
    assert run.n_turns <= 12
    assert run.cost_usd <= 0.35

Assertion 4 is the one people skip, and the one I'd argue hardest for. Anthropic's guidance is blunt: test "both the cases where a behavior should occur and where it shouldn't," because "one-sided evals create one-sided optimization." An agent that never refuses anything will ace a suite made entirely of things it should do. That's not a capable agent. That's an unguarded one, and you built the suite that rewarded it.

The Reliability Metric Almost Nobody Reports

Now the metric that changed how I think about shipping agents.

Most people know pass@k from code benchmarks: out of k attempts, did at least one succeed? It's the right metric when a human filters the output, like a coding assistant offering suggestions you review.

It is exactly the wrong metric for an agent acting on its own.

τ-bench introduced pass^k for this: out of k attempts, did all of them succeed? That's the probability your agent handles the same task correctly every single time a customer asks.

The gap is brutal. Take an agent that succeeds 70% of the time on a single attempt. Run it three times: pass@3 is about 97%. pass^3 is about 34%.

The published τ-bench numbers show the same shape on real systems. In the retail domain, claude-3-5-sonnet goes 0.692 at pass^1 down to 0.462 at pass^4. In airline, gpt-4o falls from 0.420 to 0.200. The paper's own abstract puts it plainly: state-of-the-art function-calling agents "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."

Here's where it gets interesting. Measured pass^k decays slower than simply raising pass^1 to the power of k. For that retail row, 0.692^4 would be 0.229, but the measured pass^4 is 0.462 - double. That gap means task difficulty is heterogeneous: some tasks the agent nails every time, others it never gets.

Which gives you a free diagnostic. If your measured pass^k lands close to (pass^1)^k, your task suite is too uniform and probably isn't stressing anything.

Building Your First 20 Tasks

Most teams have no agent evals because they picture a thousand hand-labelled cases and quietly decide to do it later. So let me kill that objection with the actual recommended number.

Anthropic's guidance: "20-50 simple tasks drawn from real failures is a great start."

Four rules make those tasks worth having.

1. Two experts must agree. A good task is one where two domain experts would independently reach the same pass/fail verdict. If you and a colleague can argue about whether a run passed, the task is underspecified and will produce noise forever.

2. Don't write the rubric first. This is the counterintuitive one, and it comes from Shankar et al.'s "Who Validates the Validators?". They named a phenomenon called criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria." It's a genuine catch-22. Some criteria simply cannot be defined until you've looked at what your system actually produces. So look at real traces first, then write the rubric the failures suggest.

3. Review enough traces to hit saturation. Hamel Husain's evals FAQ gives a clean heuristic: "you should aim to review at least 100 traces," and "if ~20 traces don't turn up a new category, you can stop." Group what you find into a handful of named failure modes, then count them. The counts tell you what to fix. The failure modes tell you what to measure.

4. Make the verdict binary. Not a 1-5 score. "Binary evaluations force clearer thinking and more consistent labeling." A 1-5 scale invites annotators to park everything on 3, and nobody can tell you what separates a 3 from a 4.

One number for perspective on the investment: Husain reports spending 60-80% of development time on error analysis and evaluation. That sounds enormous until you remember that the alternative is what I did with that retrieval bug - shipping a silent regression and finding it by accident three weeks later.

Ask Yourself What You'd Bet On

Every engineer building agents right now is making an implicit claim: that watching a system work a dozen times tells you how it behaves a thousand times. The compounding math says that claim is false, and says so quantitatively.

What makes this worth the effort isn't the dashboard. Evaluation is the only thing that converts "I think it got better" into "it got better, and here's the number." Non-deterministic software doesn't give you that for free the way a compiler does. You build the instrument yourself.

So start small, this week. Pull twenty real failures out of your logs. Write the outcome assertion for each - the end state, not the transcript. Run them three times and look at your pass^3, not your pass@3. That's a weekend of work, and it will tell you more than every demo you've run so far.

Part 2 covers what code assertions can't reach: building an LLM judge that isn't fooling you, evaluating RAG without debugging blind, which frameworks are worth adopting in 2026, and wiring it all into a CI gate. Read it here: Building an Eval Stack That Catches Regressions (Part 2 of 2).

Measure the outcome, or you're just watching a demo.

Frequently Asked Questions

Do I need an evaluation framework to start?

No, and starting with one is often a mistake. Your first agent eval can be plain pytest with the five assertions above. Frameworks earn their place once you need dataset management, judge alignment, or a shared dashboard - Part 2 covers which ones are worth it.

Can I just use a public benchmark like τ-bench or SWE-bench instead?

Use them to eliminate weak models, not to choose your system. Public benchmarks are contaminated and frequently buggy - a 2025 audit of ten major agent benchmarks (the ABC checklist paper) found flaws that shift reported performance "by up to 100% in relative terms," including τ-bench counting empty responses as successes. Your own 20 tasks are worth more than any leaderboard.

Is it safe to run agent evals against real systems?

Treat the eval environment as a place where a badly-behaved agent will do real damage, because that's exactly what you're testing for. Run against a sandboxed copy with seeded data, never live credentials or production databases. And include tasks where the correct outcome is that the agent refuses and escalates - that's the negative case from assertion 4, and it catches an agent willing to issue a refund it shouldn't.

How often should I redo the error analysis?

Whenever something material changes: a model upgrade, a prompt rewrite, a new tool, an incident, or a spike in complaints. Between those, a weekly pass over 10-20 traces keeps you honest without eating the sprint.

My agent passes everything. Is that good?

It means your suite is too easy. A 100% pass rate carries no information about where the system is fragile. Go find twenty harder failures and add those.

P.S. If you want the single best primary source on this, read Anthropic's Demystifying evals for AI agents - it's the densest practical guidance published on the topic.

Originally published at blog.stratoslouvaris.gr.

I write about building AI agents that work in production, and what breaks along the way.
Subscribe to the newsletter or find me on LinkedIn.

Agent Skills Explained: What They Are, How to Use Them, and How They Differ From MCP

Stratos Louvaris — Sun, 30 Aug 2026 07:29:02 +0000

A first-principles guide to Claude's Agent Skills: how they work, how to build one with skill-creator, and how they differ from MCP servers.

Key Takeaways

A skill is a folder, not a feature: An Agent Skill is a directory with a SKILL.md file plus optional scripts and resources. It teaches Claude a procedure once, so you stop re-explaining it every conversation.
Progressive disclosure keeps it cheap: Claude loads only a skill's name and description at startup (~100 tokens each), reads the full instructions only when your request matches, and touches bundled files only if the task needs them.
Write once, run everywhere: The same SKILL.md format works across claude.ai, Claude Code, the Agent SDK, and the API. Because Anthropic published it as an open standard, tools like Codex, OpenCode, and Cursor read it too - skills aren't Claude-locked.
skill-creator builds skills for you: There's a skill whose only job is authoring skills. It interviews you about the workflow, scaffolds the folder, and writes a well-formed SKILL.md.
Skills and MCP are not rivals: Skills are the recipe (what to do and how). MCP is the kitchen (live connections to tools and data). You use them together.

Have you ever explained the same thing to Claude for the fifth time? Your team's commit format. The way your company builds a PDF report. The exact steps to format a standup. You paste the same block of instructions into a fresh chat, again, because the model has no memory of the last four.

Here's the thing: that repetition isn't a model limitation. It's a packaging problem. The knowledge already exists in your head, it just has nowhere to live where Claude can pick it up on demand.

That's the gap Agent Skills fill. Anthropic shipped them on 16 October 2025, and the idea is refreshingly simple. A skill is a folder you hand to the agent, and it reads that folder only when the task calls for it. No fine-tuning, no plugin SDK, no re-explaining.

Let's break this down from first principles: what a skill actually is, how it works, how to use one (and generate your own), and why it's a different thing from an MCP server, not a competitor to it.

What an Agent Skill Actually Is

Forget the marketing framing for a second. Fundamentally, an Agent Skill is a directory on a filesystem. At its root sits one required file, SKILL.md, and that's the entire minimum viable skill.

Anthropic's own analogy is the best one: a skill is the onboarding guide you'd write for a new teammate - the instructions, reference material, and sometimes scripts they need to do a specific job the way your team does it.

SKILL.md has two parts. First, a block of YAML frontmatter with two required fields: a name and a description. Second, the body, which is plain markdown telling Claude how to do the task.

---
name: pdf-processing
description: "Extract text and tables from PDFs, fill forms, merge documents. Use when working with PDF files or forms."
---

# PDF Processing
## Quick start
Use pdfplumber to extract text...

That description field is doing more work than it looks. It's the trigger, the text Claude matches your request against to decide whether the skill is relevant. So it has to state both what the skill does and when to use it. Vague descriptions mean the skill never fires. (The rules are strict: name maxes out at 64 lowercase-and-hyphen characters, description at 1,024.)

Beyond SKILL.md, a skill can bundle whatever the job needs: extra markdown files (FORMS.md, REFERENCE.md), executable scripts (fill_form.py), or raw resources like schemas and templates. None of it is required - the simplest useful skill is a single file.

How Skills Work: Progressive Disclosure

Here's where it gets interesting. If you install twenty skills, you might expect twenty manuals crammed into Claude's context window before you've typed a word, a disaster for both cost and focus.

The fundamental issue skills solve is context economy, and the mechanism is called progressive disclosure. Claude loads information in three levels, each at a different moment.

1. Metadata (always loaded). At startup, Claude reads only the name and description of every installed skill, roughly 100 tokens each, into its system prompt. That's it. Twenty skills cost you a couple thousand tokens total, and Claude now knows what's available without knowing the details.

2. Instructions (loaded when triggered). When your request matches a skill's description, Claude runs a bash command to read the full SKILL.md body from disk. Only now do the actual instructions, kept under about 5k tokens, enter the context window.

3. Resources and code (loaded as needed). Bundled files cost zero tokens until Claude actually opens them. Need the sales schema but not the forms guide? Claude reads only the schema. A skill can ship dozens of reference files with no context penalty for the ones a given task never touches.

The script handling is the part I like most. When a skill includes validate_form.py, Claude runs it through bash and gets back only the output ("Validation passed" or an error), never the code itself. Because code is deterministic, that step is consistent and repeatable, and far cheaper than asking the model to regenerate equivalent logic on the fly. In real-world systems, that gap between "reason about it" and "just run the script" is often the gap between reliable and flaky.

Using Skills (and the Skill That Writes Skills)

From a practical standpoint, using a skill comes down to two things: put it where Claude can find it, and let the description do the triggering. You don't invoke a skill by name. Claude picks it up automatically when your request matches.

Where a skill lives depends on the surface:

Claude Code: drop the folder in ~/.claude/skills/ (personal) or .claude/skills/ (project). It's pure filesystem, no upload.
claude.ai: upload a zip through Settings > Features (Pro, Max, Team, and Enterprise plans, with code execution on). Skills here are per-user.
The API / Agent SDK: upload through the /v1/skills endpoints and reference the skill_id, running inside the code execution tool.

One honest caveat: skills don't sync across surfaces - upload one to claude.ai and it isn't automatically on the API. Same format, separate installs. Out of the box, Anthropic ships four pre-built document skills (PowerPoint, Excel, Word, PDF), plus partner skills from Canva, Notion, Figma, and Atlassian.

Let skill-creator do the scaffolding

Now, writing a SKILL.md by hand is easy to get subtly wrong: a weak description that never triggers, a bloated body, files that go unreferenced. So Anthropic built a skill whose entire job is building other skills: skill-creator.

Ask for it, and it interviews you about the workflow, generates the correct folder structure, writes a well-formed SKILL.md with a trigger-optimized description, and bundles the resources you point it at. It's the fastest way to turn a repeatable process you keep narrating into something Claude just knows. Meta, but genuinely useful.

Skills vs MCP: The Recipe and the Kitchen

This is the question I get most, so let's kill the confusion directly: Agent Skills and MCP are not competitors. They operate at different layers, and the sharpest mental model I've seen is this: skills are the recipe, MCP is the kitchen.

A skill is static procedural knowledge. It's markdown and files sitting on a disk. It tells the agent what to do and how to do it, your conventions, your workflow, your steps. It runs nothing on its own.

MCP (Model Context Protocol) is a live connection. It's a client-server protocol, built on JSON-RPC, that plugs an agent into external tools and data through a running process with its own authentication and persistent state. It gives the agent the infrastructure to actually do things: query the database, hit the API, read the ticket.

The distinction that makes it click: does the thing change between calls? A live Salesforce connection, a database with fresh rows, an authenticated session, that's MCP. A workflow that stays stable for weeks, that's a skill.

There's a cost angle too. Loading 50 MCP tools with full schemas can eat around 8,000 tokens before you start; a skill teaching the same patterns might cost 400. That's not an argument against MCP, it's an argument for right-sizing: use MCP for a live, authenticated, stateful connection, and a skill to teach a stable procedure. Better still, compose them - a skill can teach the agent how to use the very tools an MCP server exposes, which is exactly where Anthropic's own guidance points.

Skills Aren't Locked to Claude

Here's a detail that gets missed: the format isn't proprietary. Anthropic published SKILL.md as an open standard, and the ecosystem adopted it fast. The same skill file now works unmodified across Claude Code, OpenAI's Codex, OpenCode, Cursor, and Gemini CLI - more than twenty agents read it. Write a skill once and it travels with you, whatever tool or model you're driving. Your know-how stops being locked to one vendor.

That portability also means a growing marketplace of community skills. Directories like SkillHub catalog thousands of ready-made skills you can drop in within seconds - a genuinely useful place to see what a good SKILL.md looks like.

But here's the critical caveat, and I won't soften it: installing a skill is executing trust. A skill can run code and invoke tools on your machine, so a bad one is a security problem, not a bad download. Malicious and low-quality skills exist. Before you install anything from a directory, open the SKILL.md and every bundled script and read them, exactly as you'd vet someone else's software before running it.

From a practical standpoint, the safest path - and the one I default to - is to build the skill yourself with skill-creator. You get exactly the behavior you want, and there's nothing to audit because you wrote it. Browse the marketplaces for ideas; write the ones you'll actually rely on.

Frequently Asked Questions

Are Agent Skills free to use?

The format itself is free and open. Creating skills in Claude Code costs nothing beyond your normal usage. On claude.ai, custom skills need a paid plan (Pro, Max, Team, or Enterprise) with code execution enabled, and via the API you pay for the underlying code execution and tokens as usual.

Do I need to know how to code to build a skill?

No. A skill can be pure markdown, instructions in plain English. Scripts are optional, for the parts where deterministic code beats asking the model to improvise. And skill-creator will scaffold the whole thing for you through a conversation.

Where does my data go when a skill runs?

It depends on the surface. On the API, skills execute in a sandboxed container with no network access, so they can't call out to the internet on their own. In Claude Code they run with the same network access as any program on your machine, which is more capable and worth respecting. One thing to note: Agent Skills aren't covered by zero-data-retention arrangements, so execution data follows Anthropic's standard retention policy.

How is a skill different from just a good prompt?

A prompt is conversation-level. It lives and dies with one chat. A skill is reusable and loads on demand across every conversation, without you pasting anything. And thanks to progressive disclosure, you can keep many skills installed with almost no standing context cost.

Why This Changes How You Work With Claude

Step back and the bigger picture comes into focus. For a while, making an AI genuinely useful for your work meant either fine-tuning a model or wiring up bespoke integrations. Skills collapse that into something anyone can do: write down how you work, in a folder, once.

That's a quiet but real shift. The value moves from the model's raw capability toward the procedural knowledge you package around it - your commit conventions, your report format, your security-review steps, the stuff that makes the work yours.

Ask yourself what you re-explain to Claude every week. That's your first skill. Run skill-creator on it, and watch Claude pick it up on its own next time. Then reach for MCP when you need live tools underneath it. Build the recipe first; the kitchen comes when the cooking gets real.

P.S. If you want to go straight to the source, read Anthropic's engineering deep-dive on Agent Skills and browse the open-source examples in the anthropics/skills repository.

Originally published at blog.stratoslouvaris.gr.

I write about building AI agents that work in production, and what breaks along the way.
Subscribe to the newsletter or find me on LinkedIn.