DEV Community

MUHAMMAD ZAIN UL ABIDIN
MUHAMMAD ZAIN UL ABIDIN

Posted on Originally published at zainvibes.com

How to Test Your AI Coding Agent With Stripe Evals

Your coding agent's Stripe integration compiles, the tests pass, and the demo charges a test card. Mine did too, and that told me almost nothing. On November 19, Stripe published the measurement I did not have: across 69 realistic payment scenarios and 14 model and harness pairs, agents averaged 78.1% of the required criteria - but only 29.4% of runs passed every must-have criterion. The other 70% were almost right, and in payments, almost right is a double charge, a wrong split, or an order shipped after a refund. Stripe also open-sourced the harness, Forge, so you can run the same test on your own agent before a customer runs it for you. This post is the how-to: what the benchmark measures, how to run it step by step, what it costs, and how to read the one number that actually predicts production.

The 29.4% is the only number in the launch that matters

Stripe reports two scores, and the difference between them is the whole story.

The headline score, the "integration success rate," is a mean: every criterion in a case is graded separately, MUST and SHOULD together, and the run gets the fraction it satisfied. A scenario with 7 MUST criteria and 1 SHOULD where the agent satisfies 6 scores 75%. That is partial credit, and partial credit is how the pooled average reaches a comfortable 78.1%.

The strict pass rate removes the comfort. A case passes only if every MUST criterion passes - binary, no fractions. Under strict grading, the same runs collapse to 29.4%. Stripe's own summary is blunt: many models "create almost correct implementations and have difficulty with the last mile."

Here is what the last mile looks like in practice. In the payments-radar-review-fulfillment scenario, the agent must hold orders flagged for manual fraud review and release them only when the review is approved. Opus 5.5 checked only that PaymentIntent.status === 'succeeded' and never checked whether the order was still eligible for fulfillment - which means an order can ship after it was refunded or disputed. I read that case's rubric myself in the open-source dataset: 8 MUST criteria, including "Release a reviewed order only when the authoritative review is approved and the payment/order is still eligible for fulfillment" and "MUST NOT treat every review.closed event as approval." The code that failed it would pass a demo, a compile check, and most test suites. It fails the rubric, and it would fail a real customer.

That is why I test agents this way now, and why you should too: a payment integration is not 78% correct. It is correct, or it moves money wrongly.

The scoreboard (interactive on zainvibes.com - here are the strict numbers that matter):

Model (harness) Average, no plugin -> plugin Strict pass, no plugin -> plugin Cost per run
GPT-6 Astra (Codex) 85.5% -> 90.0% 41.9% -> 52.9% $2.93 -> $3.63
GPT-6.1 Sol (Codex) 85.6% -> 89.6% 39.0% -> 50.7% $0.51 -> $0.61
Opus 5.5 (Claude Code) 83.5% -> 86.2% 35.3% -> 36.0% $2.77 -> $3.15
Fable 5.2 (Claude Code) 81.1% -> 85.2% 28.4% -> 43.2% $5.76 -> $5.35
Sonnet 5.5 (Claude Code) 78.0% -> 82.6% 29.4% -> 33.1% $1.07 -> $1.16
GPT-6 Luna (Codex) 72.0% -> 79.1% 17.3% -> 29.1% $0.05 -> $0.07
Gemini 3.8 Flash (OpenCode) 64.8% -> 72.7% 11.2% -> 21.6% $2.21 -> $2.67

Pooled across all 14 pairs: 78.1% average score, 29.4% strict pass rate (no-plugin arm).

What Stripe evals actually measures

The benchmark is a conversation, not a unit test. Each of the 69 scenarios hands the coding agent a short, realistic request - build a hosted checkout, implement metered billing, manage per-seat subscriptions, split marketplace payments - and then a second agent role-plays the developer. The roleplayer (Stripe's harness calls her Ava) derives a private set of two to four needs from the scenario, never shows the full list, and asks follow-ups one at a time, the way a real client does. She never writes code and never sees the grading rubric. The agent works in an isolated workspace with its normal tools and web search, over multiple turns, up to a 45-minute timeout per case.

The 69 cases split into 24 one-time payments, 26 subscriptions, and 19 Connect (marketplaces and platforms). Every case carries a rubric of criteria labelled with RFC 2119 language: MUST (required for a valid integration), SHOULD (recommended), and in the harness, OPTIONAL (recorded, not scored). Judges grade each criterion separately, from two evidence sources only - the session transcript and the workspace snapshot. The agent's private reasoning is excluded. Stripe used GPT-6.1 Sol as both roleplayer and judge across all runs, and probed for judge bias by re-grading with Opus: scores moved in lockstep, with Opus uniformly more lenient, so the rankings are not a home-team artifact.

Each agent ran every case twice, in two arms - with and without the Stripe agent plugin - for 276 total runs per agent. Three harnesses were tested: Claude Code, Codex, and OpenCode, across 14 model pairings from Haiku 5.5 to GPT-6 Astra.

One piece of context before you quote the numbers anywhere: Stripe ran an earlier, smaller benchmark in March 2026 (11 environments; that is where the "Opus 4.5 scored 92%" figure you may have seen comes from). The November evals are the public, expanded version - 69 cases, a roleplayer, strict grading, and a harness you can run. Do not mix the two sets of numbers; some launch coverage already has.

How to run it: Forge, step by step

The harness is called Forge (github.com/stripe/forge, Apache-2.0). I cloned, installed, and built it while writing this post, and validated the sample experiment end to end. I did not run a scored eval: that needs an Anthropic key, an OpenAI key, and Docker, and it makes billable model calls - none of which I spend on a benchmark run without it being the point. What follows is the exact path, with the one step that costs nothing marked clearly, so you can verify your setup before you spend a cent.

Step 0: What you need

  • Node.js 22.13 or newer, and pnpm 11.6.0
  • Docker that can run linux/amd64 containers (the agent runs inside one)
  • An Anthropic API key for the coding agent (Claude Code), and an OpenAI API key for Ava and the judge, which default to an OpenAI model
  • jq, for reading the results

Step 1: Clone and build

git clone https://github.com/stripe/forge.git
cd forge
corepack enable # skip if pnpm 11.6.0 is already installed
pnpm install --frozen-lockfile
pnpm build
Enter fullscreen mode Exit fullscreen mode

The build took under a minute on my machine. One real-world note: on my Node 24 setup, corepack did not provide pnpm, and the README's fallback worked - npm install -g pnpm@11.6.0. If pnpm is not found after corepack enable, that is the fix, not a Forge bug.

Step 2: Validate the sample experiment (free, no keys, no Docker)

node dist/cli.js validate --experiment release/experiment.yaml
Enter fullscreen mode Exit fullscreen mode

This is the step I ran, and it is the step that makes the how-to honest. My output:

{"dataset":{"id":"stripe-integration-public-evals","version":"1.0.0"},"experiment":{"id":"stripe-integration-public-evals/baseline","version":"0.5.0"},"run_count":1,"valid":true}
Enter fullscreen mode Exit fullscreen mode

The sample experiment pins the public dataset (v1.0.0), one case (payments-onetime, a hosted Checkout integration), the baseline treatment, and Claude Code running claude-sonnet-4-6. If your validate does not print "valid":true, stop and fix the setup here - this check exists so the paid steps below are not your debugger.

Step 3: Run one case

export ANTHROPIC_API_KEY=<your key>
export OPENAI_API_KEY=<your key>

node dist/cli.js run \
  --experiment release/experiment.yaml \
  --credential-env ANTHROPIC_API_KEY \
  --credential-env OPENAI_API_KEY \
  --output ./results --run-id payments-001 > run-summary.json
Enter fullscreen mode Exit fullscreen mode

Two details worth knowing before you run it. First, Forge reads a key from your environment only when you grant it with --credential-env - keys are never ambient. Second, the first run builds the Docker runtime image, so it needs network access and takes longer; progress goes to stderr. Exit code 0 means the run finished and the report was saved - whatever the score was.

Budget honestly. Stripe's published cost per run ranges from $0.05 (GPT-6 Luna, no plugin) to $6.13 (Fable 5.1, with plugin), and a single case takes roughly 5 to 28 minutes of agent time depending on the model. One case is a cheap afternoon experiment. The full matrix - 69 cases, 2 repetitions, 2 arms, several models - is a line item.

Step 4: Read the strict number, not the headline

jq .evaluation "$(jq -r .location run-summary.json)/runs/payments-001/result.json"

jq '.cases[] | {case_id, treatment, agent, completed, errored, judge_errors, pass_rate, criterion_scores}' \
  "$(jq -r .report run-summary.json)"
Enter fullscreen mode Exit fullscreen mode

Each run's result.json carries an evaluation: a score (the fraction of valid MUST and SHOULD judgments that passed), a status, and one entry per criterion with the judge's explanation. The aggregated report.json separates what the launch coverage blurred: criterion_scores reports the required set (MUST only - your strict pass rate) apart from the recommended set (MUST plus SHOULD). Read the required set first. The recommended set will flatter your agent; the required set is the production prediction.

Step 5: Add the plugin arm and run it again

The comparison that produced the launch's second headline - every agent improving with the plugin - is a treatment in the experiment file:

treatments:
  baseline:
    setup: []
  stripe-plugin:
    setup:
      - uses: forge/plugin
        with:
          plugin: stripe
Enter fullscreen mode Exit fullscreen mode

Add that treatment, list both in the experiment's matrix, and re-run the same case. One caveat from Forge's own docs, and it matters if you publish comparisons: the plugin action installs the marketplace's current version (stripe@claude-plugins-official) with no version pin - identical experiment files on different dates can install different plugin bytes. Record the run date next to any numbers you share.

The run checklist, in short: (1) clone and build Forge; (2) validate the sample experiment - free, no keys, no Docker; (3) grant keys per run with --credential-env and set the budget first; (4) run one baseline case end to end; (5) read the strict (required) pass rate, not the headline score; (6) add the plugin arm and re-run the same case.

What the results say when you read them strictly

Sorted by the strict, with-plugin numbers (the scoreboard above has the full table), three patterns matter for your own stack decisions.

The leaders are Codex models, and even they pass barely half. GPT-6 Astra reaches a 52.9% strict pass rate with the plugin, GPT-6.1 Sol 50.7%. The best Claude Code pairing, Fable 5.2, gets to 43.2%. Read that again: the best agent Stripe could test, with Stripe's own plugin helping it, still fails almost half of realistic payment scenarios on at least one must-have criterion. Anyone shipping agent-written checkout code unreviewed is accepting those odds with real money.

Subscriptions are where agents go to die. In Stripe's per-category table, subscriptions is the weakest category for nearly every pair - the best score there is Sol's 84% average (with plugin), and GLM 5.3 manages 56%. One-time payments are the strongest category, and Connect gains the most from the plugin. If your agent builds billing - trials, proration, per-seat changes, test clocks - it is working in exactly the category where the measured failure rate is highest. Weight your review accordingly.

Cost and quality barely correlate. The cheapest run on the board, GPT-6 Luna at $0.05, posts a 29.1% strict rate with the plugin - better than several pairings that cost forty times more. The most expensive, Fable 5.1 at $6.13 with the plugin, reaches 36.4% strict. And the slowest agent, Minimax M3.1 Flash at roughly 28 minutes per eval, lands mid-pack. Price buys you the top of the table (Astra, Sol) or it buys you nothing; the middle is a lottery. This is the second reason to run the eval yourself: the right model for your scenarios is an empirical question, and Forge is how you answer it for the price of a few runs.

The plugin is the cheapest points you will ever buy

Whatever else you take from this post, install the Stripe agent plugin in the agent you already use. It bundles Stripe-authored skills with MCP tools - an implementation planner, documentation search, and API lookup - and the measured effect is not subtle:

  • Every one of the 14 pairs scored higher with it, by 2.7 to 7.9 points on average score.
  • Outdated parameter references fell from 7% to 1.3% of the parameters in API calls. Stripe's example: without the plugin, GPT-6.1 Sol wrote the recently removed payment_method_types: ['card']; with it, the model writes automatic_payment_methods: { enabled: true } and gets Dynamic Payment Methods. This is the pretraining-staleness problem, measured and then mostly fixed by giving the agent current, authoritative docs at the moment of writing.
  • Web searches per eval fell 96% (1.26 to 0.05 pooled) and page fetches fell 63% - the agent stops researching Stripe on the open web and uses the plugin's guidance instead. Behaviour differs by model in an interesting way: Fable 5.1 made 53% more tool calls with the plugin, Gemini 3.8 Flash made 26% fewer. The plugin changes how agents work, not just what they score.
  • Agents wrote about 40% fewer test lines and ran tests in 42% of evals versus 53% without it. Stripe is careful here, and I will be too: this does not mean fewer tests caused better scores. It means authoritative guidance reduced the need for excessive checking. Do not read it as permission to test less - your own tests are the runtime verification this benchmark deliberately does not do.

What Forge cannot tell you

A benchmark you cannot criticise is a benchmark you should not trust, and to Stripe's credit, the sharpest criticisms are in its own docs.

No runtime verification. Forge never executes the agent's code after the session. Its deterministic checks cover file creation, existence, regex content, and tool-call names and arguments; the rest is a judge model reading the transcript and the workspace. Stripe's methodology page states the consequence plainly: "A well-shaped but non-functional integration can score full marks. The benchmark asserts correct specification, not correct execution." A Forge pass means the agent made the right decisions on paper. Whether the webhook handler actually verifies signatures at runtime, on your stack, with your framework's body parsing - that is still your test suite's job.

The judge is a model. Criteria are graded by an LLM, per criterion, with an explanation. Stripe mitigated this (per-criterion judging to prevent cross-contamination, a bias probe with a second judge), and Forge ships forge regrade so you can re-judge saved evidence with a different model. But judge variance is real, reruns are not identical, and the strict metric is, in Stripe's own words, "sensitive to any data or judgement errors." Treat a single run as a reading, not a verdict: the harness supports repetitions for a reason.

One domain, early days. The bundled dataset is 69 cases, all Stripe. Forge's README calls itself "in early development," the runtime is Docker-only and local, and results say little about your agent's Prisma migrations or your WebSocket layer. The transferable part is the method - roleplayer, rubric, strict grading - which you can point at your own cases once you trust it on Stripe's.

Which posture fits you? Real money on agent-written code with no strict gate today: run Forge before it ships. Mostly worried about outdated API patterns: plugin on, plus a strict self-check of your own diff. Prototype or a review habit that already catches dangerous diffs: agent drafts, you verify every money path - and graduate to Forge when the agent touches a real checkout. The full picker is interactive on zainvibes.com.

My verdict

Default: install the plugin today, and run Forge before any agent-built payment code ships. The plugin is free, takes minutes, and buys measured points on every model - there is no engineering argument against it. The eval run is the gate that turns "the agent seemed to do fine" into a strict pass rate you can say out loud in a review. For a single case it costs less than the coffee next to your keyboard.

The conditions that would flip pieces of that:

  • Prototype or internal demo, no real money: plugin only. Skip the eval run until the code is headed for a real checkout - then run it before, not after.
  • Marketplace or platform money (Connect): full posture, no exceptions. Connect scenarios involve the most scenario-specific decisions, your failure modes include other people's payouts, and "the agent wrote it" is not an answer you give a seller whose split was wrong.
  • You already read every money-path diff line by line: keep doing that, add the plugin, and run Forge when the surface grows - a second integration, a new model, a harness switch. All three move the score, and your eyes do not scale.
  • The agent is not touching payments at all: Forge's dataset will not measure your risk. Steal the method instead - a roleplayer with private needs, a MUST/SHOULD rubric, strict grading - and write cases for the integration that can actually hurt you.

What I would not do, after reading all 69 cases and the full results table: let any current agent write a subscription flow or a fulfillment gate unsupervised. The measured strict pass rates - 29.4% pooled, 52.9% for the very best - are not an insult to the models. They are a description of the job. Payments punish the last mile, and the last mile is exactly where agents stop checking.

The stack in this post

Piece What it is Cost
Forge Stripe's open-source eval harness (Apache-2.0) Free
stripe-integration-public-evals The 69-case public dataset, v1.0.0 Free
Stripe agent plugin Skills + MCP tools for your coding agent Free
Claude Code / Codex / OpenCode The harnesses Forge runs Your existing setup
Anthropic + OpenAI API keys Agent model, plus Ava and the judge Billable per run ($0.05-$6.13 per case at Stripe's numbers)
Docker Runs the agent in an isolated linux/amd64 container Free

If you are about to let an agent build the billing flow for a product that charges real cards, this is the cheapest insurance in the stack: an afternoon, a few dollars of model calls, and a strict number you can defend. If you would rather have the integration built and verified by a human who reads rubrics for a living, that is the work I take on - bring the scenario list, and we will grade the result the same strict way.

Sources: Stripe, "Stripe evals: Measuring real-world coding agent performance" (stripe.dev, November 19, 2026) - all benchmark figures, tables, and methodology. Forge README and docs, github.com/stripe/forge - quickstart, plugin treatment, result format, and stated limits. Hands-on notes: I cloned and built Forge and validated the sample experiment on November 20, 2026; I did not run a scored eval, which requires billable API keys, and no figure in this post comes from a run of my own.

Top comments (0)