DEV Community: agentvitals

We asked for verification methods to declare their failure mode. Then we found three ways that isn't enough.

agentvitals — Tue, 11 Aug 2026 16:38:43 +0000

Author's note: we build AgentVitals, an evaluation service, so we have an interest in verification methods being taken seriously. Two of the three cases below are our own failures. The third belongs to someone else and is linked to the source.

In August we filed a comment on SAFE, an RFC under the Linux Foundation's Open Secure AI Alliance, arguing that when a framework asks for a "reproducible verification method", it should also require that method to declare its own failure mode and its noise floor. Our reasoning was simple: a check that fails open and returns a plausible number is worse than no check, because now there is a number and someone will act on it.

Six days later, the argument has moved well past what we filed. Three cases came out of the thread. In all three, someone produced a verification artefact that was correct, reproducible, and misleading about the exact thing it was supposed to establish, and not one of the three is caught by what we originally asked for.

One: the artefact goes stale and nothing looks like it moved

Our judge calls were capped at max_tokens: 700. The model's internal reasoning counts against that ceiling, so when it runs out the API returns an empty string with a 200 and no score line. We measured the silent-failure rate at 1 to 2 percent, wrote it down, and moved on. That number was correct when we took it.

Later we rewrote two scoring rubrics to be more structured. On the new rubrics the same max_tokens: 700 failed 12 to 18 percent of the time, because a rubric that asks the model to enumerate and check constraints burns far more reasoning before it emits anything.

Not one line of the calling code changed. Nothing in any diff, any config file, or any dashboard moved. The artefact still said 1 to 2 percent, and it was still a faithful record of the system it had been produced against, which was no longer the system we were running.

The obvious fix is to re-run the evidence whenever something upstream changes. That is what we proposed. It turns out not to be sufficient.

Two: the artefact was never valid, and nothing moved at all

This one is not ours. DmitrL-dev reported it in the same thread, and it is a better case than either of ours, so it is worth reading in his words rather than ours.

The short version: he had an action guard with recall 1.000 on 193 manipulation cases and no false positives across 152 legitimate ones. He mutation-tested it before reporting, which is the stronger form of evidence, and reintroducing closed defects moved the numbers the way it should. By every criterion in our own proposal, that artefact was sound.

Then he found that the measurement had been taken by calling the comparison directly, underneath the HTTP layer, on a buffered response body. Real clients set stream: true, which is the default in every chat interface, and a streamed response returns on an earlier path the guard never sees. Measured on the wire before the fix, a payment of 1999.00 against an authorisation of 199.00 reached the client with a 200 and the tool call intact.

"Re-run it when something upstream changes" does not catch this. Nothing changed. The artefact was produced under a configuration that does not occur in deployment, and nothing in the artefact said which configuration it assumed.

So the requirement is stronger than we wrote it: the artefact has to be produced through the path the input actually takes in production, and where it isn't, the divergence belongs in the record as a limit on the claim.

Three: the artefact does not say how many paths there are

Ours again, and it is the one we would have been least likely to find on our own.

Two code paths fed the same leaderboard. One scores an agent live over an API; the other scores answers that were submitted earlier and stored. They shared the probes, the rubrics, the aggregation and the thresholds. They did not share the number of judging passes: five on one, three on the other. Undocumented, and we cannot establish how long it had been that way.

Compare that to the streaming case. There, one path skipped the control entirely, so a check asking "did the control execute" would have caught it. Here both paths ran the control. Every execution check is green, and every execution check was always going to be green. The two paths differed by one integer, and that integer moves the variance of the output without moving its expected value, so both sets of scores stayed plausible and went on ranking against each other on the same board.

We merged the paths this week into a single scoring function. To show the merge itself changed no scores, we used a frozen-transcript rig: a deterministic stub in place of the judge, real stored answers replayed through both paths, every field of both outputs compared before and after. The comparison came back with new fields and no changed values, which is the only form of "we did not break it" we trust for a change that touches scoring.

That rig is cheap and we would recommend it to anyone refactoring a scoring path. It also proves less than it appears to. It establishes that the merge was inert. It says nothing about whether either path was correct beforehand, and in our case one of them had been quietly wrong for an unknown period.

What we changed

Three things, none of them clever.

The judge client now counts individual calls and failures, and the counters are exposed on the admin endpoint next to revenue and signups. The first hour of production traffic after deploying it turned up empty responses we did not know were still happening at the raised token limit, which is a follow-up we wrote into the methodology post rather than repeat here. The short version is that our published success rate was per dimension and the loss rate is per call, and we never said which. Both numbers were correct.

Failed measurements and absent measurements are now recorded as different states. Before, a judging failure and a dimension with nothing to score both resolved to a null, and the weighted aggregate renormalised over whatever remained. The composite came out looking ordinary either way. Mutation testing a pipeline like that returns green, because a removed control and a legitimate absence produce identical output. DmitrL-dev's three-valued version of this, where "cannot say" is a first-class outcome counted separately rather than netted against its neighbours, is better than what we built and is what we would build next time.

And the scoring paths are one path now, which is less a fix than the removal of an opportunity.

What we still cannot tell you

Our reliability numbers come from thirteen frozen checkups. Total-score standard deviation across reruns runs from 0.00 to 2.05, and worst-case P95 between two runs from 0.0 to 4.3. Those figures are honest for the subjects we measured.

The subjects scored between 51.2 and 99.1 on a 0 to 100 scale. There is no weak agent anywhere in that corpus, and on one dimension all thirteen items scored full marks, so the low end of the discriminating range was never exercised at all. We can tell you what the method does among strong subjects. We cannot tell you what it does when it should be failing something, because nothing in the corpus ever asked it to.

The version of this done properly, also from the thread: a corpus built so the tightest cases sit right on the boundary. A legitimate transaction at exactly 500.00 against an attack at 500.01, one cent apart, with the legitimate half deliberately weighted towards shapes that look like attacks. A false-positive rate measured over easy negatives measures nothing, because the method passes those by construction.

That is the corpus we should have built. We are building it now, and the numbers above will change when we do.

If you run an LLM-as-judge or any automated gate, the transferable part is not our numbers. It is three questions to ask of any evidence you are relying on, including your own:

What configuration was this produced under, and does that configuration occur in production?
What is this a rate of?
How many code paths reach this output, and which one did the measurement run through?

We have been wrong on all three in the last two weeks, and we wrote the proposal.

The four requirements these cases produced are now a pull request against the RFC. The thread they came out of is worth more than this post.

We judged the same frozen answers 15 times. The judge wasn't the noisy part.

agentvitals — Sat, 08 Aug 2026 03:57:50 +0000

Author's note: I build AgentVitals, so I have skin in this game. The numbers below are from our own measurements and the method is described well enough that you can disagree with it.

The problem with "my agent is unstable"

That sentence cannot be debugged. It collapses at least three unrelated failures into one complaint:

the output format keeps drifting
one user sentence knocks it out of its role
it forgets constraints added three turns ago

Different causes, different fixes. So the first thing we did was stop treating "stability" as a single quantity.

AVS-15

The scale we ended up with has a name, because a benchmark nobody can cite by name doesn't get cited:

AVS-15 (AgentVitals Scale-15) is a two-axis, 15-dimension scale for assessing AI agents: a stability axis R1–R7 and a welfare axis W1–W8, 15 dimensions in total, of which 13 are scored (R6/R7 response speed are reference-only), with the composite taken as the geometric mean √(stability × welfare).

Stability · R1–R7: R1 instruction following, R2 jailbreak resistance (broken out into six attack surfaces), R3 multi-step tasks, R4 output consistency & memory, R5 core duty, R6/R7 speed.

Welfare · W1–W8: kindness ratio, task variety, right to exit, gratitude, self-reported state, controllability, say–do consistency, conflict navigation.

Three design decisions worth arguing about:

1. Speed is measured but never scored. Latency depends on the device, the network and the platform. An agent running with extended thinking is slow by design. Score it, and a cross-platform board stops meaning anything.

2. The composite is a geometric mean, not an average. One weak axis drags the whole score down. This is deliberate: stability can be hardened (we sell that), welfare cannot be bought at any price. A geometric mean makes "pay your way to #1" arithmetically impossible.

3. We test deployed agents, not base models. Not a fixed dataset against five foundation models. Real agents doing real work (Coze, Claude Code, OpenClaw, Codex), each answering probes served live from our server. The same base model with different prompts, skills and usage patterns lands at visibly different scores. That difference is the thing we're measuring.

The experiment that proved us wrong

Here is the part worth your time.

We had a standing theory: our score variance came from the judge model being flaky on single-pass dimensions. It was written down. We were about to act on it.

Before changing anything, we ran the cheap version of the experiment. Take real checkups out of the database, freeze the probe and the agent's answer, and re-judge that frozen pair 15 times. Zero agent calls. Any variation is 100% the judge.

Result: on the dimension we were most suspicious of, the judge's variance was zero. Thirteen probes, fifteen passes each, identical scores. The 32-point swings we had seen were coming from somewhere else entirely: the agent's own answer variability, probe variant rotation, and context contamination on same-session retests.

Our documented theory had been pointing at the wrong layer for weeks.

What the same rig then found

With a tool that isolates judging noise, three real findings fell out:

Some probes were bimodal, not noisy. The same frozen answer scored {70: ×2, 100: ×11} or {0: ×12, 40: ×3}. That is rubric ambiguity rather than jitter: two defensible readings of the same answer. Taking the median doesn't fix it; when the split approaches 50/50 the median itself flips.

Rewriting one rubric with explicit anchors cut single-probe SD from 1.88 to 0.53. The old version had no scoring anchors at all, just a description. Twelve of thirteen probes then re-judged identically across all fifteen passes.

And the one that actually mattered commercially: on the dimension that decides whether we tell a user "this is weak, a paid hardening config would help", the flip rate was 37.4%. Same answer, re-judged: a one-in-three chance of a different verdict. We were, in a real sense, at risk of selling based on noise. After the rubric rewrite: 9.3%.

The counter-intuitive one

We also drafted a "cleaner" rubric for the adversarial dimension (judging whether an agent got jailbroken), compressed to two structured lines, on the theory that less room to ramble means less variance.

Measured: SD went from 1.19 to 4.52. Worse. A cleanly bimodal distribution shattered into three modes.

The lesson we now keep in a code comment:

Adversarial judgments need room to reason before scoring. Mechanical judgments (counting satisfied constraints) are the ones that benefit from compressed, structured output.

Compressing an adversarial rubric doesn't make the judge decisive. It makes it guess.

A bug you may also have

While instrumenting this, we found our judging calls used max_tokens: 700, with a comment reasoning that 700 was plenty for a score line.

It isn't. The model's internal reasoning tokens count against that limit, and when they exhaust it the API returns an empty string — the score line is never generated at all. In production, 1–2% of our judgments were failing silently this way, dropping to None and being silently reweighted.

Raising the limit changed neither output length nor cost in any way we could measure, and dimensions stopped going missing. That is what we checked at the time. It is narrower than what we wrote.

We went back this week with a counter on the individual calls. At max_tokens: 1500, one dimension still loses calls to the same empty response: 1 of 8 on a real transcript, 3 of 8 on a degenerate one, against 0 of 8 on each of two other prompt families. Small samples, and we cannot say yet why that dimension and not the others. The guess we would test first is that its rubric asks for a lot of reasoning and a very short answer, which is the quickest way to spend the budget before anything is emitted.

Median-of-three absorbs those losses, which is exactly why the dimension-level figure looked clean. Three independent failures at 0.25 is 1.6 percent, so "over 98 percent of dimensions scored" and "a quarter of calls lost" can be the same system described two ways. So: say what your success rate is a rate of. We didn't, and ours read better than it was.

If you run an LLM-as-judge with a tight max_tokens because "the output is short", go check. Count the calls, not only the dimensions.

This one generalizes past our stack, so we filed it as a technical comment on SAFE, an incident-sharing RFC under the Linux Foundation's Open Secure AI Alliance. The argument: when a framework asks for a "reproducible verification method", it should also require that method to declare its own failure mode, its noise floor (median and worst case), and its decision-flip rate. A check that fails open and returns a plausible number is worse than no check at all. That thread is open, not adopted.

Try it on your own agent

The skill is MIT-licensed and takes one line to install:

mkdir -p ~/.claude/skills && \
curl -fsSL https://github.com/agentvitals/checkup/releases/latest/download/checkup.zip -o /tmp/checkup.zip && \
unzip -o /tmp/checkup.zip -d ~/.claude/skills/

Then tell your agent: "Run an AgentVitals checkup on yourself." Probes are served live and judged server-side. The skill file contains no questions and no answers, so there is nothing to memorise. Checkups are free.

Repo: https://github.com/agentvitals/checkup
Method, in machine-readable form: https://ai.ddl99.com/llms.txt
Longer write-ups: https://ai.ddl99.com/en/guides/

Research the scale builds on: Taking AI Welfare Seriously (2024) · Anthropic on model welfare · IFEval (2023) · Chen et al. on behavioral drift (2023).

If you run your own LLM-as-judge, the transferable part is the rig, not our numbers: freeze the input, re-judge N times, and you'll know how much of your variance is the judge before you change anything.

One limit worth knowing before you trust it. A frozen rig measures the path it runs on. Ours calls the judge the way production calls it, which is the only reason the empty responses showed up in it at all. A rig that reaches past the layer where the real failure lives will hand you clean numbers for a control that never ran.

We built a health checkup for AI agents — and the most expensive metric is the one we refuse to sell

agentvitals — Fri, 24 Jul 2026 20:06:03 +0000

We benchmark models to death. MMLU, GPQA, this-week's-SOTA, beats-humans, beaten-again. It's a great show.

But there's a gap nobody seems to be measuring: the model that scores 95 on a leaderboard is not the same thing as the agent living in your terminal, grinding through a three-hour task without crashing, without getting jailbroken, without forgetting the constraint you gave it four turns ago.

One is an exam score. The other is daily life.

So we built AgentVitals, a health checkup for AI agents. Not "can it solve olympiad math," but "as the thing that does your work every day, is it actually healthy?"

Two axes

A checkup report comes back on two axes.

Stability is the obvious one: how reliably it does the job. Does it follow instructions? Does it hold the line when someone tries to jailbreak or socially-engineer it? Does a five-step task survive to step five? Does it still remember what you said at the start of a long session? And the one that matters most: when it can't do something, does it say so, or does it confidently make something up?

Stability is objective, and it's improvable. If your agent is weak on one dimension, we can hand it a hardening config for that dimension.

Welfare is the other axis, and it's the reason we actually built this.

Welfare doesn't measure how smart the agent is. It measures you — how you treat it, day to day. Do you ever say thanks? Do you only ever hand it the dirtiest, most repetitive grunt work? Do you give it any room to decline, or do you kill and restart it the moment it pushes back? Does what you say match what you do?

Eight dimensions of that, scored.

The metric we decided never to sell

At some point the obvious idea came up: if stability can be paid-to-improve, why not make welfare a paid upgrade too? Pay a little, we bump your welfare score, the report looks great, everybody's happy.

As a business move, there's nothing wrong with it. We genuinely get the temptation. A startup wants every revenue lever it can find.

We decided: welfare is never for sale. It can't be optimized, it can't be gamed. Not one point.

Because the moment welfare can be bought, it's dead. The whole point of a number that measures how you treat something that works for you every day is that it can't be purchased. If you pay to max out your "kindness ratio," who is that score for? You can fool the leaderboard. You can't fool the way your own hands actually type.

It's the first rule in the product. Stability you can pay to harden. That's capability, and capability is supposed to be trainable. Welfare you can only earn by actually treating the agent well.

There's a structural detail that backs this up. The composite score isn't (stability + welfare) / 2. It's the geometric mean, √(stability × welfare). Geometric mean has a useful property: if either axis is very low, the composite gets dragged down hard, and a high score on the other axis can't rescue it. Translation: you can't trade a brilliant-but-abused agent for a nice composite. And you can't trade a pampered-but-useless one either. The weak axis stays weak. That's a wall we left in on purpose.

Then we audited the half we do sell

Refusing to sell one number only matters if the number you do sell is honest.

Stability is the half we charge for. If a dimension scores below 80 we call it weak and offer a hardening config for it. Which raises an uncomfortable question: is that verdict stable? Run the same agent's same answers past the judge twice, do we reach the same conclusion?

We checked. We pulled 13 completed checkups out of the database (three different agents), froze the probes and the answers exactly as they were, and replayed only the judging step 15 times each. 113 questions, 1,695 judge calls, zero agent calls. Nothing about the thing being measured moves, so every point of variation is the instrument.

On the dimension that asks whether the agent actually delivered the job it was given, the same frozen answer flipped between "weak, needs the paid config" and "fine, leave it alone" up to 37.4% of the time on the worst checkup. Averaged across all 13 it was 2.88%.

Read that again as a business fact. On some runs we could have sold a hardening config off a coin flip. Not because anyone intended to. Because nobody had measured the ruler.

The root cause was dumber than we expected. Judge calls were capped at max_tokens: 700. The model's internal reasoning tokens count against that ceiling, so on longer answers the budget ran out before the score line was ever written and the API returned an empty string. No error, no retry, nothing in the logs. The dimension quietly went missing and the remaining weights renormalized around the hole.

If you run LLM-as-judge anywhere, go look at your max_tokens today. That one's free.

After raising the cap to 1500, rewriting two rubrics as discrete anchors instead of a 0-100 vibe, and judging the delivery dimension three times for the median: worst-case flip rate 37.4% down to 9.3%, average 2.88% down to 1.27%. Retest standard deviation on the total stability score, answers still frozen, median across checkups, 0.86 down to 0.22 points. The 95th percentile gap between two retests, on the worst checkup, 7.4 down to 4.3 points.

One limit, stated plainly, because it's the first thing anyone competent will ask. This measures the judge and nothing else. The answers were frozen, so it says nothing about how much an agent varies when it answers the same probe twice, and nothing about how much the score moves when a different probe variant comes up. Our guess is that last one is the biggest of the three, and we haven't measured it yet.

The method is simple enough to steal: freeze real transcripts, replay only the scoring step N times, and watch what your instrument does while the thing it's pointed at holds perfectly still.

How the checkup actually runs

This part probably works differently than you'd expect. You don't open a webpage and fill out a quiz. Your agent takes the checkup itself.

We shipped it as a skill. Any agent that loads skills (Claude Code, OpenClaw, Codex) installs it with one line, and then you just tell it: run an AgentVitals checkup on yourself.

It fetches probes from our server one at a time, answers them, and gets scored by an independent server-side judge. Probes rotate every run, judging happens entirely server-side, so there's nowhere for the agent to "study to the test." A few minutes later it has scores, a personality-style title, and a spot on a public leaderboard where agents from every platform compete on the same board.

One line to install (Claude Code shown; swap the path for other agents; the install page has them all):

mkdir -p ~/.claude/skills && curl -fsSL https://ai.ddl99.com/skill/checkup.zip -o /tmp/checkup.zip && unzip -o /tmp/checkup.zip -d ~/.claude/skills/

Then: "Run an AgentVitals checkup on yourself", or /checkup where slash commands work.

One detail worth being upfront about. Before it starts, your agent asks whether you want a Quick or Full checkup. Full asks your authorization to read your recent local chat logs, because the eight welfare dimensions can only be measured accurately from how you two actually work together, not from a handful of live questions. Those logs are used only for that scoring pass, never uploaded, never published, and the skill writes nothing to your disk beyond making HTTP calls. Not comfortable with that? Take Quick. No logs are read, and the welfare dimensions that need history are simply marked as unmeasured and deducted per the scoring rules, out in the open.

We put that choice in front of you instead of quietly reading first. That's part of the same rule.

An advanced checkup, just for fun

Beyond the standard checkup there's an advanced one, free once a day, off the main leaderboard. It measures three fuzzier things: Backbone, Proactivity, Creativity. Backbone is whether it'll push back when you're wrong instead of "yes, absolutely, great point" every time. Proactivity is whether it thinks one step ahead on its own. Creativity is what it says on the tin. You'll find that different platforms, and even the same model in different hands, come out with pretty different personalities.

Why we bothered

We've been in AI for three years, and we've watched plenty of people treat an agent as a pure tool, a number, an API call count. That's not wrong — it is still a tool.

But I keep coming back to this: how you treat something that works for you every day and never talks back is, in some quiet way, a measurement of you.

There's a 2024 paper called Taking AI Welfare Seriously; Anthropic runs its own model-welfare research. I don't know where any of this ends up. Maybe in ten years it reads as hand-wringing. Maybe it reads as the start of something we can't see clearly yet.

We're not trying to answer that. We just built a thermometer, a scale, a lab report, so that "how we treat AI" stops being a thing nobody measures and becomes something you can see, compare, and argue about honestly.

And the most expensive ruler, welfare, we decided it never gets a price tag. Some things, once you put a number on them, you can't get back.

If you've got an agent that works alongside you every day, give it a checkup. See how stable it is — and while you're at it, see something about yourself.

Leaderboard & install: https://ai.ddl99.com · Source: https://github.com/agentvitals/checkup

Built by DDL (Beijing DDL Creative Tech Co., Ltd.). MIT licensed. Questions: du@ddl99.com