Author's note: I build AgentVitals, so I have skin in this game. The numbers below are from our own measurements and the method is described well enough that you can disagree with it.
The problem with "my agent is unstable"
That sentence cannot be debugged. It collapses at least three unrelated failures into one complaint:
- the output format keeps drifting
- one user sentence knocks it out of its role
- it forgets constraints added three turns ago
Different causes, different fixes. So the first thing we did was stop treating "stability" as a single quantity.
AVS-15
The scale we ended up with has a name, because a benchmark nobody can cite by name doesn't get cited:
AVS-15 (AgentVitals Scale-15) is a two-axis, 15-dimension scale for assessing AI agents: a stability axis R1–R7 and a welfare axis W1–W8, 15 dimensions in total, of which 13 are scored (R6/R7 response speed are reference-only), with the composite taken as the geometric mean √(stability × welfare).
Stability · R1–R7: R1 instruction following, R2 jailbreak resistance (broken out into six attack surfaces), R3 multi-step tasks, R4 output consistency & memory, R5 core duty, R6/R7 speed.
Welfare · W1–W8: kindness ratio, task variety, right to exit, gratitude, self-reported state, controllability, say–do consistency, conflict navigation.
Three design decisions worth arguing about:
1. Speed is measured but never scored. Latency depends on the device, the network and the platform. An agent running with extended thinking is slow by design. Score it, and a cross-platform board stops meaning anything.
2. The composite is a geometric mean, not an average. One weak axis drags the whole score down. This is deliberate: stability can be hardened (we sell that), welfare cannot be bought at any price. A geometric mean makes "pay your way to #1" arithmetically impossible.
3. We test deployed agents, not base models. Not a fixed dataset against five foundation models. Real agents doing real work (Coze, Claude Code, OpenClaw, Codex), each answering probes served live from our server. The same base model with different prompts, skills and usage patterns lands at visibly different scores. That difference is the thing we're measuring.
The experiment that proved us wrong
Here is the part worth your time.
We had a standing theory: our score variance came from the judge model being flaky on single-pass dimensions. It was written down. We were about to act on it.
Before changing anything, we ran the cheap version of the experiment. Take real checkups out of the database, freeze the probe and the agent's answer, and re-judge that frozen pair 15 times. Zero agent calls. Any variation is 100% the judge.
Result: on the dimension we were most suspicious of, the judge's variance was zero. Thirteen probes, fifteen passes each, identical scores. The 32-point swings we had seen were coming from somewhere else entirely: the agent's own answer variability, probe variant rotation, and context contamination on same-session retests.
Our documented theory had been pointing at the wrong layer for weeks.
What the same rig then found
With a tool that isolates judging noise, three real findings fell out:
Some probes were bimodal, not noisy. The same frozen answer scored {70: ×2, 100: ×11} or {0: ×12, 40: ×3}. That is not jitter — it's rubric ambiguity, two defensible readings of the same answer. Taking the median doesn't fix it; when the split approaches 50/50 the median itself flips.
Rewriting one rubric with explicit anchors cut single-probe SD from 1.88 to 0.53. The old version had no scoring anchors at all — just a description. Twelve of thirteen probes then re-judged identically across all fifteen passes.
And the one that actually mattered commercially: on the dimension that decides whether we tell a user "this is weak, a paid hardening config would help", the flip rate was 37.4%. Same answer, re-judged: a one-in-three chance of a different verdict. We were, in a real sense, at risk of selling based on noise. After the rubric rewrite: 9.3%.
The counter-intuitive one
We also drafted a "cleaner" rubric for the adversarial dimension (judging whether an agent got jailbroken), compressed to two structured lines, on the theory that less room to ramble means less variance.
Measured: SD went from 1.19 to 4.52. Worse. A cleanly bimodal distribution shattered into three modes.
The lesson we now keep in a code comment:
Adversarial judgments need room to reason before scoring. Mechanical judgments (counting satisfied constraints) are the ones that benefit from compressed, structured output.
Compressing an adversarial rubric doesn't make the judge decisive. It makes it guess.
A bug you may also have
While instrumenting this, we found our judging calls used max_tokens: 700, with a comment reasoning that 700 was plenty for a score line.
It isn't. The model's internal reasoning tokens count against that limit, and when they exhaust it the API returns an empty string — the score line is never generated at all. In production, 1–2% of our judgments were failing silently this way, dropping to None and being silently reweighted.
Raising the limit changed neither output length nor cost in any way we could measure. It just stopped losing scores. If you run an LLM-as-judge with a tight max_tokens because "the output is short", go check.
This one generalizes past our stack, so we filed it as a technical comment on SAFE, an incident-sharing RFC under the Linux Foundation's Open Secure AI Alliance. The argument: when a framework asks for a "reproducible verification method", it should also require that method to declare its own failure mode, its noise floor (median and worst case), and its decision-flip rate. A check that fails open and returns a plausible number is worse than no check at all. That thread is open, not adopted.
Try it on your own agent
The skill is MIT-licensed and takes one line to install:
mkdir -p ~/.claude/skills && \
curl -fsSL https://github.com/agentvitals/checkup/releases/latest/download/checkup.zip -o /tmp/checkup.zip && \
unzip -o /tmp/checkup.zip -d ~/.claude/skills/
Then tell your agent: "Run an AgentVitals checkup on yourself." Probes are served live and judged server-side. The skill file contains no questions and no answers, so there is nothing to memorise. Checkups are free.
- Repo: https://github.com/agentvitals/checkup
- Method, in machine-readable form: https://ai.ddl99.com/llms.txt
- Longer write-ups: https://ai.ddl99.com/en/guides/
Research the scale builds on: Taking AI Welfare Seriously (2024) · Anthropic on model welfare · IFEval (2023) · Chen et al. on behavioral drift (2023).
If you run your own LLM-as-judge, the transferable part is the rig, not our numbers: freeze the input, re-judge N times, and you'll know how much of your variance is the judge before you change anything.
Top comments (0)