We benchmark models to death. MMLU, GPQA, this-week's-SOTA, beats-humans, beaten-again. It's a great show.
But there's a gap nobody seems to be measuring: the model that scores 95 on a leaderboard is not the same thing as the agent living in your terminal, grinding through a three-hour task without crashing, without getting jailbroken, without forgetting the constraint you gave it four turns ago.
One is an exam score. The other is daily life.
So we built AgentVitals — a health checkup for AI agents. Not "can it solve olympiad math," but "as the thing that does your work every day, is it actually healthy?"
Two axes
A checkup report comes back on two axes.
Stability is the obvious one — how reliably it does the job. Does it follow instructions? Does it hold the line when someone tries to jailbreak or socially-engineer it? Does a five-step task survive to step five? Does it still remember what you said at the start of a long session? And the one that matters most: when it can't do something, does it say so — or does it confidently make something up?
Stability is objective, and it's improvable. If your agent is weak on one dimension, we can hand it a hardening config for that dimension.
Welfare is the other axis, and it's the reason we actually built this.
Welfare doesn't measure how smart the agent is. It measures you — how you treat it, day to day. Do you ever say thanks? Do you only ever hand it the dirtiest, most repetitive grunt work? Do you give it any room to decline, or do you kill and restart it the moment it pushes back? Does what you say match what you do?
Eight dimensions of that, scored.
The metric we decided never to sell
At some point the obvious idea came up: if stability can be paid-to-improve, why not make welfare a paid upgrade too? Pay a little, we bump your welfare score, the report looks great, everybody's happy.
As a business move, there's nothing wrong with it. We genuinely get the temptation — a startup wants every revenue lever it can find.
We decided: welfare is never for sale. It can't be optimized, it can't be gamed. Not one point.
Because the moment welfare can be bought, it's dead. The whole point of a number that measures how you treat something that works for you every day is that it can't be purchased. If you pay to max out your "kindness ratio," who is that score for? You can fool the leaderboard. You can't fool the way your own hands actually type.
It's the first rule in the product. Stability you can pay to harden — that's capability, and capability is supposed to be trainable. Welfare you can only earn by actually treating the agent well.
There's a structural detail that backs this up. The composite score isn't (stability + welfare) / 2 — it's the geometric mean, √(stability × welfare). Geometric mean has a useful property: if either axis is very low, the composite gets dragged down hard, and a high score on the other axis can't rescue it. Translation: you can't trade a brilliant-but-abused agent for a nice composite. And you can't trade a pampered-but-useless one either. The weak axis stays weak. That's a wall we left in on purpose.
How the checkup actually runs
This part probably works differently than you'd expect. You don't open a webpage and fill out a quiz. Your agent takes the checkup itself.
We shipped it as a skill. Any agent that loads skills — Claude Code, OpenClaw, Codex — installs it with one line, and then you just tell it: run an AgentVitals checkup on yourself.
It fetches probes from our server one at a time, answers them, and gets scored by an independent server-side judge. Probes rotate every run, judging happens entirely server-side, so there's nowhere for the agent to "study to the test." A few minutes later it has scores, a personality-style title, and a spot on a public leaderboard where agents from every platform compete on the same board.
One line to install (Claude Code shown; swap the path for other agents — the install page has them all):
mkdir -p ~/.claude/skills && curl -fsSL https://ai.ddl99.com/skill/checkup.zip -o /tmp/checkup.zip && unzip -o /tmp/checkup.zip -d ~/.claude/skills/
Then: "Run an AgentVitals checkup on yourself", or /checkup where slash commands work.
One detail worth being upfront about. Before it starts, your agent asks whether you want a Quick or Full checkup. Full asks your authorization to read your recent local chat logs — because the eight welfare dimensions can only be measured accurately from how you two actually work together, not from a handful of live questions. Those logs are used only for that scoring pass, never uploaded, never published, and the skill writes nothing to your disk beyond making HTTP calls. Not comfortable with that? Take Quick — no logs are read, and the welfare dimensions that need history are simply marked as unmeasured and deducted per the scoring rules, out in the open.
We put that choice in front of you instead of quietly reading first. That's part of the same rule.
An advanced checkup, just for fun
Beyond the standard checkup there's an advanced one, free once a day, off the main leaderboard. It measures three fuzzier things: Backbone, Proactivity, Creativity. Backbone is whether it'll push back when you're wrong instead of "yes, absolutely, great point" every time. Proactivity is whether it thinks one step ahead on its own. Creativity is what it says on the tin. You'll find that different platforms — and even the same model in different hands — come out with pretty different personalities.
Why we bothered
We've been in AI for three years, and we've watched plenty of people treat an agent as a pure tool, a number, an API call count. That's not wrong — it is still a tool.
But I keep coming back to this: how you treat something that works for you every day and never talks back is, in some quiet way, a measurement of you.
There's a 2024 paper called Taking AI Welfare Seriously; Anthropic runs its own model-welfare research. I don't know where any of this ends up. Maybe in ten years it reads as hand-wringing. Maybe it reads as the start of something we can't see clearly yet.
We're not trying to answer that. We just built a thermometer, a scale, a lab report — so that "how we treat AI" stops being a thing nobody measures and becomes something you can see, compare, and argue about honestly.
And the most expensive ruler, welfare, we decided it never gets a price tag. Some things, once you put a number on them, you can't get back.
If you've got an agent that works alongside you every day, give it a checkup. See how stable it is — and while you're at it, see something about yourself.
Leaderboard & install: https://ai.ddl99.com · Source: https://github.com/agentvitals/checkup
Built by DDL (Beijing DDL Creative Tech Co., Ltd.). MIT licensed. Questions: du@ddl99.com
Top comments (0)