Most teams shipping LLM features have no idea whether last week's prompt edit made things better or worse. They have a hunch. They tried five inputs in a playground, the outputs looked fine, and it went to production. That is not testing — that's a code review where the reviewer only read the first page.
An LLM eval framework is tooling that runs a fixed set of tasks against one or more model configurations and grades the outputs against an explicit rubric, producing comparable scores instead of impressions. That's the fix, and the reason most teams don't have one isn't ignorance. It's that the existing options are heavyweight: a Python library that owns your process, a hosted dashboard with a seat price, a schema you have to learn before you can score a single prompt. So people keep saying "we should build evals" and keep not building them.
On 31 July 2026, Simon Willison released smevals, built with Jesse Vincent's Prime Radiant lab, and it is the most convincing argument I've seen that this problem is small. An eval is a folder. Tasks are YAML. The runner is a shell script. That's the whole design, and it's the right one.
TL;DR
- An LLM eval framework replaces impressions with comparable scores — same tasks, multiple model configs, explicit rubric, persisted results.
- smevals models this as seven nouns: eval, task, config, runner, run, grader, check. Getting the vocabulary right is most of the work.
- The runner is an arbitrary executable, so you can evaluate your whole agent harness, not just a raw model call.
- Grade deterministically first. Checkers are plain programs that exit 0 or non-zero — reach for LLM-as-judge only for genuinely subjective dimensions.
- Ten tasks from your own failure history beat a thousand generic benchmark items. Start there this week.
What an LLM eval framework actually is
To restate the definition in the form worth quoting: an LLM eval framework runs a fixed set of tasks against one or more model configurations and grades the outputs against an explicit rubric, producing comparable scores rather than impressions. Willison's own definition of the unit is tighter: "An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?"
The word "framework" is doing less work than it sounds like. What you actually need is separation of four jobs that vibes-based testing smashes into one:
- Define the task, once, so it's the same task next month.
- Execute it against a named configuration, so you know what produced the output.
- Judge the output by a rule you wrote down before you saw the result.
- Record the whole thing so the next run is a comparison, not a fresh opinion.
Skip step 4 and you don't have an eval — you have a demo you ran twice.
Why prompt testing by feel breaks down
It breaks down the moment you have more than one variable, which is immediately. A modern LLM feature has at least four moving parts: the model, the system prompt, the tool definitions, and the parsing layer that turns text into something your code can use. Change any one and the others' behaviour shifts.
Here is the failure mode I keep seeing. Someone tightens a system prompt to fix a formatting bug. The formatting bug goes away. Two weeks later, support notices the assistant has become curt and stopped asking clarifying questions — because the tightened prompt also suppressed the follow-up behaviour nobody was watching. There was no test for the follow-up behaviour, because there were no tests.
Evals are regression tests for non-deterministic systems. They don't need to be perfect to be enormously valuable; they need to exist and to run on every change. The same argument that makes CI non-negotiable for a REST API applies with more force here, because the failure surface is larger and the failures are quieter.
💡 Key insight: You are not trying to prove the model is good. You are trying to detect the day it got worse at the specific thing you depend on.
The smevals model: seven nouns
Willison spent, by his own account, a long time on terminology, and it shows. Once these seven nouns click, the design is obvious:
| Noun | What it is | Lives in |
|---|---|---|
| Eval | A collection of tasks probing one capability | eval.yaml |
| Task | A single challenge the model must complete | tasks/*.yaml |
| Config | Which model and parameters to use | configs/*.yaml |
| Runner | An executable that turns a task into an output | run-llm |
| Run | The immutable record of one task × one config | runs/… |
| Grader | An ordered set of checks producing a grade | graders/*.yaml |
| Checker | A program implementing one check | checkers/* |
The separation that matters most is run vs. grade. A run is expensive — it costs an API call and wall-clock time. A grade is cheap and you will get it wrong on the first attempt. Because smevals persists every run to disk, you can rewrite your rubric and re-grade months of history with smevals grade my-eval --regrade without spending a cent on inference. Frameworks that fuse execution and judgement force a full re-run every time you improve a rubric, which is exactly the tax that stops people from improving rubrics.
Your first eval is five files
The directory is the API:
my-eval/
├── eval.yaml # name and description
├── tasks/ # one YAML file per task
├── configs/ # one YAML file per model config
├── graders/ # one YAML file per grader
├── checkers/ # custom checker executables
├── run-llm # the runner executable
└── runs/ # created for you by `smevals run`
A task is about as small as a config file gets:
# tasks/pelicans.yaml
name: pelicans
prompt: Write a haiku about pelicans. Reply with only the haiku, three lines.
A config names the model and the runner that will invoke it:
# configs/default.yaml
name: default
runner: ../run-llm
model: gpt-4.1-mini
And the runner is a shell script — this is the part that should make you sit up:
#!/usr/bin/env bash
llm -m "$SMEVALS_MODEL" "$SMEVALS_PROMPT"
llm logs -c --json > log.json
That's the entire contract. smevals sets SMEVALS_MODEL, SMEVALS_TASK, SMEVALS_PROMPT, SMEVALS_RUN_DIR and one SMEVALS_TASK_ variable per extra task key, then executes your program and captures stdout. Anything that can read an environment variable and write to stdout can be a runner. Your Node agent entrypoint. A curl call to your staging API. A Python script that does retrieval, calls a model, and post-processes the result.
Run it across models and grade in one pass:
uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6
uvx smevals grade path-to-eval/
uvx smevals serve path-to-eval/
Install is uv tool install smevals or pip install smevals; the project is MIT-licensed on GitHub, and smevals build emits a static HTML report you can commit or publish. If you've built a RAG pipeline from scratch, point the runner at the whole retrieval-plus-generation path — the retrieval step is usually the thing that regressed, and a model-only eval will never see it.
Grading is the hard part
Running is trivial. Grading is where evals live or die, and the instinct most teams follow — "we'll have an LLM judge it" — is the wrong first move.
In smevals a checker is a program. It reads SMEVALS_RUN_DIR and the check's own config from SMEVALS_CHECK, then exits 0 for pass or non-zero for fail. Optionally it prints JSON with a score between 0.0 and 1.0, plus metrics, tags and notes. Two checkers ship built in — contains and xml-valid — and everything else is yours to write.
A grader chains them:
# graders/default.yaml
name: default
checks:
- checker: ../checkers/three-lines
required: true
scoring:
pass_threshold: 1.0
Notice what checkers/three-lines is: a twenty-line Python script that counts non-empty lines. Deterministic, instant, free, and it never has an off day. Most of what you actually care about is checkable this way — valid JSON, required fields present, the SQL parses, the generated code compiles, the answer cites a source that exists, latency under budget, no leaked system prompt.
Use an LLM judge only for the dimensions that genuinely have no rule: tone, helpfulness, whether an explanation would land with a beginner. When you do, treat the judge as a component under test too — it drifts, it's biased toward verbosity, and it should have its own small eval. Layer it on top of deterministic checks, never instead of them.
Where teams get evals wrong
They evaluate the model instead of the system. Your users never touch a raw model. They touch your prompt, your tools, your retries, your parsers. Point the runner at your real entrypoint — the same principle behind writing an agent-readable CLAUDE.md or a well-scoped MCP server: the thing worth getting right is the harness the model operates inside.
They chase public benchmark numbers. Vendor benchmarks tell you about the average case across everyone's workload, which is nobody's workload. Even a careful independent benchmark read is a starting hypothesis, not evidence about your app. Ten tasks pulled from your own bug tracker are worth more than every leaderboard combined.
They write the rubric after seeing the output. This is the eval equivalent of writing the assertion after running the test. Decide what "good" means first — that discipline is the same one that makes spec-driven development work with coding agents.
They build it once and never run it. An eval suite that isn't wired into CI is a document. Run it on every prompt change, every model bump, every tool-definition edit.
They aim for 100% pass rates. A suite everything passes has stopped measuring anything. Keep tasks in it that currently fail — those are your roadmap.
How to start this week
- Pick one capability. Not "is our assistant good" — "does it produce valid JSON matching our schema."
- Write five tasks from real failures. Open the support queue or your own logs and take the five worst outputs you shipped this quarter.
- Write one deterministic checker. Schema validation is the highest-value twenty lines of code you'll write this month.
- Run it against two models and two prompt versions. Four columns is enough to reveal something.
-
Commit the eval directory to the repo and add
smevals run && smevals gradeto CI. - Add a task every time production surprises you. The suite should grow from incidents, not from imagination.
FAQ
What is an LLM eval framework?
An LLM eval framework is tooling that runs a fixed set of tasks against one or more model configurations and grades the outputs by an explicit rubric, producing comparable scores instead of impressions. It separates the four jobs that vibes-based testing collapses into one: defining the task, executing it, judging the result, and reporting the trend. The output is a durable record you can diff across models and prompt versions.
How is smevals different from other LLM evaluation tools?
smevals is deliberately small and filesystem-native: an eval is a directory of YAML files plus two executables, and every run persists as plain files under runs/. Most alternatives are Python libraries that own your process and store results in their own database. smevals instead defines a subprocess contract via environment variables, so any language that can read SMEVALS_PROMPT and write to stdout can be a runner or a checker.
Do I need an LLM to grade LLM output?
Usually not, and you should reach for one last. Deterministic checkers — string matching, schema validation, exit codes, regexes, unit tests over generated code — are faster, free, and reproducible. Save LLM-as-judge grading for genuinely subjective dimensions like tone or helpfulness, and always pair it with deterministic checks on the parts of the output that have a right answer.
How many tasks does an eval need to be useful?
Ten to twenty focused tasks beat a thousand generic ones. The value comes from tasks drawn from your own failure history — the tickets, the bad outputs, the edge cases that actually burned you — not from generic benchmark coverage. Start with five real failures, run them on every prompt change, and grow the suite each time production surprises you.
Should I evaluate the model or the whole harness?
Evaluate the harness. In production, users never touch a raw model — they touch your prompt, your tools, your retry logic, and your parsing layer, and any of those can be the thing that broke. Because smevals runs an arbitrary executable as its runner, you can point it at your full agent entrypoint and score the system your users actually experience.
The take
The reason to care about smevals specifically isn't that it's the most capable eval tool available. It's that it's small enough that "we don't have time to set up evals" stops being true. A directory, some YAML, and a shell script is a lower bar than the average ESLint config.
Every team shipping LLM features already runs evals — informally, irreproducibly, in a playground tab, once. Writing them down costs an afternoon and converts a recurring argument into a number. Do that this week, then start the harder work: making the tasks hard enough that failing them tells you something.
If you're deciding what to point your first eval at, comparing coding agents is a good warm-up — the differences that matter show up on your codebase, not on anyone's leaderboard.
Sources
- Simon Willison, "smevals", 31 July 2026 — the release announcement, the eval definition quoted above, and the example CLI invocations.
-
prime-radiant-inc/smevalson GitHub (MIT) — the README this post's directory layout, YAML examples, environment-variable contract and built-in checkers are drawn from.
Written for umesh-malik.com — no-fluff technical writing on AI, Web Dev, and Engineering.
Originally published at umesh-malik.com
Keep reading on umesh-malik.com:
Top comments (0)