Every engineer who has shipped an LLM feature eventually hits the same wall. Your unit tests are green. Nothing throws. And yet the thing is quietly, obviously worse than it was last week. Someone swapped a model, someone tweaked a prompt, someone added a tool, and the output did not error, it just got dumber. No stack trace ever tells you that.
I have spent a while building tools that try to answer one question honestly: how do you actually test an AI system? Not demo it. Test it. Below is the layered strategy I landed on, mapped to five open-source projects I wrote to make each layer real.
Why testing AI is different from testing code
Two things break the assumptions you carry over from normal software.
The first is nondeterminism. The same input can produce different output on two consecutive calls. A test that asserts equality against a golden string is either flaky or a lie.
The second is deeper: there is often no single right answer. "Summarize this ticket" has a thousand acceptable outputs and a thousand bad ones, and no assertEqual can separate them. The output is not correct or incorrect, it is better or worse, more grounded or less, safe or unsafe. Traditional testing asks "did it match?" AI testing has to ask "did it regress?" and "can I trust this particular answer?"
That reframing is the whole game. You stop trying to prove correctness and start building signals: layered, honest, mostly comparative. Here is how I stacked them.
Layer 1: Evals as a build artifact
The base layer is evals, and the trick is to treat quality like something CI can measure, not vibes you check by hand.
evalgate is prompt and agent regression CI. You write a declarative eval suite that lives in version control next to your code, it runs the suite, scores it, and stores a baseline. On every pull request it re-runs, computes the quality delta against the base branch, and fails the build when the score drops, then posts the delta table as a PR comment.
The important design choice: it does not ask "is this good?" It asks "is this worse than it was?" That is the only question CI can answer objectively. It ships ten scorers, including exact match, regex, JSON-schema, embedding similarity, LLM-as-judge, latency and cost budgets, and weighted rubrics, so you can score the fuzzy stuff and the strict stuff in the same pass. And because it runs on a deterministic mock provider, the whole thing works with zero API keys, offline. A quality gate that needs a paid key to run is a quality gate people turn off.
Layer 2: The modality your text evals cannot see
Evals over text are necessary and completely blind to whole classes of failure. Voice is the sharpest example.
voiceeval exists because if you evaluate a voice agent by reading its transcript, you are evaluating a text agent that happens to have been spoken. The demo failure that made this click for me: a caller asks for a refund, the agent refunds, the transcript agrees with itself perfectly. The caller said "fifteen." The agent heard "fifty." Read the transcript and the call is flawless.
The fix is a truth field in the input: what the caller actually said, alongside what the STT heard. Without it, mis-hearing is undetectable by construction, and voiceeval has a test that documents exactly that limitation. From there it catches things a text eval scores as a perfect call: misheard numbers, consequential actions taken without a confirmation, dead air where callers hang up, responses so slow the call already failed. The lesson generalizes past voice: every modality has failures that are invisible in the representation you happen to be evaluating.
Layer 3: Record and replay
Once you have scorers, you need something to score against that reflects real behavior over time. That is replay.
Tracecase is CI for agents built on record and replay. You define a suite of agent test cases, your CI runs them against the current agent config and POSTs the results, and Tracecase diffs the new run against the previous run of the same suite. It computes what regressed (passed before, fails now) and what got flagged (any safety flag such as a tool call that was not allowed), and returns a shouldFail signal you wire straight into your CI exit code. The dashboard shows per-case REGRESSED and FIXED diffs with the offending output and tool calls, so a prompt bump that quietly breaks one case cannot slip through as an aggregate that still looks fine.
Layer 4: Trust individual answers with receipts
Regression gates protect the system over time. They say nothing about whether one specific answer, right now, can be trusted by someone who was not there.
answerproof attaches a cryptographically signed, tamper-evident receipt to every generated answer: which sources were retrieved, which the answer actually used, under whose permissions, with which model and parameters, plus a content hash of each source and a Merkle root over the retrieval set. Anyone can later verify a receipt independently, with nothing but the receipt and the library. It turns "trust our logs" into "verify it yourself," which matters when the leaked document or the compliance question shows up three months later.
I am careful about what it claims. A receipt proves integrity, source authenticity, set membership, and a transparent record of which claims are grounded in which sources. It does not prove truth. A well-grounded claim can still be wrong if the source is wrong, and the citation binding is n-gram overlap, an auditable signal, not a judge.
Layer 5: Observability over what agents actually did
The last layer is watching the runs you already have. When you fan out ten subagents and each returns a confident wall of text, generation is not the bottleneck, verification is, and you cannot verify what you cannot see.
agentrace reads Claude Code session transcripts, shows what your agents actually did, and flags the results you should not trust. No instrumentation, no SDK, because the sessions are already on disk. Every check comes from a real failure across roughly 150 research subagents: an agent concluding a company was not hiring because an API returned an empty list (that API returns empty with HTTP 200 for accounts that do not exist), hedged claims that hardened into facts downstream, twenty URLs cited and none opened, agents dying on session limits mid-sweep. Run against the session that motivated it, it flags 36 of 152 runs, and the most useful number is the 12 where I forgot to specify an output shape. Most agent tooling assumes the model is the problem. Often the prompt was.
The one honest caveat
None of these tools tell you what is true. That is not a gap I plan to close, it is the nature of the problem. evalgate tells you the score dropped, not that the new answer is wrong. agentrace and voiceeval are heuristics over text: they tell you what to go read, not what is true, and severity is deliberately conservative because a checker that cries wolf gets switched off, which is worse than no checker. answerproof proves an answer was grounded and unaltered, not that it was correct. Testing an AI system does not give you a green checkmark that means "correct." It gives you layered signals that catch getting worse, catch the invisible failure, and let someone else verify what happened. That is a smaller promise than unit tests make, and it is the honest one.
If any of this is useful, all five are open source at github.com/royalpinto007.
Top comments (1)
The representation-specific failure point is the most useful framing here. I would make each layer expose a concrete escalation artifact, not only a score: the replay diff, the receipt, or the exact unallowed tool call. That gives an operator a bounded place to investigate instead of converting an aggregate regression into another judgment call.