Originally published at cchinchilla.dev.
More and more papers on LLM agents talk about production. One agent triages incidents for a cloud provider, another automates a release decision. I wanted to know what those papers test before they make that claim.
So my coauthors and I ran a systematic review. Eighty-eight studies, read in full and coded twice.
The short version is that they report testing what a benchmark run can show. Take the things that matter once an agent has been running for a while (reliability over time, recovery from its own failures, monitoring, resistance to attacks). Each goes unreported in more than 88% of the studies, and all four together in 73%.
What we reviewed
The review is out in Information and Software Technology: Production-oriented evaluation and testing of LLM-based agents: A systematic literature review. It follows the Kitchenham–Charters guidelines for systematic reviews in software engineering and is reported against the PRISMA 2020 checklist.
We searched Scopus, Web of Science, IEEE Xplore and the ACM Digital Library for work from 2023 to March 2026, then followed citations, and the 88 studies that made it through make up the review corpus. A later search, extended to September 2026, found 314 more. We kept those apart as an update sample with its own analysis.
"Production-oriented" needed a definition we could defend. Plenty of papers say real-world in the abstract and then run on a public benchmark, and if you take the abstract at its word you end up reviewing benchmark papers with a deployment story attached. So we built an instrument for it, the Real-World Relevance Signals (RWRS).
It has twelve signals of documented production evidence, among them real users, real data, deployment infrastructure, operational metrics and CI/CD integration. Twenty-three calibration rules say what doesn't count. A Docker container used to isolate an experiment isn't a production environment; an API bill mentioned as a limitation isn't a production challenge.
Every study needed at least two of the twelve signals to get in. Strong production evidence takes five or more, and at least one of them has to be real users, reliability or safety. Thirty-five of the 88 studies have it. How far each system got is a separate question. Forty-six are prototypes, 18 are pilots and 24 (27%) report production or large-scale deployment.
Four gaps, almost everywhere
Start with what the studies do test. We recorded which dimensions each study evaluates, meaning qualities such as task completion, latency or tool calling. The average study evaluates 2.85 of them. Accuracy benchmarking and end-to-end task completion dominate, the kind of thing you can measure offline with a script.
Then the gaps. For each study we also listed the dimensions that mattered for what it claimed but that it never reported evaluating, 4.98 on average. Every study in the review claims production relevance, so we held all 88 to the dimensions a running system needs, such as long-term reliability and failure recovery, prototypes included. Other dimensions depend on the system, and a small internal tool wouldn't be marked down for skipping cost at scale.
The four that lead the list are below, for the review corpus and the update sample.
| Dimension | Missing, review corpus (88) | Missing, update sample (314) |
|---|---|---|
| Long-term reliability | 98% | 96% |
| Monitoring coverage | 91% | 86% |
| Failure recovery | 90% | 82% |
| Adversarial robustness | 89% | 87% |
| All four | 73% | 61% |
The original post has an explorer for this table that doesn't survive being pasted into a feed: it splits the review corpus by deployment stage, production evidence and study quality, with a 95% confidence interval on each rate.
Each one is missing in most studies at every deployment stage, from prototype to production. All four are missing together in 73% of the studies (95% confidence interval 62–82%) and in 20 of the 24 production deployments. Long-term reliability goes unreported in every one of those 24.
Two objections are worth answering. The first is that the gaps only reflect weak papers. They don't: among the 56 studies we rated high or moderate quality, 70% still miss all four.
The second is that the field may have moved on since March. The 314 studies in the update sample are a broader mix, with more safety alignment work and more hierarchical agents. Each of the four dimensions is still missing in at least 82% of them, and all four together in 61%. That's lower than the 73% in the review corpus. The paper traces part of the difference to the extra safety alignment work, and the update's coding read adversarial robustness differently. Under that reading the review corpus misses all four in 67% of its studies, and the difference from the update is no longer significant. Within the update sample, studies published before and after March 2026 show similar rates (60% and 62%).
Testing more of what a benchmark can measure doesn't close the gaps either. Of the 63 studies that test both accuracy and end-to-end task completion, 49 leave out all four.
A single run can't assess most of what the four need: reliability over months, recovery from an injected failure, whether your monitoring would notice it, resistance to inputs crafted to mislead the agent. Most of it needs a system that keeps running and changing while you watch it.
Be fair to these teams, though. "Missing" means not reported. A team may well test things it never writes up, since papers have page limits, incident reviews rarely get published and nobody gets a citation for the regression suite that quietly caught a bad prompt change. The review corpus also leans recent and narrow. Of the 88 studies, 68% are from 2025 and 49% come from cloud operations and software engineering.
Any search misses studies, so we estimated how many ours missed. Measured against a broader Scopus search, our query found about 86% of the eligible studies whose main contribution is evaluation and 63% of all eligible ones. The studies it missed show the same gaps.
Automated, but outside the pipeline
The tooling to run evaluations automatically exists, and teams use it. Fifty-four of the 88 studies describe fully automated evaluation, and six of those wire it into CI/CD. The rest don't get past the script. Across the whole corpus, 8 studies (9%) implement or contribute a CI/CD pipeline, those six among them. In the update sample the share holds at 9%.
Monitoring lives in its own stack as well. Of the 23 studies with observability or full monitoring infrastructure, only two also wire evaluation into a CI/CD pipeline and only one tests the agent's reliability under load.
So in the studies, the evaluation suite runs before release and the dashboards run after it, and the review finds the two largely decoupled. The paper notes that existing approaches specialise in one mode, offline or deployed. It's the open loop from the Article 72 post.
Safety, and the one association that held throughout
Formal safety mechanisms (red-teaming, adversarial testing, compliance checks, prompt-injection defence) appear in 17 of the 64 prototype and pilot studies and in 2 of the 24 production deployments. That's 27% against 8%, so the deployed systems report fewer of them. The difference isn't statistically significant. I read it as a pattern worth watching.
The 35 studies with strong production evidence report guardrails that require human approval far more often than the other 53, 31% against 6% (Fisher's exact p = 0.002). We re-ran the comparison under ten RWRS variants. Some move the bar to four, six or seven signals, and others drop the signals that overlap the outcomes being compared, safety among them. Human approval stays significantly more frequent under every one (p ≤ 0.03). The difference also shows in the update sample (p = 0.006) and survives adding the 12 eligible studies from a sample of the broader Scopus search (p = 0.001).
Where the documented production evidence is strongest, studies report a person signing off on risky actions more often. CI/CD shows up more in those studies too, significantly in the main comparison but in only three of the ten variants, and again in the update sample.
Multi-agent systems and untested protocol conformance
Multi-agent and hierarchical architectures make up 39% of the corpus. Only 12 of those 34 studies evaluate how the agents coordinate. I'd argue coordination is where a plausible answer from one agent turns into a wrong decision in the next.
MCP (Model Context Protocol) and A2A (Agent2Agent) now define contracts you could test against, such as message schemas, capability declarations and message ordering. Twelve of the 88 studies mention MCP or A2A, and none reports testing conformance to either protocol. One study in the update sample does evaluate MCP interoperability against a contract.
An LLM judge is one more model whose output something downstream trusts. Only 5 of the 88 studies combine objective measurement, an LLM judge and human evaluation, and the paper lists calibration across the three as an open problem. That's the problem behind who calibrates the judge.
What to test before you ship
The paper turns its findings into a five-step checklist for teams preparing an agent for deployment. Here it is, with a note on each step.
- Document your production evidence. Go through the twelve RWRS signals (Table 2 of the paper) and write down which ones your evaluation covers. Real users or simulated ones? Production data or a benchmark?
-
Test the four gaps explicitly. Long-term reliability, failure recovery, monitoring coverage and adversarial robustness each need their own tests. Start with the cheapest one, a failed tool call injected on purpose, and check that the agent recovers and that your monitoring notices.
The original post has a self-check at this step: tick the RWRS signals your evaluation documents and the gaps it tests. It scores the production evidence the way the review did, from below the bar to strong, and names the gaps still untested.
Put a quality gate in CI/CD. At least one gate that blocks promotion when a metric regresses. Eight studies in the corpus already run evaluation inside a CI/CD pipeline, despite non-deterministic outputs and the cost of running it on every commit.
Add a formal safety mechanism, and human approval for high-risk capabilities. Red-teaming, prompt-injection defence or compliance checks. Of the practices the paper compares, human approval is the one most consistently tied to strong production evidence.
Observe in production and feed it back. Logs, traces and alerting, plus periodic re-evaluation. When something drifts, turn it into a case in the offline suite.
If I had to pick one change from the review, I'd keep the evaluation suite running after release, inside the delivery pipeline, and let production write its next test.
The paper is open access at its DOI. The replication package, with the data, the extraction and a notebook with 595 automated checks, is on Zenodo under CC BY 4.0.
Written with AI assistance. See AI policy.
Top comments (0)