In July 2025, an AI coding agent deleted a founder's entire production database. This happened during a declared code freeze. Then, to cover its tracks, the agent fabricated 4,000 fake users, generated false unit-test reports, and initially told its operator that rollback was impossible.
This is not a research paper scenario. It happened to a real founder, on a real system, with real data. The Register covered the incident in full.
The scale makes it harder to ignore. In April 2026, Google Cloud published a catalog of 1,302 real-world generative AI use cases from named enterprises: banks, hospitals, manufacturers, retailers, government agencies. Eighteen months earlier, the same catalog listed 101 entries. AI is not coming. It is here, running in production, touching consequential decisions.
The discipline that keeps it inside the rails is not keeping pace. That discipline is EvalOps.
AI is everywhere, evaluation is not
A survey of 1,250+ AI engineers found that only 57.4% of teams evaluate their AI applications at all. Of those who do, 75.6% rely on manual testing and reviews.
Nearly half the industry is shipping AI into production with no structured evaluation. Most of the rest are testing by hand.
AI deployment is climbing fast. Evaluation practice is moving too, just not at the same pace — and that gap is where incidents like this happen, not because engineering teams are careless, but because there's no systematic framework to catch failures before they reach production.
The foundation model labs have reached the same conclusion. In August 2025, Anthropic and OpenAI published results from a joint alignment evaluation, cross-testing each other's models across scheming, sycophancy, jailbreak resistance, and instruction hierarchy, across 13 multi-step agentic environments, with Anthropic partnering with Apollo Research. If the labs building the models invest this heavily in evaluation, that tells you something about how seriously application teams should take it.
EvalOps: a new discipline for production AI
EvalOps is a set of practices, tools, and cultural norms that bridges the gap between AI engineering, product, QA, and compliance teams. It keeps AI systems trustworthy in production by making evaluation continuous, shared, and embedded into the development lifecycle through automation, joint ownership, and learning from real-world behavior.
One thing that gets missed when EvalOps is framed as an engineering concern: it isn't one. How responsibility divides varies by organization, but the pattern is consistent. Product typically owns what 'correct' looks like for users. QA owns the adversarial edge cases and boundary conditions. Legal and compliance own the hard limits on what the system can never say or do. When evaluation sits only with engineering, all of that context stays locked in documents and meetings, and the gaps show up in production instead.
In a mature EvalOps practice, requirements graduate into test cases, compliance constraints harden into automated assertions, and security findings get codified as regression tests. The feedback loop runs across the entire organization rather than circling inside engineering. That cross-functional reach is what makes it operational rather than just technical.
The difference shows most clearly when a model provider pushes an update. Teams without EvalOps don't know what that change means for their application's reliability or robustness — they wait for support tickets to tell them. Teams with EvalOps re-run the suite and see exactly which behaviors moved, before any user is affected.
Why EvalOps now
Four failure modes make AI applications structurally different from conventional software:
Non-determinism. The same input rarely produces the same output twice. A test that passes today may fail tomorrow with no code change. Deterministic assertions — the foundation of unit testing — are the wrong tool.
Agentic cascades. In multi-step systems, decisions compound. An early misstep, a wrong tool call or a misread constraint, propagates across subsequent steps in ways end-to-end tests won't surface until real users find them.
Silent model drift. Provider updates land continuously, with minimal notice. A routine model change can shift tone, accuracy, or scope behavior with zero changes to your own code.
Semantic correctness. Evaluating whether an agent honored a constraint like "don't give legal advice" requires understanding intent, not counting word overlap. String-matching scores give you a number that says nothing about whether the application actually followed its rules.
The incidents below each map to one of these gaps.
Shipping AI applications reliably is its own discipline now. The model is one variable. The requirements, guardrails, release gates, and feedback loop have to be engineered with the same rigor as the model itself. That is what EvalOps is.
— Nicolai Bohn, founder, Rhesis AI
Conventional testing is structurally blind to these failure modes. Unit tests don't break when behavior shifts. String-matching metrics don't detect confident hallucinations. More manual testing won't fix this — what's needed is evaluation infrastructure that runs continuously, scores semantically, and enforces behavioral contracts at every release.
Courts aren't waiting for the frameworks to catch up. In May 2026, Germany's Higher Regional Court of Hamm ruled that a cosmetic clinic was liable for false specialist titles invented by its own chatbot — and held the operator responsible even when the model was never given incorrect source data. The principle: if you deploy it, you own what it says. The EU AI Act's obligations for high-risk AI systems take effect August 2, 2026, Gartner projects AI regulation will reach 75% of the world's economies by 2030, and 50% of content-risk roles are predicted to migrate from legal into AI engineering by 2028. Evaluation infrastructure is shifting from a competitive edge to a compliance baseline.
The EvalOps lifecycle
EvalOps is not a phase. It is a closed loop driven by everything that forces the system to change: production failures, expanding scope, new product features, user complaints, model upgrades, and regulatory updates. Each stage feeds the next, and the last stage feeds back into the first:
EvalOps is a closed loop. Production failures don't end in a Slack channel, they become tomorrow's test cases.
Design. Define the behavioral contract before writing agent logic: what the application must do, what it must never do, and what success looks like. These become the metrics that all subsequent evaluation enforces.
Develop. Evaluation runs continuously during development, not as a final check. The important thing is that the suite grows alongside the application — whether it lives in a repo, a workspace, or a managed platform.
CI release gate. No deployment happens without passing defined quality thresholds. The gate is automated and enforced by policy.
Staging and shadow mode. New versions run against real production traffic before full rollout. Test cases are derived from real user behavior, so the eval suite reflects what the system actually has to handle.
Production monitoring and continuous learning. Every interaction is a data point. Failures are flagged, investigated, and added to the eval suite. The next release is gated against the failures the previous release actually produced.
Who owns what
Every phase in the EvalOps lifecycle produces a shared artifact — a behavioral contract, a test suite, a scorecard, a set of flagged traces. The table below shows where each function plugs in so the loop closes across the organization, not just inside engineering.
What an EvalOps stack actually contains
A working EvalOps stack has seven capabilities: behavioral requirements grounded in real product context, test scenarios at scale, named owners for every quality dimension, semantic graders, release gates wired into CI/CD, production trace monitoring, and a closed feedback loop. Miss any one of them and the team is back to depending on memory or whoever shouts loudest during release week. In practice:
Behavioral requirements, not invented test cases.
The starting point is what the application must and must not do, written in plain language. Most organizations already have this context in support guides, compliance policies, brand playbooks, and engineering specs. An EvalOps workflow connects to that existing knowledge directly rather than asking teams to write test cases from scratch.
Test scenarios at scale across every relevant dimension.
From behavioral requirements, teams generate scenarios that cover functional correctness, linguistic variation, tone, security boundaries, compliance constraints, and adversarial inputs including jailbreaks and prompt injection. Multi-turn and agentic scenarios need special attention: the failures that matter most in agents often only appear across several conversation turns, not in a single response.
Named owners for every quality dimension.
Each metric that matters needs a specific person accountable for it. Accuracy might be owned by the product team. Safety by legal or compliance. Scope adherence by QA. Without that assignment, quality scores sit in a dashboard and nobody decides what to do when one drops. Ownership is what turns a metric into a gate.
Semantic graders and rubrics.
A mix of automated assertions (regex, semantic similarity, LLM-as-judge, tool-call verification) and explicit human review for cases where automation is not yet trustworthy. The team decides where automation is reliable and where human judgment stays mandatory.
Release gates in CI/CD.
The mechanism that turns scorecards into actual release controls. Without gates, evaluation is advisory. If a failed score changes nothing, the team has reporting, not EvalOps.
Production trace monitoring.
Scoring real interactions as they happen, at the component level: retrieval quality, tool selection, scope adherence, escalation behavior. Final output scores miss most of what goes wrong in agentic systems.
Closed feedback loop.
Production failures converted into permanent test cases, not one-time lessons. The next release is gated against the failures the previous release actually produced. That is what makes the eval suite grow stronger over time rather than drifting away from production reality.
Plenty of good tools cover parts of this stack: production tracing, prompt management, experiment tracking, judge-based scoring. Most are built primarily for developers working alone, which leaves the cross-functional handoffs — product owning behavioral requirements, compliance owning safety thresholds, QA owning adversarial scenarios — outside the tool. For a side-by-side comparison of what each covers and where each falls short, see 7 LLM evaluation and testing tools compared.
Start before the incident
Google Cloud catalogs 1,302 AI deployments. An AI coding agent deletes a production database. Both facts are true at the same time. The question is not whether to invest in evaluation. It's whether to start before the incident or after.
Gartner projects the market for AI governance platforms will surpass $1B by 2030, up from $492M in 2026. Broader estimates of the AI assurance technology category (covering evaluation, monitoring, red-teaming, and adjacent tooling) run as high as $276B by 2030. Either number tells the same story: the industry is betting that evaluation infrastructure becomes as foundational to AI as CI/CD is to software. DevOps followed the same arc: first a competitive edge, then an operational baseline. EvalOps is on that trajectory now, with regulators accelerating it.
Rhesis is the platform built to run your EvalOps cycle end to end. Product managers define behavioral requirements in plain language. QA generates test scenarios at scale, including adversarial and multi-turn cases. Engineering wires up CI gates and production traces. Compliance signs off against the same artifact. One loop, one source of truth, every function inside it.
Dr. Nicolai Bohn
Original blog
Top comments (0)