Agent evals in CI/CD: 4 gates that catch silent failures before customers do (2026)
Summary. LangChain's State of Agent Engineering, published 12 June 2026 from 1,340 responses, found that 89% of organisations have some form of agent observability but only 52.4% run offline evaluations on test sets. Among teams already in production the split is starker: 94% have observability and 71.5% have full tracing. Observability tells you what your agent did after it did it. An eval gate stops the build. That asymmetry is why agent regressions reach customers quietly: nothing crashes, no alert fires, and the failure is a plausible-sounding wrong answer. Quality is the top barrier to production for one third of respondents, ahead of latency at 20%. This guide builds the gate in four stages, with real costs: a 150-case judge run costs about $0.27 on gpt-5.4-mini versus $1.80 on gpt-5.5 at OpenAI's July 2026 list prices, and the tools to wire it (DeepEval, Braintrust, GitHub Actions) ship defaults that will silently let a failing suite pass.
The gap: almost everyone watches, half of them test
The numbers from LangChain's survey are worth sitting with, because they describe a specific engineering failure and not a general lack of maturity. The survey ran from 18 November to 2 December 2025 and collected 1,340 responses, 63% of them from technology companies.
57.3% of respondents have agents running in production, up from 51% the prior year, with another 30.4% actively building toward deployment. So agents are shipping. Of those already in production, 94% have observability in place and 71.5% have detailed tracing that shows individual steps and tool calls.
Evals lag badly. Just 52.4% run offline evaluations against test sets. Online evals sit at 37.3%, rising to 44.8% among teams with production agents. Even among production teams, 22.8% run no evals at all.
The tooling split matters too. Of teams running any evals, human review leads at 59.8% and LLM-as-judge follows at 53.3%. Traditional ML metrics like ROUGE and BLEU see limited use, which makes sense for open-ended agent output where several answers are equally valid.
Read those two lines together. 94% of production teams can reconstruct what an agent did. Roughly half can tell you, before merge, whether a change made it worse. That is the gap where silent failures live.
Why agent failures are silent
A conventional web service fails loudly. It throws a 500, the error rate spikes, the pager goes off. An agent that regresses does none of this. It returns HTTP 200 with a fluent, well-formed, confidently wrong answer.
Three properties make this worse than ordinary flakiness.
Non-determinism compounds. Every model call inside an agent has its own variance. Once a tool result or a retrieval changes, the downstream state diverges, so the same input can produce a different trajectory on the next run. A single passing test proves very little.
The failure surface is semantic, not structural. Your JSON schema validator is happy. The tool call is well-formed. The refund amount is wrong. Exact-match assertions, the backbone of normal CI, cannot see this, because the model returns varying text that is all technically valid.
Regressions arrive without a deploy. A prompt tweak, a retrieval index rebuild, or a provider silently updating a model checkpoint can move behaviour without a single line of your code changing. LangChain's respondents at 10,000-plus employee organisations named hallucinations and output consistency as their biggest quality challenge in write-in answers.
This is the case for a gate rather than a dashboard. A dashboard tells you about the regression on Tuesday. A gate refuses the merge on Monday.
The four gates
Run them in this order. Cost and speed increase down the list, so fail fast and cheap.
| Gate | What it checks | Blocks the merge? |
|---|---|---|
| 0. Unit tests | Your code: parsers, routers, retries, schema | Yes, as today |
| 1. Assertions | Deterministic properties of agent output: schema, required tool call, cost and latency budget, forbidden strings | Yes |
| 2. Golden set | A small curated dataset of core features, past bugs, known edge cases | Yes |
| 3. LLM-as-judge | Semantic qualities assertions cannot reach: factual correctness, completeness, tone | Yes, on borderline cases |
| 4. Baseline compare | This run against the last marked-good run, per scorer | Yes, on regression |
Gate 1: deterministic assertions run first
The strongest advice in this area comes from Hamel Husain, founder of the ML consultancy Parlance Labs and co-creator, with Shreya Shankar, of the AI Evals for Engineers and PMs course. Writing on the difference between CI and production evaluation, he is blunt about what belongs in CI: "Favor assertions or other deterministic checks over LLM-as-judge evaluators."
That instruction cuts against most vendor marketing, and it is correct. A deterministic assertion is free, instant, and never disagrees with itself. Before you pay a judge model to reason about quality, check the things that have exactly one right answer:
def test_refund_agent_calls_the_right_tool(golden):
trace = run_agent(golden.input)
# Structural properties: no model needed
assert trace.tool_calls[0].name == "lookup_order"
assert trace.output_json["currency"] in {"INR", "USD"}
assert trace.output_json["refund_amount"] <= golden.order_total
assert " language model" not in trace.output_text
assert trace.total_tokens < 8000 # cost budget
assert trace.wall_clock_ms < 12000 # latency budget
Every one of those catches a real production bug class, and none of them costs a token. Refund amounts above the order total, a router calling the wrong tool, a prompt leak, a runaway loop burning 40,000 tokens on one request: assertions catch all of it deterministically.
Gate 2: a small, curated golden set
Husain's guidance on CI datasets is specific: "Test datasets for CI are small (in many cases 100+ examples) and purpose-built. Examples cover core features, regression tests for past bugs, and known edge cases. Since CI tests are run frequently, the cost of each test has to be carefully considered (that's why you carefully curate the dataset)."
The instinct to build a 5,000-case eval set is wrong for CI. It is slow, it is expensive on every pull request, and it dilutes signal. A hundred well-chosen cases that each encode a real failure you have actually seen will catch more regressions than thousands of synthetic ones.
The rule that keeps the set honest: every production incident becomes a test case. Husain describes CI and production monitoring as complementary systems, where new failure patterns found in production get added to the CI dataset to prevent recurrence. That is the loop. Without it your golden set decays into a museum of the bugs you had in month one.
DeepEval models this as an EvaluationDataset of Golden objects, loaded from Confident AI, a CSV, a JSON file, or straight from code:
from deepeval.dataset import Golden, EvaluationDataset
dataset = EvaluationDataset(goldens=[
Golden(input="What is the return window for product X1000?"),
Golden(input="Refund my order 88213 to a different card"), # ticket #4471
])
Gate 3: LLM-as-judge, only where assertions cannot reach
Some qualities have no deterministic check. Whether an answer is factually supported by the retrieved context, whether it actually completed the user's task, whether it held the right tone: these need a model to grade them.
DeepEval, which is Apache 2.0 licensed, exposes these as metrics you attach to a test case, including TaskCompletionMetric, AnswerRelevancyMetric, and FaithfulnessMetric (which requires retrieval_context to be populated). Multi-turn agents get TurnRelevancyMetric, KnowledgeRetentionMetric, RoleAdherenceMetric, and ConversationCompletenessMetric.
The distinction that matters for a gate is between an evaluator and a guardrail. Husain draws the line clearly: guardrails run inline, before the response reaches the user, and can redact, refuse, or regenerate, so their false positives are production bugs. Evaluators run after a response is produced, feed dashboards and regression tests, and do not block the original answer. Judges are usually run asynchronously or in batch precisely because they are heavy. Inline judging is possible only when the latency budget allows it.
Keep the judge scoped. It should grade what assertions cannot, on the subset of cases where structural checks already passed.
| Property | Assertion | LLM-as-judge |
|---|---|---|
| Marginal cost | Zero | $0.0005 to $0.012 per case, July 2026 list prices |
| Determinism | Exact, repeatable | Varies run to run; needs repeats to trust |
| Catches wrong tool call | Yes | Yes, but you paid for it |
| Catches unsupported claim | No | Yes |
| Catches wrong tone | No | Yes |
| Fails a build cleanly | Yes | Only above a set threshold |
| Disagrees with itself | Never | Position, verbosity and self-preference bias are known issues |
Gate 4: compare against a marked baseline
A score in isolation means nothing. 0.83 is only information if you know the last good run scored 0.80.
Both major tools implement this. DeepEval lets you designate a run as the official baseline that future runs are compared against for regressions:
deepeval test run test_llm_app.py --official
The docs are explicit that this is "typically done on your main branch so every baseline reflects merged, production-ready code". Marking a run official requires a CONFIDENT_API_KEY, since official runs live on Confident AI.
Braintrust's GitHub Action does the equivalent and posts the delta straight into the pull request. Its own README shows the report shape:
| Score | Average | Improvements | Regressions |
|---|---|---|---|
| Levenshtein | 0.83 (+3pp) | 8 | 4 |
| Duration | 1s (0s) | 16 | 1 |
That table is the whole point of gate 4. An average that moved up three percentage points while four individual cases regressed is exactly the signal a single aggregate score hides.
What it costs to judge every pull request
Judge cost is the reason teams skip gate 3, so price it properly rather than guessing.
Take a realistic CI suite: 150 curated cases, roughly 1,500 input tokens per judge call (rubric, input, agent output, retrieved context) and about 150 output tokens for the verdict and its reasoning. That is 225,000 input and 22,500 output tokens per run. Against OpenAI's published list prices as of July 2026, and assuming 40 pipeline runs a day on a busy repository:
| Judge model and tier | Price per 1M tokens (input / output) | Cost per 150-case run | Cost per month at 40 runs/day |
|---|---|---|---|
| gpt-5.5, standard | $5.00 / $30.00 | $1.80 | $2,160 |
| gpt-5.5, Batch API | $2.50 / $15.00 | $0.90 | $1,080 |
| gpt-5.4-mini, standard | $0.75 / $4.50 | $0.27 | $324 |
| gpt-5.4-mini, Batch API | $0.375 / $2.25 | $0.14 | $162 |
| gpt-5.4-nano, standard | $0.20 / $1.25 | $0.07 | $88 |
| gpt-5.4-nano, Batch API | $0.10 / $0.625 | $0.04 | $44 |
Prices are OpenAI list as of July 2026; the per-run and per-month figures are calculated from the token assumptions above, not measured from a specific repository.
Three things fall out of that table.
The frontier judge is 6.7 times the cost of the mini judge for the same suite, and $2,160 a month is real money for a check that runs on every push. Most CI grading does not need frontier reasoning, because you have already spent gates 1 and 2 removing the cases a small model would get wrong.
The Batch API halves the bill but cannot gate a pull request, because batch jobs are asynchronous and a merge gate has to answer now. Batch belongs in the nightly full-suite run, not the blocking one.
Prompt caching is the real lever, and it is underused. The rubric is identical across all 150 cases, so it is the same prefix every time. Cached input on gpt-5.4-mini is $0.075 per 1M against $0.75 standard, a 10x reduction on the largest part of the payload. Braintrust builds this into its action: setting use_proxy (which defaults to true) points OPENAI_BASE_URL at https://braintrustproxy.com/v1, which its README says "will automatically cache repetitive LLM calls and run your evals faster".
The real cost of an eval gate is rarely the tokens. It is the afternoon someone spends curating the hundred cases that matter.
The defaults that quietly disable your gate
This is the part that turns a green pipeline into a false sense of safety, and both popular tools ship a version of it.
Braintrust's eval-action has a terminate_on_failure option. Its README states the default plainly: "Either true or false. If set to true, the evaluation process will stop when an error occurs. Defaults to false." A team that copies the quickstart YAML gets a workflow that runs evals, posts a nice comment with scores, and does not stop on error. The check is decorative until you change it.
DeepEval has the same trap at the API level, and its FAQ is refreshingly direct about it: "assert_test() is built for CI/CD and raises an assertion error when a metric falls below its threshold, while evaluate() is for scripts and notebooks and collects results without failing. Use assert_test() for gating."
If your CI calls evaluate(), your build cannot fail. Ever. You have bought a dashboard and called it a gate.
The same document warns against reaching for plain pytest: "The plain pytest command works but is highly not recommended. deepeval test run adds a range of functionalities on top of Pytest for unit-testing LLMs, enabled by 8+ optional flags", covering async behaviour, error handling, repeats, and identifiers.
Check three things in your own pipeline this week. Does a deliberately broken agent fail the build? Are you calling the asserting function rather than the collecting one? Does an error in the eval process stop the run, or get swallowed? If you have never watched your gate go red on purpose, you do not have a gate.
A working GitHub Actions setup
DeepEval's own CI/CD documentation gives the shape. This is their pattern, with the ordering discipline from gates 1 through 4 applied:
name: Agent evals
on:
pull_request:
branches: [main]
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Install dependencies
run: |
curl -sSL https://install.python-poetry.org | python3 -
echo "$HOME/.local/bin" >> $GITHUB_PATH
poetry install --no-root
# Gate 1: deterministic assertions. Cheap, fast, no tokens.
- name: Assertions
run: poetry run pytest tests/test_assertions.py
# Gates 2 and 3: golden set plus judged metrics. Fails on threshold.
- name: Agent evals
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
run: poetry run deepeval test run test_llm_app.py
The test file itself, following DeepEval's documented pattern for an instrumented agent:
import pytest
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe, update_current_trace
dataset = EvaluationDataset(goldens=[
Golden(input="What is the return window for product X1000?"),
])
@observe()
def my_ai_agent(query: str) -> str:
answer = support_agent.run(query)
update_current_trace(input=query, output=answer)
return answer
@pytest.mark.parametrize("golden", dataset.goldens)
def test_llm_app(golden: Golden):
my_ai_agent(golden.input)
assert_test(golden=golden, metrics=[TaskCompletionMetric()])
Two details worth noting. You do not need a Confident AI account for the gate itself: DeepEval's FAQ confirms that an LLM judge key such as OPENAI_API_KEY runs the tests entirely locally, and CONFIDENT_API_KEY is optional and only sends results to the cloud. And it is not GitHub-specific. Any provider that runs a shell step works, including GitLab CI, CircleCI, and Jenkins.
Once the app is instrumented with @observe, you can attach metrics to individual spans rather than the whole trace, grading retrievers, tool calls, and sub-agents separately. That is where a failing gate stops being a mystery: instead of "the answer was bad", you get "the retriever returned nothing for this query".
India-specific considerations
For teams building agents from India, or serving Indian users, two things change the calculation.
Data residency has a price now. OpenAI's pricing page states that regional processing endpoints for data residency "are charged a 10% uplift for models released on or after March 5, 2026, that are eligible for data residency". If your eval harness ships customer conversations to a judge model, those payloads carry the same personal data your production path handles. Treat them under the same Digital Personal Data Protection Act 2023 (DPDP) controls you apply in production, rather than assuming a test suite sits outside them. Applied to the table above, a gpt-5.4-mini gate at $324 a month becomes roughly $356 with regional processing. That is a rounding error against a compliance finding.
The cheaper answer is often to not send the data at all. Gate 1 assertions run entirely inside your own CI runner. A golden set built from synthesised or redacted inputs, rather than raw production transcripts, keeps most of your eval suite outside the judge model's context. This is where the ordering of the four gates pays off twice: the deterministic checks that cost no tokens also move no personal data.
For teams designing agent systems against Indian regulatory requirements, our guides on AI agent security and prompt injection guardrails and enterprise AI agent governance layers cover the surrounding controls. If judge spend is your constraint, free tools to measure LLM costs and our LLM token counter help you size the bill before you commit to a suite.
What to do this week
Start with the smallest thing that can go red. Pick the three failures that most recently reached your users, write them as deterministic assertions, and put them in the pipeline. Confirm the build fails. Then add a golden set from the next ten incidents, and only then attach a judge to the qualities the assertions cannot see.
The 52.4% number from LangChain's survey is not a story about tooling gaps. Every tool described here exists, is documented, and is largely free to start. The gap is that observability can be bought and switched on, while evals have to be curated by someone who understands the product. That work does not have a vendor.
FAQ
What is an agent eval gate in CI/CD?
An eval gate runs a curated test suite against your agent on every pull request and fails the build when quality drops below a threshold. Unlike observability, which reports what happened after deployment, a gate blocks the merge. LangChain's June 2026 survey found only 52.4% of teams run these offline evaluations.
Why do agent failures go unnoticed without evals?
Agents fail semantically, not structurally. The response returns HTTP 200, the JSON validates, and the tool call is well-formed, but the content is confidently wrong. No error rate spikes and no alert fires. Exact-match assertions used in normal CI cannot detect this, because valid agent output varies between runs.
How many test cases should a CI eval suite have?
Hamel Husain of Parlance Labs recommends small, purpose-built CI datasets, in many cases around 100 examples, covering core features, regression tests for past bugs, and known edge cases. Because CI runs frequently, each test carries a cost, so curation beats volume. Add a case for every production incident you find.
Should I use LLM-as-judge for every CI check?
No. Husain advises favouring assertions or other deterministic checks over LLM-as-judge evaluators in CI. Assertions cost nothing, run instantly, and never disagree with themselves. Reserve the judge for qualities assertions cannot reach, such as factual support, task completion, and tone, on cases that already passed structural checks.
What does it cost to run judged evals on every pull request?
At OpenAI's July 2026 list prices, a 150-case suite using roughly 1,500 input and 150 output tokens per case costs about $0.27 per run on gpt-5.4-mini and $1.80 on gpt-5.5. At 40 runs a day that is $324 versus $2,160 monthly. Prompt caching cuts input cost tenfold.
What is the difference between assert_test and evaluate in DeepEval?
DeepEval's documentation states that assert_test() is built for CI/CD and raises an assertion error when a metric falls below its threshold, while evaluate() is for scripts and notebooks and collects results without failing. Use assert_test() for gating. Calling evaluate() in a pipeline means your build can never fail.
Do I need a paid platform to run agent evals in CI?
No. DeepEval is Apache 2.0 licensed and its FAQ confirms that providing an LLM judge key such as OPENAI_API_KEY runs tests entirely locally. A CONFIDENT_API_KEY is optional and only sends results to the cloud, though marking an official baseline run for regression comparison does require it.
Does DPDP affect how I run agent evals in India?
If your eval suite sends customer conversations to a judge model, that payload carries the same personal data as your production path. Treat it under the same DPDP 2023 controls you apply in production. OpenAI charges a 10% uplift for regional data residency processing on eligible models released from 5 March 2026.
How eCorpIT can help
eCorpIT builds and hardens production AI agent systems for teams that have moved past the prototype and need the deployment path to be safe. Our senior engineering teams set up the eval harness described here: deterministic assertion suites, curated golden sets drawn from your real incident history, judged metrics scoped to what assertions cannot check, and a baseline comparison wired into your existing pipeline on GitHub Actions, GitLab CI, or Jenkins. We design these systems aligned with DPDP 2023 requirements, keeping personal data out of judge payloads where the check can be made deterministically instead. If your agents are in production without a gate, talk to our team.
References
- LangChain, State of Agent Engineering, 12 June 2026
- Hamel Husain, LLM Evals: Everything You Need to Know
- Hamel Husain, Your AI Product Needs Evals
- DeepEval documentation, Unit Testing in CI/CD
- Braintrust, eval-action GitHub Action repository
- OpenAI API pricing
- Parlance Labs, team
- DeepEval documentation, Flags and Configs
- DeepEval documentation, Component-Level Evals
- DeepEval documentation, Metrics introduction
- DeepEval, LLM-as-a-Judge in 2026: evaluation techniques and best practices
- Braintrust eval on GitHub Marketplace
- Maven, AI Evals for Engineers and PMs by Hamel Husain and Shreya Shankar
- DeepEval guides, Regression Testing LLM Systems in CI/CD
Last updated: 16 July 2026.
Top comments (0)