TL;DR. A quality gate that grades every pull request with an LLM-as-judge metric is buying a judge call per PR, per metric, forever. At forty PRs a day and a dozen judge-graded metrics, that is real money and, worse, a merge queue now coupled to a paid API's price, latency, and rate limits. I wired six open-source eval frameworks into the same GitHub Actions gate and watched the invoice, not the feature list. The split that matters for cost is simple: does the check run in-process and return an exit code, or does it call a model. Promptfoo assertions, MLflow's heuristic metrics, and Future AGI's deterministic metrics can gate without a single judge call. RAGAS ships deterministic metrics too (BLEU, ROUGE, string checks), but the RAG metrics you actually adopt it for are judge calls. DeepEval and Phoenix sit in the middle, judge-first by default but drivable in a cheaper mode. Rankings and the arithmetic below.
The invoice
I was not auditing cost. Finance was. Someone forwarded me a line item, a few hundred dollars against an OpenAI project key I did not recognize, tagged ci. It was the eval gate. We had added an LLM-as-judge relevance metric to the merge queue four months earlier, felt good about the coverage, and moved on. Nobody connected "we grade every PR with a model" to "we pay for every PR we grade."
A few hundred a month is noise against an engineering payroll. What bothered me was the shape of it: the bill grew with our merge volume, which means the better the quarter, the more the gate cost, and the gate itself did nothing you could not have paid for once. And the second I looked, I realized the merge queue now had a dependency nobody had reviewed. If that API rate-limited us at 9am on a busy day, PRs would stall on a gate that had nothing to do with the code in them.
So I did the boring thing and measured it.
The cost formula
The number that drives cost is not metric quality or accuracy. It is how many model calls one gate run makes.
A deterministic check makes zero. contains, equals, regex, is-json, a schema validator, a required-field assert, a golden-file diff. These run in-process, return in milliseconds, and cost nothing per PR. An LLM-as-judge metric makes one model call each, every run, and bills for the tokens.
So the monthly cost of a gate is close to:
PRs/day x judge calls per run x price per judge call x workdays
Everything else is a rounding error. The chart is that formula, nothing more.
[DIAGRAM: https://lh3.googleusercontent.com/d/1bVKW6DCpbvxco0nlIh6O06aJZLe_ttMh]
At forty PRs a day, a gate running a dozen judge-graded metrics adds roughly twenty dollars a month by this arithmetic (a stated ~$0.002 per call; check current token prices before you quote it). A single-judge gate is a few dollars. A deterministic gate is flat at zero, no matter how many PRs you merge. The absolute numbers are small. The point is the slope, and the fact that only one of these lines is flat.
The ranking
All licenses, metric counts, and defaults are as of mid-2026. Check each repo before you rely on any of this, because all six move.
Promptfoo (MIT). The cheapest to gate on, because its assertion library is deterministic by design: contains, equals, regex, is-json, starts-with, plus cost and latency asserts. An LLM-judge assertion is available but opt-in. You can build a real blocking gate that never makes a model call. CLI exit code, JSON output, maintained Action.
MLflow evaluate (Apache-2.0). mlflow.evaluate() ships heuristic metrics (exact match, token overlap, and similar) alongside optional genai judge metrics. Gate on the heuristic set and you pay nothing per run. The eval is logged next to the experiment, which is the reason to reach for it.
Future AGI (Apache-2.0). Its ai-evaluation SDK runs through one evaluate() call from Python or TypeScript and mixes deterministic checks with judge-based metrics. Gate on the deterministic ones and, like Promptfoo and MLflow, it makes no model call, so it lands on the cheap end. Gate on the judge metrics and it bills like the rest.
DeepEval (Apache-2.0). Its determinism comes from the runner. You inherit pytest exit codes and JUnit XML for free, which is why it is pleasant in CI, but most of its metric catalog is judge calls. Run it on plain pytest asserts and it is cheap. Run its default metrics on every PR and you are paying per metric, per PR.
Arize Phoenix (Elastic License 2.0). run_evals() in a script, a handful of evaluators, judge-based. The draw is tracing plus eval in one tool, not a deterministic gate. As a pure cost line it bills like any judge-first tool.
RAGAS (Apache-2.0). It does ship deterministic metrics (BLEU, ROUGE, chrF, string and tool-call checks), so a gate built on those makes no model call. But nobody adds RAGAS for BLEU. The reason it is on your stack is its RAG metrics (faithfulness, answer relevancy, context precision), and those are judge calls almost by definition. Gate on what you came to RAGAS for, and every check is a model call.
Beyond the dollars
The money was the small problem. Two bigger ones showed up once the gate depended on an API.
Rate limits became merge-queue outages. A judge-graded gate that 429s does not fail gracefully, it fails your PR, and the fix is retry-with-backoff logic you now own inside CI. A deterministic assert never 429s.
Price is not yours to control. We had modeled the gate at one token price. Model prices move, sometimes down, sometimes up, and a gate you pay per-run for is a line item that reprices without asking you. A gate that runs in-process is priced once, in compute you already have.
None of this means judge metrics are useless. I still run them. I just stopped putting them in the blocking path. Everything deterministic gates the merge. Everything judge-graded runs on the same PR, posts as a non-blocking comment, and a human reads it. The gate got cheaper and the signal did not get worse, because the judge was never a good binary gate anyway.
What I'd check first
If your CI bill has a mystery line item, or you are about to add an eval gate:
- Count the model calls in one gate run (metrics x PRs/day). If that number is not zero and you expected a fixed cost, that is your line item.
- Move every deterministic check you have (schema, required fields, forbidden strings, one golden diff) into the blocking path, and demote every judge metric to a non-blocking PR comment.
- Before you gate on any judge metric, ask whether it survives a 429 at 9am. If a rate limit would block a clean PR, it is a coupling you did not mean to buy.
Top comments (0)