DEV Community

Cover image for A Practical Cost Model for LLM Evaluation Pipelines
Eric Thompson
Eric Thompson

Posted on

A Practical Cost Model for LLM Evaluation Pipelines

In our illustrative model, a first year of evaluation costs about $29,000 for an internal assistant, $96,000 for a customer-facing support agent, and $365,000 for a high-stakes application in a regulated field. Later years run lower, at about $19,000, $64,000 and $174,000, once the test set is built.

The model calls are the cheap part. Running a 1,500-case test set through generation and an LLM judge costs about $26 per run at batch prices. Across all three programs, compute is only 2–6% of the first-year cost. The other 91–95% is people: experts labeling test cases, reviewers reading real outputs, and engineers keeping the pipeline running.
So the budget question is not "how many runs can we afford" but "whose time do we need, and how expert must they be." The answer should scale with what a wrong output costs you.

The five components of an evaluation pipeline

An evaluation pipeline tells you, before and after every change, whether your LLM application still does its job. It has five cost components.
The test set. A collection of realistic inputs with reference answers or grading rubrics, written or checked by people who know what a correct answer looks like. Mostly a one-time cost, with periodic additions.

Automated runs. Running the test set through the application and scoring the outputs, using code checks where possible and an LLM judge where judgment is needed. A recurring compute cost.

Judge calibration. Comparing the LLM judge's scores with human scores on a sample, so you know whether the judge can be trusted. A recurring people cost.
Production sampling. People reviewing a sample of real outputs, catching problems the test set does not cover and feeding new cases back into it.
Tooling and engineering. A platform to store datasets, run experiments and compare results, plus the engineers who build and maintain the pipeline.
Earlier articles in this series treat evaluation as one line in a larger budget. This one breaks it down, because for many teams it becomes one of the larger recurring costs of an AI feature.

Building the test set

The test set is the most valuable evaluation asset and, for expert domains, the largest single cost. Its price is set by three numbers: how many cases, how long each takes, and whose time it is.

The formula

\text{Labeling cost} = \text{cases} \times (1 + \text{double-label share}) \times \frac{\text{minutes per case}}{60} \times \text{hourly rate}
Double-labeling a share of cases, having two people grade the same item, is how you measure whether your rubric is clear enough to be applied consistently.

Sourced rates

Second Talent compiled published rates for AI training and evaluation work in October 2026 (Second Talent):
Generalist annotators on US-focused platforms: about 25–50 an hour; Prolific recommends a minimum of $12
Licensed lawyers: 100–150 an hour (Mercor listings)
Board-certified physicians: 115–150 an hour (Mercor)
Investment banking: 100–130; software engineering: 70–150 (Mercor)
Mercor states an average contracted rate of $97 an hour; Alignerr states $80
These are platform listings, not guaranteed rates. Your own staff may be the right labelers for internal use cases; their cost is their loaded hourly rate.

Worked examples (illustrative)

300 cases, internal experts at $50/h, 10 minutes each, 20% double-labeled: 360 labels × 10 ÷ 60 × $50 = $3,000
1,500 cases, support specialists at $50/h, 12 minutes each, 20% double-labeled: 1,800 × 12 ÷ 60 × $50 = $18,000
3,000 cases, licensed domain experts at $125/h, 20 minutes each, 30% double-labeled: 3,900 × 20 ÷ 60 × $125 = $162,500

Ways to keep it affordable

Start from real traffic. Sample actual user inputs instead of inventing them; experts then write or check the answers.
Let a model draft, people verify. Having an LLM propose reference answers and experts correct them is often faster than writing from scratch. Measure the time saved rather than assuming it.
Grow it over time. Start with a few hundred cases that cover the most common and most dangerous situations, then add cases from production failures.

Running evaluations: code checks, LLM judges and batch pricing

Automated runs are cheap enough that cost should never be the reason to skip one. The main design choice is how each output gets scored.

Three ways to score an output

Code checks. Exact matches, format validation, required fields, forbidden phrases, whether the right tool was called. Effectively free; use them wherever the answer is objective.
LLM-as-judge. A model grades each output against a rubric (correctness, faithfulness to sources, tone). In the paper that popularized the method, strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level humans reach with each other. The authors also documented position, verbosity and self-enhancement biases (Zheng et al., 2023).
Human grading. The most reliable and the most expensive. Best used to calibrate the judge and to review cases the judge flags as uncertain.

Cost per run (illustrative)

Each test case needs the application to generate an answer, then the judge to score it. At Claude Sonnet 5.5 batch prices ($1 input, $5 output per million tokens):
Generation: 3,500 input × $1 ÷ 10⁶ + 400 output × $5 ÷ 10⁶ = $0.0055
Judging, three criteria: 3 × (2,500 × $1 ÷ 10⁶ + 300 × $5 ÷ 10⁶) = $0.012
Total per case per run: about $0.0175
So a 300-case run costs about $5, a 1,500-case run about $26 and a 3,000-case run about $53.

How often to run

On every meaningful change: prompts, retrieval settings, model version, tools
On a schedule, weekly for example, to catch drift from upstream changes you did not make
Before and after model migrations, which providers force on a schedule (see our maintenance article)
Batch APIs halve the cost because evaluation results are rarely needed instantly. Use them for scheduled runs.

Human review and production sampling

Test sets go stale as users find new ways to use the application. Reading real outputs is how you find the failures your test set does not yet contain.

Production sampling

A reviewer reads a sample of real conversations or agent actions each week, grades them against the same rubric, and turns each new failure into a test case.

Internal assistant: 2 hours a week at $50 = $5,200 a year

Customer-facing agent: 4 hours a week at $50 = $10,400 a year (the same figure as our maintenance article)
Regulated application: 8 hours a week of licensed-expert time at $125 = $52,000 a year
Sample more heavily after launches, model changes and incidents.

Judge calibration

An LLM judge is only useful if it agrees with your experts on your rubric. Each quarter, have people grade a sample the judge has also graded, and measure agreement.
Internal assistant: 100 cases a quarter at 5 minutes and $50/h = about $1,667 a year
Customer-facing agent: 200 cases a quarter = about $3,333 a year
Regulated application: 300 cases a quarter at 8 minutes and $125/h = $20,000 a year
If agreement drops, fix the rubric or the judge prompt before trusting automated scores again.

Privacy

Production samples may contain personal data. Reviewers need access controls, and samples may need redacting before review. That setup is part of the engineering cost.

Tooling: evaluation and observability platforms

Platform costs are small next to people costs, usually hundreds or a few thousand dollars a year for small teams.

Sourced prices

A 2026 CostBench comparison lists (CostBench):
LangSmith Plus: $39 per seat per month
Braintrust Pro: $249 a month, including $249 of model credits
Usage-based charges (traces, data storage) apply above plan limits, and enterprise tiers are quoted separately. Check each vendor's pricing page directly.

What the tooling should do

Store versioned test sets and rubrics
Run experiments and compare scores between versions side by side
Link evaluation results to production traces
Support human review queues and record reviewer grades
Export everything, so your test set is not locked into one vendor
Engineering
Tools do not build the pipeline. Someone writes the scorers and judge prompts, wires runs into the release process, and maintains it all as the application changes. In our model:
Initial build: 80 to 320 hours at $87 an hour (senior, Latin America rate from our staffing article) = $6,960 to $27,840
Ongoing upkeep: from 0.1 FTE mid-level (10,560ayear)forasimpleassistantto0.5FTEseniorAIengineer(83,520 a year) for a regulated application

Cost model: three evaluation programs (illustrative)

First-year evaluation cost ranges from about $29,000 to $365,000 across the three programs. In each one, people account for 91–95% of the total.
Program A: internal knowledge assistant
300-case test set; 102 runs a year (weekly plus 50 change runs).
Test set: $3,000
Automated runs: 102 × 300 × $0.0175 = $536
Production sampling: $5,200
Judge calibration: $1,667
Tooling: LangSmith, 2 seats × $39 × 12 = $936
Pipeline build: 80 h × $87 = $6,960
Upkeep: 0.1 × $8,800 × 12 = $10,560
First year: $28,858. Later years, without the test set and build: $18,898

Program B: customer-facing support agent

1,500-case test set; 202 runs a year (weekly plus 150 change runs).
Test set: $18,000
Automated runs: 202 × 1,500 × $0.0175 = $5,303
Production sampling: $10,400
Judge calibration: $3,333
Tooling: Braintrust Pro, $249 × 12 = $2,988
Pipeline build: 160 h × $87 = $13,920
Upkeep: 0.25 × $13,920 × 12 = $41,760
First year: $95,704. Later years: $63,784

Program C: regulated, high-stakes application

3,000-case test set written by licensed experts; 302 runs a year (weekly plus 250 change runs).
Test set: $162,500
Automated runs: 302 × 3,000 × $0.0175 = $15,855
Production sampling: $52,000
Judge calibration: $20,000
Tooling: Braintrust Pro (enterprise tier likely needed; quote required) $2,988
Pipeline build: 320 h × $87 = $27,840
Upkeep: 0.5 × $13,920 × 12 = $83,520
First year: $364,703. Later years: $174,363

What the comparison shows

Compute is not the constraint. Even Program C runs its full test set over 300 times a year for under $16,000.
Expertise drives cost. Program C's test set costs nine times Program B's, for twice the cases, because licensed experts cost more per hour and need more time per case.
Later years are not free. The "later years" figures exclude adding new test cases. Budget a share of the original labeling cost each year to keep the set current.

How much evaluation is enough

Size evaluation to the cost of a wrong output, not to a percentage of the build budget.

Match the program to the risk

Low stakes (internal drafts, summaries a person always reads): a few hundred cases, internal reviewers and a small quarterly calibration sample are usually enough. Program A's scale.
Customer-facing (answers customers act on, actions an agent takes): a larger set covering common and edge cases, weekly production sampling and a larger quarterly calibration sample. Program B's scale.
High stakes (medical, legal, financial or safety-relevant outputs): expert-written cases, expert review of production samples, frequent calibration and documented results for audit. Program C's scale.

Signs you are under-investing

Nobody can say what the current accuracy is
Model or prompt changes ship without a test run
Users find problems before the team does
The test set has not grown since launch
The LLM judge has never been checked against people
Signs you are over-investing
Hundreds of near-duplicate test cases that never fail
Expert reviewers grading outputs that code checks could score
Running the full suite on trivial changes when a smaller smoke test would do

Where the effort pays back

Evaluation is what makes the other costs in this series manageable. It turns model migrations into a test run, lets you try cheaper models and fewer retrieved chunks safely, and gives the evidence needed to approve a pilot for production.

Where CodingCops fits

Disclosure: the author works with CodingCops. This section uses only the company's published descriptions and includes no CodingCops pricing, client results or savings claims.
CodingCops is a Chicago-headquartered software company founded in 2010 (Built In Chicago). Its company profile lists AI and machine learning, cloud and DevOps, custom software development, and dedicated engineering teams among its services (Outsource Accelerator), and it offers engagements for specific projects, short-term initiatives or long-term development (codingcops.com).
The engineering line in this article, building scorers, judge prompts and run automation, is work an external team can take on. The test set and the quality bar it encodes should stay yours: insist on owning every dataset, rubric and result, in an exportable format.

FAQs

How much does LLM evaluation cost?

In our illustrative model, about $29,000 in the first year for an internal assistant, $96,000 for a customer-facing agent and $365,000 for a regulated, high-stakes application. Later years cost less once the test set exists.

What is the most expensive part of evaluation?

People. Labeling test cases, reviewing production outputs, calibrating judges and maintaining the pipeline make up 91–95% of first-year cost in our scenarios. Model calls are a few percent.

How much does an LLM-as-judge run cost?

About $0.0175 per test case at Claude Sonnet 5.5 batch prices, including generating the answer and scoring it on three criteria. A 1,500-case run costs about $26.

Can I trust an LLM judge?

Often, but check. Research found strong LLM judges matched human preferences over 80% of the time, as often as humans agree with each other, while also showing position, verbosity and self-enhancement biases. Calibrate against your own experts regularly.

How big should my test set be?

Start with a few hundred cases covering common and high-risk situations, then grow it from real failures. Customer-facing and regulated applications typically need more, and expert-written ones.

How much do expert annotators cost?

Published 2026 listings show about 25–50 an hour for generalists and 100–150 for licensed lawyers and physicians.

Top comments (0)