Under the assumptions in this article, an AI evaluation pipeline costs about $400 a month in non-people spend for a pilot, about $5,300 a month for a single production AI feature, and about $35,000 a month for an enterprise program judging hundreds of thousands of cases. Add engineering time and the all-in figures become roughly $4,500, $25,500 and $84,000 a month with US in-house staff.
Three findings hold across every scenario:
• People cost more than tokens. Eval engineering and human review are 76–98% of the modeled total, depending on scale.
• Judge model choice is the biggest technical lever. A small judge model costs about 1/20th of a mid-tier one per call, at current list prices.
• Batch APIs halve offline eval spend. OpenAI, Anthropic and Google all list a 50% batch discount.
Pricing period: list prices checked October 16, 2026, in USD, US region, before negotiated discounts. Cost guides like this go stale fast; treat it as due for a quarterly update.
Scenario figures are illustrative estimates built from sourced prices and stated assumptions, not results from a real project. Swap in your own token counts and volumes.
What an evaluation pipeline includes (the cost components)
An eval pipeline has six cost lines; most budgets only count the first two. Each is listed below with what drives it.
- System-under-test inference. Re-running your app or agent on the test set for each offline run. Driven by test cases × runs per month × tokens per case.
- Offline judge calls. LLM-as-judge scoring of those outputs against rubrics or references. Driven by case-runs × criteria judged × judge model price.
- Online (production) evals. Judging a sample of live traffic for drift, safety and quality. Driven by production volume × sample rate × judge price.
- Human review. Expert or crowd labeling to build golden sets and calibrate judges. Driven by cases reviewed × minutes per case × hourly rate.
- Eval and observability tooling. Tracing, datasets, experiment tracking and dashboards. Driven by seats, traces or billable units, and retention.
- Engineering time. Building datasets, writing rubrics, maintaining CI gates and triaging failures. Driven by FTEs × loaded cost or vendor rate. Two items are often double-counted. Online evals do not pay for system-under-test inference: production already paid for that response. And CI runner compute is usually negligible next to API calls, so this model leaves it out.
Sourced pricing inputs (October 2026)
Every price below comes from a vendor pricing page checked on October 16, 2026. All are list prices in USD per million tokens unless noted.
Judge model token prices
Prices are shown as input / output, then batch input / batch output.
• Frontier tier: Claude Opus 5.5 at $4.00 / $20.00; batch $2.00 / $10.00.
• Mid tier: Claude Sonnet 5.5 and gpt-6.1-sol at $2.00 / $10.00; batch $1.00 / $5.00. Gemini 3.1 Pro Preview at $2.00 / $12.00 for prompts up to 200k tokens; batch $1.00 / $6.00.
• Small/mid tier: Gemini 3.8 Flash at $0.75 / $3.75; batch $0.375 / $1.875.
• Small tier: Claude Haiku 5.5 (prompts up to 100k tokens) and gpt-6-luna at $0.10 / $0.50; batch $0.05 / $0.25.
Price notes that change the math:
• Batch: all three vendors list a 50% batch discount, which suits nightly and pre-release eval runs.
• Caching: Anthropic lists cache reads at 0.1× base input (0.05× on Opus and Sonnet 5.5). OpenAI lists cached input for gpt-6.1-sol at $0.10. A rubric repeated on every judge call is an ideal cache candidate.
• Scheduled increases: Google lists Gemini 3.8 Flash rising to $1.50 input / $7.50 output from January 1, 2027.
Eval and observability platform pricing
• Langfuse Cloud: Core $29, Pro $199, Enterprise $2,499 a month. Each includes 100k billable units; overage is $8 per 100k units, lower at volume.
• Braintrust: Pro is $249 a month with 5 GB of processed data and 50k scores. Overage is $3/GB and $1.50 per 1,000 scores.
• LangSmith: Plus is $39 per seat a month with 10k base traces, then pay-as-you-go.
LangSmith no longer publishes a dollar rate per 1,000 traces on its pricing page. A third-party tracker reports about $5.00 per 1,000 base traces after a July 2026 metering change; confirm with LangChain before budgeting on it.
Compute, labor and annotation inputs
• GPU compute: an on-demand NVIDIA H100 SXM is $3.99 per GPU-hour (Lambda).
• US engineering pay: median pay was $135,980 a year for software developers and $104,300 for QA analysts and testers in May 2025 (BLS).
• Benefits load: benefits were 30.0% of private-industry compensation in June 2026 (BLS ECEC).
• Outsourced senior developer rates: Asia $31–$41/hr, Latin America $60–$75/hr, Europe $64–$76/hr (Accelerance).
• Crowd annotation floor: $8/hr minimum and $12/hr recommended participant pay (Prolific).
A transparent cost model you can rerun
Monthly eval cost is the sum of the six lines above. Written out:
C_{month} = N_{case} R \, c_{sut} + N_{case} R \, J_{off} \, c_{judge}^{batch} + V S \, J_{on} \, c_{judge} + H \, \frac{t}{60} \, w + T + F \, L
Where N_case = test cases, R = offline runs per month, J = judge calls per case, V = production requests, S = sample rate, H = human-reviewed cases, t = minutes per review, w = reviewer hourly rate, T = tooling, F = FTEs and L = loaded monthly cost per FTE.
Assumptions (illustrative, not measured)
• Judge call size: 2,000 input + 300 output tokens (rubric, input, candidate output and reference, plus a short scored rationale).
• System-under-test call size: 1,500 input + 500 output tokens, typical of a single-turn RAG or assistant response. Agents run far larger.
• Default judge and app model: mid tier at $2 / $10 per million tokens, matching Claude Sonnet 5.5 and gpt-6.1-sol list prices.
• Offline runs: Batch API at 50% off, since nightly and pre-release runs tolerate batch latency.
• Online judging: standard pricing, because teams usually want same-day drift signals.
• Human review: 4 minutes per case at $45/hr. The domain-expert rate is an assumption; the crowd floor is $12/hr per Prolific.
• Tooling: Langfuse list prices at 5–7 billable units per judged case (one trace plus spans and scores), ignoring volume discounts.
• Loaded US FTE cost: $16,188 a month = BLS median $135,980 ÷ 0.70 wage share ÷ 12. AI specialists usually earn more.
Unit costs these assumptions produce
• Small-tier judge call: $0.00035 standard, $0.000175 batch.
• Mid-tier judge call: $0.0070 standard, $0.0035 batch.
• Frontier-tier judge call: $0.0140 standard, $0.0070 batch.
• Mid-tier system-under-test call: $0.0080 standard, $0.0040 batch.
Worked example, mid-tier batch judge: 2,000 × $1.00/M + 300 × $5.00/M = $0.0020 + $0.0015 = $0.0035 per call.
Three budget scenarios
Pilot. 500 golden cases, 20 offline runs a month, 1 judge call per case, no live traffic judged, 100 human-reviewed cases and 0.25 eval engineer.
Production (one AI feature). 2,000 cases, 60 runs a month, 2 judge calls per case, 50,000 live requests judged (5% of 1M), 1,000 human-reviewed cases and 1.25 FTEs.
Enterprise program. 10,000 cases, 60 runs a month, 3 judge calls per case, 400,000 live requests judged (2% of 20M), 5,000 human-reviewed cases and 3 FTEs.
Monthly results, line by line:
• Pilot: system-under-test inference $40, offline judging $35, online judging $0, human review $300, tooling $29. Non-people subtotal $404. Engineering $4,047. All-in $4,451 a month, $53,412 a year.
• Production: inference $480, offline judging $840, online judging $700, human review $3,000, tooling $273. Non-people subtotal $5,293. Engineering $20,235. All-in $25,528 a month, $306,336 a year.
• Enterprise: inference $2,400, offline judging $6,300, online judging $8,400, human review $15,000, tooling $3,051. Non-people subtotal $35,151. Engineering $48,564. All-in $83,715 a month, $1,004,580 a year.
Calculation check, Production offline judging: 2,000 cases × 60 runs × 2 calls × $0.0035 = $840. Production tooling: (120,000 + 50,000) judged cases × 6 units = 1.02M units; $199 + 9.2 × $8 = $273.
Sensitivity: the judge model decides the token bill
Monthly judging spend (offline plus online) by judge tier:
• Small judge: $2 pilot, $77 production, $735 enterprise.
• Mid judge: $35 pilot, $1,540 production, $14,700 enterprise.
• Frontier judge: $70 pilot, $3,080 production, $29,400 enterprise.
Moving the enterprise program from a frontier to a small judge saves about $28,700 a month on list prices. That saving is only real if the small judge agrees with human labels well enough; measure agreement on your own golden set before switching.
Self-hosting a judge: when does a GPU pay off?
One on-demand H100 at $3.99/hr running all month costs about $2,913 (730 hours). That equals roughly 832,000 mid-tier batch judge calls, or 16.6 million small-tier batch calls. Self-hosting an open-weight judge only beats the API when volume is high, utilization stays near full, and someone is paid to run the serving stack. For most teams below the enterprise scenario, API judges are cheaper.
People cost: in-house vs offshore vs nearshore vs dedicated team
Staffing model moves the Production scenario's all-in cost from about $25,500 to $11,500–$20,500 a month. The people line is where hiring decisions matter most.
Who you need
• Eval / ML engineer: builds the harness, writes rubrics and judge prompts, wires CI gates, tracks judge-human agreement.
• QA or annotation lead: curates golden sets, runs human review, writes labeling guidelines.
• Domain reviewers: part-time experts (clinicians, lawyers, support leads) who label the hard cases.
• Product owner: decides which failures block a release. Usually an existing role, so not costed here.
Production scenario (1.25 FTE) by staffing model
All-in figures add the $5,293 non-people subtotal.
• US in-house. $16,188 loaded per FTE gives $20,235 a month, or $25,528 all-in. Deepest product context, but the slowest and costliest to hire.
• Nearshore, Latin America (senior). $60–$75/hr × 160 hours gives $12,000–$15,000, or $17,293–$20,293 all-in. Strong working-hour overlap with US teams.
• Europe (senior). $64–$76/hr gives $12,800–$15,200, or $18,093–$20,493 all-in. Deep engineering pools with partial US overlap.
• Offshore, Asia (senior). $31–$41/hr gives $6,200–$8,200, or $11,493–$13,493 all-in. Lowest rate; needs explicit handoff and review processes.
• Hybrid: 0.25 US lead + 1.0 dedicated Asia engineer. $9,007–$10,607, or $14,300–$15,900 all-in. Keeps rubric ownership in-house and outsources build and upkeep.
Rates come from the Accelerance 2026 guide summary, which covers general software development. Specialists in LLM evaluation often bill above those ranges, so treat them as a floor.
The comparison is not like-for-like. Vendor rates include the vendor's overhead and benefits. The in-house figure excludes recruiting, management time and equipment. Outsourced models need internal oversight, which this table does not cost.
Engagement models for eval work
• Fixed-scope project. Best for building the first harness and golden set. Predictable one-off cost, but the pipeline decays after handover if nobody owns it.
• Dedicated team (monthly). Best for ongoing eval upkeep across several AI features. Flat monthly cost, with low utilization if the AI roadmap stalls.
• Time and materials. Best for uncertain scope and research-heavy agents. Flexible but harder to forecast; define release gates early to contain scope.
• Staff augmentation. Best for filling one eval engineer seat in an existing team. Billed as rate × hours; knowledge stays with the individual unless it is documented.
Pilot-to-production: where the eval budget jumps
In this model, non-people eval spend rises about 13× from pilot to production ($404 to $5,293) while the test set grows only 4×. Three changes cause the jump, and each is a budgeting decision rather than an accident.
- Run frequency triples. A pilot runs evals when someone remembers. Production wires them into CI and nightly jobs, so case-runs go from 10,000 to 120,000 a month.
- Online judging starts. Sampling live traffic adds a cost line that did not exist in the pilot ($700 a month here), and it scales with usage, not with your test set.
- Human review becomes routine. Judges drift and need recalibration against human labels. Human review is the largest non-people line in both the production and enterprise scenarios. For production-ready AI, budget the eval pipeline as a share of the feature's running cost, not as a one-off build. Enterprise AI implementation budgets that fund the model and the app but not ongoing evaluation risk shipping regressions they cannot measure.
Cost-reduction levers, largest first
• Right-size the judge. Use small judges for format, schema and simple factuality checks; save mid or frontier judges for nuanced rubric scoring. In the enterprise scenario, the gap between small and frontier judges is about $28,700 a month.
• Batch everything offline. The 50% batch discount is already in this model. Teams running offline evals on standard pricing pay double for the same scores.
• Cache the rubric. The rubric and instructions repeat on every judge call. Anthropic lists cache reads at 5–10% of base input price, which cuts a large share of judge input cost.
• Run tiered suites. Run a small smoke set on every pull request and the full golden set nightly or before release. Halving runs halves the offline lines.
• Sample live traffic by risk. Judge 100% of flagged or high-value sessions and a low random rate of the rest, instead of one flat sample rate.
• Spend human time on disagreements. Route only cases where judges disagree or score low confidence to reviewers. Human review is the largest line you control outside salaries.
• Watch tooling meters. Retention upgrades and per-trace overages add up at enterprise volume. Negotiate annual commitments once volumes are stable.
FAQ
How much does LLM-as-judge evaluation cost per test case?
At October 2026 list prices and a 2,000-in / 300-out token judge call, one judgment costs about $0.00035 on a small model, $0.007 on a mid-tier model and $0.014 on a frontier model. Batch pricing halves each figure.
Is the API bill the biggest cost of AI evaluation?
Usually not. In all three scenarios here, engineering time and human review make up 76–98% of the total. Token spend becomes material only at high volume with mid or frontier judges.
What should a startup budget for evals on its first AI feature?
Under this model, a pilot needs roughly $400 a month in non-people spend plus a quarter of an engineer's time. Moving that feature to production raises non-people spend to about $5,300 a month.
Should we self-host an open-weight judge model to save money?
Only at high, steady volume. One on-demand H100 costs about $2,913 a month at $3.99/hr, equal to about 832,000 mid-tier batch judge calls. Below that, or with a small API judge, the API is cheaper and needs no serving team.
Is it cheaper to outsource eval engineering?
On rates alone, yes. For 1.25 FTEs, published senior rates put Asia at $6,200–$8,200 a month and Latin America at $12,000–$15,000, against about $20,200 loaded for US in-house staff. Factor in oversight time, timezone overlap and who owns the rubrics.
Which eval platform is cheapest?
It depends on your meter. Langfuse bills per unit, Braintrust per GB and score, and LangSmith per seat and trace. Estimate your monthly traces and scores, then price all three against their current pages.
How often should this cost estimate be updated?
Quarterly. Vendors in this article have changed prices in 2026, and Google has already scheduled a Gemini 3.8 Flash increase for January 2027.
Top comments (0)