DEV Community: Eric Thompson

Estimating the Cost of Running AI Evaluation Pipelines

Eric Thompson — Fri, 16 Oct 2026 15:06:58 +0000

Under the assumptions in this article, an AI evaluation pipeline costs about $400 a month in non-people spend for a pilot, about $5,300 a month for a single production AI feature, and about $35,000 a month for an enterprise program judging hundreds of thousands of cases. Add engineering time and the all-in figures become roughly $4,500, $25,500 and $84,000 a month with US in-house staff.

Three findings hold across every scenario:

• People cost more than tokens. Eval engineering and human review are 76–98% of the modeled total, depending on scale.
• Judge model choice is the biggest technical lever. A small judge model costs about 1/20th of a mid-tier one per call, at current list prices.
• Batch APIs halve offline eval spend. OpenAI, Anthropic and Google all list a 50% batch discount.
Pricing period: list prices checked October 16, 2026, in USD, US region, before negotiated discounts. Cost guides like this go stale fast; treat it as due for a quarterly update.
Scenario figures are illustrative estimates built from sourced prices and stated assumptions, not results from a real project. Swap in your own token counts and volumes.

What an evaluation pipeline includes (the cost components)

An eval pipeline has six cost lines; most budgets only count the first two. Each is listed below with what drives it.

System-under-test inference. Re-running your app or agent on the test set for each offline run. Driven by test cases × runs per month × tokens per case.
Offline judge calls. LLM-as-judge scoring of those outputs against rubrics or references. Driven by case-runs × criteria judged × judge model price.
Online (production) evals. Judging a sample of live traffic for drift, safety and quality. Driven by production volume × sample rate × judge price.
Human review. Expert or crowd labeling to build golden sets and calibrate judges. Driven by cases reviewed × minutes per case × hourly rate.
Eval and observability tooling. Tracing, datasets, experiment tracking and dashboards. Driven by seats, traces or billable units, and retention.
Engineering time. Building datasets, writing rubrics, maintaining CI gates and triaging failures. Driven by FTEs × loaded cost or vendor rate. Two items are often double-counted. Online evals do not pay for system-under-test inference: production already paid for that response. And CI runner compute is usually negligible next to API calls, so this model leaves it out.

Sourced pricing inputs (October 2026)

Every price below comes from a vendor pricing page checked on October 16, 2026. All are list prices in USD per million tokens unless noted.
Judge model token prices
Prices are shown as input / output, then batch input / batch output.
• Frontier tier: Claude Opus 5.5 at $4.00 / $20.00; batch $2.00 / $10.00.
• Mid tier: Claude Sonnet 5.5 and gpt-6.1-sol at $2.00 / $10.00; batch $1.00 / $5.00. Gemini 3.1 Pro Preview at $2.00 / $12.00 for prompts up to 200k tokens; batch $1.00 / $6.00.
• Small/mid tier: Gemini 3.8 Flash at $0.75 / $3.75; batch $0.375 / $1.875.
• Small tier: Claude Haiku 5.5 (prompts up to 100k tokens) and gpt-6-luna at $0.10 / $0.50; batch $0.05 / $0.25.
Price notes that change the math:
• Batch: all three vendors list a 50% batch discount, which suits nightly and pre-release eval runs.
• Caching: Anthropic lists cache reads at 0.1× base input (0.05× on Opus and Sonnet 5.5). OpenAI lists cached input for gpt-6.1-sol at $0.10. A rubric repeated on every judge call is an ideal cache candidate.
• Scheduled increases: Google lists Gemini 3.8 Flash rising to $1.50 input / $7.50 output from January 1, 2027.

Eval and observability platform pricing

• Langfuse Cloud: Core $29, Pro $199, Enterprise $2,499 a month. Each includes 100k billable units; overage is $8 per 100k units, lower at volume.
• Braintrust: Pro is $249 a month with 5 GB of processed data and 50k scores. Overage is $3/GB and $1.50 per 1,000 scores.
• LangSmith: Plus is $39 per seat a month with 10k base traces, then pay-as-you-go.
LangSmith no longer publishes a dollar rate per 1,000 traces on its pricing page. A third-party tracker reports about $5.00 per 1,000 base traces after a July 2026 metering change; confirm with LangChain before budgeting on it.

Compute, labor and annotation inputs

• GPU compute: an on-demand NVIDIA H100 SXM is $3.99 per GPU-hour (Lambda).
• US engineering pay: median pay was $135,980 a year for software developers and $104,300 for QA analysts and testers in May 2025 (BLS).
• Benefits load: benefits were 30.0% of private-industry compensation in June 2026 (BLS ECEC).
• Outsourced senior developer rates: Asia $31–$41/hr, Latin America $60–$75/hr, Europe $64–$76/hr (Accelerance).
• Crowd annotation floor: $8/hr minimum and $12/hr recommended participant pay (Prolific).

A transparent cost model you can rerun

Monthly eval cost is the sum of the six lines above. Written out:
C_{month} = N_{case} R \, c_{sut} + N_{case} R \, J_{off} \, c_{judge}^{batch} + V S \, J_{on} \, c_{judge} + H \, \frac{t}{60} \, w + T + F \, L
Where N_case = test cases, R = offline runs per month, J = judge calls per case, V = production requests, S = sample rate, H = human-reviewed cases, t = minutes per review, w = reviewer hourly rate, T = tooling, F = FTEs and L = loaded monthly cost per FTE.

Assumptions (illustrative, not measured)

• Judge call size: 2,000 input + 300 output tokens (rubric, input, candidate output and reference, plus a short scored rationale).
• System-under-test call size: 1,500 input + 500 output tokens, typical of a single-turn RAG or assistant response. Agents run far larger.
• Default judge and app model: mid tier at $2 / $10 per million tokens, matching Claude Sonnet 5.5 and gpt-6.1-sol list prices.
• Offline runs: Batch API at 50% off, since nightly and pre-release runs tolerate batch latency.
• Online judging: standard pricing, because teams usually want same-day drift signals.
• Human review: 4 minutes per case at $45/hr. The domain-expert rate is an assumption; the crowd floor is $12/hr per Prolific.
• Tooling: Langfuse list prices at 5–7 billable units per judged case (one trace plus spans and scores), ignoring volume discounts.
• Loaded US FTE cost: $16,188 a month = BLS median $135,980 ÷ 0.70 wage share ÷ 12. AI specialists usually earn more.

Unit costs these assumptions produce

• Small-tier judge call: $0.00035 standard, $0.000175 batch.
• Mid-tier judge call: $0.0070 standard, $0.0035 batch.
• Frontier-tier judge call: $0.0140 standard, $0.0070 batch.
• Mid-tier system-under-test call: $0.0080 standard, $0.0040 batch.
Worked example, mid-tier batch judge: 2,000 × $1.00/M + 300 × $5.00/M = $0.0020 + $0.0015 = $0.0035 per call.

Three budget scenarios

Pilot. 500 golden cases, 20 offline runs a month, 1 judge call per case, no live traffic judged, 100 human-reviewed cases and 0.25 eval engineer.
Production (one AI feature). 2,000 cases, 60 runs a month, 2 judge calls per case, 50,000 live requests judged (5% of 1M), 1,000 human-reviewed cases and 1.25 FTEs.
Enterprise program. 10,000 cases, 60 runs a month, 3 judge calls per case, 400,000 live requests judged (2% of 20M), 5,000 human-reviewed cases and 3 FTEs.

Monthly results, line by line:

• Pilot: system-under-test inference $40, offline judging $35, online judging $0, human review $300, tooling $29. Non-people subtotal $404. Engineering $4,047. All-in $4,451 a month, $53,412 a year.
• Production: inference $480, offline judging $840, online judging $700, human review $3,000, tooling $273. Non-people subtotal $5,293. Engineering $20,235. All-in $25,528 a month, $306,336 a year.
• Enterprise: inference $2,400, offline judging $6,300, online judging $8,400, human review $15,000, tooling $3,051. Non-people subtotal $35,151. Engineering $48,564. All-in $83,715 a month, $1,004,580 a year.
Calculation check, Production offline judging: 2,000 cases × 60 runs × 2 calls × $0.0035 = $840. Production tooling: (120,000 + 50,000) judged cases × 6 units = 1.02M units; $199 + 9.2 × $8 = $273.

Sensitivity: the judge model decides the token bill

Monthly judging spend (offline plus online) by judge tier:
• Small judge: $2 pilot, $77 production, $735 enterprise.
• Mid judge: $35 pilot, $1,540 production, $14,700 enterprise.
• Frontier judge: $70 pilot, $3,080 production, $29,400 enterprise.
Moving the enterprise program from a frontier to a small judge saves about $28,700 a month on list prices. That saving is only real if the small judge agrees with human labels well enough; measure agreement on your own golden set before switching.

Self-hosting a judge: when does a GPU pay off?

One on-demand H100 at $3.99/hr running all month costs about $2,913 (730 hours). That equals roughly 832,000 mid-tier batch judge calls, or 16.6 million small-tier batch calls. Self-hosting an open-weight judge only beats the API when volume is high, utilization stays near full, and someone is paid to run the serving stack. For most teams below the enterprise scenario, API judges are cheaper.
People cost: in-house vs offshore vs nearshore vs dedicated team
Staffing model moves the Production scenario's all-in cost from about $25,500 to $11,500–$20,500 a month. The people line is where hiring decisions matter most.

Who you need

• Eval / ML engineer: builds the harness, writes rubrics and judge prompts, wires CI gates, tracks judge-human agreement.
• QA or annotation lead: curates golden sets, runs human review, writes labeling guidelines.
• Domain reviewers: part-time experts (clinicians, lawyers, support leads) who label the hard cases.
• Product owner: decides which failures block a release. Usually an existing role, so not costed here.

Production scenario (1.25 FTE) by staffing model

All-in figures add the $5,293 non-people subtotal.
• US in-house. $16,188 loaded per FTE gives $20,235 a month, or $25,528 all-in. Deepest product context, but the slowest and costliest to hire.
• Nearshore, Latin America (senior). $60–$75/hr × 160 hours gives $12,000–$15,000, or $17,293–$20,293 all-in. Strong working-hour overlap with US teams.
• Europe (senior). $64–$76/hr gives $12,800–$15,200, or $18,093–$20,493 all-in. Deep engineering pools with partial US overlap.
• Offshore, Asia (senior). $31–$41/hr gives $6,200–$8,200, or $11,493–$13,493 all-in. Lowest rate; needs explicit handoff and review processes.
• Hybrid: 0.25 US lead + 1.0 dedicated Asia engineer. $9,007–$10,607, or $14,300–$15,900 all-in. Keeps rubric ownership in-house and outsources build and upkeep.
Rates come from the Accelerance 2026 guide summary, which covers general software development. Specialists in LLM evaluation often bill above those ranges, so treat them as a floor.
The comparison is not like-for-like. Vendor rates include the vendor's overhead and benefits. The in-house figure excludes recruiting, management time and equipment. Outsourced models need internal oversight, which this table does not cost.

Engagement models for eval work

• Fixed-scope project. Best for building the first harness and golden set. Predictable one-off cost, but the pipeline decays after handover if nobody owns it.
• Dedicated team (monthly). Best for ongoing eval upkeep across several AI features. Flat monthly cost, with low utilization if the AI roadmap stalls.
• Time and materials. Best for uncertain scope and research-heavy agents. Flexible but harder to forecast; define release gates early to contain scope.
• Staff augmentation. Best for filling one eval engineer seat in an existing team. Billed as rate × hours; knowledge stays with the individual unless it is documented.

Pilot-to-production: where the eval budget jumps

In this model, non-people eval spend rises about 13× from pilot to production ($404 to $5,293) while the test set grows only 4×. Three changes cause the jump, and each is a budgeting decision rather than an accident.

Run frequency triples. A pilot runs evals when someone remembers. Production wires them into CI and nightly jobs, so case-runs go from 10,000 to 120,000 a month.
Online judging starts. Sampling live traffic adds a cost line that did not exist in the pilot ($700 a month here), and it scales with usage, not with your test set.
Human review becomes routine. Judges drift and need recalibration against human labels. Human review is the largest non-people line in both the production and enterprise scenarios. For production-ready AI, budget the eval pipeline as a share of the feature's running cost, not as a one-off build. Enterprise AI implementation budgets that fund the model and the app but not ongoing evaluation risk shipping regressions they cannot measure.

Cost-reduction levers, largest first

• Right-size the judge. Use small judges for format, schema and simple factuality checks; save mid or frontier judges for nuanced rubric scoring. In the enterprise scenario, the gap between small and frontier judges is about $28,700 a month.

• Batch everything offline. The 50% batch discount is already in this model. Teams running offline evals on standard pricing pay double for the same scores.

• Cache the rubric. The rubric and instructions repeat on every judge call. Anthropic lists cache reads at 5–10% of base input price, which cuts a large share of judge input cost.

• Run tiered suites. Run a small smoke set on every pull request and the full golden set nightly or before release. Halving runs halves the offline lines.

• Sample live traffic by risk. Judge 100% of flagged or high-value sessions and a low random rate of the rest, instead of one flat sample rate.

• Spend human time on disagreements. Route only cases where judges disagree or score low confidence to reviewers. Human review is the largest line you control outside salaries.

• Watch tooling meters. Retention upgrades and per-trace overages add up at enterprise volume. Negotiate annual commitments once volumes are stable.

FAQ

How much does LLM-as-judge evaluation cost per test case?

At October 2026 list prices and a 2,000-in / 300-out token judge call, one judgment costs about $0.00035 on a small model, $0.007 on a mid-tier model and $0.014 on a frontier model. Batch pricing halves each figure.

Is the API bill the biggest cost of AI evaluation?

Usually not. In all three scenarios here, engineering time and human review make up 76–98% of the total. Token spend becomes material only at high volume with mid or frontier judges.

What should a startup budget for evals on its first AI feature?

Under this model, a pilot needs roughly $400 a month in non-people spend plus a quarter of an engineer's time. Moving that feature to production raises non-people spend to about $5,300 a month.

Should we self-host an open-weight judge model to save money?

Only at high, steady volume. One on-demand H100 costs about $2,913 a month at $3.99/hr, equal to about 832,000 mid-tier batch judge calls. Below that, or with a small API judge, the API is cheaper and needs no serving team.

Is it cheaper to outsource eval engineering?

On rates alone, yes. For 1.25 FTEs, published senior rates put Asia at $6,200–$8,200 a month and Latin America at $12,000–$15,000, against about $20,200 loaded for US in-house staff. Factor in oversight time, timezone overlap and who owns the rubrics.

Which eval platform is cheapest?

It depends on your meter. Langfuse bills per unit, Braintrust per GB and score, and LangSmith per seat and trace. Estimate your monthly traces and scores, then price all three against their current pages.

How often should this cost estimate be updated?

Quarterly. Vendors in this article have changed prices in 2026, and Google has already scheduled a Gemini 3.8 Flash increase for January 2027.

Calculating Token, Retrieval, and Observability Costs for a RAG Service

Eric Thompson — Thu, 15 Oct 2026 15:09:49 +0000

A typical RAG query (2,000 tokens of prompt, 2,500 tokens of retrieved context, and 400 tokens of output) on a mid-tier model costs about $0.013 in usage. Once retrieval, reranking, and observability are added, that comes to roughly $13 per 1,000 queries at production volume.

In our model, a service handling 10,000 queries a month costs about $203 in usage, or about $234 with a 15% buffer for retries, query rewrites, and evaluation runs. At 100,000 queries a month, the bill is about $1,369, or $1,574 with the buffer. At 1 million queries a month, it reaches about $13,113, or $15,079 with the buffer. Ten times the traffic doesn't mean ten times the bill at lower volumes, because fixed fees such as plan minimums are shared across more queries.

*Three findings matter more than the totals:
*
LLM tokens are ~75–80% of usage cost at scale. Vector search is under 1%.
Reranking is the surprise second line item (~15% at 100K+ queries).
At low volume, people cost more than tokens. Half an AI engineer maintaining the service costs $3,300–$11,250 a month depending on location, 15–50× the usage bill at 10K queries.

Every number below is either linked to a source or labelled as an illustrative assumption.

The cost equation

A RAG service has three kinds of cost:

Monthly cost =
queries × (LLM + query embedding + vector read + rerank + observability per query)

fixed platform fees (plan minimums, seats, storage)
corpus maintenance (re-embedding, writes)
people (build, evaluation, on-call, iteration)

The rest of this article prices each term.

1. Token costs: where most of the money goes

*The per-query token budget
*
We modelled a knowledge-assistant style RAG endpoint. These are illustrative assumptions; measure your own with real traffic before you trust any forecast.

Each request starts with a 1,500-token system prompt of instructions. Because it's identical on every request, it can be cached and billed at a discounted rate. Next comes the user's message plus a short conversation history, about 500 tokens. Then the system adds the retrieved context: five chunks of 500 tokens each, or 2,500 tokens in total. Both of these change on every request, so they're billed at the full input price. Finally, the model writes an answer of about 400 tokens, billed as output. That's 4,900 tokens per request.

Retrieved context is the largest input. Every extra chunk you stuff into the prompt is paid for on every single query.

Current list prices (October 2026)

**
Model prices fall into four broad tiers, quoted per million tokens.

Small tier (Claude Haiku 5.5 for prompts up to 100K tokens, and OpenAI's gpt-6-luna): $0.10 for input, $0.50 for output and $0.01 for cached input.
Mid tier (Claude Sonnet 5.5 and gpt-6.1-sol): $2.00 for input, $10.00 for output and $0.10 for cached input. The two vendors price this tier identically, so the choice between them comes down to quality on your own evaluation set.
Large tier: Claude Opus 5.5 charges $4.00 for input, $20.00 for output and $0.20 for cached input. gpt-6-astra charges $10.00, $50.00 and $1.00.

Across every tier, output costs five times as much as input. That's why long answers cost more than long prompts of the same length.

Sources: Anthropic and OpenAI official pricing pages, captured October 15, 2026. Both vendors offer roughly 50% off for asynchronous batch processing, which suits offline evaluation and re-indexing but not live chat.

Per-query calculation (mid tier, with prompt caching)

System prompt: 1,500 × [0.95 × $0.10 + 0.05 × $2.50] / 1M = $0.00033
Other input: 3,000 × $2.00 / 1M = $0.00600
Output: 400 × $10.00 / 1M = $0.00400
Total ≈ $0.01033 per query

Without caching, the same query costs $0.0130. Caching saves about 20% here because the system prompt is only 30% of input; the retrieved context changes every query and can't be cached the same way.

Running the same 4,900-token request through each tier shows how much the price list matters. On a small-tier model, a query costs about $0.00052, or $0.52 per 1,000 queries. On a mid-tier model, it's about $0.01033, or $10.33 per 1,000. Claude Opus 5.5 brings it to $0.02066, or $20.66 per 1,000, and gpt-6-astra to $0.05236, or $52.36 per 1,000. Nothing about the query changes between these figures; only the model does. For budgeting, multiply the per-1,000 figure by your expected traffic in thousands.

The 100× spread between small and large tiers is the single biggest decision in your budget. Many teams route simple queries to a small model and escalate only hard ones; whether that works for you depends on your evaluation results, not on price alone.

2. Retrieval costs: embeddings, vector database and reranking

*Embeddings: cheap enough to ignore (almost)
*
OpenAI's text-embedding-3-small is listed at $0.02 per 1M tokens.

Initial corpus: 500M tokens × $0.02 / 1M = $10.00 one-off
Monthly refresh: 10% of corpus = $1.00
Per query: 50 tokens × $0.02 / 1M = $0.000001

Embedding cost is negligible. The real cost of embeddings is engineering time spent on chunking strategy and re-indexing when you change models.

*Vector database: small, but watch the pricing model
*
We used Pinecone serverless as the example because its pricing is usage-based and documented. The key rule, from Pinecone's cost documentation: a query consumes 1 read unit per GB of namespace size, with a 0.25 RU minimum, regardless of top_k.

Index size: 1M vectors × (1,536 dims × 4 bytes + 1 KB metadata) ≈ 7.14 GB
Read units: ≈ 7.14 RU per query
Read cost: 7.14 × $16 / 1M ≈ $0.000114 per query
Storage: 7.14 GB × $0.33 ≈ $2.36 / month
Plan minimum: $50 / month (Standard)

Unit rates are from a secondary source and should be confirmed in Pinecone's calculator; the $50 minimum and RU rule are from Pinecone's own pages.

At 10K and 100K queries per month, you'll pay the $50 minimum. Even at 1M queries, the vector database costs about $120/month.

*Two practical consequences:
*
Namespace size drives read cost. Partitioning by tenant or document type cuts RUs per query proportionally.
If you already run Postgres, pgvector on existing infrastructure may cost nothing incremental at this scale. The trade-off is operational: you own tuning, scaling and backups.
*Reranking: the line item people forget
*
A cross-encoder reranker takes the top 50 candidates and picks the best 5. Cohere defines one search as one query with up to 100 documents, and Rerank 4 Fast is listed at $2.00 per 1,000 searches (secondary source, verify).

Per query: $0.002
At 100K queries/month: $200
At 1M queries/month: $2,000

That's about 15% of usage cost at scale, and more than the vector database by an order of magnitude. Reranking often improves answer quality enough to let you send fewer chunks to the LLM, which can pay for itself. Test both configurations.

3. Observability costs: per-unit vs per-trace pricing

You can't run RAG in production without tracing retrieval quality, latency and token spend. The two common hosted options price differently.

Langfuse Cloud charges per unit, where every trace, observation and score you record counts as one unit. Its Core plan costs $29 a month and includes 100,000 units, with additional usage at $8 per 100,000 units (lower at higher volumes). LangSmith charges per seat and per trace. Its Plus plan costs $39 per seat per month and includes 10,000 base traces, with additional traces at about $2.50 per 1,000, kept for 14 days.

In practice, a small team with lots of traffic tends to favour unit pricing, while a large team with modest traffic may find per-trace pricing easier to predict.

Sources: Langfuse and LangChain official pricing pages. LangSmith overage rate from a June 2026 secondary source; verify before budgeting.

*Translating that to per-query cost
*
In Langfuse, a single RAG request typically produces one trace plus several observations (embed, retrieve, rerank, generate, guardrail, post-process) and maybe an automated score. Assumption: 8 units per query.

At 10,000 queries a month, that's 80,000 units, inside Langfuse's included allowance, so the bill is just the $29 plan fee. LangSmith comes to $117, which is three seats at $39 each. At 100,000 queries, Langfuse rises to about $85 and LangSmith to about $342, as per-trace charges kick in above 10,000 traces. At 1 million queries, Langfuse reaches about $661 at list rates, while LangSmith reaches about $2,592. The gap between the two widens as traffic grows.

Illustrative; both vendors discount at volume and both can be self-hosted, which swaps subscription cost for infrastructure and engineering time.

Observability can quietly reach 5–20% of usage cost. Sampling (tracing 10–20% of production traffic in full while logging tokens for 100%) is a common way to control it.

4. Three monthly scenarios, line by line

These scenarios use the mid-tier model with prompt caching on, and Langfuse for observability.

At 10,000 queries a month, LLM tokens cost $103.30, reranking $20.00 and observability $29.00. The vector database costs $50.00, which is simply Pinecone's plan minimum. Query embeddings cost a cent, and refreshing a tenth of the corpus each month costs $1.00. The total is $203.31, or $20.33 per 1,000 queries, rising to $233.81 with the 15% buffer. At this volume, the vector database is a quarter of the bill, purely because of its minimum fee.

At 100,000 queries a month, LLM tokens rise to $1,033.00 and reranking to $200.00. Observability comes to $85.00, and the vector database stays at its $50.00 minimum. Embeddings cost $0.10 and corpus refresh $1.00. The total is $1,369.10, or $13.69 per 1,000 queries, and $1,574.46 with the buffer.

At 1 million queries a month, LLM tokens reach $10,330.00 and reranking $2,000.00. Observability is $661.00, and the vector database finally rises above its minimum to $119.52. Embeddings cost $1.00 and corpus refresh $1.00. The total is $13,112.52, or $13.11 per 1,000 queries, and $15,079.40 with the buffer. LLM tokens now account for almost four-fifths of spending and reranking for most of the rest, while embeddings and storage barely register.

*What happens if you change the model
*
Holding everything else constant at 1 million queries a month, the model choice dominates the bill. With a small-tier model, LLM cost falls to about $524 and the usage total to about $3,300. The mid-tier model gives the $10,330 LLM cost and ~$13,100 total above. Claude Opus 5.5 pushes LLM cost to $20,660 and the total to about $23,400, and gpt-6-astra to $52,363 and about $55,100. Each total includes the same $2,783 of retrieval, reranking and observability, so the difference is entirely the LLM line. Moving from the cheapest tier to the most expensive one makes the bill roughly 17 times larger.

Per-1,000-query cost falls as volume grows only because fixed fees get spread thinner. After roughly 100K queries, the bill becomes almost entirely variable.

5. The line item that dwarfs tokens at low volume: people

A production RAG service needs ongoing work: evaluation sets, chunking and prompt changes, regression testing when models update, cost monitoring, and incident response. Assumption: 0.5 FTE AI engineer for maintenance, roughly 20 hours a week.

Done in-house in the US, at a fully loaded cost of about $150 an hour, that half-engineer costs about $11,250 a month. Outsourced to Central and Eastern Europe at about $80.50 an hour, it costs about $6,440. In Latin America, at about $77.63 an hour, it's about $6,210. In Asia, at about $41.40 an hour, it's about $3,312.

The US figure is what a company pays its own employee including benefits, while the outsourced figures are vendor-billed rates with overhead and margin already included. The cheapest location isn't automatically the best value, because time-zone overlap, review effort and rework can narrow the gap.

US rate derived from a $185K base salary and BLS's June 2026 benefits share (wages at 68.5% of compensation for full-time private-industry workers). Outsourced rates from Accelerance 2026 senior bands plus a 15% AI premium. Illustrative.

*What this means for the build-vs-buy-vs-outsource decision:
*
At 10K queries/month, usage is ~$200 and maintenance labour is $3,300–$11,250. Your real cost question is who maintains the system, not which model you pick.
At 1M queries/month, usage (~$13K) exceeds the labour cost in every location, and model routing and prompt efficiency start paying for an engineer's time.
A hybrid model is common: one internal owner for evaluation and architecture, with build and maintenance capacity from a nearshore or offshore team. Keep the evaluation dataset and the prompt/retrieval configuration under your own version control regardless of who writes the code.

6. Sensitivity: which levers move the bill?

To see which decisions matter most, we changed one setting at a time in the 100,000-query mid-tier scenario, which has a base cost of about $1,369 a month.

Switching to a small-tier model has by far the largest effect, cutting about $980 a month, or 72%, because tokens are the dominant cost. Retrieving three chunks instead of five removes 1,000 input tokens per query and saves $200, or 15%. Shortening answers from 400 to 250 tokens saves $150, or 11%, since output tokens cost five times as much as input. Dropping the reranker would also save $200, or 15%, but is likely to hurt answer quality.

Some changes raise costs instead. Turning off prompt caching adds about $270, or 20%, because the system prompt is then billed at the full rate. Switching observability to LangSmith with three seats adds about $257 through seat and per-trace charges. Partitioning the vector index into smaller namespaces saves essentially nothing at this volume, because you're paying Pinecone's $50 minimum either way.

These are effects on cost only. Retrieving fewer chunks or removing the reranker may lower answer quality, so test those changes before adopting them.

The lesson: optimise tokens first, reranking second, everything else later.

FAQ

**How much does a RAG query cost?
**In our model, a mid-tier RAG query costs about $0.013 all-in (~$13 per 1,000 queries) at 100K+ queries/month. With a small model it can drop below $0.003; with a frontier model it can exceed $0.05.

**What is the biggest cost in a RAG system?
**LLM tokens at scale — specifically input tokens from retrieved context. At low volume, engineering time to maintain and evaluate the system is larger than all usage costs combined.

**How much does a vector database cost for RAG?
**For a ~7 GB index (1M chunks), serverless vector search costs roughly $50–$120/month at 10K–1M queries in our model. Plan minimums dominate at low volume.

**Does prompt caching reduce RAG costs?
**Yes, but less than in chatbots with long static prompts, because retrieved context changes each query. In our example, caching cut LLM cost by about 20%.

**How much should I budget for LLM observability?
**Roughly $30–$350/month at 10K–100K queries and $650–$2,600/month at 1M queries in our model, depending on tool and seats. Sampling traces reduces this.

**Should I self-host the vector database and observability stack?
**Self-hosting removes subscription fees but adds infrastructure and engineering time. At the volumes modelled here, the subscription line items are small relative to a half-FTE engineer, so self-hosting usually makes sense for data-residency or compliance reasons rather than cost.