A typical RAG query (2,000 tokens of prompt, 2,500 tokens of retrieved context, and 400 tokens of output) on a mid-tier model costs about $0.013 in usage. Once retrieval, reranking, and observability are added, that comes to roughly $13 per 1,000 queries at production volume.
In our model, a service handling 10,000 queries a month costs about $203 in usage, or about $234 with a 15% buffer for retries, query rewrites, and evaluation runs. At 100,000 queries a month, the bill is about $1,369, or $1,574 with the buffer. At 1 million queries a month, it reaches about $13,113, or $15,079 with the buffer. Ten times the traffic doesn't mean ten times the bill at lower volumes, because fixed fees such as plan minimums are shared across more queries.
*Three findings matter more than the totals:
*
LLM tokens are ~75–80% of usage cost at scale. Vector search is under 1%.
Reranking is the surprise second line item (~15% at 100K+ queries).
At low volume, people cost more than tokens. Half an AI engineer maintaining the service costs $3,300–$11,250 a month depending on location, 15–50× the usage bill at 10K queries.
Every number below is either linked to a source or labelled as an illustrative assumption.
The cost equation
A RAG service has three kinds of cost:
Monthly cost =
queries × (LLM + query embedding + vector read + rerank + observability per query)
- fixed platform fees (plan minimums, seats, storage)
- corpus maintenance (re-embedding, writes)
- people (build, evaluation, on-call, iteration)
The rest of this article prices each term.
1. Token costs: where most of the money goes
*The per-query token budget
*
We modelled a knowledge-assistant style RAG endpoint. These are illustrative assumptions; measure your own with real traffic before you trust any forecast.
Each request starts with a 1,500-token system prompt of instructions. Because it's identical on every request, it can be cached and billed at a discounted rate. Next comes the user's message plus a short conversation history, about 500 tokens. Then the system adds the retrieved context: five chunks of 500 tokens each, or 2,500 tokens in total. Both of these change on every request, so they're billed at the full input price. Finally, the model writes an answer of about 400 tokens, billed as output. That's 4,900 tokens per request.
Retrieved context is the largest input. Every extra chunk you stuff into the prompt is paid for on every single query.
**
Current list prices (October 2026)
**
Model prices fall into four broad tiers, quoted per million tokens.
Small tier (Claude Haiku 5.5 for prompts up to 100K tokens, and OpenAI's gpt-6-luna): $0.10 for input, $0.50 for output and $0.01 for cached input.
Mid tier (Claude Sonnet 5.5 and gpt-6.1-sol): $2.00 for input, $10.00 for output and $0.10 for cached input. The two vendors price this tier identically, so the choice between them comes down to quality on your own evaluation set.
Large tier: Claude Opus 5.5 charges $4.00 for input, $20.00 for output and $0.20 for cached input. gpt-6-astra charges $10.00, $50.00 and $1.00.
Across every tier, output costs five times as much as input. That's why long answers cost more than long prompts of the same length.
Sources: Anthropic and OpenAI official pricing pages, captured October 15, 2026. Both vendors offer roughly 50% off for asynchronous batch processing, which suits offline evaluation and re-indexing but not live chat.
Per-query calculation (mid tier, with prompt caching)
System prompt: 1,500 × [0.95 × $0.10 + 0.05 × $2.50] / 1M = $0.00033
Other input: 3,000 × $2.00 / 1M = $0.00600
Output: 400 × $10.00 / 1M = $0.00400
Total ≈ $0.01033 per query
Without caching, the same query costs $0.0130. Caching saves about 20% here because the system prompt is only 30% of input; the retrieved context changes every query and can't be cached the same way.
Running the same 4,900-token request through each tier shows how much the price list matters. On a small-tier model, a query costs about $0.00052, or $0.52 per 1,000 queries. On a mid-tier model, it's about $0.01033, or $10.33 per 1,000. Claude Opus 5.5 brings it to $0.02066, or $20.66 per 1,000, and gpt-6-astra to $0.05236, or $52.36 per 1,000. Nothing about the query changes between these figures; only the model does. For budgeting, multiply the per-1,000 figure by your expected traffic in thousands.
The 100× spread between small and large tiers is the single biggest decision in your budget. Many teams route simple queries to a small model and escalate only hard ones; whether that works for you depends on your evaluation results, not on price alone.
2. Retrieval costs: embeddings, vector database and reranking
*Embeddings: cheap enough to ignore (almost)
*
OpenAI's text-embedding-3-small is listed at $0.02 per 1M tokens.
Initial corpus: 500M tokens × $0.02 / 1M = $10.00 one-off
Monthly refresh: 10% of corpus = $1.00
Per query: 50 tokens × $0.02 / 1M = $0.000001
Embedding cost is negligible. The real cost of embeddings is engineering time spent on chunking strategy and re-indexing when you change models.
*Vector database: small, but watch the pricing model
*
We used Pinecone serverless as the example because its pricing is usage-based and documented. The key rule, from Pinecone's cost documentation: a query consumes 1 read unit per GB of namespace size, with a 0.25 RU minimum, regardless of top_k.
Index size: 1M vectors × (1,536 dims × 4 bytes + 1 KB metadata) ≈ 7.14 GB
Read units: ≈ 7.14 RU per query
Read cost: 7.14 × $16 / 1M ≈ $0.000114 per query
Storage: 7.14 GB × $0.33 ≈ $2.36 / month
Plan minimum: $50 / month (Standard)
Unit rates are from a secondary source and should be confirmed in Pinecone's calculator; the $50 minimum and RU rule are from Pinecone's own pages.
At 10K and 100K queries per month, you'll pay the $50 minimum. Even at 1M queries, the vector database costs about $120/month.
*Two practical consequences:
*
Namespace size drives read cost. Partitioning by tenant or document type cuts RUs per query proportionally.
If you already run Postgres, pgvector on existing infrastructure may cost nothing incremental at this scale. The trade-off is operational: you own tuning, scaling and backups.
*Reranking: the line item people forget
*
A cross-encoder reranker takes the top 50 candidates and picks the best 5. Cohere defines one search as one query with up to 100 documents, and Rerank 4 Fast is listed at $2.00 per 1,000 searches (secondary source, verify).
Per query: $0.002
At 100K queries/month: $200
At 1M queries/month: $2,000
That's about 15% of usage cost at scale, and more than the vector database by an order of magnitude. Reranking often improves answer quality enough to let you send fewer chunks to the LLM, which can pay for itself. Test both configurations.
3. Observability costs: per-unit vs per-trace pricing
You can't run RAG in production without tracing retrieval quality, latency and token spend. The two common hosted options price differently.
Langfuse Cloud charges per unit, where every trace, observation and score you record counts as one unit. Its Core plan costs $29 a month and includes 100,000 units, with additional usage at $8 per 100,000 units (lower at higher volumes). LangSmith charges per seat and per trace. Its Plus plan costs $39 per seat per month and includes 10,000 base traces, with additional traces at about $2.50 per 1,000, kept for 14 days.
In practice, a small team with lots of traffic tends to favour unit pricing, while a large team with modest traffic may find per-trace pricing easier to predict.
Sources: Langfuse and LangChain official pricing pages. LangSmith overage rate from a June 2026 secondary source; verify before budgeting.
*Translating that to per-query cost
*
In Langfuse, a single RAG request typically produces one trace plus several observations (embed, retrieve, rerank, generate, guardrail, post-process) and maybe an automated score. Assumption: 8 units per query.
At 10,000 queries a month, that's 80,000 units, inside Langfuse's included allowance, so the bill is just the $29 plan fee. LangSmith comes to $117, which is three seats at $39 each. At 100,000 queries, Langfuse rises to about $85 and LangSmith to about $342, as per-trace charges kick in above 10,000 traces. At 1 million queries, Langfuse reaches about $661 at list rates, while LangSmith reaches about $2,592. The gap between the two widens as traffic grows.
Illustrative; both vendors discount at volume and both can be self-hosted, which swaps subscription cost for infrastructure and engineering time.
Observability can quietly reach 5–20% of usage cost. Sampling (tracing 10–20% of production traffic in full while logging tokens for 100%) is a common way to control it.
4. Three monthly scenarios, line by line
These scenarios use the mid-tier model with prompt caching on, and Langfuse for observability.
At 10,000 queries a month, LLM tokens cost $103.30, reranking $20.00 and observability $29.00. The vector database costs $50.00, which is simply Pinecone's plan minimum. Query embeddings cost a cent, and refreshing a tenth of the corpus each month costs $1.00. The total is $203.31, or $20.33 per 1,000 queries, rising to $233.81 with the 15% buffer. At this volume, the vector database is a quarter of the bill, purely because of its minimum fee.
At 100,000 queries a month, LLM tokens rise to $1,033.00 and reranking to $200.00. Observability comes to $85.00, and the vector database stays at its $50.00 minimum. Embeddings cost $0.10 and corpus refresh $1.00. The total is $1,369.10, or $13.69 per 1,000 queries, and $1,574.46 with the buffer.
At 1 million queries a month, LLM tokens reach $10,330.00 and reranking $2,000.00. Observability is $661.00, and the vector database finally rises above its minimum to $119.52. Embeddings cost $1.00 and corpus refresh $1.00. The total is $13,112.52, or $13.11 per 1,000 queries, and $15,079.40 with the buffer. LLM tokens now account for almost four-fifths of spending and reranking for most of the rest, while embeddings and storage barely register.
*What happens if you change the model
*
Holding everything else constant at 1 million queries a month, the model choice dominates the bill. With a small-tier model, LLM cost falls to about $524 and the usage total to about $3,300. The mid-tier model gives the $10,330 LLM cost and ~$13,100 total above. Claude Opus 5.5 pushes LLM cost to $20,660 and the total to about $23,400, and gpt-6-astra to $52,363 and about $55,100. Each total includes the same $2,783 of retrieval, reranking and observability, so the difference is entirely the LLM line. Moving from the cheapest tier to the most expensive one makes the bill roughly 17 times larger.
Per-1,000-query cost falls as volume grows only because fixed fees get spread thinner. After roughly 100K queries, the bill becomes almost entirely variable.
5. The line item that dwarfs tokens at low volume: people
A production RAG service needs ongoing work: evaluation sets, chunking and prompt changes, regression testing when models update, cost monitoring, and incident response. Assumption: 0.5 FTE AI engineer for maintenance, roughly 20 hours a week.
Done in-house in the US, at a fully loaded cost of about $150 an hour, that half-engineer costs about $11,250 a month. Outsourced to Central and Eastern Europe at about $80.50 an hour, it costs about $6,440. In Latin America, at about $77.63 an hour, it's about $6,210. In Asia, at about $41.40 an hour, it's about $3,312.
The US figure is what a company pays its own employee including benefits, while the outsourced figures are vendor-billed rates with overhead and margin already included. The cheapest location isn't automatically the best value, because time-zone overlap, review effort and rework can narrow the gap.
US rate derived from a $185K base salary and BLS's June 2026 benefits share (wages at 68.5% of compensation for full-time private-industry workers). Outsourced rates from Accelerance 2026 senior bands plus a 15% AI premium. Illustrative.
*What this means for the build-vs-buy-vs-outsource decision:
*
At 10K queries/month, usage is ~$200 and maintenance labour is $3,300–$11,250. Your real cost question is who maintains the system, not which model you pick.
At 1M queries/month, usage (~$13K) exceeds the labour cost in every location, and model routing and prompt efficiency start paying for an engineer's time.
A hybrid model is common: one internal owner for evaluation and architecture, with build and maintenance capacity from a nearshore or offshore team. Keep the evaluation dataset and the prompt/retrieval configuration under your own version control regardless of who writes the code.
6. Sensitivity: which levers move the bill?
To see which decisions matter most, we changed one setting at a time in the 100,000-query mid-tier scenario, which has a base cost of about $1,369 a month.
Switching to a small-tier model has by far the largest effect, cutting about $980 a month, or 72%, because tokens are the dominant cost. Retrieving three chunks instead of five removes 1,000 input tokens per query and saves $200, or 15%. Shortening answers from 400 to 250 tokens saves $150, or 11%, since output tokens cost five times as much as input. Dropping the reranker would also save $200, or 15%, but is likely to hurt answer quality.
Some changes raise costs instead. Turning off prompt caching adds about $270, or 20%, because the system prompt is then billed at the full rate. Switching observability to LangSmith with three seats adds about $257 through seat and per-trace charges. Partitioning the vector index into smaller namespaces saves essentially nothing at this volume, because you're paying Pinecone's $50 minimum either way.
These are effects on cost only. Retrieving fewer chunks or removing the reranker may lower answer quality, so test those changes before adopting them.
The lesson: optimise tokens first, reranking second, everything else later.
FAQ
**How much does a RAG query cost?
**In our model, a mid-tier RAG query costs about $0.013 all-in (~$13 per 1,000 queries) at 100K+ queries/month. With a small model it can drop below $0.003; with a frontier model it can exceed $0.05.
**What is the biggest cost in a RAG system?
**LLM tokens at scale — specifically input tokens from retrieved context. At low volume, engineering time to maintain and evaluate the system is larger than all usage costs combined.
**How much does a vector database cost for RAG?
**For a ~7 GB index (1M chunks), serverless vector search costs roughly $50–$120/month at 10K–1M queries in our model. Plan minimums dominate at low volume.
**Does prompt caching reduce RAG costs?
**Yes, but less than in chatbots with long static prompts, because retrieved context changes each query. In our example, caching cut LLM cost by about 20%.
**How much should I budget for LLM observability?
**Roughly $30–$350/month at 10K–100K queries and $650–$2,600/month at 1M queries in our model, depending on tool and seats. Sampling traces reduces this.
**Should I self-host the vector database and observability stack?
**Self-hosting removes subscription fees but adds infrastructure and engineering time. At the volumes modelled here, the subscription line items are small relative to a half-FTE engineer, so self-hosting usually makes sense for data-residency or compliance reasons rather than cost.
Top comments (0)