DEV Community: Eric Thompson

Self-Hosted vs Managed Vector Databases: A Cost Modeling Guide

Eric Thompson — Fri, 13 Nov 2026 10:04:50 +0000

For most companies under about 10 million vectors, a managed vector database costs $50 to $1,500 a month and is cheaper than self-hosting once engineering time is counted. Self-hosting starts to win at high query volume, at 50–100 million vectors and above, or when you can reuse an existing Postgres or platform team.

Three variables decide the bill:

Memory footprint: vector count multiplied by dimensions, plus index overhead. This sets the size of the machines you need when self-hosting, and the storage line when you buy managed.
Query and write volume: usage-priced services such as Pinecone Serverless charge per read and write, so busy applications pay more each month. Capacity-priced options, including self-hosting, charge the same whether you run one query or a million.
People: a self-hosted cluster needs someone to patch, scale, back up, tune and be on call for it. That cost exists even when the infrastructure is cheap.
The common mistake is comparing a managed price per gigabyte with the raw cost of a virtual machine. A fair comparison counts replicas, backups, index rebuilds and the engineer who keeps it all running.
All figures in this guide are planning estimates based on published list prices as of November 2026. Vendors changed pricing several times during 2026, so check current rates before you commit.

Step one: size your vectors

Before comparing prices, work out how much memory and storage your vectors need. Every cost model starts from bytes, not from vector counts.

The basic math

Each dimension of a standard float32 vector takes 4 bytes. A 1,536-dimension embedding is about 6.1 KB.
An HNSW index, the most common search index, adds graph structure on top. A vector index sizing guide suggests roughly doubling raw vector size to account for it.
Add metadata (document IDs, text chunks, filters) and replicas for availability.

What that means at common scales (1,536 dimensions)

1 million vectors: about 6 GB of raw vectors, roughly 12–25 GB in memory with the index.
10 million vectors: about 61 GB raw. One analysis of pgvector in production budgets 20–25 KB per vector with the graph, or 200–250 GB, if nothing is compressed.
100 million vectors: over 600 GB raw before the index. At this scale, ClickHouse's engineering guidance suggests evaluating whether standard in-memory HNSW is still the right structure.

Compression changes everything

Quantization shrinks vectors with a small loss in recall:
Half precision (float16, or halfvec in pgvector) halves memory.
Scalar quantization to int8 cuts memory by about 75%.
Binary or product quantization can cut memory by up to 90% or more, usually paired with re-ranking to recover accuracy.
Smaller embedding models help too. A 768-dimension model needs half the memory of a 1,536-dimension one, and many newer models let you request shorter vectors directly.
Compression is the single biggest cost lever in vector search, and it applies whether you self-host or buy managed.

How managed vector databases are priced

Managed vector databases use three main pricing models: usage-based, per-dimension and capacity-based. The same workload can cost very different amounts depending on which model you are on.

Usage-based: Pinecone Serverless

Pinecone's pricing page lists storage at $0.33 per GB per month, with Standard plan read units from $16 per million and write units from $4 per million, plus a $50 monthly minimum. Enterprise starts at $500 a month with higher unit rates.
The detail that drives cost: according to UsagePricing's breakdown, a query uses roughly 1 read unit per GB of namespace searched, with a minimum of 0.25, and a write uses 1 write unit per KB. In practice, large single namespaces make every query expensive, while splitting data into many smaller namespaces (for example, one per customer) keeps reads cheap.
Best fit: small to mid-size indexes, spiky or low query volume, multi-tenant apps that split data by namespace.

Per-dimension: Weaviate Cloud

Weaviate Cloud bills mainly on vector dimensions stored. After a January 2026 restructure, UsagePricing reports a Flex plan at $45 a month with an entry rate of about $0.00465 per million dimensions, with Premium plans from $400 a month. Higher-dimension embeddings cost proportionally more, so compression and smaller models reduce the bill directly.
Best fit: teams that want hybrid keyword and vector search built in, with a predictable monthly bill.

Capacity-based: Qdrant Cloud, Zilliz Cloud and managed Postgres

Qdrant Cloud bills hourly for the vCPU, memory and storage you allocate, with no per-query fees. Zilliz Cloud (managed Milvus) uses compute units. Managed Postgres with pgvector, such as AWS RDS, Supabase or Neon, bills for the database instance.
Capacity pricing gets cheaper per query as traffic grows, until you need a bigger node.
Best fit: steady, high query volume, and teams that already run Postgres.

Light-load comparisons can mislead

Published comparisons often assume light traffic. One 2026 comparison at 10 million 768-dimension vectors put Qdrant Cloud at about $65 a month, Pinecone at $70, pgvector on RDS at $120 and Weaviate Cloud at $135. Those numbers are useful for small workloads but say little about cost at production query volume, which is where the models diverge.

What self-hosting really costs

Self-hosting replaces a vendor bill with three costs: compute, operations and people. The software is free; running it reliably is not.
Compute and storage
You run Qdrant, Milvus, Weaviate or Postgres with pgvector on your own virtual machines or Kubernetes cluster. Memory-optimized instances are the main cost:
A 32 GB memory-optimized cloud instance costs roughly 150–250 a month on demand.
A 128 GB instance costs roughly 600–900 a month.
A 256 GB instance costs roughly 1,200–1,800 a month.
Plan for at least two replicas for availability, plus disk, snapshots and backups. Reserved instances or committed-use discounts typically cut compute by 30–50%.

Operations

Index builds and rebuilds: HNSW builds are memory-hungry. A guide to scaling pgvector estimates about 60 GB of working memory to build an index on 5 million 1,536-dimension vectors, so builds may need temporary larger machines.
Re-embedding: switching embedding models means re-embedding and re-indexing the whole corpus, which needs spare capacity.
Scaling and sharding: past a single node, you manage shards, replicas and rebalancing.
Security: network isolation, encryption, access control and patching.
Monitoring: latency, recall, memory pressure and disk usage.

People

Running a production vector database needs part of an engineer's time for upgrades, tuning, incidents and on-call. As a planning range:
Small single-node deployment: about 10% of one engineer.
Mid-size replicated cluster: about 25% of one engineer.
Large sharded cluster: about 50% of one engineer or more.
At a fully loaded cost of $200,000 a year, that is roughly $1,700 to $8,300 a month, often more than the infrastructure itself.

The big exception: you already run Postgres

If your team already operates Postgres in production, adding pgvector for a few million vectors can cost almost nothing extra. Backups, monitoring, access control and on-call already exist. This is why pgvector is often the cheapest option for small and mid-size RAG workloads.

A cost model at three scales

At small scale, managed services win clearly. At mid scale, the answer depends on query volume and whether you can share engineering time. At very large scale, self-hosting usually wins on cost if you have the team.
The assumptions: 1,536-dimension embeddings, a RAG workload with 1 million queries a month at small scale and 5 million at mid and large scale, and monthly re-indexing of about 10% of vectors. Engineering time is costed at $200,000 a year fully loaded.

Scale 1: 1 million vectors

Pinecone Serverless: about 7 GB stored costs about $2; at roughly 7 read units per query, 1 million queries cost about $112; writes are minor. Total about $120 a month.
Weaviate Cloud Flex: about 1.5 billion dimensions at the entry rate is a few dollars, so the $45 plan minimum applies. Total about 45–100 a month.
Self-hosted on new infrastructure: two 32 GB instances plus storage cost about $400, plus about 10% of an engineer at about $1,700. Total about $2,100 a month.
pgvector on existing Postgres: often under $200 a month in extra capacity.
Verdict: managed, or pgvector on Postgres you already run.

Scale 2: 10 million vectors

Pinecone Serverless in one large namespace: about 70 GB stored costs about $23, but at roughly 70 read units per query, 5 million queries cost about $5,600. Total about $5,700 a month. Split into many smaller namespaces, reads drop sharply.
Weaviate Cloud: about 15 billion dimensions at the entry rate is about $71 a month in dimension charges, plus plan minimums and any compute or compression add-ons.
Self-hosted with int8 quantization: two 128 GB instances plus storage cost about $1,600, plus about 25% of an engineer at about $4,200. Total about $5,800 a month.
Qdrant Cloud or similar capacity-priced managed service: typically a few hundred to a few thousand dollars a month depending on the cluster, with no per-query fees.
Verdict: a capacity-priced managed service or a well-partitioned usage-based index is usually cheapest. Self-hosting only wins here if engineering time is shared with an existing platform team.

Scale 3: 100 million vectors

Pinecone Serverless in large namespaces: read costs become very large at millions of queries a month. Dedicated read capacity, aggressive namespace partitioning or an enterprise agreement becomes necessary.
Self-hosted with quantization and sharding: four 256 GB instances (two shards, two replicas) plus storage cost about $6,500, plus about 50% of an engineer at about $8,300. Total about $15,000 a month.
Managed capacity-priced services: one 2026 comparison estimated about $480 a month on Qdrant Cloud versus about $280 self-hosted at 100 million vectors and 10 million queries a month, but that assumes compact vectors and a lean configuration. Real costs scale with dimensions, replicas and recall targets.
Verdict: self-hosting or a negotiated enterprise contract usually wins, and compression, disk-based indexes such as DiskANN, and partitioning matter more than the vendor.

What the model shows

Usage-based pricing is cheapest at low volume and becomes expensive when large indexes meet high query volume.
Capacity-based pricing, managed or self-hosted, gets cheaper per query as traffic grows.
People cost dominates self-hosting below roughly 50 million vectors unless the engineers are already on payroll for other infrastructure.
Treat these as illustrative numbers; plug in your own vector count, dimensions, query volume and salaries.

How to decide, and how to cut costs

Choose managed when speed and low operational load matter most, and self-host when scale, data control or existing infrastructure make it cheaper. Whichever you pick, a few levers cut cost on any platform.
Choose managed when
You have fewer than about 10 million vectors, or traffic that is spiky or unpredictable.
You do not have database or platform engineers with spare capacity.
Speed to production matters more than unit cost.
You need built-in features such as hybrid search, reranking or multi-tenancy without building them.

Choose self-hosted when

You already run Postgres, Kubernetes or search infrastructure with an experienced team.
You have 50–100 million vectors or more, or very high, steady query volume.
Data residency, air-gapped deployment or strict compliance rules out third-party hosting.
You need custom indexing, filtering or hardware choices that managed services do not offer.

Eight ways to cut vector database costs

Quantize: use half precision, int8 or binary quantization with re-ranking. It is the biggest saving available.
Use smaller embeddings: choose a 768-dimension or shorter output where retrieval quality holds up in your evaluations.
Partition by tenant or topic: smaller namespaces or collections make each query cheaper on usage-based pricing and faster everywhere.
Index only what you search: keep raw text and rarely queried metadata in cheaper storage, linked by ID.
Delete stale vectors: remove outdated document versions and duplicates on a schedule.
Re-embed in batches: plan model upgrades as batch jobs on temporary capacity, not on your production cluster.
Cache frequent queries: many RAG workloads repeat the same questions.
Start on pgvector if you already run Postgres, and move to a dedicated vector database only when measured latency, scale or features require it.

Frequently asked questions

How much does a vector database cost?

A vector database typically costs $50 to $1,500 a month for up to 10 million vectors on a managed service, and several thousand dollars a month at larger scale. Self-hosting has lower infrastructure cost but adds engineering time that often exceeds the hardware.

Is self-hosting a vector database cheaper than Pinecone?

It can be at high query volume or above roughly 50 million vectors, especially if your team already runs similar infrastructure. Below that, Pinecone or another managed service is usually cheaper once engineering time is included.

How much memory do I need for 1 million vectors?

About 6 GB for 1 million 1,536-dimension float32 vectors, and roughly 12–25 GB once an HNSW index and metadata are added. Quantization can reduce this by 50–90%.

Is pgvector good enough for production?

Yes, for many workloads up to several million vectors, especially if you already run Postgres. Past about 10 million high-dimension vectors, you will need quantization, memory tuning and possibly partitioning, or a dedicated vector database.

Why is my Pinecone bill higher than expected?

Read costs scale with the size of the namespace each query searches, not just the number of queries. Large single namespaces make every query expensive; splitting data into smaller namespaces and reducing vector size usually lowers the bill.

What is the cheapest vector database?

For small projects, free tiers from Pinecone, Qdrant and Weaviate, or pgvector on an existing Postgres database, cost little or nothing. At large scale, self-hosted open-source options such as Qdrant, Milvus or pgvector are cheapest if you have the team to run them.

Should I switch vector databases later?

Plan for it. Keep embeddings and source documents in your own storage so you can re-index into another database. Switching usually costs a re-embedding or re-indexing run plus testing, not a rebuild of your application.

Get a vector database cost model for your workload

The cheapest option depends on your vector count, query patterns and team. CodingCops designs and runs retrieval infrastructure for RAG and AI agents, managed or self-hosted, and sizes it against real workloads before you commit. Talk to CodingCops for a cost model built on your data.

How to Estimate the Infrastructure Cost of a Production RAG Pipeline

Eric Thompson — Wed, 11 Nov 2026 11:32:07 +0000

In our illustrative model, a production RAG pipeline costs about $1,350 a month for an internal assistant answering 100,000 questions, $6,550 for a support assistant answering 500,000, and $28,350 for enterprise search answering 2 million. That is infrastructure only, before the engineers who run it.
The surprise for most teams is where the money goes. LLM generation is 78–84% of the bill in every scenario, because each answer sends several thousand tokens of retrieved text to the model. Embedding the whole corpus costs between about $1 and $115, once. The vector database is often the smallest line, until the index grows large enough that every query scans tens of gigabytes.
So the biggest savings come from sending fewer, better chunks to the model, not from shopping for a cheaper vector store. The method below shows each component and its formula, so you can plug in your own corpus size and query volume.

The six cost components of a RAG pipeline

A RAG pipeline has two halves: building and maintaining the index, and answering each query. Most of the cost sits in the second half.

Index side (mostly one-time, plus churn)

Ingestion and chunking. Parsing documents, splitting them into chunks and attaching metadata. Usually compute you already run.
Embedding the corpus. Converting every chunk into a vector, once at the start and again whenever content changes or you switch embedding models.
Vector storage and writes. Storing the vectors and their metadata, and paying to write or overwrite them.

Query side (every request)

Query embedding and vector search. Embedding the user's question and searching the index.
Reranking. Re-scoring the top results with a more accurate model, so only the best chunks reach the LLM. Optional but common.
LLM generation. Sending the question plus retrieved chunks to the model and generating the answer.
On top of these sit operating costs that teams often leave out: evaluation, observability, re-indexing projects and the engineers who maintain the pipeline. Those are covered later.

Ingestion and embedding costs

Embedding a corpus is far cheaper than most teams expect. At current API prices, even a five-billion-token document collection costs about $115 to embed.

Sourced prices

OpenAI prices text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13 per million. Both support a dimensions parameter that shortens the stored vector; text-embedding-3-large produces up to 3,072 dimensions (OpenAI, Jan 2024; current price confirmed on OpenAI's model page).

The formula

\text{Embedding cost} = \frac{\text{corpus tokens} \times (1 + \text{overlap})}{10^6} \times \text{price per million tokens}
Worked examples (illustrative, text-embedding-3-small, 15% overlap)
50 million tokens (a mid-sized internal knowledge base): 50 × 1.15 × $0.02 = $1.15
200 million tokens (a product's full help center, tickets and manuals): $4.60
5 billion tokens (an enterprise document archive): 5,000 × 1.15 × $0.02 = $115
With text-embedding-3-large the same figures are 6.5 times higher, still small.

Where the real ingestion cost sits

Parsing. PDFs with tables, scanned documents and slide decks need extraction or OCR, which can cost more in compute and engineering time than the embeddings themselves.
Re-embedding when you change models. Switching embedding models means re-embedding everything, because vectors from different models cannot be mixed in one index. The API cost is small; the migration work and testing are not.
Ongoing churn. At 10% of content changing per month, re-embedding costs pennies to about $12 a month in our scenarios. The pipeline that detects and processes those changes is the real cost.

Vector storage and search

Managed vector databases are cheap for small indexes and get expensive in one specific way: on usage-based pricing, query cost can grow with index size, not just with query count.

Sourced pricing: Pinecone as an example

Pinecone's on-demand Standard plan has a $50 monthly minimum, then charges $0.33 per GB-month of storage, $4 to $4.50 per million write units and $16 to $18 per million read units, varying by cloud and region (Pinecone pricing). Its cost documentation sets the rules (Pinecone docs):
Storage per record: ID bytes + metadata bytes + (dimensions × 4 bytes) for dense vectors
Writes: 1 write unit per KB of upsert request, minimum 5 per request
Queries: 1 read unit per GB of namespace queried, minimum 0.25 per query, regardless of how many results are returned
Other managed and self-managed options (pgvector on an existing Postgres, open-source vector databases, search engines with vector support) price differently, often by instance size rather than usage. The sizing math below still applies.

Sizing an index (illustrative)

One chunk stored at 1,536 dimensions with 1,000 bytes of metadata: 8 + 1,536 × 4 + 1,000 = 7,152 bytes.
100,000 chunks: about 0.72 GB
400,000 chunks: about 2.86 GB
10 million chunks: about 71.5 GB

Why index size drives query cost

Under the read-unit rule, a query against a 71.5 GB namespace costs 71.5 read units. At $16 per million, that is about $0.0011 per query, or $2,289 for 2 million queries a month. The same query against a 0.72 GB index costs one-hundredth as much.
The practical fix is partitioning. If each query only needs one department's, customer's or product line's documents, store them in separate namespaces so each search scans only the relevant slice.

Query-time costs: embedding, reranking and generation

Every question triggers three paid steps. The first is negligible, the second is moderate, and the third dominates.

Query embedding

A 30-token question at $0.02 per million tokens costs $0.0000006. Even 2 million questions a month cost about $1.20.

Reranking

Rerankers re-score the top 20 or so search results so only the best few reach the LLM. Pricing is in flux:
Cohere previously charged $2.00 per 1,000 searches for Rerank 4 Fast and $2.50 for Rerank 4 Pro.
According to UsagePricing's capture of Cohere's rate card, Rerank 5 (released October 2026) moved to per-token billing at a reported $0.02 per million tokens for Fast and $0.05 for Pro (UsagePricing, Oct 2026). Verify on Cohere's own pricing page before budgeting.
Under per-token pricing, cost depends on how much text you rerank: 20 chunks of 500 tokens is 10,000 tokens per query. Short snippets get cheaper; long passages may not. This article uses the older $2 per 1,000 searches as a conservative estimate.

LLM generation

This is the line that matters. Each answer sends the system prompt, the question and the retrieved chunks to the model.
\text{Generation cost} = \frac{\text{queries} \times \text{input tokens}}{10^6} \times p_{in} + \frac{\text{queries} \times \text{output tokens}}{10^6} \times p_{out}
With 5 chunks of 500 tokens, plus a 500-token system prompt and the question, a typical request carries about 3,500 input tokens and returns about 400 output tokens. At $2 and $10 per million (Claude Sonnet 5.5 standard, as used in our batch vs real-time article):
Per query: 3,500 × $2 ÷ 10⁶ + 400 × $10 ÷ 10⁶ = $0.007 + $0.004 = $0.011

Per 100,000 queries: $1,100

Retrieved context, not the answer, is most of the input. Every extra chunk adds about $0.001 per query at these prices.

Cost model: three RAG deployments (illustrative)

In all three scenarios, LLM generation is 78–84% of monthly infrastructure cost and the vector database is under 10%. All use the assumptions above.
Scenario A: internal knowledge assistant
50 million-token corpus (100,000 chunks, 0.72 GB index); 100,000 questions a month.
One-time setup: embedding $1.15 + initial writes $2.86 = about $4
Vector database: storage $0.24 + reads $1.14 + churn writes $0.29 = $1.67, billed at the $50 minimum
Content churn re-embedding: $0.11
Query embedding: $0.06
Reranking: 100 × $2 = $200
LLM generation: 100,000 × $0.011 = $1,100
Monthly total: about $1,350, of which generation is 81%

Scenario B: customer support assistant

200 million-token corpus (400,000 chunks, 2.86 GB index); 500,000 questions a month.
One-time setup: embedding $4.60 + initial writes $11.44 = about $16
Vector database: storage $0.94 + reads $22.89 + churn writes $1.14 = $24.97, billed at the $50 minimum
Content churn re-embedding: $0.46
Query embedding: $0.30
Reranking: 500 × $2 = $1,000
LLM generation: 500,000 × $0.011 = $5,500
Monthly total: about $6,551, of which generation is 84%

Scenario C: enterprise document search

5 billion-token corpus (10 million chunks, 71.5 GB index in one namespace); 2 million questions a month.
One-time setup: embedding $115 + initial writes $286 = about $401
Vector database: storage $23.60 + reads $2,288.64 + churn writes $28.61 = $2,341
Content churn re-embedding: $11.50
Query embedding: $1.20
Reranking: 2,000 × $2 = $4,000
LLM generation: 2,000,000 × $0.011 = $22,000
Monthly total: about $28,354, of which generation is 78%
What changes the picture
Partitioning Scenario C into ten equal namespaces would cut vector read cost about tenfold, to roughly $229 a month.
Per-token reranking, if the reported Rerank 5 prices hold, would cut reranking to about $20, $100 and $400 a month in A, B and C.
A cheaper model for generation moves the total far more than any retrieval change. Test quality first.

Costs teams forget

The infrastructure bill above is the smaller part of running RAG in production. People and process add more.
Engineering. Someone owns ingestion pipelines, chunking strategy, retrieval tuning, access controls and upgrades. Using our maintenance article's figures, half a mid-level AI engineer costs about $6,336 a month from a Latin America team ($76,032 a year) or $11,913 in-house in the US ($142,955 a year). In Scenario A that is several times the infrastructure bill; in Scenario B it roughly equals it (Latin America) or nearly doubles it (in-house).
Evaluation. RAG quality fails quietly: the wrong chunk is retrieved and the model answers confidently anyway. Budget a test set of real questions with known correct sources, rerun on every change, plus human review of sampled answers.
Re-indexing projects. Changing chunk size, embedding model or metadata schema means rebuilding the index. The API bill is small; the engineering and testing time is not.
Access control. Enterprise RAG must respect who is allowed to see which document. Mapping permissions into metadata filters and keeping them in sync is real work, and a mistake is a data leak.
Observability. Tracing which chunks were retrieved for each answer is how you debug wrong answers. See our maintenance article for tool pricing.
Document parsing at scale. Tables, scans and slides may need specialist extraction services or OCR.
Environments. A separate staging index for testing changes doubles storage and write costs, though at these prices that is usually small.

Levers that cut RAG cost without hurting quality

Because generation dominates, the most effective savings reduce the tokens sent to the model. Test every change against your evaluation set.
Retrieve fewer, better chunks. Reranking 20 results and sending the best 3 instead of 5 cuts about 1,000 input tokens per query: roughly $2 per 1,000 queries at the prices above, often with no loss in answer quality.
Right-size chunks. Smaller chunks mean less irrelevant text per retrieved passage. Too small and context is lost; test a few sizes.
Cache the system prompt. It repeats on every request. Prompt caching cuts its cost sharply on providers that support it (see our batch vs real-time article).
Route by difficulty. Send simple, high-confidence questions to a smaller, cheaper model and reserve the larger one for hard cases.
Cache frequent answers. Many support and internal questions repeat. Serving a stored answer for a repeated question costs nothing in generation.
Cap output length. Ask for concise answers with citations, not essays. Output tokens cost five times input tokens at the prices used here.
Partition the index. Separate namespaces by tenant, department or product line, so each search scans only what it needs.
Shorten embeddings if quality allows. Fewer dimensions mean smaller storage and, on read-unit pricing, cheaper queries. Test recall first.
Batch offline work. Re-indexing, evaluation runs and summary generation can use batch APIs at half price.

Where CodingCops fits

Disclosure: the author works with CodingCops. This section uses only the company's published descriptions and includes no CodingCops pricing, client results or savings claims.
CodingCops is a Chicago-headquartered software company (Built In Chicago). Its company profile lists AI and machine learning, cloud and DevOps, custom software development, and dedicated engineering teams among its services (Outsource Accelerator), and it offers engagements for specific projects, short-term initiatives or long-term development (codingcops.com).
The ongoing engineering behind a RAG pipeline, ingestion, retrieval tuning, access control and evaluation, is the kind of work a dedicated team can cover. Whoever builds it, ask for a written cost estimate using the components above, an evaluation set you own, and documentation of how permissions map into retrieval.

FAQs

How much does a RAG pipeline cost per month?

In our illustrative model, about $1,350 a month for 100,000 questions, $6,550 for 500,000, and $28,350 for 2 million, infrastructure only. Engineering and evaluation come on top.

What is the most expensive part of RAG?

LLM generation: 78–84% of infrastructure cost in every scenario we modeled, because each answer sends thousands of tokens of retrieved text to the model.

How much does it cost to embed documents?

Very little. At $0.02 per million tokens, embedding 5 billion tokens costs about $115 once. Parsing messy documents usually costs more than the embeddings.

How much does a vector database cost?

For small and mid-sized indexes, often just the plan minimum, such as Pinecone's $50 a month. On read-unit pricing, query cost grows with index size, so very large single indexes can cost thousands a month unless partitioned.

Do I need a reranker?

Not always, but reranking lets you send fewer, better chunks to the LLM, which usually saves more on generation than the reranker costs. Test on your own questions.

How can I reduce RAG costs?

Send fewer chunks, cache the system prompt and frequent answers, route easy questions to a smaller model, cap output length and partition large indexes.

The Cost of Self-Hosting Open-Source LLMs: GPUs, Staffing, and Operations

Eric Thompson — Fri, 06 Nov 2026 14:13:22 +0000

Self-hosting an open-source LLM in production costs roughly $150,000 to $2 million or more per year in 2026. GPUs are only part of the bill: for most companies, the engineers who run the stack cost as much as or more than the hardware. Self-hosting usually beats paying for an API only at high, steady volume or when data rules require it.

• Small deployment, $150K–$300K a year: a 7B–32B model on 1–2 rented GPUs with business-hours load. GPUs run $15K–$45K a year; staffing is part of 1–2 engineers' time, about $100K–$200K.
• Mid-size deployment, $400K–$900K a year: a 70B-class model on 4–8 rented H100s or H200s with high availability. GPUs run $90K–$300K a year; staffing is 2–3 engineers, about $250K–$500K.
• Large deployment, $1.2M–$3M a year: a large mixture-of-experts model on 16–32 H200s or B200s across regions. GPUs run $500K–$1.5M a year; staffing is 4–6 engineers, about $600K–$1.2M.

These are planning estimates as of November 2026, using specialist GPU cloud rates and US-market salaries. Teams in lower-cost regions, or those that share engineers with existing platform teams, can land well below them.

The most common mistake is pricing a GPU-hour and stopping there. A self-hosted model also needs a serving stack, autoscaling, monitoring, security reviews, model upgrades and an on-call rotation. Those costs exist whether you serve one request a day or a million.

GPU costs: renting vs buying

For most companies, renting GPUs from a specialist cloud is the cheapest way to start self-hosting, and buying only pays off at near-constant utilization. Prices vary more by provider than by chip, so shopping around matters more than picking the newest GPU.

Cloud rental prices in late 2026

• H100 (80GB): roughly $2.00–$4.00 per GPU-hour on specialist clouds such as Lambda, CoreWeave, RunPod and Crusoe. GMI Cloud's 2026 analysis puts hyperscaler H100 bundles at $5–$8 per GPU-hour before commitments.
• H200 (141GB): roughly $2.60–$6.30 per GPU-hour on specialist clouds. The extra memory makes it the better fit for 70B-class models and long context.

• B200: roughly $4.00–$8.60 per GPU-hour, with capacity still tight in 2026. AWS Capacity Blocks list about $5.19 per GPU-hour for H100 and $10.58 for B200.

• Smaller cards (L40S, A100): about $1–$2.50 per GPU-hour, often enough for 7B–14B models.

Reserved and committed contracts typically cut 20–40% off on-demand rates. Spot or preemptible capacity is cheaper still, but only suits batch work that can tolerate interruption.

Buying GPUs outright

• A single H100 costs about $25,000–$40,000, and a full 8-GPU H200 server about $370,000, according to 2026 market data.
• 8-GPU B200 servers run roughly $400,000–$500,000, per IntuitionLabs' 2026 price index.

• Add colocation or data center space, power, cooling, networking and spares. An 8-GPU server can draw 10+ kilowatts.
The break-even is longer than it looks. One worked example from Spheron shows a $35,000 H100 needs about 17,400 hours of use at a $2.01 rental rate to pay for itself: roughly two years at 24/7 utilization, or six years at eight hours a day. Hardware also loses value as each new GPU generation arrives.

How many GPUs does your model need?

Memory decides the minimum, and traffic decides the rest.
• 7B–14B models: one 24–48GB card, or one H100 for high throughput.
• 32B models: one H100 at FP8, or two smaller cards.
• 70B models: one 80GB H100 at FP8 just fits, as Spheron's H100 benchmarks note; two GPUs or one H200 give headroom for longer context and more users.

• Large mixture-of-experts models (DeepSeek-class, gpt-oss-120B and larger): a full 8-GPU node or more.

Then double it for production. High availability means at least two replicas, and peak traffic needs spare capacity on top.

Utilization is the number that matters

Cost per token depends on how busy the GPUs are. SemiAnalysis InferenceX benchmarks show Llama 3.3 70B on one H100 serving about 1,465 tokens per second at interactive speeds, roughly $0.25 per million tokens. That figure assumes the GPU is fully loaded. At 20% average utilization, the real cost per token is five times higher, which is where many self-hosted deployments quietly lose money.

Staffing costs: the line item most budgets miss

People are usually the largest cost of self-hosting, because a production LLM needs engineers who can run GPUs, serving software and on-call support around the clock. With an API, the provider carries that work; when you self-host, you do.

The roles you need

• ML infrastructure or platform engineer: runs GPU clusters, Kubernetes, autoscaling and the inference engine (vLLM, SGLang or TensorRT-LLM).

Second Talent's 2026 benchmarks put US salaries at $150,000–$220,000.

• MLOps engineer: handles model deployment, versioning, evaluation, monitoring and upgrades. The same source puts US pay at $130,000–$200,000, and KORE1's 2026 guide notes senior roles at top firms pass $300,000 in total compensation.

• Applied ML engineer (optional): tunes prompts, fine-tunes models and runs quality evaluations.

• Share of SRE, security and FinOps time: incident response, access controls, cost tracking.

Add 25–40% on top of salary for benefits, taxes, equipment and recruiting to get the fully loaded cost.

How many engineers?

• Small deployment: 0.5–1.5 engineers, often borrowed from an existing platform team.

• Mid-size production deployment: 2–3 engineers, enough for a real on-call rotation.

• Large or multi-model platform: 4–6 or more, with dedicated owners for serving, evaluation and infrastructure.

A 24/7 on-call rotation is hard to staff sustainably with fewer than three people. This is the hidden floor on self-hosting cost: even a lightly used model needs someone awake when it breaks.

The hiring timeline is a cost too

GPU infrastructure skills are scarce, and senior hires often take three to six months to land. During that time the project either waits or runs on an API anyway. Many companies close the gap with a partner team that runs the stack until in-house staff are ready to take over.

Operations costs beyond GPUs and people

Operations typically add 15–30% on top of GPU spend, covering the serving stack, storage, networking, monitoring, security and the steady work of upgrading models. None of it is optional in production.

Serving stack and orchestration

Open-source inference engines such as vLLM, SGLang and TensorRT-LLM are free to use, but not free to run. Expect to budget engineering time for Kubernetes or a managed GPU platform, a load balancer and API gateway, request queuing, autoscaling and an OpenAI-compatible API layer so applications can switch models easily. Managed control planes and enterprise support contracts, where used, typically add $1,000–$10,000 a month.

Storage, networking and egress

Model weights for a 70B model take about 140GB at 16-bit precision, and teams usually keep several versions. Add fast local NVMe for loading, object storage for checkpoints and logs, and data egress fees if traffic leaves the cloud. This line often lands at $500–$5,000 a month.

Monitoring and evaluation

You need to see latency, throughput, GPU utilization, error rates and output quality. Tools such as Prometheus, Grafana, Datadog, Langfuse or Arize cost $200–$3,000 a month at typical scale, plus the work of maintaining evaluation datasets so each model upgrade can be checked before release.

Security and compliance

Self-hosting is often chosen for data control, but that control has to be built: network isolation, encryption, access logging, vulnerability patching of inference containers, and audits for SOC 2, HIPAA or GDPR. Budget security review time for every new model, since open weights from public repositories need vetting before production use.

Model upgrades

Open-source models improve every few months. Each upgrade means benchmarking, re-quantizing, load testing and rolling out without downtime. Teams that skip upgrades save effort but fall behind API models on quality, which erodes the case for self-hosting over time.

Power and facilities (owned hardware only)

If you buy GPUs, add colocation space, power and cooling. A single 700W H100 draws about 6,100 kWh a year; with server overhead and cooling, an 8-GPU node can cost $15,000–$30,000 a year in power and space at typical colocation rates.

Self-hosting vs API: where is the break-even?

Self-hosting a 70B-class model usually breaks even with a comparable API somewhere around 20–50 billion tokens a month, depending mostly on how much staff time the deployment needs. Below that, the fixed cost of engineers outweighs any savings on tokens.

The worked example

Take a company that wants to run a 70B-class open model in production, with high availability, on four rented H100s.

• GPUs: 4 H100s at $2.50 an hour, running 24/7, cost about $7,300 a month.
• Operations: serving, storage, monitoring and security at about 20% of GPU spend add roughly $1,460 a month.

• Staffing: 2.5 engineers at a $200,000 fully loaded cost each come to about $41,700 a month.

• Total: roughly $50,500 a month, or about $600,000 a year.

At 50% average utilization, those four GPUs serve on the order of 7–8 billion tokens a month. At that volume, a comparable small or mid-tier API at a blended $2 per million tokens would cost about $15,000 a month. The API is more than three times cheaper.

Finding your break-even

The math changes as volume grows, because each extra GPU is cheap compared with the fixed team.

• Marginal cost of self-hosting: each additional H100 at 50% utilization adds about 1.9 billion tokens a month for roughly $2,200 including operations, or about $1.15 per million tokens.

• Marginal cost of the API: about $2 per million tokens in this example.

• Break-even: the $41,700 monthly staffing cost divided by the $0.85 saving per million tokens gives about 49 billion tokens a month, or roughly 1.6 billion tokens a day.

• With shared staff: if one engineer from an existing platform team can run the stack, the break-even drops to about 20 billion tokens a month.

As a formula:

\text{Break-even tokens per month} = \frac{\text{Fixed monthly staff and ops cost}}{\text{API price per token} - \text{Self-hosted marginal cost per token}}
What moves the number
• Utilization: doubling average GPU utilization roughly halves self-hosted marginal cost.

• API pricing: API prices have been falling fast. Epoch AI found the price to reach a given performance level has dropped between 9x and 900x a year depending on the task, which keeps pushing the break-even higher.

• Model quality: comparing a self-hosted 70B model to a frontier API is not like for like. If the task truly needs frontier quality, the comparison is against a much pricier API, but the open model may not do the job.

• Engineering reuse: companies that already run GPU platforms for other workloads add LLM serving at a fraction of the staffing cost.

All figures here are illustrative planning numbers; plug in your own token volume, utilization and salaries before deciding.

When self-hosting is worth it

Self-hosting makes sense when at least one of four conditions holds: very high steady volume, strict data residency, a need for deep customization, or latency an API cannot meet. If none apply, an API or a managed open-model endpoint is almost always cheaper.

Good reasons to self-host

• Sustained high volume: billions of tokens a day with predictable traffic, where GPUs stay busy.

• Data control: regulated data, government or defense work, or contracts that forbid sending data to third-party model providers.

• Customization: fine-tuned or distilled models, custom decoding or architectures that hosted APIs do not offer.

• Latency and locality: on-premise or edge deployments, or tight latency budgets close to your users.

• Vendor independence: a strategic need to avoid dependence on any single model provider.

Weak reasons to self-host

• "Open-source is free": the weights are free; running them is not.
• Low or spiky traffic: idle GPUs cost the same as busy ones.
• Cost savings at prototype stage: early products rarely have the volume to justify a team.

The middle path: managed open-model endpoints

Providers such as Together AI, Fireworks, Groq, AWS Bedrock and Azure AI host open models like Llama, Qwen and DeepSeek and charge per token. You keep model choice and avoid most staffing cost. Many teams start here, then move their highest-volume workload in-house once traffic proves the case.

Seven ways to cut self-hosting cost

Quantize: FP8 or INT4 versions of a model often cut GPU memory needs by half or more with little quality loss.
Right-size the model: a well-tuned 8B–32B model often beats a generic 70B model on a narrow task, at a fraction of the cost.
Batch aggressively: continuous batching in vLLM or SGLang raises throughput per GPU sharply.
Scale to demand: autoscale replicas down overnight and at weekends instead of running peak capacity 24/7.
Use reserved capacity for the baseline and on-demand for peaks.
Route by difficulty: send easy requests to a small self-hosted model and hard ones to a frontier API.
Track cost per million tokens, not cost per GPU-hour, and review it every month.

Frequently asked questions

How much does it cost to self-host an LLM?
Self-hosting an open-source LLM in production typically costs $150,000 to $3 million a year in 2026. GPUs account for $15,000 to $1.5 million of that, and the engineers who run the stack usually cost as much or more.

How much does it cost to run Llama 70B?

Running a 70B-class model like Llama 3.3 70B with high availability on four rented H100s costs about $7,300 a month in GPUs alone, or roughly $88,000 a year. Adding operations and a small engineering team brings the total to around $600,000 a year.

Is self-hosting an LLM cheaper than using an API?

Only at high, steady volume. For a 70B-class model, self-hosting typically breaks even with a comparable API at about 20–50 billion tokens a month. Below that, APIs are cheaper because you are not paying a team to run infrastructure.

How much does an H100 GPU cost per hour?

An H100 rents for roughly $2 to $4 per GPU-hour on specialist clouds in late 2026, and $5 to $8 or more on hyperscalers before discounts. Buying one costs about $25,000 to $40,000.

Should I rent or buy GPUs for LLM inference?

Rent unless you can keep GPUs busy around the clock for two years or more. Buying adds power, cooling, space and depreciation, and a $35,000 H100 takes roughly two years of 24/7 use to beat rental prices.

How many engineers do I need to self-host an LLM?

Plan for 2–3 engineers for a single production model with 24/7 support, and 4–6 for a multi-model platform. Small internal deployments can run on part of one engineer's time if an existing platform team supports them.

What is the cheapest way to use open-source LLMs?

For most companies, a managed open-model endpoint such as Together AI, Fireworks or AWS Bedrock is cheapest. You pay per token, keep the freedom to switch models, and avoid GPU and staffing costs until volume justifies self-hosting.

Comparing Batch and Real-Time LLM Inference Economics

Eric Thompson — Wed, 04 Nov 2026 12:11:17 +0000

Batch inference costs 50% less than real-time inference at Anthropic, OpenAI and Google, in exchange for results that can take up to 24 hours. For a workload of one million document classifications a month on a mid-tier model, that is roughly $7,000 a month in real time versus $3,500 in batch, about $42,000 a year saved (illustrative calculation below).
The catch is that only work nobody is waiting on can move to batch: enrichment, classification, evaluations, reports, backfills. Anything a user sees live, such as chat, search answers or agents acting in a session, stays real-time. For those, prompt caching is the main cost lever, and at Anthropic its discount stacks with batch.
The practical move for most companies is to split their AI traffic in two. Run interactive requests in real time with caching, and route everything else through a batch queue. The rest of this article shows how to size each part.

How batch and real-time inference differ

The price difference reflects one tradeoff: the provider decides when your request runs.
Real-time (standard) inference returns a response in seconds, usually streamed. You pay full list price, and your throughput is capped by per-minute rate limits.
Batch inference accepts a file or list of many requests, processes them when capacity is free, and returns all results together. Anthropic's Message Batches API, for example, accepts up to 100,000 requests or 256 MB per batch. Most batches finish within an hour, but any request not done within 24 hours expires, and results stay downloadable for 29 days (Anthropic docs).
Middle tiers now exist between the two. OpenAI's flex processing charges Batch API rates for ordinary synchronous calls that are slower and can be refused when capacity is short; refused requests return an error and are not charged (OpenAI docs). Google's Gemini API offers a Flex tier at 50% off with a 1–15 minute target, and a Priority tier that costs 75–100% more for the fastest, non-sheddable service (Google docs).
So the real choice is not two modes but a range. Pay more for guaranteed speed, standard price for normal interactive use, or half price for work that can wait minutes to hours.

Provider pricing: batch discounts and caching (sourced)

All three major API providers price batch at half of standard, so the decision is about workload fit, not vendor choice.
Anthropic: Message Batches cost 50% of standard prices on input and output tokens. Prompt caching discounts stack with batch pricing, though cache hits in batch are best-effort, typically 30–98% depending on traffic (Anthropic batch docs). Example batch prices: Claude Sonnet 5.5 at $1 input and $5 output per million tokens; Claude Opus 5.5 at $2 and $10. Standard prices are double.
OpenAI: launched its Batch API in April 2024 at a 50% discount with a completion window of up to 24 hours (Simon Willison). Flex processing gives Batch API rates on synchronous calls, and prompt caching can add further discounts (OpenAI docs).
Google Gemini: Batch mode is 50% off with a target turnaround of up to 24 hours and input files up to 2 GB. Flex is also 50% off. Context caching gives a 90% discount on cached tokens plus a storage charge (Google docs, updated October 2026).

How prompt caching changes the numbers

When many requests share a long prefix, such as a system prompt, policy document or product catalog, the provider can reuse it. Reading cached tokens costs a small fraction of normal input price: 10% of base input on standard Anthropic models, with lower multipliers reported for some newer models, and a premium of 1.25 to 2 times base input to write the cache. This model uses the conservative 10% read price. Verify current multipliers before budgeting.
List prices change frequently and enterprise contracts often differ. Treat every figure here as a November 2026 snapshot.
*Cost model: three workloads (illustrative)
**Batch halves the bill for work that can wait; caching does most of the saving for work that cannot. All three scenarios use Claude Sonnet 5.5 list prices (standard $2 input and $10 output per million tokens; batch $1 and $5). Volumes are invented examples, and cache-write charges are left out because they are small when one prefix serves many requests.
**The formula:
*\text{Monthly cost} = \frac{\text{requests} \times \text{input tokens}}{10^6} \times p_{in} + \frac{\text{requests} \times \text{output tokens}}{10^6} \times p_{out}

Scenario 1: document classification and enrichment

1 million documents a month, 2,000 input tokens and 300 output tokens each. Nobody waits on individual results.
Input: 2,000 million tokens. Output: 300 million tokens.
Real-time: 2,000 × $2 + 300 × $10 = $4,000 + $3,000 = $7,000 a month
Batch: 2,000 × $1 + 300 × $5 = $2,000 + $1,500 = $3,500 a month
Saving: $3,500 a month, $42,000 a year.

Scenario 2: customer support assistant

500,000 conversations a month, 4,000 input tokens each (of which 3,000 are a shared system prompt and knowledge snippet), and 400 output tokens. Customers wait live, so batch is not an option.
Input: 2,000 million tokens, of which 1,500 million are cacheable. Output: 200 million tokens.
Real-time, no caching: 2,000 × $2 + 200 × $10 = $4,000 + $2,000 = $6,000 a month
Real-time with caching: 1,500 × $0.20 + 500 × $2 + 200 × $10 = $300 + $1,000 + $2,000 = $3,300 a month
Saving: $2,700 a month (45%), with no change to user experience.

Scenario 3: nightly evaluation and reporting pipeline

200,000 requests a month, run overnight, 10,000 input tokens each (8,000 shared context) and 1,000 output tokens. Results are needed by morning.
Input: 2,000 million tokens, of which 1,600 million are shared. Output: 200 million tokens.
Real-time: 2,000 × $2 + 200 × $10 = $6,000 a month
Batch: 2,000 × $1 + 200 × $5 = $3,000 a month
Batch plus caching, assuming a 50% cache hit rate on the shared context (Anthropic describes batch cache hits as best-effort): 800 million cached × $0.10 + 1,200 million uncached × $1 + 200 × $5 = $80 + $1,200 + $1,000 = $2,280 a month

Saving against real-time: $3,720 a month (62%).

Notice what dominates once input is discounted: output tokens. In Scenario 3, output is 44% of the batch-plus-cache bill. Shorter, structured outputs are often the next lever.

Self-hosted inference: utilization decides the math

If you run an open-weight model on rented GPUs, you pay for GPU hours whether or not requests arrive. Batch workloads matter here for a different reason: they keep GPUs busy.
Sourced GPU prices. An October 2026 survey of H100 rental prices found specialist clouds listing roughly $2 to $4.50 per GPU-hour on demand, and hyperscalers roughly $7 to $12 (IntuitionLabs). Several providers raised list prices in 2026.
Illustrative calculation. Assume a $3 GPU-hour and a model that sustains 1,000 output tokens per second per GPU at full load. That throughput is an assumption for illustration; real figures vary widely by model size, hardware and serving software.
At full load: 1,000 × 3,600 = 3.6 million tokens an hour, so $3 ÷ 3.6 = $0.83 per million tokens
At 70% utilization: $0.83 ÷ 0.70 = $1.19 per million
At 30% utilization: $0.83 ÷ 0.30 = $2.78 per million
Interactive traffic is spiky, so real-time clusters are sized for peaks and sit partly idle. Batch jobs fill those idle hours, which is why self-hosting teams run backfills and evaluations overnight on the same hardware.
These per-token numbers are not directly comparable to API prices. They use a different model, and they exclude the engineers who keep the serving stack running, monitoring, and redundancy. Self-hosting usually makes economic sense only at sustained high volume, or when data cannot leave your environment.

Which workloads belong where

Ask one question of every AI call: is a person or a live process waiting for this answer right now?

Good batch candidates

Classifying, tagging or extracting fields from documents, tickets and records
Enriching CRM or product catalogs
Generating embeddings or summaries for search indexes
Running evaluation suites and regression tests on prompts or models
Overnight reports, digests and analytics narratives
Backfilling historical data after a prompt or model change
Generating synthetic data or test cases

Must stay real-time

Chat assistants and customer support
Search answers and in-app copilots
Agents taking actions inside a user's session
Fraud, moderation or routing decisions that block a transaction

Middle ground: flex tiers

Work that can wait minutes but not hours, such as a background summary a user checks later, fits flex tiers. They give batch-level prices on ordinary synchronous calls, but requests can be refused or delayed, so your code needs a fallback to standard.

The hybrid pattern

Most production systems end up with three lanes. Interactive traffic runs real-time with prompt caching. Deferrable traffic goes into a queue that submits batches on a schedule. Anything urgent that a flex call cannot serve falls back to standard. The engineering is modest, and the savings apply to every token in the deferrable lane.

Engineering costs and risks beyond the token bill

The batch discount is only real if the pipeline around it is reliable; budget engineering time for these pieces.
Queue and scheduler. Requests must be collected, grouped within batch size limits, and submitted on a schedule.
Result handling. Results arrive together, so each one must be matched back to its source record by ID and written to the right place.
Expiry and retries. Requests not completed within the provider's window expire. You need logic to detect and resubmit them, and an alert if a whole batch fails.
Partial failures. Some requests in a batch can fail while others succeed. Track status per request, not per batch.
Result retention. Anthropic keeps batch results for 29 days. Download and store them before then.
Freshness. A result produced overnight reflects yesterday's data. Check that downstream users accept that.
Cost visibility. Tag requests by lane and team so finance sees what batch is actually saving.
Two further risks apply to every mode. Provider prices and model versions change often, so a budget built on one month's list prices can drift. And a cheaper token bill does not help if output quality drops; rerun your evaluation set whenever you change model, mode or prompt.

Where CodingCops fits

Disclosure: the author works with CodingCops. This section uses only the company's published descriptions and includes no CodingCops pricing, client results or savings claims.
CodingCops is a Chicago-headquartered software company founded in 2010 (Built In Chicago). Its company profile lists AI and machine learning, cloud and DevOps, custom software development, and dedicated engineering teams among its services (Outsource Accelerator).
The work described in this article, separating interactive and deferrable traffic, building a batch queue with retries, and tracking cost per lane, sits at the intersection of AI engineering and DevOps. Teams without that capacity in-house can add it through a dedicated team or staff augmentation, which CodingCops offers for specific projects, short-term initiatives or ongoing development (codingcops.com).
[Marketing to verify before publishing] Confirm the description wording. Do not add token-cost savings or client results without evidence and approval.

FAQs

How much cheaper is batch LLM inference?

Anthropic, OpenAI and Google all price batch at 50% of standard rates, as of November 2026.

How long does a batch job take?

Up to 24 hours at all three providers. Anthropic says most batches finish within an hour; requests not done within 24 hours expire.

Can I combine batch with prompt caching?

At Anthropic, yes: the discounts stack, though cache hits in batch are best-effort. Check each provider's current docs.

What is flex processing?

A synchronous tier priced like batch. Calls are slower and may be refused when capacity is short, so you need a fallback. OpenAI and Google both offer one.

Is self-hosting cheaper than batch APIs?

Only at sustained high volume with high GPU utilization, and after counting the engineers who run it. Idle GPUs make self-hosted tokens expensive quickly.

What share of AI workloads can move to batch?

It depends entirely on the product. Audit your own traffic: anything nobody waits on is a candidate.

Dedicated Engineering Team vs Specialist Contractor for an AI Integration

Eric Thompson — Wed, 28 Oct 2026 15:04:44 +0000

hire a specialist contractor when your own engineers can do the integration work and you need one scarce skill, such as retrieval design or evaluation, for a few weeks. Choose a dedicated engineering team when you lack integration capacity, when the work runs past about three months, or when the AI feature becomes something you must keep improving. On price, one senior freelance AI specialist at $165–210 an hour costs about the same per week as a three-person offshore pod in India: about 6,600–8,400 versus $6,330 before fees and overhead. So the contractor route only saves money when one person is genuinely enough. Our scenarios show a 6-week feature costing about 42,000–53,000 with one contractor or 46,000–60,000 with a three-person pod. A 12-month capability costs 333,000–423,000 for one contractor, versus $365,000 for a three-person pod in India. All figures are illustrative estimates from stated assumptions.

What is the difference, and what does an AI integration involve?

The two options differ less in hourly price than in who owns delivery, continuity and the work around the AI model.

The two options

Specialist contractor. An independent expert, usually hired directly or through a freelance platform, paid by the hour or the milestone. You direct the work, supply the surrounding engineering and own project risk. Best at deep, narrow problems: retrieval design, prompt and agent architecture, evaluation, model selection.
Dedicated engineering team. A vendor-employed group, typically a lead plus engineers and QA, that works only on your product. The vendor handles hiring, replacement and day-to-day delivery; you set priorities. Best when the integration touches many parts of your product, or when the work continues after launch.

What an AI integration actually involves

The model call is the small part. A typical integration also needs:
Data work: finding, cleaning, permissioning and connecting the data the model will use.
Application work: APIs, UI changes, authentication, logging and error handling inside your existing product.
AI-specific work: retrieval, prompts or agent logic, model choice, and guardrails.
Evaluation: a test set and measurements that show whether the output is good enough, rerun on every change.
Operations: monitoring quality, cost and latency after launch, and updating as models change.
A specialist contractor usually covers item 3 and part of 4. Items 1, 2 and 5 still need engineers. That gap is where most of the cost difference in the scenarios below comes from. It also matches what MIT NANDA's preliminary 2025 research found: most enterprise AI tools that fail do so because of brittle workflows and poor fit with daily operations, not model quality (MIT NANDA abstract).

What do AI contractors and dedicated teams cost in 2026?

Freelance AI specialists charge roughly two to five times the hourly rate of offshore team engineers, so rate comparisons only make sense per unit of capacity.

Freelance AI specialists

US freelance LLM developers: $125–175/h mid-level, $175–250/h senior, $250–350/h lead, with a reported senior median of $210/h (Second Talent, 2026).
Freelance AI tiers, global: $130–200/h for retrieval (RAG) or fine-tuning engineers, $180–300/h for AI agent developers, $250–500/h for PhD or strategic consultants (Index.dev, 2026).
Platform fees: Upwork is reported to charge clients a 5% fee; agencies in Eastern Europe and Latin America add 30–50% over a developer's take-home pay (Index.dev).
Both sources are hiring platforms, and their published ranges vary. Treat them as indicative and get quotes.

Dedicated teams

India: $28–45/h mid-level, $45–70/h senior.
Eastern Europe: $40–60/h mid-level, $60–90/h senior.
Latin America: $45–65/h mid-level, $60–85/h senior.
AI specialists within teams: a 20–40% premium over generalist rates.
Overhead: coordination, QA and rework add 15–25% to the labour line.
Source: Kanerika's 2026 synthesis of the Accelerance 2026 Rates & Trends Guide.
In-house, for reference
The US median software developer wage was $135,980 in May 2025, about $65 an hour before benefits and overhead (BLS). Specialist AI engineers earn more, and hiring takes months, so in-house rarely fits a single integration project.

The comparison that matters

At our reference rates, a three-person India pod costs $6,330 a week. That buys about 38 hours of a $165/h contractor. In other words: one week of one specialist contractor costs about the same as one week of a three-person offshore team. The question is whether you need one expert or three people's worth of delivery.
Cost model: three AI integrations priced both ways
The contractor is cheaper only when one person can do the job and your own team covers the rest; as scope and duration grow, the dedicated team delivers more capacity per dollar. All figures below are illustrative estimates.

How we priced each option

Contractor cost = hours × hourly rate × 1.05 (platform fee). Two rates: $165/h (mid-range freelance RAG engineer) and $210/h (US senior LLM freelancer median).
Dedicated pod cost = weeks × weekly pod cost × 1.20 (overhead). The pod is 3.0 FTE: tech lead 0.5, AI engineer 1.0, full-stack 1.0, QA 0.5.
Weekly pod cost, India: (20 h × $57.50) + (40 h × $57.50 × 1.30) + (40 h × $36.50) + (20 h × $36.50) = $1,150 + $2,990 + $1,460 + $730 = $6,330.
Weekly pod cost, Eastern Europe: (20 × $75) + (40 × $75 × 1.30) + (40 × $50) + (20 × $50) = $1,500 + $3,900 + $2,000 + $1,000 = $8,400.

Scenario A — a bounded AI feature (6 weeks)

Example: add document summarisation or ticket classification to an existing product with a strong in-house team.
Contractor, 240 hours: $41,580 at $165/h; $52,920 at $210/h.
India pod: 6 × $6,330 × 1.20 = $45,576.
Eastern Europe pod: 6 × $8,400 × 1.20 = $60,480.
Verdict: contractor, if your engineers can absorb the application and QA work. The pod costs a similar amount but brings capacity you may not need.

Scenario B — a retrieval integration (12 weeks)

Example: an assistant that answers from your knowledge base, with a data pipeline, permissions and an evaluation set.
Contractor route, two freelancers: AI specialist 480 h × $165 = $79,200, plus a full-stack freelancer 480 h × $55 (midpoint of Index.dev's $40–70 Eastern Europe range) = $26,400. Total $105,600; with 5% fee, $110,880. QA and project management still fall on you.
India pod: 12 × $6,330 × 1.20 = $91,152, including QA and a lead.
Eastern Europe pod: 12 × $8,400 × 1.20 = $120,960.
Verdict: dedicated team, unless you already have spare engineers and a project manager. Assembling contractors costs about the same and leaves coordination with you.

Scenario C — ongoing AI capability (12 months)

Example: an AI feature set that keeps expanding: new workflows, agent steps, evaluation runs, model upgrades.
One contractor, 1,920 hours: $332,640 at $165/h; $423,360 at $210/h.
India pod: 48 × $6,330 × 1.20 = $364,608, for three people.
Eastern Europe pod: 48 × $8,400 × 1.20 = $483,840, for three people.
Verdict: dedicated team. For about the cost of one senior contractor you get three times the capacity and continuity if someone leaves. One contractor for a full year also raises classification and key-person risks, covered next.

What the scenarios show

The cheapest option flips with scope: contractor for narrow, short work; team once the work spans data, application and operations.
Duration matters more than rate. Over 12 months, small weekly differences add up, and the value of continuity grows.
Your own spare capacity is the swing factor. If your engineers are already fully booked, the contractor route has a hidden cost: their time.
Risks and hidden costs that change the answer
The biggest costs of either option rarely show up on the rate card; they show up when someone leaves, when knowledge has to be handed over, or when the work outgrows the original plan.

Risks that weigh more on the contractor route

Key-person dependency. One person holds the retrieval design, prompts and evaluation logic. If they take another engagement, progress stops and a replacement has to relearn the system.
Knowledge transfer. AI integrations accumulate unwritten decisions: why a chunking strategy was chosen, which test cases matter. Without a written handover, you pay to rediscover them.
Coordination load. You become the project manager. Your engineers' time spent integrating, reviewing and testing is a real cost, even if it never appears on an invoice.
Worker classification. In the US, the IRS assesses whether someone is a contractor or employee across behavioural control, financial control and the relationship between the parties. Misclassification can lead to back taxes, fines and interest (Deel summary of the IRS test). A full-time, open-ended, closely directed contractor raises that risk. This is general information, not legal advice; check with counsel for your jurisdiction.

Risks that weigh more on the dedicated-team route

Ramp-up time. A team needs time to learn your codebase and domain. Expect slower output in the first weeks.
Paying for capacity you do not use. If the work is genuinely small, a team adds cost without adding value.
Seniority dilution. A vendor may staff the AI role with someone less experienced than promised. Name the roles and minimum experience in the contract.
Coordination overhead. Index.dev estimates distributed teams lose 10–15% of productive time to time-zone coordination (Index.dev, 2026). Overlap matters more for AI work, where outputs need frequent review.

Risks common to both

Data and IP. Both need written IP assignment, limits on using your data, and access controls on production data.
Model and price changes. Model releases can change the best design mid-project. Agree up front how such changes are scoped and charged.
Converting a contractor into a hire later. Employing someone abroad through an Employer of Record adds 15–25% on top of base pay, per Index.dev. Factor that in if you expect to keep a contractor long term.

How to decide: a five-question framework

Answer five questions; if three or more point the same way, that is your default.
Do your engineers have spare capacity for data, application and testing work? Yes points to a contractor; no points to a team.
Will the work last more than about three months? No points to a contractor; yes points to a team.
Is the gap one specific skill, or broad delivery capacity? One skill points to a contractor; broad capacity points to a team.
Will the AI feature need ongoing improvement after launch? No points to a contractor; yes points to a team.
Can you manage the project day to day? Yes points to a contractor; no points to a team.

The hybrid option

A practical middle path combines both: a senior specialist contractor for two to four weeks to set the architecture, evaluation approach and model choice, followed by a dedicated team that builds, integrates and operates it. The specialist can stay on for a few hours a week as a reviewer. This pairs scarce expertise with continuity, and it limits the contractor's hours to where they add the most value.

Contract terms to insist on

For a contractor: a defined deliverable or milestone, written handover documentation (design decisions, prompts, evaluation set), IP assignment on payment, and a notice period.
For a dedicated team: named roles and minimum seniority, a replacement guarantee with handover time, a monthly scaling clause, retention data, and IP assignment.
For both: no training on your data, access controls on production systems, and a clear process for pricing model or provider changes.

Where CodingCops fits

Disclosure: this article is published by CodingCops, and the author is affiliated with the company.
CodingCops describes itself as a full-cycle technology partner that helps businesses design, build, scale and maintain software systems, working with startups, growing companies and enterprises across custom software development, AI and machine learning, cloud and DevOps, and dedicated engineering teams. That covers the dedicated-team route described here. This article includes no CodingCops pricing or client results. If you are comparing options, ask contractors and vendors, including us, to price the same scenario from this guide so the quotes are comparable.

FAQs

Is a specialist contractor cheaper than a dedicated team for AI work?

Per hour, no: freelance AI specialists charge about $130–250 an hour, versus roughly $37–75 for offshore team engineers. Per project, a contractor can be cheaper for short, narrow work when your own engineers handle the rest. In our 6-week scenario, one contractor costs 41,580–52,920 versus $45,576 for a three-person India pod.

How much does a freelance AI engineer cost in 2026?

Published ranges put US freelance LLM developers at $175–250 an hour for senior work, and global freelance retrieval or fine-tuning engineers at $130–200. AI agent developers are quoted at $180–300. Sources are hiring platforms, so get quotes.

How much does a dedicated AI engineering team cost?

Our three-person pod costs about $6,330 a week in India and $8,400 in Eastern Europe before overhead. Kanerika puts a five-engineer team at 24,000–44,000 a month depending on region.

When should I switch from a contractor to a team?

When the work passes about three months, when your engineers become the bottleneck, or when the AI feature needs continuous improvement after launch.

Can I use both?

Yes. A short specialist engagement to set architecture and evaluation, followed by a dedicated team to build and run it, combines expertise with continuity.

What is the biggest risk with a single AI contractor?

Key-person dependency. If the contractor leaves, the knowledge behind the retrieval design, prompts and evaluation goes with them unless it is documented.

When Is a Managed LLM Gateway Worth Its Cost?

Eric Thompson — Tue, 27 Oct 2026 14:53:45 +0000

A managed LLM gateway is worth its cost in two situations: almost always when it is priced per log or per seat, and only below a spend threshold when it takes a percentage of your model spend.
Log-priced gateways are cheap relative to engineering time. Portkey's Production plan, for example, works out to about $130 a month at one million logged requests. That is less than two hours of a fully loaded US engineer.
Percentage-fee gateways scale with your bill. At a 5.5% platform fee, a managed gateway costs more than self-hosting once your monthly model spend passes roughly $35,000 if self-hosting would take 20 hours a month of US engineering time, or roughly $13,000 if that work is done at offshore rates (illustrative; infrastructure excluded).
Above those lines, self-hosting an open-source gateway, using your own provider keys, or negotiating an enterprise contract usually costs less. The rest of this article shows the numbers, so you can rerun them with your own.

Disclosure, scope and freshness

Affiliation: Published by CodingCops, which builds AI systems and supplies engineering teams. [Editor to confirm before publishing: CodingCops has no reseller or paid relationship with any gateway named here.]
Scope: US dollars; US-based buyer; vendor prices as listed on each vendor's pricing page on October 27, 2026; US engineering cost from BLS May 2025 wages and June 2026 benefit data. Gateway pricing changes often, so this guide is reviewed quarterly.
Labels: Sourced figures come from the cited page. Illustrative figures are this article's calculations, with assumptions stated. Model token prices are excluded throughout: they are the same whichever gateway you use, unless a gateway marks them up.

What an LLM gateway does, and three ways to get one

An LLM gateway is a proxy between your application and model providers. It gives you one API across providers, plus the controls you would otherwise build yourself: key management, retries and fallbacks, routing and load balancing, caching, rate limits and budgets, request logging, cost attribution, and guardrails.

You can get one in three ways:

Managed SaaS. The vendor runs it. You pay a subscription, a per-log fee, or a percentage of model spend.
Self-hosted open source. You run a free gateway (LiteLLM and Portkey both publish open-source versions) on your own infrastructure. No licence fee; you pay in infrastructure and engineering time.
Enterprise licence. A vendor's enterprise edition, often deployable in your own cloud or VPC, with SSO, audit logs and support. Priced by quote. The cost question is really a question about which bill grows fastest as you scale: a vendor fee, your cloud bill, or your engineers' time. Managed gateway pricing in 2026: the two pricing families Managed gateways price in one of two ways, and which one you face matters more than the headline number: a percentage of model spend, which grows with your bill, or a fee per log, seat or plan, which grows with request volume and usually stays small.

Percentage-of-spend pricing (sourced)

• OpenRouter: 5.5% platform fee on the Standard pay-as-you-go plan and 8% on Business; enterprise fee discounts by negotiation. With your own provider keys (BYOK), the first $25,000 of list-price inference a month carries no fee, then 5% (OpenRouter pricing).
• Cloudflare AI Gateway, Unified Billing: a 5% fee on credit purchases, with provider inference passed through at the provider's per-token rate. Using your own provider keys avoids this fee; core gateway features (analytics, caching, rate limiting) are free (Cloudflare AI Gateway pricing).

Plan and log-based pricing (sourced)

• Portkey: Developer plan free (10,000 recorded logs a month; the vendor says it is not suitable for production). Production plan $49 a month with 100,000 logs, then $9 per additional 100,000. Enterprise by quote, with SSO, VPC hosting and compliance options (Portkey pricing).
• Helicone: Hobby free (10,000 requests, 7-day retention); Pro $79 a month; Team $799 a month; Enterprise custom. Usage-based charges apply beyond included volumes, but per-unit rates are not published on the pricing page (Helicone pricing).
• Cloudflare logs: older gateways include 100,000 stored logs on Workers Free and 10 million per gateway on Workers Paid; gateways created on or after September 24, 2026 follow Workers Logs pricing.

Self-hosted and enterprise licences (sourced)

• LiteLLM: open-source edition free with no licence fee, including virtual keys, budgets, rate limits, fallbacks and logging. Enterprise is an annual custom quote based on request capacity, deployment architecture and support, and the vendor states it is never priced per token. SSO, audit logs, RBAC and air-gapped deployment are enterprise-only (LiteLLM pricing).
Two things to check on any plan: whether hitting a log limit stops logging or stops traffic (Portkey says requests keep flowing), and whether the features you need, usually SSO, audit logs and data residency, sit only in the enterprise tier.

What self-hosting really costs

The licence for an open-source gateway is free; the real cost is the engineer who keeps it running, and that cost depends heavily on where that engineer sits.

Engineering rates used (sourced inputs)

• US in-house: $95.44 an hour fully loaded. BLS puts the median software developer wage at $135,980 (May 2025); wages were 68.5% of total employer cost for full-time private-industry workers in June 2026, so $135,980 ÷ 2,080 hours ÷ 0.685 (BLS OOH, BLS ECEC).
• Nearshore (Latin America): $67.50 an hour, the midpoint of the $60–$75 senior band in Accelerance's 2026 survey (via BEON.tech).
• Offshore (Asia): $36 an hour, the midpoint of the $31–$41 senior band in the same survey.

Where the hours go

• Setup: deployment, secrets management, provider configuration, high availability, log storage, dashboards, access control.
• Upkeep: version upgrades, new providers and models, incident response, cost-map updates when provider prices change, log retention, security patching.
• Compliance work: audit evidence, access reviews, data-residency controls; heavier in regulated industries.
This article assumes 20 hours a month of upkeep for one production gateway and 80 hours a month (half an engineer) for an enterprise deployment. These are illustrative; replace them with your own estimate.

Monthly upkeep cost (illustrative)

• 20 hours a month: US $1,909 · nearshore $1,350 · offshore $720
• 80 hours a month: US $7,635 · nearshore $5,400 · offshore $2,880
Infrastructure (compute, database, log storage, network) comes on top. It is left as a variable here because it depends on your cloud, region, redundancy and log volume; price it from your own cloud bill.

The break-even model

A percentage-fee gateway is cheaper than self-hosting only while its fee is below what self-hosting would cost you each month:
f \times S < I + H \times r
where f is the gateway fee rate, S is monthly model spend through the gateway, I is monthly self-hosting infrastructure, H is monthly upkeep hours and r is the loaded hourly rate. Solving for the spend at which the two are equal:
S^{*} = \frac{I + H \times r}{f}
Break-even monthly spend at 20 upkeep hours, infrastructure excluded (illustrative)
• 5.5% fee (OpenRouter Standard): US engineer $34,705 · nearshore $24,545 · offshore $13,091
• 5% fee (Cloudflare Unified Billing; OpenRouter BYOK above its free allowance): US $38,176 · nearshore $27,000 · offshore $14,400
Example: with a US engineer, $1,909 ÷ 0.055 = $34,705. Adding infrastructure raises every threshold. Each $500 a month of infrastructure moves the 5.5% break-even up by about $9,100 of spend.

Two adjustments specific to vendors:

• OpenRouter BYOK charges nothing on the first $25,000 a month of list-price inference, so its fee is (S − 25,000) × 5%. Below $25,000 a month, the fee is zero and self-hosting cannot win on fees alone.
• Log-priced gateways don't fit the formula because their cost doesn't track spend. Portkey Production is $49 + $9 per extra 100,000 logs: about $130 a month at 1 million logs and $310 at 3 million. Against a 20-hour upkeep budget, self-hosting does not beat that on cost at any of the three engineering rates.

Budget scenarios: pilot, production, enterprise

The right gateway changes as an AI product moves from pilot to production: managed is the clear winner in a pilot, log-priced managed or BYOK wins in production, and at enterprise scale the real choice is an enterprise quote versus your own team. All figures are monthly and illustrative; model token costs excluded.

Scenario 1 — Pilot: $2,000 model spend, 50,000 requests

• OpenRouter with credits (5.5%): $110
• OpenRouter BYOK: $0 (under the $25,000 free allowance)
• Cloudflare with your own keys: $0 for core features; $100 if using Unified Billing
• Portkey Production: $49 (within 100,000 logs)
• Helicone Pro: $79 base
• Self-hosted, 20 hours upkeep: $720–$1,909 plus infrastructure
Read: Self-hosting costs roughly 7 to 40 times more than the paid managed options here. In a pilot, engineering hours belong on the product, not the plumbing.

Scenario 2 — Production: $40,000 model spend, 1 million requests

• OpenRouter with credits (5.5%): $2,200
• OpenRouter BYOK: $750 ((40,000 − 25,000) × 5%)
• Cloudflare Unified Billing (5%): $2,000; own keys: core features free
• Portkey Production: $130
• Self-hosted, 20 hours upkeep: US $1,909 · nearshore $1,350 · offshore $720, each plus infrastructure
Read: This is where percentage fees start to hurt: $2,200 a month of credits fees is more than a US engineer's upkeep time. Switching to your own keys, or to a log-priced gateway, keeps managed cheaper than self-hosting.

Scenario 3 — Enterprise: $250,000 model spend, 10 million requests, SSO, audit logs and VPC required

• OpenRouter with credits (5.5%): $13,750
• OpenRouter BYOK: $11,250
• Cloudflare Unified Billing (5%): $12,500
• Portkey or LiteLLM enterprise: custom quote (Portkey's Production plan stops at 3 million requests; SSO and VPC sit in enterprise tiers)
• Self-hosted open source, 80 hours upkeep: US $7,635 · nearshore $5,400 · offshore $2,880, each plus infrastructure, plus the cost of building whatever enterprise features the open-source edition lacks
Read: Percentage fees at this scale cost $135,000–$165,000 a year. That approaches the loaded cost of a full-time US engineer (about $198,500 a year) and exceeds a full-time nearshore senior (about $129,600 at 1,920 hours), so the decision becomes an enterprise licence quote versus self-hosting with your own (or a partner's) engineers. Compare the quote to 80 hours a month plus infrastructure plus the build cost of SSO, audit and compliance controls.

Savings claims to test before you count them

Gateways are often sold on the savings they create, not just their price. Those savings can be real, but they depend on your traffic, so measure them on your own workload before putting them in a business case.
• Caching. Exact-match caching only saves money when identical requests repeat; semantic caching widens the match but can return a stale or wrong answer. Measure your hit rate in a trial before assuming any saving.
• Model routing. Sending simple requests to cheaper models can cut spend, but the saving depends on how much of your traffic can tolerate a smaller model. Validate with your own evaluation set.
• Fallbacks and retries. These protect uptime more than they cut cost; retries can increase spend if they re-send long prompts.
• Budgets and rate limits. Their value is avoiding runaway bills from a bug or abuse. They prevent losses rather than reduce normal spend.
• Cost visibility. Per-team and per-feature attribution is often the most useful output for a CFO, and it is available in both managed and open-source gateways.
None of these savings are exclusive to managed gateways: open-source gateways offer caching, routing, budgets and fallbacks too. What you pay a managed vendor for is time and operational risk, not the features themselves.

Decision framework: managed, self-hosted or enterprise

Pick the option whose cost grows slowest at the scale you expect in 12 months, not the one that is cheapest today.

Choose a managed gateway when:

• You are in a pilot or early production and model spend is below the break-even for your engineering rate.
• The gateway is priced per log or plan, not per dollar of spend.
• You can bring your own provider keys, so fees stay low or zero.
• You have no engineer with time to own another piece of infrastructure.

Choose self-hosted open source when:

• Monthly spend is well above break-even and you are paying a percentage fee.
• Data must stay inside your own cloud, and the open-source edition covers your controls.
• You already run platform infrastructure and can absorb upkeep inside an existing team.

Choose an enterprise licence when:

• You need SSO, audit logs, RBAC, air-gapped or VPC deployment, and vendor support, and building them yourself would cost more than the quote.
• Compliance requires a vendor contract, SLAs or a BAA.

Who should run a self-hosted gateway

If you self-host, the engineering rate is the biggest variable in the break-even, so staffing matters:
• In-house US platform team: highest cost per hour but deepest context; best when the gateway sits next to other internal platform services.
• Nearshore dedicated engineers: about 30% below the US loaded rate on these benchmarks, with most of the US workday in overlap for incidents.
• Offshore dedicated engineers: the lowest rate, which lowers break-even the most, but plan for limited overlap during US-hours incidents.
CodingCops is one option for that last mile. It describes itself as an AI-native technology and engineering partner whose work spans AI integration, custom LLM applications, cloud and DevOps, and dedicated engineering teams (CodingCops company profile, G2). Evaluate any partner, CodingCops included, against the break-even numbers above.

FAQ

How much does an LLM gateway cost per month?

From $0 to a few hundred dollars for most small and mid-size workloads on log- or plan-priced gateways (for example, Portkey Production is $49 a month for 100,000 logs). Percentage-fee gateways cost 5–8% of model spend unless you bring your own keys. Enterprise editions are quoted individually.

Is a self-hosted LLM gateway free?

The licence can be; the operation is not. At 20 hours a month of upkeep, the labour alone runs about $720 (offshore) to $1,909 (US in-house) a month, before infrastructure.

At what spend should I stop paying a percentage fee?

At a 5.5% fee and 20 upkeep hours a month, roughly $13,000 of monthly model spend with offshore engineering, $25,000 with nearshore and $35,000 with US in-house, before infrastructure. Use the formula above with your own numbers.

Does a gateway mark up model prices?

Check each vendor. Cloudflare states provider inference is passed through at the provider's per-token rate and charges 5% on credits. OpenRouter charges a platform fee; its pricing page did not show its markup FAQ answer when checked, so confirm with the vendor.

Do I need a gateway for an AI pilot?

Not necessarily, but a free or low-cost managed tier gives you logging and cost tracking from day one, which makes the production budget easier to estimate.

What features usually force an enterprise plan?

SSO, audit logs, role-based access, VPC or air-gapped deployment, data-retention controls and compliance agreements. On the pricing pages reviewed, these sit in enterprise tiers.

Build vs Managed Vector Database: A Total-Cost Framework

Eric Thompson — Wed, 21 Oct 2026 15:08:05 +0000

For most teams under roughly 200 million vectors, a managed vector database costs less in total than self-hosting, because people cost more than servers. In our model, a 20M-vector production workload runs about $21K–$27K a year on Weaviate Cloud or Pinecone, including staff time to manage the vendor. Self-hosting the same workload costs $58K–$114K in year one, depending on whether the operators are offshore, nearshore or US-based. The cloud bill for self-hosting is only about $13.7K of that.

Self-hosting starts to win at large scale when your query pattern is expensive on usage-based pricing, when data residency rules out SaaS, or when a platform team already runs stateful clusters and can absorb the work. The single biggest swing factor is not build vs buy. It is how your data is partitioned, which can move a managed bill by more than 100x.

Year-one total cost at a glance

P*ilot (1M vectors, 0.5M queries a month)*. Managed costs about $10.2K on Weaviate Cloud and $10.3K on Pinecone. Self-hosting costs $26.8K with offshore operators, $36.9K nearshore and $52.0K with a US in-house team.

Growth (20M vectors, 10M queries a month). Managed costs about $21.4K on Weaviate Cloud and $26.9K on Pinecone with partitioned data, but $300.9K on Pinecone if every query scans one namespace. Self-hosting costs $58.3K offshore, $80.7K nearshore and $114.1K in-house.

Scale (200M vectors, 100M queries a month). Managed costs from about $55.1K on Weaviate Cloud (lowest published rates; a quote is needed) and $90.3K–$152.1K on Pinecone with partitioned data. Self-hosting costs $168.1K offshore, $214.9K nearshore and $284.9K in-house.

All figures are modeled estimates built from sourced unit prices and the documented assumptions in Part 1. Managed totals include vendor-management staff time; self-host totals include cloud infrastructure, initial build and ongoing operations. Embedding, LLM and application costs are excluded because they are the same either way.

What "total cost" means for a vector database

A vector database bill has five lines, and only two of them show up on an invoice.

Platform or infrastructure: the managed subscription, or EC2/VM compute, block storage and snapshots.
Usage: read units, write units, stored dimensions or egress on managed plans; cross-AZ traffic when self-hosting.
Initial build: cluster design, infrastructure-as-code, backups, monitoring, load testing and recall evaluation.
Ongoing operations: upgrades, re-indexing, capacity planning, incident response, security patching.
Vendor management: cost monitoring, quota tuning, support tickets and contract reviews on managed plans.

Self-hosting shrinks lines 1–2 and grows lines 3–4. Managed services do the opposite. The decision comes down to which side grows faster for your workload.

Managed vector database pricing (October 2026)

• Pinecone Standard: usage-based on reads, writes and storage. $16–18 per 1M read units, $4–4.50 per 1M write units and $0.33/GB-month, with a $50/month minimum. No uptime SLA is listed.

• Pinecone Enterprise: the same model at $24–27 per 1M read units and $6–6.75 per 1M write units, with a $500/month minimum and a 99.95% SLA.

• Weaviate Cloud Flex: priced on stored dimensions and storage, from $0.00465 per 1M dimensions, $0.12/GiB storage and $0.03036/GiB backup. The minimum is $45/month and the SLA is 99.5%.

• Weaviate Cloud Premium: a prepaid contract from $400/month. Shared clusters start at $0.003875 per 1M dimensions and dedicated at $0.002718, with SLAs up to 99.95%.

• Qdrant Cloud: billed on provisioned CPU, RAM and disk after a free tier. Rates appear only in its pricing calculator; Standard carries a 99.5% SLA and Premium higher.

• Zilliz Cloud: compute units on dedicated plans or vCUs on serverless, after a free tier. Storage is $0.040/GB-month from January 2026 and internet egress starts at $0.09/GB.

Two details matter more than headline rates. First, Pinecone charges reads by namespace size: each query costs 1 read unit per GB of the namespace searched, with a 0.25 RU minimum. Second, Weaviate prices on stored dimensions with no per-query fee in its published table, and its rates vary with index type, compression and region. Zilliz's own engineering blog shows the same workload pricing at about $7,000/month on dedicated and about $10,800 on serverless. The pricing model you pick can matter as much as the vendor.

Self-hosted infrastructure cost

Self-hosting Qdrant, Weaviate or Milvus is mostly a memory problem. Qdrant's capacity-planning guide gives a rough formula for keeping float32 vectors in RAM:

\text{RAM} = \text{vectors} \times \text{dimensions} \times 4\ \text{bytes} \times 1.5

Quantizing to int8 replaces the 4 bytes with 1 and keeps full-precision originals on disk for rescoring. That is the design we assume from 20M vectors up.

AWS us-east-1 rates used in the model:

• r7g.xlarge (4 vCPU, 32 GiB): $0.2142/hour on demand, about $156/month, or $0.1552/hour on a 1-year Compute Savings Plan.

• r7g.2xlarge (8 vCPU, 64 GiB): $0.4284/hour, about $313/month, or $0.3104/hour on the Savings Plan.

• r7g.4xlarge (16 vCPU, 128 GiB): $0.8568/hour, about $625/month, or $0.6208/hour on the Savings Plan.

• EBS gp3 block storage: $0.08/GB-month. EBS snapshots: $0.05/GB-month.

The people line: in-house vs nearshore vs offshore

Engineering time is the largest cost of self-hosting at every scale we modeled. We price it three ways.

• US in-house hire: $101.18/hour, or $194,257 a year per FTE. This is the BLS median software developer wage of $135,980 (May 2025, reported here) divided by 0.70, because wages are 70.0% of private-industry compensation per BLS.

• Nearshore senior engineer (Latin America): $67.50/hour, or $129,600 a year per FTE. This is the midpoint of the $60–75/hour senior range in FullStack's 2026 report.

• Offshore senior engineer (South Asia): $45/hour, or $86,400 a year per FTE. This is the midpoint of the $40–50/hour India senior range in Full Scale's 2026 guide.

The US figure is conservative for this work. SRE total compensation has a reported Levels.fyi median of about $200,200 before benefits, per KORE1's 2026 guide. Vendor rates for nearshore and offshore teams already include the vendor's overhead, so no benefits multiplier is applied to them.

Scenario 1 — Pilot: 1M vectors, 500K queries/month, 1M upserts/month

• Pinecone: reads in one namespace are 500K queries × 7.15 RU × $16/M = $57.20. Writes are 1M upserts × 7 WU × $4/M = $28.00, and storage is 7.15 GB × $0.33 = $2.36. That totals $87.56 a month, which falls under the $50 minimum once the data is split into 10 or more namespaces.

• Weaviate Flex: 1,536M dimensions × $0.00465 = $7.14 plus under $1 of storage, so the $45 minimum applies.

• Self-host infrastructure: three r7g.xlarge nodes at $156.37 each cost $469; the 8.6 GiB of vectors fits easily. Adding 150 GB of gp3 ($12) and snapshots ($0.36), then the 15% allowance, gives about $554 a month.

• Self-host operations: 0.15 FTE costs $2,428 a month in-house, $1,620 nearshore or $1,080 offshore.

• Initial build: 4 engineer-weeks cost $16,188 in-house, $10,800 nearshore or $7,200 offshore, once.

At pilot scale the managed bill is close to its minimum. The self-hosted three-node cluster alone costs more than ten times the managed subscription before anyone touches it. Verdict: managed, without much debate.

Scenario 2 — Growth: 20M vectors, 10M queries/month, 10M upserts/month

• Pinecone reads: one 143 GB namespace costs 10M queries × 143 RU × $16/M = $22,880 a month. Split into 500 tenant namespaces, it costs 10M × 0.286 RU × $16/M = $45.76.

• Pinecone writes, storage and support: 70M WU × $4/M = $280, 143 GB × $0.33 = $47.19, plus the $250 Pro support add-on. The total is $623 partitioned or $23,457 in a single namespace.

• Weaviate Flex: 30,720M dimensions × $0.00465 = $142.85, plus 133.2 GiB × $0.12 = $15.98 storage and $4.04 backup, for about $163 a month.

• Self-host infrastructure: three r7g.2xlarge nodes cost $938 ($680 on a 1-year Savings Plan). Int8 vectors take 42.9 GiB, or 85.8 GiB with two replicas, out of 192 GiB. Adding $48 of EBS and $7 of snapshots, then 15%, gives about $1,142 a month.

• Self-host operations: 0.35 FTE costs $5,666 a month in-house, $3,780 nearshore or $2,520 offshore.

• Initial build: 8 engineer-weeks cost $32,376 in-house, $21,600 nearshore or $14,400 offshore, once.

Weaviate Flex is a shared cluster with a 99.5% SLA. A team that needs SSO, private networking or a higher SLA moves to Premium at $400/month or more. Even then, managed stays well under the self-host total. Verdict: managed, provided your data partitions cleanly.

Scenario 3 — Scale: 200M vectors, 100M queries/month, 50M upserts/month

• Pinecone Enterprise reads: with 2,000 namespaces, 100M × 0.715 RU × $24/M = $1,716 a month. With 500 namespaces, 100M × 2.86 RU × $24/M = $6,864.

• Pinecone writes and storage: 350M WU × $6/M = $2,100 and 1,430 GB × $0.33 = $471.90. The total is $4,288–$9,436 a month for partitioned data; a single 1.4 TB namespace is not viable on usage pricing.

• Weaviate Premium (shared, lowest published rates): 307,200M dimensions × $0.003875 = $1,190, plus $133 storage and $28 backup, for about $1,352 a month. A quote is required.

• Self-host infrastructure: eight r7g.4xlarge nodes cost $5,004 ($3,625 on a 1-year Savings Plan). Int8 vectors take 429 GiB, or 858 GiB with two replicas, out of 1,024 GiB. Adding $320 of EBS and $72 of snapshots, then 15%, gives about $6,204 a month.

• Self-host operations: 0.75 FTE costs $12,141 a month in-house, $8,100 nearshore or $5,400 offshore.

• Initial build: 16 engineer-weeks cost $64,752 in-house, $43,200 nearshore or $28,800 offshore, once.

At this scale the answer depends on query shape. With fine-grained tenant partitions, managed stays cheaper. If queries must search large shared namespaces, an offshore-operated cluster ($139K/year steady state) undercuts Pinecone's 500-namespace case ($152K/year with vendor management).

Weaviate's figure uses the lowest published rate and will rise with HNSW indexes, dedicated tenancy or specific regions. Verdict: get vendor quotes and run the break-even test below.

The read-unit trap: partitioning changes the bill 100x

On Pinecone serverless, cost per query scales with the size of the namespace searched. The same 20M-vector, 10M-query workload costs:

• One namespace of 143 GB: 143 read units per query, about $23,457 a month.

• Ten namespaces of 14.3 GB: 14.3 read units per query, about $2,865 a month.

• 500 namespaces of 0.29 GB: 0.286 read units per query, about $623 a month.

Multi-tenant SaaS products fit this model well, since each customer's queries hit only their own namespace. A global semantic search across one large corpus does not. Check your query pattern before you compare vendors; it is the most consequential input in this framework.

Break-even: when self-hosting wins

Self-hosting pays off when the managed bill exceeds the infrastructure cost plus the extra engineering time it takes to run the cluster:

\text{Break-even managed bill per month} = \text{self-host infra per month} + \Delta\text{FTE} \times \text{monthly cost of 1 FTE}

ΔFTE is self-host operations effort minus managed vendor-management effort. One FTE costs $16,188/month in-house in the US, $10,800 nearshore and $7,200 offshore at the rates above.

• Growth (20M vectors): infrastructure of $1,142 plus 0.25 extra FTE puts the break-even at about $5,189 a month with a US team, or $2,942 with an offshore team.

• Scale (200M vectors): infrastructure of $6,204 plus 0.55 extra FTE puts the break-even at about $15,107 a month with a US team, or $10,164 with an offshore team.

If your managed quote sits above these lines for a steady workload, self-hosting deserves a serious look. A 1-year Savings Plan lowers both break-even points by roughly a quarter of the compute line.

Non-cost reasons that override the math

• Data residency or air-gapped requirements: self-host, or use bring-your-own-cloud options from Pinecone, Qdrant Hybrid Cloud or Zilliz BYOC.

• Predictable spend: provisioned models (Qdrant Cloud, Zilliz dedicated, self-hosted) do not spike with query volume.

• Latency control: self-hosting lets you co-locate the index with the application and tune HNSW and quantization directly.

• Team focus: every hour spent on re-indexing is an hour not spent on retrieval quality, which usually matters more to the product.

Team composition and timelines for a self-hosted build

• Pilot (1M vectors): a part-time senior platform or SRE engineer leads, with part-time ML help for recall testing and a security checklist. Modeled effort is 4 engineer-weeks.

• Growth (20M vectors): a senior platform engineer leads, one backend engineer handles ingestion and migration, ML help is part-time, and security gets a review. Modeled effort is 8 engineer-weeks.

• Scale (200M vectors): a senior platform lead plus an on-call rotation, one or two backend engineers, a dedicated ML or data engineer for quantization tuning, and a formal security review. Modeled effort is 16 engineer-weeks.

The build effort figures are illustrative estimates, not benchmarks. A 16-week Scale build spread across two or three engineers typically fits a quarter, and the on-call rotation needs at least two trained people to be sustainable.

Engagement models compared

• In-house hire: best for long-lived platform teams already running stateful systems. The risk is hiring time and dependence on one person.

• Dedicated nearshore team: best for real-time collaboration with US teams, with 4–6 hours of overlap per Full Scale. The trade-off is a higher rate than offshore.

• Dedicated offshore team: best for well-specified builds and follow-the-sun operations. The risk is handoff discipline across time zones.

• Fixed-price migration project: best for one-off moves between vendors or to self-hosting. The risk is scope creep once production traffic is live.

• Managed service plus an advisory retainer: best for teams that want managed infrastructure but need help with schema, partitioning and cost tuning. Its value depends on clear goals.

A five-step decision framework

Model your query shape first. Write down vectors, dimensions, queries per month, upserts per month and the size of the partition each query searches. On usage-priced plans the last number can swing the bill by two orders of magnitude.
Get three real quotes. Price one usage-based service (Pinecone), one dimension-based service (Weaviate Cloud) and one provisioned service (Qdrant Cloud or Zilliz dedicated) for your exact workload.
Price self-hosting with people in it. Use the RAM formula to size the cluster, then add build effort and an honest ops fraction. If nobody on the team has run a stateful cluster through an upgrade, use the high end.
Run the break-even test. If the best managed quote sits below infrastructure plus the extra engineering cost, stay managed and revisit at the next scale step.
Apply the non-cost filters. Residency, compliance, latency and lock-in can justify self-hosting or a bring-your-own-cloud option even when the math says otherwise.

Pilot-to-production budgeting

Most AI teams move through three budget stages. A pilot fits inside managed minimums of $45–$50 a month, so the real cost is engineering time on ingestion and retrieval quality. A first production launch typically adds a support plan and a higher-SLA tier, which in our Growth scenario still lands under $1,000 a month on managed plans. Enterprise scale is where contracts, committed-use discounts and dedicated capacity enter, and where a self-hosted or bring-your-own-cloud option is worth pricing properly.

Budget the vector database as one line in a larger retrieval stack.
Embedding generation, reranking, LLM inference and evaluation tooling are excluded from this model and often cost more than the database itself.

Frequently asked questions

How much does a managed vector database cost per month?

At pilot scale, roughly $45–$90 a month on Weaviate Cloud Flex or Pinecone Standard as of October 2026. At 20M vectors and 10M queries a month, our modeled range is about $160–$620 when data is partitioned well, and over $23,000 on Pinecone if every query scans one large namespace.

Is self-hosting a vector database cheaper than Pinecone?

The cloud bill usually is; the total cost usually is not. In our 20M-vector scenario, self-hosted infrastructure costs about $1,142 a month, but operations and build effort bring year one to $58K–$114K depending on where the team sits.

How much RAM does a vector database need?

A common estimate is vectors × dimensions × 4 bytes × 1.5 for float32 vectors held in memory. One million 1,536-dimension vectors need about 8.6 GiB before replication. Int8 quantization cuts that by roughly four times.
When does self-hosting a vector database make sense? When managed quotes exceed the break-even line (about $3K–$5K a month at 20M vectors and $10K–$15K at 200M in our model), when data must stay in your own environment, or when an existing platform team can absorb the operations work.

Does an offshore or nearshore team change the decision?

It lowers the break-even point. At the rates used here, an offshore-operated cluster costs about 41–49% less in year one than the same cluster run by a US in-house team. It does not remove the need for at least two people who can handle an incident.

How often should these numbers be re-checked?

Quarterly. Weaviate changed its price book in September 2026, and Qdrant restructured its tiers in 2026, so cost guides in this category go stale quickly.

Estimating the Cost of Running AI Evaluation Pipelines

Eric Thompson — Fri, 16 Oct 2026 15:06:58 +0000

Under the assumptions in this article, an AI evaluation pipeline costs about $400 a month in non-people spend for a pilot, about $5,300 a month for a single production AI feature, and about $35,000 a month for an enterprise program judging hundreds of thousands of cases. Add engineering time and the all-in figures become roughly $4,500, $25,500 and $84,000 a month with US in-house staff.

Three findings hold across every scenario:

• People cost more than tokens. Eval engineering and human review are 76–98% of the modeled total, depending on scale.
• Judge model choice is the biggest technical lever. A small judge model costs about 1/20th of a mid-tier one per call, at current list prices.
• Batch APIs halve offline eval spend. OpenAI, Anthropic and Google all list a 50% batch discount.
Pricing period: list prices checked October 16, 2026, in USD, US region, before negotiated discounts. Cost guides like this go stale fast; treat it as due for a quarterly update.
Scenario figures are illustrative estimates built from sourced prices and stated assumptions, not results from a real project. Swap in your own token counts and volumes.

What an evaluation pipeline includes (the cost components)

An eval pipeline has six cost lines; most budgets only count the first two. Each is listed below with what drives it.

System-under-test inference. Re-running your app or agent on the test set for each offline run. Driven by test cases × runs per month × tokens per case.
Offline judge calls. LLM-as-judge scoring of those outputs against rubrics or references. Driven by case-runs × criteria judged × judge model price.
Online (production) evals. Judging a sample of live traffic for drift, safety and quality. Driven by production volume × sample rate × judge price.
Human review. Expert or crowd labeling to build golden sets and calibrate judges. Driven by cases reviewed × minutes per case × hourly rate.
Eval and observability tooling. Tracing, datasets, experiment tracking and dashboards. Driven by seats, traces or billable units, and retention.
Engineering time. Building datasets, writing rubrics, maintaining CI gates and triaging failures. Driven by FTEs × loaded cost or vendor rate. Two items are often double-counted. Online evals do not pay for system-under-test inference: production already paid for that response. And CI runner compute is usually negligible next to API calls, so this model leaves it out.

Sourced pricing inputs (October 2026)

Every price below comes from a vendor pricing page checked on October 16, 2026. All are list prices in USD per million tokens unless noted.
Judge model token prices
Prices are shown as input / output, then batch input / batch output.
• Frontier tier: Claude Opus 5.5 at $4.00 / $20.00; batch $2.00 / $10.00.
• Mid tier: Claude Sonnet 5.5 and gpt-6.1-sol at $2.00 / $10.00; batch $1.00 / $5.00. Gemini 3.1 Pro Preview at $2.00 / $12.00 for prompts up to 200k tokens; batch $1.00 / $6.00.
• Small/mid tier: Gemini 3.8 Flash at $0.75 / $3.75; batch $0.375 / $1.875.
• Small tier: Claude Haiku 5.5 (prompts up to 100k tokens) and gpt-6-luna at $0.10 / $0.50; batch $0.05 / $0.25.
Price notes that change the math:
• Batch: all three vendors list a 50% batch discount, which suits nightly and pre-release eval runs.
• Caching: Anthropic lists cache reads at 0.1× base input (0.05× on Opus and Sonnet 5.5). OpenAI lists cached input for gpt-6.1-sol at $0.10. A rubric repeated on every judge call is an ideal cache candidate.
• Scheduled increases: Google lists Gemini 3.8 Flash rising to $1.50 input / $7.50 output from January 1, 2027.

Eval and observability platform pricing

• Langfuse Cloud: Core $29, Pro $199, Enterprise $2,499 a month. Each includes 100k billable units; overage is $8 per 100k units, lower at volume.
• Braintrust: Pro is $249 a month with 5 GB of processed data and 50k scores. Overage is $3/GB and $1.50 per 1,000 scores.
• LangSmith: Plus is $39 per seat a month with 10k base traces, then pay-as-you-go.
LangSmith no longer publishes a dollar rate per 1,000 traces on its pricing page. A third-party tracker reports about $5.00 per 1,000 base traces after a July 2026 metering change; confirm with LangChain before budgeting on it.

Compute, labor and annotation inputs

• GPU compute: an on-demand NVIDIA H100 SXM is $3.99 per GPU-hour (Lambda).
• US engineering pay: median pay was $135,980 a year for software developers and $104,300 for QA analysts and testers in May 2025 (BLS).
• Benefits load: benefits were 30.0% of private-industry compensation in June 2026 (BLS ECEC).
• Outsourced senior developer rates: Asia $31–$41/hr, Latin America $60–$75/hr, Europe $64–$76/hr (Accelerance).
• Crowd annotation floor: $8/hr minimum and $12/hr recommended participant pay (Prolific).

A transparent cost model you can rerun

Monthly eval cost is the sum of the six lines above. Written out:
C_{month} = N_{case} R \, c_{sut} + N_{case} R \, J_{off} \, c_{judge}^{batch} + V S \, J_{on} \, c_{judge} + H \, \frac{t}{60} \, w + T + F \, L
Where N_case = test cases, R = offline runs per month, J = judge calls per case, V = production requests, S = sample rate, H = human-reviewed cases, t = minutes per review, w = reviewer hourly rate, T = tooling, F = FTEs and L = loaded monthly cost per FTE.

Assumptions (illustrative, not measured)

• Judge call size: 2,000 input + 300 output tokens (rubric, input, candidate output and reference, plus a short scored rationale).
• System-under-test call size: 1,500 input + 500 output tokens, typical of a single-turn RAG or assistant response. Agents run far larger.
• Default judge and app model: mid tier at $2 / $10 per million tokens, matching Claude Sonnet 5.5 and gpt-6.1-sol list prices.
• Offline runs: Batch API at 50% off, since nightly and pre-release runs tolerate batch latency.
• Online judging: standard pricing, because teams usually want same-day drift signals.
• Human review: 4 minutes per case at $45/hr. The domain-expert rate is an assumption; the crowd floor is $12/hr per Prolific.
• Tooling: Langfuse list prices at 5–7 billable units per judged case (one trace plus spans and scores), ignoring volume discounts.
• Loaded US FTE cost: $16,188 a month = BLS median $135,980 ÷ 0.70 wage share ÷ 12. AI specialists usually earn more.

Unit costs these assumptions produce

• Small-tier judge call: $0.00035 standard, $0.000175 batch.
• Mid-tier judge call: $0.0070 standard, $0.0035 batch.
• Frontier-tier judge call: $0.0140 standard, $0.0070 batch.
• Mid-tier system-under-test call: $0.0080 standard, $0.0040 batch.
Worked example, mid-tier batch judge: 2,000 × $1.00/M + 300 × $5.00/M = $0.0020 + $0.0015 = $0.0035 per call.

Three budget scenarios

Pilot. 500 golden cases, 20 offline runs a month, 1 judge call per case, no live traffic judged, 100 human-reviewed cases and 0.25 eval engineer.
Production (one AI feature). 2,000 cases, 60 runs a month, 2 judge calls per case, 50,000 live requests judged (5% of 1M), 1,000 human-reviewed cases and 1.25 FTEs.
Enterprise program. 10,000 cases, 60 runs a month, 3 judge calls per case, 400,000 live requests judged (2% of 20M), 5,000 human-reviewed cases and 3 FTEs.

Monthly results, line by line:

• Pilot: system-under-test inference $40, offline judging $35, online judging $0, human review $300, tooling $29. Non-people subtotal $404. Engineering $4,047. All-in $4,451 a month, $53,412 a year.
• Production: inference $480, offline judging $840, online judging $700, human review $3,000, tooling $273. Non-people subtotal $5,293. Engineering $20,235. All-in $25,528 a month, $306,336 a year.
• Enterprise: inference $2,400, offline judging $6,300, online judging $8,400, human review $15,000, tooling $3,051. Non-people subtotal $35,151. Engineering $48,564. All-in $83,715 a month, $1,004,580 a year.
Calculation check, Production offline judging: 2,000 cases × 60 runs × 2 calls × $0.0035 = $840. Production tooling: (120,000 + 50,000) judged cases × 6 units = 1.02M units; $199 + 9.2 × $8 = $273.

Sensitivity: the judge model decides the token bill

Monthly judging spend (offline plus online) by judge tier:
• Small judge: $2 pilot, $77 production, $735 enterprise.
• Mid judge: $35 pilot, $1,540 production, $14,700 enterprise.
• Frontier judge: $70 pilot, $3,080 production, $29,400 enterprise.
Moving the enterprise program from a frontier to a small judge saves about $28,700 a month on list prices. That saving is only real if the small judge agrees with human labels well enough; measure agreement on your own golden set before switching.

Self-hosting a judge: when does a GPU pay off?

One on-demand H100 at $3.99/hr running all month costs about $2,913 (730 hours). That equals roughly 832,000 mid-tier batch judge calls, or 16.6 million small-tier batch calls. Self-hosting an open-weight judge only beats the API when volume is high, utilization stays near full, and someone is paid to run the serving stack. For most teams below the enterprise scenario, API judges are cheaper.
People cost: in-house vs offshore vs nearshore vs dedicated team
Staffing model moves the Production scenario's all-in cost from about $25,500 to $11,500–$20,500 a month. The people line is where hiring decisions matter most.

Who you need

• Eval / ML engineer: builds the harness, writes rubrics and judge prompts, wires CI gates, tracks judge-human agreement.
• QA or annotation lead: curates golden sets, runs human review, writes labeling guidelines.
• Domain reviewers: part-time experts (clinicians, lawyers, support leads) who label the hard cases.
• Product owner: decides which failures block a release. Usually an existing role, so not costed here.

Production scenario (1.25 FTE) by staffing model

All-in figures add the $5,293 non-people subtotal.
• US in-house. $16,188 loaded per FTE gives $20,235 a month, or $25,528 all-in. Deepest product context, but the slowest and costliest to hire.
• Nearshore, Latin America (senior). $60–$75/hr × 160 hours gives $12,000–$15,000, or $17,293–$20,293 all-in. Strong working-hour overlap with US teams.
• Europe (senior). $64–$76/hr gives $12,800–$15,200, or $18,093–$20,493 all-in. Deep engineering pools with partial US overlap.
• Offshore, Asia (senior). $31–$41/hr gives $6,200–$8,200, or $11,493–$13,493 all-in. Lowest rate; needs explicit handoff and review processes.
• Hybrid: 0.25 US lead + 1.0 dedicated Asia engineer. $9,007–$10,607, or $14,300–$15,900 all-in. Keeps rubric ownership in-house and outsources build and upkeep.
Rates come from the Accelerance 2026 guide summary, which covers general software development. Specialists in LLM evaluation often bill above those ranges, so treat them as a floor.
The comparison is not like-for-like. Vendor rates include the vendor's overhead and benefits. The in-house figure excludes recruiting, management time and equipment. Outsourced models need internal oversight, which this table does not cost.

Engagement models for eval work

• Fixed-scope project. Best for building the first harness and golden set. Predictable one-off cost, but the pipeline decays after handover if nobody owns it.
• Dedicated team (monthly). Best for ongoing eval upkeep across several AI features. Flat monthly cost, with low utilization if the AI roadmap stalls.
• Time and materials. Best for uncertain scope and research-heavy agents. Flexible but harder to forecast; define release gates early to contain scope.
• Staff augmentation. Best for filling one eval engineer seat in an existing team. Billed as rate × hours; knowledge stays with the individual unless it is documented.

Pilot-to-production: where the eval budget jumps

In this model, non-people eval spend rises about 13× from pilot to production ($404 to $5,293) while the test set grows only 4×. Three changes cause the jump, and each is a budgeting decision rather than an accident.

Run frequency triples. A pilot runs evals when someone remembers. Production wires them into CI and nightly jobs, so case-runs go from 10,000 to 120,000 a month.
Online judging starts. Sampling live traffic adds a cost line that did not exist in the pilot ($700 a month here), and it scales with usage, not with your test set.
Human review becomes routine. Judges drift and need recalibration against human labels. Human review is the largest non-people line in both the production and enterprise scenarios. For production-ready AI, budget the eval pipeline as a share of the feature's running cost, not as a one-off build. Enterprise AI implementation budgets that fund the model and the app but not ongoing evaluation risk shipping regressions they cannot measure.

Cost-reduction levers, largest first

• Right-size the judge. Use small judges for format, schema and simple factuality checks; save mid or frontier judges for nuanced rubric scoring. In the enterprise scenario, the gap between small and frontier judges is about $28,700 a month.

• Batch everything offline. The 50% batch discount is already in this model. Teams running offline evals on standard pricing pay double for the same scores.

• Cache the rubric. The rubric and instructions repeat on every judge call. Anthropic lists cache reads at 5–10% of base input price, which cuts a large share of judge input cost.

• Run tiered suites. Run a small smoke set on every pull request and the full golden set nightly or before release. Halving runs halves the offline lines.

• Sample live traffic by risk. Judge 100% of flagged or high-value sessions and a low random rate of the rest, instead of one flat sample rate.

• Spend human time on disagreements. Route only cases where judges disagree or score low confidence to reviewers. Human review is the largest line you control outside salaries.

• Watch tooling meters. Retention upgrades and per-trace overages add up at enterprise volume. Negotiate annual commitments once volumes are stable.

FAQ

How much does LLM-as-judge evaluation cost per test case?

At October 2026 list prices and a 2,000-in / 300-out token judge call, one judgment costs about $0.00035 on a small model, $0.007 on a mid-tier model and $0.014 on a frontier model. Batch pricing halves each figure.

Is the API bill the biggest cost of AI evaluation?

Usually not. In all three scenarios here, engineering time and human review make up 76–98% of the total. Token spend becomes material only at high volume with mid or frontier judges.

What should a startup budget for evals on its first AI feature?

Under this model, a pilot needs roughly $400 a month in non-people spend plus a quarter of an engineer's time. Moving that feature to production raises non-people spend to about $5,300 a month.

Should we self-host an open-weight judge model to save money?

Only at high, steady volume. One on-demand H100 costs about $2,913 a month at $3.99/hr, equal to about 832,000 mid-tier batch judge calls. Below that, or with a small API judge, the API is cheaper and needs no serving team.

Is it cheaper to outsource eval engineering?

On rates alone, yes. For 1.25 FTEs, published senior rates put Asia at $6,200–$8,200 a month and Latin America at $12,000–$15,000, against about $20,200 loaded for US in-house staff. Factor in oversight time, timezone overlap and who owns the rubrics.

Which eval platform is cheapest?

It depends on your meter. Langfuse bills per unit, Braintrust per GB and score, and LangSmith per seat and trace. Estimate your monthly traces and scores, then price all three against their current pages.

How often should this cost estimate be updated?

Quarterly. Vendors in this article have changed prices in 2026, and Google has already scheduled a Gemini 3.8 Flash increase for January 2027.

Calculating Token, Retrieval, and Observability Costs for a RAG Service

Eric Thompson — Thu, 15 Oct 2026 15:09:49 +0000

A typical RAG query (2,000 tokens of prompt, 2,500 tokens of retrieved context, and 400 tokens of output) on a mid-tier model costs about $0.013 in usage. Once retrieval, reranking, and observability are added, that comes to roughly $13 per 1,000 queries at production volume.

In our model, a service handling 10,000 queries a month costs about $203 in usage, or about $234 with a 15% buffer for retries, query rewrites, and evaluation runs. At 100,000 queries a month, the bill is about $1,369, or $1,574 with the buffer. At 1 million queries a month, it reaches about $13,113, or $15,079 with the buffer. Ten times the traffic doesn't mean ten times the bill at lower volumes, because fixed fees such as plan minimums are shared across more queries.

*Three findings matter more than the totals:
*
LLM tokens are ~75–80% of usage cost at scale. Vector search is under 1%.
Reranking is the surprise second line item (~15% at 100K+ queries).
At low volume, people cost more than tokens. Half an AI engineer maintaining the service costs $3,300–$11,250 a month depending on location, 15–50× the usage bill at 10K queries.

Every number below is either linked to a source or labelled as an illustrative assumption.

The cost equation

A RAG service has three kinds of cost:

Monthly cost =
queries × (LLM + query embedding + vector read + rerank + observability per query)

fixed platform fees (plan minimums, seats, storage)
corpus maintenance (re-embedding, writes)
people (build, evaluation, on-call, iteration)

The rest of this article prices each term.

1. Token costs: where most of the money goes

*The per-query token budget
*
We modelled a knowledge-assistant style RAG endpoint. These are illustrative assumptions; measure your own with real traffic before you trust any forecast.

Each request starts with a 1,500-token system prompt of instructions. Because it's identical on every request, it can be cached and billed at a discounted rate. Next comes the user's message plus a short conversation history, about 500 tokens. Then the system adds the retrieved context: five chunks of 500 tokens each, or 2,500 tokens in total. Both of these change on every request, so they're billed at the full input price. Finally, the model writes an answer of about 400 tokens, billed as output. That's 4,900 tokens per request.

Retrieved context is the largest input. Every extra chunk you stuff into the prompt is paid for on every single query.

Current list prices (October 2026)

**
Model prices fall into four broad tiers, quoted per million tokens.

Small tier (Claude Haiku 5.5 for prompts up to 100K tokens, and OpenAI's gpt-6-luna): $0.10 for input, $0.50 for output and $0.01 for cached input.
Mid tier (Claude Sonnet 5.5 and gpt-6.1-sol): $2.00 for input, $10.00 for output and $0.10 for cached input. The two vendors price this tier identically, so the choice between them comes down to quality on your own evaluation set.
Large tier: Claude Opus 5.5 charges $4.00 for input, $20.00 for output and $0.20 for cached input. gpt-6-astra charges $10.00, $50.00 and $1.00.

Across every tier, output costs five times as much as input. That's why long answers cost more than long prompts of the same length.

Sources: Anthropic and OpenAI official pricing pages, captured October 15, 2026. Both vendors offer roughly 50% off for asynchronous batch processing, which suits offline evaluation and re-indexing but not live chat.

Per-query calculation (mid tier, with prompt caching)

System prompt: 1,500 × [0.95 × $0.10 + 0.05 × $2.50] / 1M = $0.00033
Other input: 3,000 × $2.00 / 1M = $0.00600
Output: 400 × $10.00 / 1M = $0.00400
Total ≈ $0.01033 per query

Without caching, the same query costs $0.0130. Caching saves about 20% here because the system prompt is only 30% of input; the retrieved context changes every query and can't be cached the same way.

Running the same 4,900-token request through each tier shows how much the price list matters. On a small-tier model, a query costs about $0.00052, or $0.52 per 1,000 queries. On a mid-tier model, it's about $0.01033, or $10.33 per 1,000. Claude Opus 5.5 brings it to $0.02066, or $20.66 per 1,000, and gpt-6-astra to $0.05236, or $52.36 per 1,000. Nothing about the query changes between these figures; only the model does. For budgeting, multiply the per-1,000 figure by your expected traffic in thousands.

The 100× spread between small and large tiers is the single biggest decision in your budget. Many teams route simple queries to a small model and escalate only hard ones; whether that works for you depends on your evaluation results, not on price alone.

2. Retrieval costs: embeddings, vector database and reranking

*Embeddings: cheap enough to ignore (almost)
*
OpenAI's text-embedding-3-small is listed at $0.02 per 1M tokens.

Initial corpus: 500M tokens × $0.02 / 1M = $10.00 one-off
Monthly refresh: 10% of corpus = $1.00
Per query: 50 tokens × $0.02 / 1M = $0.000001

Embedding cost is negligible. The real cost of embeddings is engineering time spent on chunking strategy and re-indexing when you change models.

*Vector database: small, but watch the pricing model
*
We used Pinecone serverless as the example because its pricing is usage-based and documented. The key rule, from Pinecone's cost documentation: a query consumes 1 read unit per GB of namespace size, with a 0.25 RU minimum, regardless of top_k.

Index size: 1M vectors × (1,536 dims × 4 bytes + 1 KB metadata) ≈ 7.14 GB
Read units: ≈ 7.14 RU per query
Read cost: 7.14 × $16 / 1M ≈ $0.000114 per query
Storage: 7.14 GB × $0.33 ≈ $2.36 / month
Plan minimum: $50 / month (Standard)

Unit rates are from a secondary source and should be confirmed in Pinecone's calculator; the $50 minimum and RU rule are from Pinecone's own pages.

At 10K and 100K queries per month, you'll pay the $50 minimum. Even at 1M queries, the vector database costs about $120/month.

*Two practical consequences:
*
Namespace size drives read cost. Partitioning by tenant or document type cuts RUs per query proportionally.
If you already run Postgres, pgvector on existing infrastructure may cost nothing incremental at this scale. The trade-off is operational: you own tuning, scaling and backups.
*Reranking: the line item people forget
*
A cross-encoder reranker takes the top 50 candidates and picks the best 5. Cohere defines one search as one query with up to 100 documents, and Rerank 4 Fast is listed at $2.00 per 1,000 searches (secondary source, verify).

Per query: $0.002
At 100K queries/month: $200
At 1M queries/month: $2,000

That's about 15% of usage cost at scale, and more than the vector database by an order of magnitude. Reranking often improves answer quality enough to let you send fewer chunks to the LLM, which can pay for itself. Test both configurations.

3. Observability costs: per-unit vs per-trace pricing

You can't run RAG in production without tracing retrieval quality, latency and token spend. The two common hosted options price differently.

Langfuse Cloud charges per unit, where every trace, observation and score you record counts as one unit. Its Core plan costs $29 a month and includes 100,000 units, with additional usage at $8 per 100,000 units (lower at higher volumes). LangSmith charges per seat and per trace. Its Plus plan costs $39 per seat per month and includes 10,000 base traces, with additional traces at about $2.50 per 1,000, kept for 14 days.

In practice, a small team with lots of traffic tends to favour unit pricing, while a large team with modest traffic may find per-trace pricing easier to predict.

Sources: Langfuse and LangChain official pricing pages. LangSmith overage rate from a June 2026 secondary source; verify before budgeting.

*Translating that to per-query cost
*
In Langfuse, a single RAG request typically produces one trace plus several observations (embed, retrieve, rerank, generate, guardrail, post-process) and maybe an automated score. Assumption: 8 units per query.

At 10,000 queries a month, that's 80,000 units, inside Langfuse's included allowance, so the bill is just the $29 plan fee. LangSmith comes to $117, which is three seats at $39 each. At 100,000 queries, Langfuse rises to about $85 and LangSmith to about $342, as per-trace charges kick in above 10,000 traces. At 1 million queries, Langfuse reaches about $661 at list rates, while LangSmith reaches about $2,592. The gap between the two widens as traffic grows.

Illustrative; both vendors discount at volume and both can be self-hosted, which swaps subscription cost for infrastructure and engineering time.

Observability can quietly reach 5–20% of usage cost. Sampling (tracing 10–20% of production traffic in full while logging tokens for 100%) is a common way to control it.

4. Three monthly scenarios, line by line

These scenarios use the mid-tier model with prompt caching on, and Langfuse for observability.

At 10,000 queries a month, LLM tokens cost $103.30, reranking $20.00 and observability $29.00. The vector database costs $50.00, which is simply Pinecone's plan minimum. Query embeddings cost a cent, and refreshing a tenth of the corpus each month costs $1.00. The total is $203.31, or $20.33 per 1,000 queries, rising to $233.81 with the 15% buffer. At this volume, the vector database is a quarter of the bill, purely because of its minimum fee.

At 100,000 queries a month, LLM tokens rise to $1,033.00 and reranking to $200.00. Observability comes to $85.00, and the vector database stays at its $50.00 minimum. Embeddings cost $0.10 and corpus refresh $1.00. The total is $1,369.10, or $13.69 per 1,000 queries, and $1,574.46 with the buffer.

At 1 million queries a month, LLM tokens reach $10,330.00 and reranking $2,000.00. Observability is $661.00, and the vector database finally rises above its minimum to $119.52. Embeddings cost $1.00 and corpus refresh $1.00. The total is $13,112.52, or $13.11 per 1,000 queries, and $15,079.40 with the buffer. LLM tokens now account for almost four-fifths of spending and reranking for most of the rest, while embeddings and storage barely register.

*What happens if you change the model
*
Holding everything else constant at 1 million queries a month, the model choice dominates the bill. With a small-tier model, LLM cost falls to about $524 and the usage total to about $3,300. The mid-tier model gives the $10,330 LLM cost and ~$13,100 total above. Claude Opus 5.5 pushes LLM cost to $20,660 and the total to about $23,400, and gpt-6-astra to $52,363 and about $55,100. Each total includes the same $2,783 of retrieval, reranking and observability, so the difference is entirely the LLM line. Moving from the cheapest tier to the most expensive one makes the bill roughly 17 times larger.

Per-1,000-query cost falls as volume grows only because fixed fees get spread thinner. After roughly 100K queries, the bill becomes almost entirely variable.

5. The line item that dwarfs tokens at low volume: people

A production RAG service needs ongoing work: evaluation sets, chunking and prompt changes, regression testing when models update, cost monitoring, and incident response. Assumption: 0.5 FTE AI engineer for maintenance, roughly 20 hours a week.

Done in-house in the US, at a fully loaded cost of about $150 an hour, that half-engineer costs about $11,250 a month. Outsourced to Central and Eastern Europe at about $80.50 an hour, it costs about $6,440. In Latin America, at about $77.63 an hour, it's about $6,210. In Asia, at about $41.40 an hour, it's about $3,312.

The US figure is what a company pays its own employee including benefits, while the outsourced figures are vendor-billed rates with overhead and margin already included. The cheapest location isn't automatically the best value, because time-zone overlap, review effort and rework can narrow the gap.

US rate derived from a $185K base salary and BLS's June 2026 benefits share (wages at 68.5% of compensation for full-time private-industry workers). Outsourced rates from Accelerance 2026 senior bands plus a 15% AI premium. Illustrative.

*What this means for the build-vs-buy-vs-outsource decision:
*
At 10K queries/month, usage is ~$200 and maintenance labour is $3,300–$11,250. Your real cost question is who maintains the system, not which model you pick.
At 1M queries/month, usage (~$13K) exceeds the labour cost in every location, and model routing and prompt efficiency start paying for an engineer's time.
A hybrid model is common: one internal owner for evaluation and architecture, with build and maintenance capacity from a nearshore or offshore team. Keep the evaluation dataset and the prompt/retrieval configuration under your own version control regardless of who writes the code.

6. Sensitivity: which levers move the bill?

To see which decisions matter most, we changed one setting at a time in the 100,000-query mid-tier scenario, which has a base cost of about $1,369 a month.

Switching to a small-tier model has by far the largest effect, cutting about $980 a month, or 72%, because tokens are the dominant cost. Retrieving three chunks instead of five removes 1,000 input tokens per query and saves $200, or 15%. Shortening answers from 400 to 250 tokens saves $150, or 11%, since output tokens cost five times as much as input. Dropping the reranker would also save $200, or 15%, but is likely to hurt answer quality.

Some changes raise costs instead. Turning off prompt caching adds about $270, or 20%, because the system prompt is then billed at the full rate. Switching observability to LangSmith with three seats adds about $257 through seat and per-trace charges. Partitioning the vector index into smaller namespaces saves essentially nothing at this volume, because you're paying Pinecone's $50 minimum either way.

These are effects on cost only. Retrieving fewer chunks or removing the reranker may lower answer quality, so test those changes before adopting them.

The lesson: optimise tokens first, reranking second, everything else later.

FAQ

**How much does a RAG query cost?
**In our model, a mid-tier RAG query costs about $0.013 all-in (~$13 per 1,000 queries) at 100K+ queries/month. With a small model it can drop below $0.003; with a frontier model it can exceed $0.05.

**What is the biggest cost in a RAG system?
**LLM tokens at scale — specifically input tokens from retrieved context. At low volume, engineering time to maintain and evaluate the system is larger than all usage costs combined.

**How much does a vector database cost for RAG?
**For a ~7 GB index (1M chunks), serverless vector search costs roughly $50–$120/month at 10K–1M queries in our model. Plan minimums dominate at low volume.

**Does prompt caching reduce RAG costs?
**Yes, but less than in chatbots with long static prompts, because retrieved context changes each query. In our example, caching cut LLM cost by about 20%.

**How much should I budget for LLM observability?
**Roughly $30–$350/month at 10K–100K queries and $650–$2,600/month at 1M queries in our model, depending on tool and seats. Sampling traces reduces this.

**Should I self-host the vector database and observability stack?
**Self-hosting removes subscription fees but adds infrastructure and engineering time. At the volumes modelled here, the subscription line items are small relative to a half-FTE engineer, so self-hosting usually makes sense for data-residency or compliance reasons rather than cost.