"How much will this cost to run?" is the question that kills more AI side projects than any technical problem. People build a working RAG chatbot in a weekend, then freeze because they have no idea whether serving it costs 5 dollars a month or 5,000. This post breaks the bill into its real parts so you can estimate before you commit.
I'll use round numbers and current-ish prices. Yours will differ. The point is the shape of the bill, not the exact digits.
The bill has four line items
A retrieval-augmented chatbot spends money in four places:
- Embedding your documents (one time, plus updates)
- Storing the vectors
- Embedding each incoming question
- Generating each answer
Three of those four are close to free. One of them is the whole bill. Guess which before you read on.
1. Embedding your documents
You embed each chunk of your documents once. Embedding is cheap. Most providers charge a few cents per million tokens.
Say you have 200 documents averaging 3,000 words each. That's roughly 800,000 tokens. At current embedding prices that is a fraction of a cent to a few cents, total, one time. Even 10,000 documents lands in the low single-dollar range.
If your documents never change, you pay this once and forget it. If they change daily, you re-embed the changed ones, which is still pennies.
Line item: negligible.
2. Storing the vectors
Each chunk becomes a vector, usually 768 or 1536 numbers. A few thousand chunks is a few megabytes. You can hold it in memory, in a local file, or in a hosted vector database.
For a small project, a local store (Chroma, SQLite with a vector extension, FAISS on disk) costs nothing beyond the machine you already run. Hosted vector databases start free and stay cheap until you're storing millions of vectors.
Line item: zero to a few dollars a month until you're at real scale.
3. Embedding each question
Every question a user asks gets embedded so you can search. A question is short, maybe 20 tokens. At embedding prices that is far less than a hundredth of a cent per question.
A thousand questions a day is still cents per month.
Line item: negligible.
4. Generating the answer
Here's the bill.
Every answer is a call to a language model, and the input to that call is not just the question. It's the question plus the chunks you retrieved plus your instructions. That's the expensive part, because generation models charge per token and you're feeding them a few thousand tokens of context every single time.
Rough math. Say each answer sends 2,000 tokens of context and gets back 400 tokens. On a mid-tier model at a few dollars per million input tokens and a bit more for output, one answer costs somewhere around half a cent to two cents depending on the model.
Now multiply by traffic:
- 100 answers a day: roughly 1 to 6 dollars a month
- 1,000 answers a day: roughly 15 to 60 dollars a month
- 10,000 answers a day: roughly 150 to 600 dollars a month
That range is wide because model choice swings it more than anything else. A frontier model can cost 10 to 20 times what a smaller model costs for the same job. For a lot of support and lookup use cases, a smaller model with good retrieval beats a frontier model with sloppy retrieval, and costs a tenth as much.
What actually moves the number
Once you see that generation is the bill, the levers are obvious:
- Model choice. The single biggest lever. Try the cheapest model that passes your quality bar before reaching for the expensive one.
- Context size. Retrieving 15 chunks instead of 5 triples your input tokens for often worse answers. Fewer, better chunks cost less and read better.
- Caching. If the same questions come up again and again, and in support they do, cache the answers. A cache hit costs nothing.
- Trimming instructions. A 600-token system prompt sent on every call adds up. Tighten it.
A quick estimate you can do right now
Take your expected answers per day, multiply by 30 for the month, multiply by roughly 1 cent. That's your rough monthly generation bill on a mid-tier model. Everything else on the bill is rounding error at small scale.
So a chatbot answering 300 questions a day costs on the order of 10 dollars a month to run. That's the number that unfreezes the project.
Where this fits
If you'd rather not stand up the embedding, storage, and generation plumbing yourself and meter each piece, that's the kind of thing LLMGraph handles. You build the workflow, it runs the retrieval and the model calls, and you get a deployed API and chat widget. The cost shape above is still what's happening underneath, so it's worth understanding either way.
The takeaway: a RAG chatbot is cheap to run at small scale, generation is the only line item that matters, and model choice is the dial that turns a 10-dollar bill into a 200-dollar one. Estimate before you build, and the "what will this cost" freeze goes away.
Top comments (0)