DEV Community: BoilerShelf

RAG Evaluation Metrics: How to Test a RAG Chatbot Before You Ship It

BoilerShelf — Sat, 14 Nov 2026 04:14:02 +0000

A RAG chatbot fails quietly. It still answers, in confident prose, even when retrieval has returned the wrong passage or the model has ignored the context. Without a way to measure that, every change to chunk size, embedding model, prompt or provider is a guess. This guide covers how to build an evaluation you can repeat, and how to describe its results without overstating them.

Evaluate retrieval and generation separately

A wrong answer has two possible causes, and they need different fixes. Either the right passage never reached the model (a retrieval problem), or it did and the model used it badly (a generation problem). If you only score the final answer you cannot tell which, so measure both stages.

Retrieval metrics

Hit rate or context recall. For a question with a known source passage, did that passage appear in the top results? This is the first number to look at, because nothing downstream can recover a passage that was never retrieved.
Precision. How much of what was retrieved is relevant? Low precision floods the model with noise, which raises cost and tempts it to blend unrelated facts.
Ranking quality. Measures such as mean reciprocal rank reward putting the right passage first, which matters when you pass only a few passages to the model.

Tune the number of passages and any similarity cut-off against these numbers rather than by feel, and remember that a cut-off that is too strict produces confident refusals, while one that is too loose produces confident nonsense.

Generation metrics

Groundedness (faithfulness). Is every claim in the answer supported by the retrieved passages? This is the closest measurable proxy for hallucination.
Answer relevance and completeness. Does the answer address the question that was asked, and does it include the parts that matter?
Citation correctness. A citation that points at a real retrieved passage is not the same as a citation that supports the sentence it follows. Checking that the reference exists is easy and worth doing, but it only proves the format, not the truth.
Refusals. When the documents do not contain the answer, the bot should say so. Include questions that cannot be answered and check that it declines.
Forbidden claims. List statements the bot must never make, such as promising a refund the policy does not allow, and check that they never appear.

Build a test set that earns its name

Thirty to one hundred questions is a useful starting size. Draw them from real sources: support tickets, search logs, questions the client's staff actually ask. Avoid questions you wrote by copying a sentence from a document, because they are trivially easy. For each question record the passage or passages that should be retrieved, and where it helps, the key facts a correct answer must contain. Add unanswerable questions and tricky ones: negations, numbers, questions that span two documents, and wording that does not match the document's vocabulary.

Keep the test set stable. Changing questions after a failure so the numbers recover is the quickest way to make an evaluation meaningless.

Choose checks you can repeat

Two families of check complement each other.

Deterministic checks are cheap and stable: the expected source was retrieved, a forbidden phrase is absent, every citation tag refers to a retrieved passage, the answer arrived within a time limit. They run on every change and never disagree with themselves.
Model-graded checks use a second language model as a judge of groundedness or relevance. They catch nuance a string match cannot, but a judge has its own biases and varies between runs. Calibrate it by comparing its verdicts with human labels on a sample, and record which judge model and prompt you used.

Whichever you use, add periodic human review of a random sample. It is the only check that can catch what both automated methods miss.

Make it a regression test

The real value of an evaluation is comparing runs. Freeze and record the whole configuration with each run: chunk size and overlap, embedding model, number of passages, similarity cut-off, system prompt, generation model and its settings. A short fingerprint of that configuration lets you see at a glance whether two results are comparable. Then make a rule that a change to any of those settings does not ship until the suite has run and the differences are understood.

Measure latency and cost per question

Record the time each question took and report percentiles, not just an average. For cost, use the token counts that the provider returns with each response, multiplied by the provider's current prices. A flat per-answer estimate looks tidy in a report but hides the expensive cases.

Reporting results to a client

A pass rate is only meaningful with its definition beside it. A useful report states the test set size and where the questions came from, the configuration that was tested, what each check means, which cases failed and why, and the date. It also states limits. A passing score means the bot did well on those questions with those documents. It does not guarantee behavior on questions nobody wrote, and a citation check shows that an answer cites a source, not that the answer is correct.

Common mistakes

A tiny test set where a single case moves the score by several points.
Only easy, document-derived questions, and no unanswerable ones.
Scoring the final answer alone, so retrieval and generation faults cannot be told apart.
Treating a model judge as ground truth without calibrating it.
Re-running until the numbers look good, or editing the test set after a failure.

Where to go next

If you are choosing a starting point for a RAG project, the RAG boilerplates category lists the options, and this comparison explains how two of them differ. Whatever you choose, build your own test set first and judge any starter by how it performs on it.

The idea in short

A fixed test set of questions runs through the same pipeline as live traffic. For each question the run records which passages were retrieved, the answer produced and the time taken, then applies retrieval checks, answer checks and a configuration fingerprint, so two runs can be compared after any change.

Originally published at boilershelf.com.

AI SaaS Architecture: How to Structure a FastAPI and Next.js Product

BoilerShelf — Fri, 13 Nov 2026 09:50:04 +0000

An AI SaaS looks like any other SaaS from the outside: a login, a dashboard, a billing page. The differences sit underneath. Every request can cost real money, latency is measured in seconds rather than milliseconds, you depend on providers you do not control, and customers hand you sensitive data. This guide walks through the decisions that shape a FastAPI and Next.js product, in the order they usually bite.

1. Split the frontend and backend on purpose

Use Next.js for what it is good at: rendering, routing, SEO and the user interface. Keep the business logic, data access and every call to a language model in the backend. FastAPI suits that role because it is asynchronous, typed through Pydantic, and produces an OpenAPI description of your API for free. That description is your contract: the frontend, the documentation and your tests can all be generated or checked against it.

The rule that saves the most pain: the browser never talks to an LLM provider directly and never holds a provider key. All model traffic goes through your backend.

2. Authentication that survives production

Most early mistakes here come from convenience. Store short-lived access tokens, and keep the longer-lived refresh token in an HttpOnly cookie so scripts on the page cannot read it. Rotate the refresh token on every use and reject a token that has already been used, because a replayed token is the sign of a theft. Hash passwords with a slow algorithm, verify email addresses, and put roles behind every administrative route rather than hiding the buttons in the interface.

Decide early whether the frontend framework or the backend owns the session. Splitting responsibility between them is a common source of subtle bugs, such as a user who is logged in on one side and logged out on the other.

3. Choose your tenancy model before you have customers

There are three common shapes: single users with their own data, workspaces that a few people share, and organizations with roles, invitations and billing. Retrofitting tenancy into a product that assumed single users means touching every table and every query, so pick the shape you expect to need.

For shared databases, put a tenant identifier on every tenant-owned record and enforce it in one data-access layer instead of trusting each endpoint to remember. Then test it from the attacker's side: sign in as a member of tenant A and try to read tenant B's records by guessing identifiers. Every route should answer with a refusal.

4. Put an LLM gateway between your code and the providers

The gateway is a small internal service layer with one interface for chat and embedding calls, behind which sit the individual providers. It is where the AI-specific concerns live:

Credentials. Store provider keys encrypted at rest, scoped to the tenant that owns them, and never return them to the browser.
Model choice. Keep an allow-list of models per plan so a customer cannot select an expensive one by editing a request.
Resilience. Apply timeouts, retries with backoff and a fallback provider, and translate each provider's errors into your own small set.
Accounting. Record the model, input tokens and output tokens for every call. Use the usage numbers the provider returns, not an estimate based on text length.
Limits. Enforce per-tenant rate limits and monthly quotas here, so no feature can bypass them.

Without a gateway, these concerns end up copied into every feature, and the first provider change becomes a refactor.

5. Move slow work to background jobs

A web request that waits twenty seconds for a document to be processed will time out behind a proxy and tie up a worker. Anything beyond a few seconds, such as document ingestion, batch runs and report generation, belongs in a queue with separate workers. Make tasks idempotent so a retry cannot do the work twice, store a status the frontend can poll, and decide up front what happens when a task fails halfway: retry, mark as failed with a reason, or alert a person.

6. Meter usage separately from billing

Keep two ledgers. Usage events record what happened: who, when, which model, how many tokens. Billing records what you charged. Deriving one from the other later is far easier than reconstructing usage from invoices. When a payment provider sends webhooks, verify the signature, process each event once, and grant entitlements only from the server side. Never trust a value the browser sends about what a customer has paid for.

7. Observability and cost control

Give every request an identifier and carry it through the logs, the queue and the provider call. Track latency percentiles instead of averages, because a slow tail is what users notice. Set a budget alert for provider spend, and cache only where the same input is guaranteed to deserve the same answer.

8. Deployment and secrets

Package the backend and the frontend as containers, run database migrations as an explicit step instead of at start-up, keep secrets in your platform's secret store rather than in the repository, and rehearse a restore from backup before you need one.

Build it or start from a codebase

None of these pieces is exotic, but together they take months to get right. If you would rather begin with them in place, compare the options by asking the same questions of any starter: how does it handle refresh tokens, how is tenant isolation enforced, where do provider keys live, what runs in the background, and what is not included? BoilerShelf's AI SaaS Core is one such starting point, and the SaaS boilerplates category lists the alternatives. Whichever you pick, run it and check its answers yourself.

The idea in short

A request enters through the Next.js frontend, reaches the FastAPI backend with a short-lived access token, is checked against the caller's role and tenant, and goes through an internal LLM gateway that picks the provider, applies quotas and records token usage. Anything slow is queued and handled by a worker, and the result is reported back through a status endpoint or an event.

Originally published at boilershelf.com.