Before you build a RAG PoC: a 10-point checklist

Smart Choice — Thu, 29 Oct 2026 12:57:39 +0000

Most RAG proofs of concept don't fail because the model is bad. They fail because nobody agreed what "good" means, the documents were messier than expected, or the demo worked on ten hand-picked questions and fell apart on the eleventh.

A RAG PoC is cheap to start and easy to misjudge. Below is the checklist I go through with a team before any code is written. None of it is exotic. All of it saves weeks.

First, what RAG actually is

RAG (retrieval-augmented generation) is a way to make a language model answer from your documents instead of only from what it learned during training. It works in two steps:

Retrieve. The user's question is used to search a knowledge base (usually a vector index built from your documents split into chunks, often combined with keyword search) and pull back the few most relevant passages.
Generate. Those passages are put into the prompt together with the question, and the LLM writes an answer grounded in them, ideally with citations to the sources.

The model itself is not retrained. You get answers that reflect current, private data, can point to where they came from, and can be updated just by re-indexing documents. The flip side: the answer can only be as good as what the retrieval step finds, which is why most of the checklist below is about data and retrieval, not about the model.

1. Name the process and its current cost

"We want a chatbot over our docs" is not a goal. "Support agents spend ~6 minutes per ticket finding the right policy paragraph, on ~1,500 tickets a month" is. With a baseline you can later say whether the PoC moved anything, and whether it's worth paying for in production.

Also ask the uncomfortable question early: does this need an LLM at all? Sometimes a good search index with filters solves 80% of the problem for a fraction of the cost.

2. Inventory the data: where it lives, who owns it, how fresh it is

List every source: Confluence spaces, PDFs on a shared drive, a ticketing system, a product database. For each one, write down:

format (born-digital PDF, scanned PDF, HTML, tables, slides),
owner and access permissions,
update frequency,
whether it may leave your infrastructure.

Scanned PDFs and slide decks are where projects quietly lose a week. If 30% of the knowledge is in tables inside scanned PDFs, your parsing step is the project, and you want to know that on day one.

3. Check that the PoC sample is representative

It's tempting to start with the cleanest folder. Don't. Pick a sample that includes the ugly cases: outdated versions of the same document, contradictory policies, multilingual content, very long files. A PoC that only sees clean data tells you very little about production.

4. Decide on a chunking strategy, then test it, not just pick it

Chunk size and boundaries change retrieval quality more than most model choices do. Reasonable starting points:

split on document structure (headings, sections) before falling back to token windows,
keep chunks around 300–800 tokens with a small overlap,
store metadata with every chunk: source, section title, date, access level.

Metadata isn't decoration. It lets you filter ("only current policies", "only documents this user can see") and cite sources in answers, which users trust far more than bare text.

Treat chunking as a parameter you'll compare on the eval set (point 5), not a one-off decision.

5. Build an evaluation set before the pipeline

This is the single most skipped step and the most valuable one. Sit down with domain experts and collect 50–150 real questions, each with:

the expected answer (or key facts it must contain),
the document(s) and section(s) where the answer lives,
a few questions that cannot be answered from the corpus.

The unanswerable ones matter: you want to measure whether the system says "I don't know" instead of inventing something.

A plain JSONL file is enough:

{"id": "q017", "question": "How many days do customers have to return a damaged item?", "relevant_ids": ["returns-policy-2025#sec-3"], "must_contain": ["30 days"]}
{"id": "q018", "question": "What is the CEO's home address?", "relevant_ids": [], "must_contain": [], "expect_refusal": true}

6. Measure retrieval separately from generation

When an answer is wrong, you need to know whether the right chunk was never retrieved, or it was retrieved and the model ignored it. Those are different fixes. So measure retrieval on its own, with simple metrics like recall@k and MRR:

import json

def load_eval(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

def retrieval_metrics(eval_set, retrieve, k=5):
    hits, rr_sum, n = 0, 0.0, 0
    for item in eval_set:
        relevant = set(item["relevant_ids"])
        if not relevant:
            continue  # unanswerable questions are scored elsewhere
        n += 1
        retrieved = [doc_id for doc_id, _score in retrieve(item["question"], k=k)]
        if relevant & set(retrieved):
            hits += 1
        for rank, doc_id in enumerate(retrieved, start=1):
            if doc_id in relevant:
                rr_sum += 1 / rank
                break
    return {"recall@k": hits / n, "mrr": rr_sum / n, "k": k, "n": n}

# metrics = retrieval_metrics(load_eval("eval.jsonl"), my_retriever.search, k=5)

Twenty lines of code, and now "let's try hybrid search" or "let's try smaller chunks" becomes a measurable experiment instead of a debate. For generation, start with simple checks (does the answer contain the required facts, does it cite a retrieved source, does it refuse when it should) before reaching for LLM-as-a-judge.

7. Set latency and cost budgets up front

Write down two numbers before building: acceptable response time (e.g. p95 under 4 seconds for an internal assistant) and acceptable cost per query. Then estimate:

tokens per query = system prompt + retrieved chunks (k × chunk size) + question + answer,
queries per month from the baseline in point 1.

Retrieving 10 chunks of 800 tokens means ~8k input tokens per question before the model says anything. Multiply by monthly volume and the price list of the model you're considering. This often changes the design: fewer, better chunks, a reranker, or a smaller model for simple questions.

8. Agree on guardrails and data boundaries

Questions to settle with security and legal before the PoC, not after the demo:

What data may be sent to an external model API, and under which terms? Or does everything have to stay in your own cloud project?
How are document permissions enforced in retrieval? (Filter by access-level metadata before the LLM sees anything; never rely on the prompt to hide content.)
How do you handle prompt injection hidden inside documents?
Who reviews outputs in high-stakes cases, and who is accountable for a wrong answer?

9. Define success criteria and the decision that follows

Write them down, with numbers, and agree on them with whoever signs off the next budget:

retrieval recall@5 ≥ 0.85 on the eval set,
≥ 80% of answers contain the required facts,
≥ 90% correct refusals on unanswerable questions,
p95 latency and cost per query within budget,
a time saving measured with real users on real tickets.

Equally important: what happens next? "If we hit these numbers, we fund a production rollout for team X; if not, we stop or narrow the scope." A PoC without a pre-agreed decision tends to turn into an endless demo.

10. Sketch the path to production now

A PoC notebook and a production service are different animals. Before starting, have rough answers to:

Ingestion: how do new and updated documents get re-parsed, re-chunked and re-embedded? A scheduled pipeline (Airflow, Cloud Scheduler, a queue) beats a manual script.
Monitoring: log questions, retrieved chunk IDs, answers and user feedback, so the eval set can grow from real traffic.
Regression testing: re-run the eval set on every change to prompts, chunking, models or embeddings.
Ownership: who maintains it when the PoC team moves on?

You don't need to build any of this during the PoC. You need to know it's feasible, so the PoC architecture doesn't have to be thrown away.

A quick self-test

If you can't answer at least half of these points for your use case, the best next step is probably not a PoC but a short discovery phase: collect questions, look at the data, agree on metrics. It's a few days of work that can save a few months.

I keep a shorter, business-oriented version of this list, 10 questions before starting an AI PoC, for conversations with non-technical stakeholders.

What would you add? I'm especially curious how others build and maintain eval sets once the system is live.

Bolesław Badziąg has spent 20+ years building IT products and runs Smart Choice, helping teams take AI, data and cloud projects from audit and prototype to production.

DEV Community: Smart Choice