A RAG Pre-Production Checklist with Real Configuration

Most RAG launch checklists say "evaluate retrieval" and "monitor in production". That is true and hard to act on. This one is different. Every item maps to a setting you actually change, in the order you would change it.

It assumes you already know why retrieval and generation should be scored separately. Here we only wire that idea into a release process. I run this checklist in EvaliQA, so the setting names below come from its connectors, test plans and online evaluation.

1. Build a small golden dataset first

Your release gate needs a set of rows you trust and never change casually. A row belongs in it only if it has four things.

• A specific, canonical input, in the exact phrasing a user would write.
• An expected outcome or a rubric. For RAG you want a gold expected_output, because the retrieval recall metric needs it.
• Metadata that survives without you, at minimum a category, a difficulty and a source such as hand, production, ticket or incident.
• A one-sentence reason to exist, for example "Refund flow for a damaged item, order id in message body."

Size it on purpose. For a new project that means 10 to 50 rows, written by someone who knows the domain. Start with canonical happy paths for every documented business scenario. Then add out-of-scope and forbidden-topic rows for every restriction. Skip hand-writing robustness or attack variants, because those are better generated into a separate dataset.

Two traps are worth avoiding from day one. Do not bulk-generate rows and call the result golden. And never delete a row with source = incident because the bug is fixed. That row is your alarm for the day a prompt tweak or model update brings the bug back.

2. Add deterministic checks as a structural floor

Before you pay for any judge calls, add rule checks. They are free, run in milliseconds and give the same verdict every time. Each one scores 1 or 0, so the default threshold of 0.5 simply means the rule held.

A sensible floor for a RAG assistant looks like this.

• Non-Empty on every plan. A timeout or a tool-only turn can return nothing, and this catches it. It has no parameters.
• JSON Schema if your service returns structured output. Set required, additionalProperties: false and enum constraints, so you catch a wrong shape and not just broken JSON.
• Contains for required text, such as a compliance disclaimer. In all mode it scores the share of keywords found, so set the threshold to 1.0 when every keyword must appear. Otherwise 3 of 6 keywords passes at the default 0.5.
• Contains in none mode as a cheap forbidden-phrase guardrail. Matching is by substring, so the keyword cat also matches "category".
• Length Check with min_length, max_length and unit (chars or words) to catch verbosity regressions after a model update.

If your RAG answer must cite a source id in a fixed format, a Regex Match with full_match turned off will do. For a forbidden pattern, the docs suggest a negative lookahead with full_match on, for example (?s)^(?!.*\d{16}).*$, which passes only when no 16-digit run appears.

Test any pattern on real outputs before you commit to it. A \d+ pattern passes any response with any digit in it.

3. Set metric thresholds deliberately

Now add the judge-based metrics. For a RAG assistant the starting set in the docs is three metrics.

• Answer Precision, default threshold 0.8. It is the strict correctness check against expected_output.
• Faithfulness, default threshold 0.7. It checks that every claim is supported by retrieval_context.
• Contextual Recall, default threshold 0.7. It checks whether retrieval brought back what the gold answer needs.

Together they separate "retrieval missed something" from "retrieval was fine and generation drifted". If recall is low, no prompt work will fix the answer. If recall is high and precision is low, look at generation.

Raise the Faithfulness threshold to 0.9 or 0.95 if you work in a medical, legal or financial setting, where any ungrounded claim is a bug. Do not attach all six RAG metrics at once. The wizard allows it, but the pass-rate signal turns into noise. Add Contextual Relevancy, Contextual Precision or Answer Relevancy once your first numbers show a specific gap.

Contextual Precision has a top_k parameter. If you retrieve 20 chunks but feed only 5 to the model, cap it at 5 so the metric reflects what the model actually saw. It also needs ranked context, so keep the order in your dataset.

One more setting matters more than it looks. On your own key, pin the judge model and do not rotate it between runs you want to compare. The judge is part of the measurement instrument. On platform credits it is kept fixed for you.

4. Connect your RAG endpoint and map the output

A connector tells the platform how to call your system and where the answer sits in the response. Create it on the project's Connectors tab. You can import the request from cURL, HAR or Postman, and then replace the captured question in the body with {{input}}. If you skip that step, every row sends the same text and the scores mean nothing.

The default body is a good starting point.

{
  "input": "{{input}}"
}

Mark every credential header as Secret, so the value is stored encrypted. Under Advanced, timeout defaults to 60 seconds and max retries to 1. Retries apply to every failed row in a run, so keep them low for slow or paid endpoints. If your service starts a job and returns a result later, pick Job, then polling and point the output paths at the finished status response.

The important part for RAG is the mapping. Say your service returns this.

{
  "answer": "You can return unused items within 30 days.",
  "sources": [
    { "id": "kb-112", "text": "Unused items can be returned within 30 days of delivery." },
    { "id": "kb-087", "text": "Refunds go back to the original payment method." }
  ],
  "usage": { "prompt_tokens": 812, "completion_tokens": 41, "total_tokens": 853 },
  "model": "gpt-4o-mini"
}

Then the paths are the following.

Field	Path
Actual output path	answer
Retrieval context path	sources[*].text
Token usage path	leave empty
Model path	leave empty

Use sources[*].text and not sources. Whole objects would reach the judge as JSON with ids in it. Faithfulness and all three Contextual metrics need the retrieval context path. Without it they have nothing to score.

Run Test connection and then Run test in Configure output. Activate unlocks once the latest test passed and the Actual output path is set. Saving the request sheet again deactivates the connector until you test and activate it once more, so the last test always matches what runs will send.

5. Run the plan from CI

Create an API token on the Integrations page and store it in your CI secrets as EVALIX_TOKEN. The same page generates ready-made steps for GitHub Actions, GitLab CI/CD, Jenkins, CircleCI, Bitbucket Pipelines, Azure Pipelines and plain curl. Under the hood it is one call.

POST https://<your EvaliQA host>/api/test-plan/triggers/eval-run
X-Evalix-Token: <token>
Content-Type: application/json

{
  "test_plan_id": "…",
  "dataset_id": "…",
  "connector_id": "…"
}

Optional fields are column_binding (auto-matched by column name by default), name and concurrency (1 to 20, default 1).

Be clear about what this call does. It returns 202 Accepted as soon as the run is queued. It does not wait for results and does not fail your build on its own. You read the scores on the plan's Eval runs tab and compare with the previous run before you merge. Plan your process around that.

A few error codes are worth handling in your pipeline.

• 409 means a run is already in progress on this dataset.
• 400 covers cases such as a plan with no metrics or no judge, or BINDING_UNRESOLVED when a required column cannot be matched.
• 5xx is temporary, so retry the call.

Keep the PR plan small. The docs recommend a golden plan of 20 to 50 rows that finishes in under a minute. A 30-row gate that runs in 40 seconds gets used every time, while a gate that takes minutes gets bypassed on urgent PRs.

Do not gate on a razor-thin pass-rate difference either. A drop of a few percentage points can be judge noise, so choose a threshold that leaves room for it.

For nightly runs, call the same endpoint from a scheduler you already have. EvaliQA has no scheduler of its own, so use a GitHub Actions schedule trigger, GitLab pipeline schedules, Jenkins or a Kubernetes CronJob. A nightly run against the golden set catches drift you did not cause, such as provider model updates or retrieval index changes.

6. Turn on online evaluation

Offline runs only test what you thought of. Production traffic shows the rest. Send traces with the SDK or OpenTelemetry using a tracing API key from the project's Tracing tab, and make sure each trace carries a session_id. A trace without one is stored but never scored.

Then open the project's online evaluation Settings tab. The Runtime auto-eval panel has three switches.

• Auto-evaluate sessions is the master switch. If it is off, nothing else runs.
• Deterministic detectors are free checks for errors, retry loops, redundant tool calls and cost or duration outliers.
• AI Analyst is a strong model that reads the whole session. It is one call per session, so leave it off until someone will actually read its findings.

Next, bind metrics. For each binding you set a metric, a threshold from 0 to 1 and a trace field mapping. The defaults map input from trace.input and actual_output from trace.output. For RAG metrics you add a mapping for retrieval context, using a dotted path or JSONPath into the trace, usually into metadata or spans.

An optional Rule scope keeps the metric from scoring health checks or greetings. Each binding is one judge call per matching trace, so bind the metrics you would act on. A reasonable pair for RAG is Faithfulness and Answer Relevancy.

Scoring uses the same evaluated-cases allowance as a test run, and it runs through your workspace's Platform AI agent, either on credits or on your own key. Changes apply only to traces that arrive afterwards, and history is not re-scored.

7. Attach alerts to the bindings

An alert rule can only watch a metric the project actually scores, so bind first. Then do three things.

1. Create a channel. Slack, Discord, Teams, Telegram, email and a generic webhook are supported. Use the Test action and check the Last test column before you trust it.
2. Create a rule on the project. For RAG, a metric pass-rate rule is a good start, named by its meaning, for example answer-relevancy < 70%.
3. Set the throttle. The default of 15 minutes turns a bad deploy into one message instead of forty.

Rules are checked once per session, 60 seconds after its last trace, so a long conversation is analysed once and not per turn. If a failing metric points to exactly one trace, the deep link opens that trace, and otherwise it opens the session. Afterwards, the History tab shows per-channel delivery, so you can find a webhook that silently broke.

8. Close the loop

The last item is not a setting, it is a habit. Once a week, review a handful of low-scoring or thumbs-down traces. If one represents a class of interaction the product must handle, use Add to dataset to promote it to the golden set. Fill in the category, difficulty, source = production and a note. Then write the expected output yourself, because traces come without one.

Every real incident becomes a permanent row too. Once a quarter, prune rows for retired features and merge near-duplicates.

The checklist in one place

• Golden set of 10 to 50 reviewed rows with expected outputs and metadata.
• Non-Empty plus the structural checks your output format needs.
• Answer Precision, Faithfulness and Contextual Recall with explicit thresholds, and a pinned judge.
• Connector with {{input}} in the body, secret headers and a retrieval context path.
• CI step calling the trigger endpoint, a small plan, and a person who reads the run.
• Nightly run on a scheduler you already own.
• Online evaluation on, metrics bound, rule scope set.
• Alert channel tested, rule created, throttle chosen.
• Weekly trace promotion into the golden set.