Performance Testing RAG Applications: Complete Engineering Guide

In this blog post, we will see how to performance test a RAG (Retrieval-Augmented Generation) application properly, covering both speed and correctness, and how to wire both into a CI/CD pipeline so regressions get caught before they reach production.

Performance testing a RAG application requires two separate testing gates: one for speed and one for answer quality. Traditional load testing tools measure response times but cannot detect hallucinations, where a model returns fast but factually incorrect answers grounded in fabricated context rather than retrieved documents.

The guide demonstrates using k6 for load testing end-to-end latency and DeepEval for evaluating faithfulness and answer relevancy using an LLM-as-judge approach. Both gates are integrated into a GitHub Actions CI/CD pipeline so regressions in either performance or output quality are caught automatically on every pull request before reaching production.

If you've come from a JMeter or k6 background like I have, your first instinct with a RAG endpoint is probably to point a load test at it and check response times. That gets you halfway there. A RAG app can return a fast, confident, completely wrong answer, and a plain load test will never tell you that. You need two testing surfaces, not one: performance and quality. This guide covers both, using a single running example throughout: a documentation assistant that answers "How do I run JMeter in non-GUI mode?" against a small knowledge base.

Why RAG breaks traditional load testing assumptions

A conventional API returns a complete response and you measure the round trip. A RAG endpoint does two expensive things before it answers: it retrieves context from a vector store or search index, then it streams a generated response token by token. That second part matters a lot. A single request can stream hundreds of tokens over several seconds, so "request duration" as a single number hides two very different problems: how long the model took to start answering, and how fast it generated once it started.

A system with slow startup but fast generation feels broken to someone typing in a chat UI. A system with fast startup but slow generation is fine for a quick question but painful for a long document summary. Averaging those together tells you nothing useful.

The two testing surfaces: performance and quality

I think of RAG testing as two separate gates that happen to run against the same endpoint.

Performance answers: how fast is it, and does it hold up under load? This is k6's job, same as any other API load test, just with LLM-specific metrics layered on.

Quality answers: is the answer actually grounded in what got retrieved, or did the model make something up? This is where DeepEval comes in, scoring faithfulness and relevancy on every response using an LLM as the judge.

Neither gate alone tells the full story. A fast RAG app that hallucinates is worse than a slow one that's accurate, and a perfectly grounded app that takes eight seconds to respond will lose users regardless of correctness.

Metrics that actually matter

Performance metrics

Metric	What it tells you
TTFT (Time to First Token)	How long a user stares at a blank screen before anything appears
ITL (Inter-Token Latency)	How smoothly tokens stream once generation starts
Tokens/sec	Generation speed, matters most for long-form answers
p95 / p99 latency	The tail experience, not the average one

TTFT is the most user-visible number in the whole system, and it's also the metric most classic load testing tools weren't built to isolate, since they were designed for atomic request/response cycles, not streams.

Quality metrics

Metric	What it tells you
Faithfulness	Is the answer grounded in the retrieved context, or invented
Answer relevancy	Does the answer address the actual question, or just sound plausible
Context precision	Did retrieval return the right chunks, ranked correctly
Context recall	Did retrieval miss anything the answer needed

These four metrics carry most of the diagnostic weight in RAG evaluation. Faithfulness and answer relevancy live on the generation side; context precision and recall live on the retrieval side. When faithfulness is low but context recall is high, the retriever did its job and the model ignored it that's a prompting problem, not a retrieval problem. Worth knowing the difference before you go tuning the wrong component.

Hallucination detection with DeepEval

I'm using DeepEval here instead of RAGAS mainly because DeepEval treats evaluations as pytest test cases with pass/fail thresholds, which is exactly the shape you need for a CI/CD gate. It also accepts any LLM as the judge model, so it isn't locked to one vendor even though our example app happens to use Gemini.

Here's what a test case looks like against our JMeter doc-assistant example:

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.models import GeminiModel

judge_model = GeminiModel(
    model="gemini-3.5-flash",
    api_key=os.getenv("GEMINI_API_KEY"),
)

faithfulness_metric = FaithfulnessMetric(threshold=0.75, model=judge_model)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.8, model=judge_model)

def test_jmeter_non_gui_mode_answer():
    question = "How do I run JMeter in non-GUI mode?"
    result = query_rag_app(question)

    test_case = LLMTestCase(
        input=question,
        actual_output=result["answer"],
        retrieval_context=result["retrieved_chunks"],
    )

    for metric in [faithfulness_metric, answer_relevancy_metric]:
        metric.measure(test_case)
        status = "PASS" if metric.success else "FAIL"
        print(f"[{status}] {metric.__class__.__name__}: {metric.score:.3f}")

    failed = [m for m in [faithfulness_metric, answer_relevancy_metric] 
              if not m.success]
    if failed:
        names = ", ".join(m.__class__.__name__ for m in failed)
        raise AssertionError(f"Metrics below threshold: {names}")

Run this with pytest, and it either passes or fails like any other test. That's the whole point it turns a fuzzy "does the AI sound right" question into a binary CI/CD signal.

The test suite includes retry logic to handle transient Gemini API 503 errors, automatically retrying up to 3 times with exponential backoff. DeepEval generates both JUnit XML and HTML reports, making it trivial to wire into any CI system that understands pytest output.

Load testing with k6 (and why you can't measure TTFT yet)

Here's where things get frustrating if you came here looking for a clean TTFT measurement story: the k6 SSE extension (xk6-sse) is not compatible with k6 v2. It targets go.k6.io/k6 v1, and until it gets updated, you're stuck choosing between k6 v2's improved architecture or the ability to measure streaming metrics properly.

So the companion repo does the pragmatic thing: it tests the /chat/complete endpoint instead of the /chat streaming endpoint, using k6's built-in http module. No custom binary, no extensions, just standard k6. The tradeoff is you lose true TTFT measurement, because /chat/complete waits for the full response before returning. What you get instead is end-to-end latency, which is still useful it tells you if the system is slow, just not why it's slow.

Here's what the test looks like:

import http from 'k6/http';
import { Trend, Counter } from 'k6/metrics';
import { check } from 'k6';

const totalDuration = new Trend('total_duration_ms', true);
const tokensPerSecond = new Trend('tokens_per_second');

const BASE_URL = __ENV.RAG_APP_URL || 'http://localhost:8080';

export const options = {
  scenarios: {
    rag_chat: {
      executor: 'ramping-vus',
      stages: [
        { duration: '30s', target: 10 },
        { duration: '1m', target: 10 },
        { duration: '30s', target: 0 },
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<6000'],
    total_duration_ms: ['p(95)<6000'],
  },
};

export default function () {
  const startTime = Date.now();

  const res = http.post(
    `${BASE_URL}/chat/complete`,
    JSON.stringify({ query: 'How do I run JMeter in non-GUI mode?' }),
    {
      headers: { 'Content-Type': 'application/json' },
      timeout: '30s',
    },
  );

  const duration = Date.now() - startTime;

  check(res, {
    'status 200': (r) => r.status === 200,
    'has answer': (r) => JSON.parse(r.body).answer !== undefined,
  });

  totalDuration.add(duration);

  // Rough tokens/sec estimate from word count
  const words = JSON.parse(res.body).answer.trim().split(/\s+/).length;
  tokensPerSecond.add((words / duration) * 1000);
}

The test ramps from 0 to 10 virtual users over 30 seconds, holds for a minute, then ramps back down. Thresholds are set at p95 < 6000ms for both http_req_duration and the custom total_duration_ms metric.

When should you switch back to SSE? Watch the xk6-sse repo. Once it adds k6 v2 support, swap the endpoint from /chat/complete to /chat, add the SSE extension to your Dockerfile, and you'll get true TTFT measurement. Until then, this is the most pragmatic path forward standard k6, no custom builds, just with the caveat that you're measuring end-to-end latency rather than streaming behavior.

The companion repo includes both endpoints in the Express app so you can switch when you're ready:

Endpoint	Response	Status
POST /chat	SSE stream	Ready for when xk6-sse supports k6 v2
POST /chat/complete	Full JSON	Used by k6 and DeepEval today

Wiring both gates into CI/CD

Once both tests run locally, wiring them into GitHub Actions is mostly plumbing: start the app, wait for it to be healthy, run the k6 gate, run the DeepEval gate, both in parallel since they're independent.

name: RAG CI

on: [pull_request]

jobs:
  performance-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Write app env file
        run: |
          cat > app/.env << EOF
          GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
          GEMINI_MODEL=gemini-3.5-flash
          FILE_SEARCH_STORE_NAME=${{ secrets.FILE_SEARCH_STORE_NAME }}
          PORT=8080
          EOF

      - name: Start RAG app
        run: docker compose up -d --build app

      - name: Wait for health
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:8080/health; do sleep 2; done'

      - name: Run k6 load test
        run: docker compose --profile perf run --rm k6

  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Write app env file
        run: |
          cat > app/.env << EOF
          GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
          GEMINI_MODEL=gemini-3.5-flash
          FILE_SEARCH_STORE_NAME=${{ secrets.FILE_SEARCH_STORE_NAME }}
          PORT=8080
          EOF

      - name: Start RAG app
        run: docker compose up -d --build app

      - name: Wait for health
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:8080/health; do sleep 2; done'

      - name: Run DeepEval tests
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: docker compose --profile quality run --rm deepeval

Both jobs run on every pull request. A PR that slows down response time and a PR that quietly makes the model hallucinate get caught the same way, before either reaches a reviewer's eyeballs, let alone production.

You'll need to add two secrets to your GitHub repo before the workflow will pass:

Secret	Value
GEMINI_API_KEY	Your Gemini API key from https://aistudio.google.com/apikey
FILE_SEARCH_STORE_NAME	The store name from setup-store.js (format: fileSearchStores/your-store-id)

Setting SLOs

I'm deliberately not giving you one universal latency number to target. I've seen guidance ranging from sub-second targets for chat-style RAG apps to 3-5 second budgets for more complex document analysis, and the right number for you depends entirely on your retrieval backend, your model, and what your users are actually doing. Run the load test against your own baseline first, then set thresholds off that baseline, not off a number from a blog post (including this one).

The example repo uses p95 < 6000ms as a starting point because that's what the test Gemini File Search RAG app achieves at 10 concurrent users with gemini-3.5-flash. Your mileage will vary dramatically based on:

• Model choice (flash vs pro, size of context window actually used)
• Retrieval backend (vector DB query time, number of chunks retrieved)
• Document size and complexity
• Network latency to your LLM provider

What you should track regardless of the exact number:

• p95 and p99 latency, not just the median. The tail experience is what users complain about.
• Latency at your expected concurrency, not at 1 user. RAG apps often degrade non-linearly under load because of retrieval bottlenecks.
• Faithfulness and answer relevancy trending over time, not just pass/fail on one run. A metric that's consistently 0.90 dropping to 0.78 is a signal even if both pass the 0.75 threshold.

Wrap-up

RAG performance testing is really two disciplines wearing one trench coat: classic load testing with LLM-aware metrics, and LLM-as-judge quality scoring that classic load testing tools were never built to do. Run them both, gate on both, and you'll catch the regressions that a speed-only test walks right past.

The current state of tooling isn't perfect you can't measure TTFT with k6 v2 without writing your own SSE client, and LLM-as-judge scoring has its own consistency quirks but it's good enough to catch regressions before production, which is the whole point of a CI/CD gate.

Head to the companion GitHub repo for the full working app, k6 script, DeepEval tests, Docker Compose setup, and GitHub Actions workflow you can clone and run locally in under five minutes.

Happy Testing!

Have you run into hallucination regressions that a pure load test missed? I'd like to hear how you caught them reply on X or open an issue on the companion repo.

Comments

[Alex Shev]

RAG performance testing has to include answer quality under load, not just latency and token cost. A fast pipeline that silently drops retrieval quality is harder to debug than a slow one.