RAG Evaluation and Testing

Interview answer (say this first). Evaluation is how you know a RAG system works and keeps working. You keep a labelled offline set built from real user questions plus reviewed synthetic ones, evaluate components separately for attribution and end-to-end for the user outcome, and run the suite in CI with thresholds so a regression fails the build. In production you watch online signals. When something breaks, the metrics tell you whether retrieval or generation is at fault instead of leaving you to guess.

Why this exists

You ship a RAG feature. The demo works. Then, over three weeks, four things happen:

• Someone swaps the embedding model. Answers get subtly worse, and nobody can prove it.
• Someone edits the prompt. The answer style improves and three factual questions now fail.
• The corpus is re-indexed. Recall drops because a parser changed, but the change shipped on a Tuesday.
• A customer asks a question with no answer in the corpus. The model invents one, confidently.

Every one of these is invisible without evaluation. A demo proves that one happy path works once. It says nothing about the next thousand queries, or about the query that changed because a document was reformatted.

The deeper problem is attribution. When an answer is wrong, you have two very different suspects:

• Retrieval did not put the evidence in the context.
• Generation had the evidence and still answered badly.

These have different fixes, different costs, and different owners. Without measurement, teams spend days editing prompts when the real bug is that the retriever never returned the right chunk. Evaluation exists to answer one question fast: which stage failed, and by how much?

Note:

The one-sentence purpose. Evaluation turns “it feels better” into a repeatable number, and splits every failure into a retrieval fault or a generation fault.

Start from zero

Three distinctions do most of the work:

• Offline vs online. Offline is fast and controlled and can be wrong about what users care about. Online is real and slow and noisy. You need both, in that order.
• Component vs end-to-end. Components tell you where a failure is. End-to-end tells you whether users are served. Run both.
• Average vs slice. A single average hides the failures that matter. Always look at the hardest slice.

The core idea

Think about how a backend team tests a web service. They do not click around the UI and call it tested. They write unit tests for each function, integration tests for the wiring, and a smoke test for the real endpoint. CI runs all of it on every commit. Production has dashboards for errors and latency. That is a mature test strategy, and RAG needs exactly the same shape.

RAG evaluation has two loops that connect:

flowchart TD
    subgraph Offline["Offline (every change, in CI)"]
        Q["Eval set<br/>real + synthetic questions"] --> R["Run pipeline"]
        R --> M["Retrieval metrics<br/>+ generation metrics"]
        M --> G{"Gate: did any metric<br/>regress below threshold?"}
        G -->|yes| F["Fail the build<br/>block the merge"]
        G -->|no| P["Merge and deploy"]
    end
    P --> O["Production traffic"]
    O --> S["Online signals<br/>thumbs, escalations,<br/>refusal rate, latency"]
    S --> A["Collect failures<br/>into new eval questions"]
    A --> Q

The arrow from production back into the eval set is what makes the system improve. Every real failure becomes a permanent test, so the same bug cannot return. This is exactly how a good software team turns an incident into a regression test.

Inside the pipeline, measurement points give you attribution. This is the part interviewers care about most:

flowchart LR
    Q["Question"] --> R["Retrieve"]
    R --> RM["Retrieval metrics<br/>Recall@K, NDCG@K"]
    RM --> RR["Rerank"]
    RR --> C["Build context"]
    C --> CM["Context relevance"]
    CM --> G["Generate"]
    G --> GM["Faithfulness,<br/>answer relevance"]
    GM --> V{"Wrong answer?"}
    V -->|recall low| F1["Retrieval fault:<br/>fix index, chunking, query"]
    V -->|recall good,<br/>faithfulness low| F2["Generation fault:<br/>fix prompt, model, context"]

That decision tree is the single most useful thing on this page. Check recall first. It is cheap, it is upstream, and it eliminates a whole class of causes.

How it works

1. Define what “good” means before you build the set. Pick the metrics (recall, NDCG, faithfulness, answer relevance), the thresholds, and the slices. Metrics chosen after seeing results are how teams accidentally measure what they built instead of what they wanted.

2. Collect questions from three sources. Real user questions from logs and support tickets (the most valuable), domain experts writing edge cases, and synthetic questions generated from documents and then human-reviewed. Mix them so the set covers real phrasing and rare-but-important cases.

3. Label each question. Store the relevant document IDs for retrieval metrics and, where possible, a reference answer or a set of acceptable answers. Also mark unanswerable questions explicitly. Keep labels independent of the system you are testing.

4. Stratify the set. Tag each question by intent and difficulty: exact lookup, multi-hop, comparison, acronym, no-answer. Report metrics per slice, not only overall. A set that is 90% easy lookups will look great and predict nothing.

5. Evaluate components. Run the retriever alone on the labelled queries and compute Recall@K and NDCG@K. Run the reranker on the retriever’s candidate set. Run the generator with a fixed context and measure faithfulness. Fixed inputs make each stage’s number meaningful.

6. Evaluate end-to-end. Run the real pipeline, question to answer, and measure both retrieval and generation metrics on the same run. This catches integration bugs that component tests miss.

7. Store results as a baseline. Save the metric values (and the raw outputs) from a known-good run. Every later run is compared to this snapshot, so “regression” has a concrete meaning.

8. Add statistical honesty. A small set gives a noisy mean. Bootstrap your per-query scores to get a confidence interval, and compare two runs on the same questions. A one-point drop on 20 questions is usually noise.

9. Run it in CI with a gate. Re-run the suite on changes to prompts, models, chunking, embeddings, or parsing. Fail the build when a metric drops below its threshold. Keep the fast subset fast so the gate does not block every commit for an hour.

10. Watch online signals after deploy. Thumbs up and down, escalation to a human, follow-up-question rate, refusal rate, latency, and cost. Online is the ground truth the offline set is trying to predict.

11. Do error analysis on a schedule. Read 20–50 real failures by hand. Group them into causes. This is where you find the failure mode no metric was defined for.

12. Feed failures back into the set. Every confirmed bug becomes a new eval case. Version the eval set like code, and note why each case was added.

A useful rule for the gate: gate on the metric, not on the average alone. Check the overall number and the worst slice. A change that raises the average while destroying one slice is not an improvement.

The syntax you will use

One eval case as data. Keep the question, the labels, and the expected answer together. A dataclass is enough; you do not need a framework to start.

from dataclasses import dataclass

@dataclass
class Item:
    question: str
    relevant: set[str]        # labelled relevant document IDs
    answer: str               # the system's answer
    context: str              # the context it was given

A retriever you can swap. The harness calls an interface, not a specific vector store, so you can evaluate any retriever.

def retrieve(question: str, k: int) -> list[str]:
    return FAKE_INDEX[question][:k]      # replace with your real retriever

Retrieval metrics. The functions from the retrieval chapter, reused unchanged.

def recall_at_k(retrieved, relevant, k):
    if not relevant:
        return None                 # unanswerable item: no denominator, skip
    return sum(1 for d in retrieved[:k] if d in relevant) / len(relevant)

def precision_at_k(retrieved, relevant, k):
    return sum(1 for d in retrieved[:k] if d in relevant) / k

A per-query loop that attributes the fault. The order of the checks is the whole trick: retrieval before generation.

def fault_for(recall, faith):
    if recall < 1.0:
        return "retrieval"        # the evidence never arrived
    if faith < 1.0:
        return "generation"       # the evidence arrived, the answer ignored it
    return "ok"

The evaluation loop. It calls the retrieval metrics from the retrieval chapter, the faithfulness function from the generation chapter, and records a fault label on every row.

def evaluate(items, k):
    rows = []
    for item in items:
        retrieved = retrieve(item.question, k)
        faith, _ = faithfulness(item.answer, item.context)
        if not item.relevant:
            # Unanswerable item: retrieval metrics have no denominator. Skip them
            # (the guarded functions from chapter 17 return None) and score abstention separately.
            rows.append({
                "question": item.question,
                "recall": None,
                "precision": None,
                "mrr": None,
                "ndcg": None,
                "faithfulness": faith,
                "fault": "unanswerable",
            })
            continue
        recall = recall_at_k(retrieved, item.relevant, k)
        rows.append({
            "question": item.question,
            "recall": recall,
            "precision": precision_at_k(retrieved, item.relevant, k),
            "mrr": reciprocal_rank(retrieved, item.relevant),
            "ndcg": ndcg_at_k(retrieved, {d: 1 for d in item.relevant}, k),
            "faithfulness": faith,
            "fault": fault_for(recall, faith),
        })
    return rows

Bootstrap confidence interval. Resample the per-query scores with replacement and read the 2.5th and 97.5th percentiles.

import random
from statistics import mean

def bootstrap_ci(scores, n=2000, seed=0, alpha=0.05):
    rng = random.Random(seed)
    means = sorted(mean(rng.choices(scores, k=len(scores))) for _ in range(n))
    return means[int(alpha / 2 * n)], means[int((1 - alpha / 2) * n) - 1]

seed makes the run reproducible, which a CI gate needs. rng.choices(..., k=len(scores)) samples with replacement, which is what “pretend we collected a new set like this” means.

A CI gate. Express it as an assertion so pytest and the build pipeline understand it.

THRESHOLDS = {"recall": 0.9, "faithfulness": 0.9}

def gate(summary):
    for metric, threshold in THRESHOLDS.items():
        assert summary[metric] >= threshold, f"{metric}={summary[metric]:.3f} < {threshold}"

A pytest wrapper. Mark the full suite slow and run the fast subset on every commit.

def test_retrieval_gate():
    rows = evaluate(GOLDEN, k=4)
    # Skip None from unanswerable items so a zero-relevant case cannot crash the gate.
    summary = {m: mean(r[m] for r in rows if r[m] is not None) for m in THRESHOLDS}
    gate(summary)

A synthetic-question prompt. Generate from a document, then review. Never label synthetic data with the model that produced it.

Read the document below. Write 3 questions a real user might ask that
this document answers, and 1 question it does NOT answer.
For each, list the sentence(s) that support the answer.

Document: {chunk}

Ask for the supporting sentences so a reviewer can check the label cheaply, and ask for one unanswerable question so the set covers abstention.

Examples: simple to real

Example 1 — the smallest useful check.

One query, one labelled document, one metric:

question: "refund window"
relevant: {d1}
retrieved: [d1, d9, d3, d8]
Recall@4 = 1/1 = 1.0

This already catches the worst bug: the document is not found at all. It takes ten lines of code.

Example 2 — grow it into a dataset and a mean.

Four questions with labels, run together. The harness prints one line per query:

refund window  recall=1.00 faith=0.50 ndcg=0.92 -> generation
trial length   recall=1.00 faith=1.00 ndcg=0.63 -> ok
support hours  recall=0.50 faith=1.00 ndcg=0.61 -> retrieval
cancel policy  recall=1.00 faith=1.00 ndcg=1.00 -> ok

Now failures have names. The refund answer invented a shipping refund the context explicitly denied; that is a generation fault. The support answer missed a relevant document; that is a retrieval fault. Two different tickets, two different fixes.

Example 3 — averaging without hedging is a trap.

The same four rows give these means:

recall       mean=0.875  95% CI=[0.625, 1.000]
precision    mean=0.312  95% CI=[0.250, 0.438]
mrr          mean=0.875  95% CI=[0.625, 1.000]
ndcg         mean=0.791  95% CI=[0.622, 0.960]
faithfulness mean=0.875  95% CI=[0.625, 1.000]

The recall mean is 0.875, but the confidence interval spans 0.625 to 1.000. On four questions, that interval is so wide it barely rules anything out. This is the honest way to present a small eval set: the mean, plus how uncertain it is. A dashboard that shows 0.875 with no interval invites teams to over-react to noise.

Example 4 — the CI gate catches the regression.

With thresholds of 0.9 for recall and faithfulness, the gate raises on the first metric that fails:

GATE FAIL: recall=0.875 < 0.9

The build fails, the merge is blocked, and the pull request must either fix the regression or justify lowering the threshold in review. Faithfulness is also below 0.9 in this run, so a gate that reports every failure rather than stopping at the first gives a more complete picture. Either way, the threshold is a decision made in advance, not a story told afterwards.

Example 5 — a gate that is honest about noise.

A better gate compares the new run to the stored baseline and only fails on a drop that is larger than the interval:

def regressed(baseline_scores, new_scores, tolerance=0.0):
    base_lo, _ = bootstrap_ci(baseline_scores)
    new_mean = mean(new_scores)
    return new_mean < base_lo - tolerance

This fails only when the new mean falls below the baseline’s lower bound. It is slower to react to small true improvements and much less likely to fail the build on noise, which matters because a gate that cries wolf gets ignored.

Example 6 — error analysis in one table.

Read the raw outputs, not just the numbers, and group them. A simple log does the job:

Two failures, two owners, two fixes. Without the fault column, both look like “the AI is wrong”.

In production

• Start the eval set before you need it. The best time to label 30 questions is while building the feature, when you still remember what users ask. Retrofitting an eval set is slow and political.
• Draw questions from real traffic. Synthetic questions inherit the phrasing of the source chunk and are too easy. Mix in real queries with typos, ambiguity, and multi-hop structure.
• Always include unanswerable questions. They test abstention, which is the behaviour users trust most and which naive faithfulness metrics punish. Score refusals as their own category, and skip retrieval metrics for them (there is no relevant document to recall).
• Gate on slices as well as the average. A change that improves easy lookups and destroys multi-hop questions can still raise the overall mean. Check the worst slice.
• Make the fast gate fast. A ten-minute eval on every commit gets bypassed. Split into a small fast suite for every change and a full suite nightly or before release.
• Version the eval set and the baseline together. A metric is meaningless without the set it was computed on. When you add cases, recompute the baseline in the same commit.
• Resist tuning on the test set. If you change the pipeline until the held-out set improves, it is no longer held out. Keep a private, rarely used set for the final number.
• Expect the set to drift. Question styles and documents change. Audit labels quarterly, and retire cases that no longer reflect the product.
• Do not let one metric stand in for all quality. Recall, precision, faithfulness, and relevance disagree on purpose. Report a small panel, not one number.
• Log enough to reproduce failures. Save the question, retrieved IDs, scores, prompt version, model version, and the final answer. Without the retrieved IDs you cannot attribute the fault later.
• Watch cost and latency as first-class metrics. A retrieval change that raises recall by three points and doubles latency may still be a regression. Gate on p95 latency too.
• Treat online signals as noisy but real. A thumbs-down rate can move for reasons unrelated to quality. Use it to find candidates for error analysis, not as a precise score.

Interview questions

1. Where do evaluation questions come from?

Answer. Three sources. Real user questions from logs and support tickets, which are the most valuable because they capture real phrasing. Expert-written questions for edge cases the logs do not reach yet. And synthetic questions generated from documents, then human-reviewed, to bootstrap coverage quickly. Mix them, and label the relevant documents for each.

Follow-up: “What is wrong with purely synthetic questions?” They are generated from a chunk, so they often reuse the chunk’s wording and are easier than real questions. They also miss the messy phrasing, typos, and multi-hop structure that users actually produce. They are useful for coverage, not for realism.

Trap. Generating questions and labels with the same model you are evaluating. The model will agree with itself, the score will look great, and you will have measured nothing.

2. What is the difference between component and end-to-end evaluation?

Answer. Component evaluation tests one stage in isolation: the retriever against labelled relevant documents, the reranker against the candidate set, the generator against a fixed context. It gives fast, attributable feedback. End-to-end evaluation runs the whole pipeline and measures the user-facing outcome. Component scores tell you where the problem is; end-to-end tells you whether the system works. You need both.

Follow-up: “Why not just do end-to-end?” Because a bad end-to-end number does not tell you which of four stages broke. Without component metrics you debug by guessing. Component metrics narrow the search to one stage.

Trap. Testing components with hand-picked clean inputs and declaring victory, then being surprised when the glued-together pipeline fails on real data. Component tests must use realistic inputs.

3. How do you decide whether retrieval or generation is at fault?

Answer. Check retrieval first. If Recall@candidate_K is low, the evidence never reached the model, so it is a retrieval fault: fix indexing, chunking, query transformation, or the candidate K. If recall is good but faithfulness is low, the model had the evidence and went beyond it, so it is a generation fault: fix the prompt, the model, or the context construction. If both are good and users still complain, the problem is usually relevance or UX, not correctness.

Follow-up: “What if recall is good and faithfulness is good but the answer is still wrong?” Then either the retrieved context itself is wrong or outdated, or the question needs reasoning across documents rather than extraction. That points at the corpus and at the agent’s multi-step reasoning, not at the retriever.

Trap. Jumping straight to prompt edits. Prompt changes are slow to validate and often fix a retrieval problem at the wrong layer.

4. How do you set up a CI gate without constant false alarms?

Answer. Compare each run to a stored baseline on the same questions, and include a statistical margin. Bootstrap the per-query scores and fail only when the new mean drops below the baseline’s lower confidence bound. Threshold the overall metric and the worst slice, keep the fast suite small enough to run on every commit, and pin the judge and dataset versions so the comparison is fair.

Follow-up: “What makes a gate untrustworthy?” Flaky metrics, a changing eval set, a silent model upgrade, or a threshold no one believes. Once a team learns to ignore red builds, the gate is worse than no gate.

Trap. Gating on a single absolute number with no baseline. The number drifts with the dataset and the judge, so the gate either never fires or fires constantly.

5. Offline versus online evaluation: what does each give you?

Answer. Offline is fast, cheap, reproducible, and safe; it is how you catch regressions before release. Online is the real distribution, real phrasing, and real consequences; it is how you learn what offline missed. Offline cannot tell you if users are satisfied, and online is too slow and noisy to catch a subtle regression before release. Use offline to decide whether to ship, and online to confirm that shipping helped.

Follow-up: “How do you connect them?” Feed production failures back into the offline set, and use offline metrics to predict online signals. If offline says a change is better but online signals do not move, your eval set is measuring the wrong thing.

Trap. Treating a thumbs-up rate as a precise metric. It is noisy, biased toward vocal users, and driven by factors beyond answer quality.

6. How many evaluation examples do you need?

Answer. It depends on the metric and the size of the effect you care about. More examples per metric than you think: a mean over 20 questions can swing several points from noise alone. Compute a bootstrap confidence interval and size the set so the interval is narrow enough to detect the change you care about. For per-slice decisions you need examples within each slice, not only overall.

Follow-up: “What is the cheapest way to get more examples?” Harvest real queries and review labels in batches, and add every confirmed production failure as a permanent case. Both grow the set while keeping it realistic.

Trap. Reporting a mean with no uncertainty and treating a one-point difference as real. On a small set, that difference is usually noise.

7. How do you evaluate abstention and unanswerable questions?

Answer. Include questions whose answer is not in the corpus and label them as unanswerable. The correct behaviour is a clear refusal or a statement that the documents do not cover it. Score refusals as their own category, and separately measure over-refusal (refusing answerable questions) and under-refusal (inventing answers to unanswerable ones). Those two errors have very different costs.

Follow-up: “Why do naive metrics mishandle refusals?” Faithfulness and answer-relevance scores reward producing a grounded answer, so a correct refusal can look like a low score. Without a refusal category, you punish the safest behaviour.

Trap. Only testing answerable questions. That trains the system to always answer, which is exactly the failure mode users lose trust over.

8. How do you keep the evaluation set from going stale or being overfit?

Answer. Version it like code, audit labels on a schedule, and retire cases that no longer match the product. Feed in fresh real queries regularly. Keep a private, rarely used held-out set for the final number, and never tune on it. Add every confirmed production failure as a new case. When the eval set changes, recompute the baseline in the same change.

Follow-up: “What is the sign that the set is overfit?” Offline metrics keep improving while online signals stay flat. That means the system is learning the quirks of the set rather than getting better at the task.

Trap. Tuning until the test set improves, then calling that number a held-out result. It is now a training metric, however much you liked the old name.

Remember this

• Build the eval set from real questions, add reviewed synthetic ones for coverage, label unanswerable cases, and version the whole set.
• Components for attribution, end-to-end for truth. Check recall first, then faithfulness: that one split names the failing stage.
• Gate in CI with a baseline and a confidence interval, not a bare average, so the gate catches real regressions instead of noise.
• Offline decides whether to ship; online confirms whether it helped. Feed production failures back into the set.
• Read the failures. Error analysis finds the bug no metric was written for.