RAG Evaluation: Metrics and Methods to Measure Retrieval Quality (2026)

Your retrieval-augmented generation app demos beautifully, ships to production, and then starts making things up. The model sounds confident, the UI looks clean, and users quietly lose trust. The uncomfortable truth is that most teams never actually measure whether their RAG system retrieves the right context or grounds its answers in that context. RAG evaluation is the discipline that closes this gap, and in 2026 it is the difference between a demo and a dependable product.

This guide breaks down what RAG evaluation is, the metrics that matter, how to build an evaluation set, and a practical workflow you can run in continuous integration. It sits alongside our broader developer's guide to LLM evaluation, which covers the wider evals landscape.

What is RAG evaluation?

RAG evaluation is the process of scoring how well a retrieval-augmented generation pipeline finds relevant information and uses it to produce accurate, grounded answers. Unlike a plain language model, a RAG system has two moving parts: a retriever that pulls documents from a knowledge base, and a generator that writes an answer conditioned on those documents. A good evaluation strategy measures both parts, because a failure in either one produces a bad response for a different reason.

The reason RAG evaluation gets skipped is that it feels harder than testing normal software. There is no single correct output, answers are free text, and the right retrieved chunk is a matter of degree. But the field has matured. Reference-free metrics, LLM-as-a-judge scoring, and open frameworks like RAGAS and TruLens now make it practical to put a number on retrieval quality and generation quality without hand-labeling thousands of examples.

The two halves you must evaluate separately

The single most common mistake in RAG evaluation is treating the pipeline as one black box. When the final answer is wrong, you cannot tell whether the retriever failed to surface the right passage or the generator ignored a passage it was handed. Diagnosing that requires splitting the pipeline.

Retrieval quality asks a simple question: given the user's query, did the system fetch the chunks that actually contain the answer? Generation quality asks a second question: given those chunks, did the model produce an answer that is faithful to them and relevant to the query? A response can be fluent and completely ungrounded, or perfectly grounded in retrieved text that was never relevant in the first place. You need metrics for each stage to know which lever to pull.

Core RAG evaluation metrics

Retrieval metrics

Retrieval is an information retrieval problem, so many of its metrics predate large language models. The classics still apply.

Context precision measures how much of the retrieved context is actually relevant. If you retrieve five chunks and only one is useful, precision is low and you are feeding the model noise. Context recall measures the opposite: of all the information needed to answer the query, how much did retrieval actually surface. Low recall means the answer simply is not in the context, and no amount of prompt tuning will fix that.

Rank-aware metrics add another dimension. Mean reciprocal rank (MRR) rewards putting the first relevant chunk near the top, and normalized discounted cumulative gain (nDCG) rewards ordering all relevant chunks ahead of irrelevant ones. Hit rate, the fraction of queries where at least one relevant chunk appears in the top k, is the blunt but useful headline number many teams track first.

Generation metrics

Once retrieval is solid, the generation metrics take over. Faithfulness, sometimes called groundedness, measures whether every claim in the answer can be traced back to the retrieved context. This is the metric that catches hallucinations directly: if the model asserts something the context does not support, faithfulness drops. Answer relevance measures whether the response actually addresses the user's question rather than wandering into related but unhelpful territory. Answer correctness, when you have a reference answer, compares the generated response against ground truth for factual accuracy.

Reference-free vs reference-based evaluation

Evaluation methods split into two camps. Reference-based evaluation compares outputs against a curated set of ideal answers. It is precise but expensive, because every query needs a human-written gold answer that ages as your knowledge base changes. Reference-free evaluation uses the query, the retrieved context, and the generated answer to score quality without any gold labels, usually by prompting a strong model to judge the relationship between the three.

Most production teams run a hybrid. They keep a small, carefully labeled reference set for regression testing, and layer reference-free metrics on top so they can score live traffic continuously. Reference-free scoring is what makes it feasible to evaluate thousands of production queries a day without a labeling team.

LLM-as-a-judge for RAG

The engine behind most reference-free RAG metrics is a language model acting as an evaluator. You give a capable model the query, the retrieved passages, and the answer, then ask it to rate faithfulness or context relevance on a defined scale with a rubric. This technique deserves its own treatment, and we cover the pitfalls, prompt design, and bias controls in our piece on using an LLM as a judge.

The short version: judge models are convenient but not neutral. They favor longer answers, prefer their own writing style, and can be inconsistent across runs. Anchoring the judge with a clear rubric, few-shot examples, and a low temperature meaningfully improves agreement with human raters. Treat the judge as a noisy sensor you calibrate, not an oracle.

Building a RAG evaluation set

You cannot evaluate what you cannot represent. A useful RAG evaluation set is a collection of queries paired with the passages that should be retrieved and, ideally, a reference answer. Start with real questions from your logs rather than invented ones, because synthetic queries rarely capture how users actually phrase things.

Aim for coverage across the failure surface: easy factual lookups, multi-hop questions that need several chunks, questions whose answer is genuinely absent from the knowledge base, and adversarial phrasings. That last category matters. A RAG system that confidently answers a question it has no source for is worse than one that says it does not know, and only an evaluation set with unanswerable questions will surface that behavior.

Frameworks can bootstrap this. RAGAS can generate synthetic question-context-answer triples from your documents to seed an evaluation set, which you then prune and correct by hand. That mix of automation plus review gets you a defensible set in hours instead of weeks.

A practical RAG evaluation workflow

Here is a minimal RAGAS-based workflow that scores a batch of RAG outputs on the core metrics. It assumes you have already collected questions, the contexts your retriever returned, and the answers your generator produced.

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import Dataset

# Each row is one evaluated interaction from your RAG pipeline
samples = {
    "question": questions,          # list[str]
    "contexts": retrieved_contexts, # list[list[str]] retrieved chunks per query
    "answer": generated_answers,    # list[str]
    "ground_truth": reference_answers,  # list[str] for reference-based metrics
}

dataset = Dataset.from_dict(samples)

result = evaluate(
    dataset,
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
    ],
)

print(result)
# {'context_precision': 0.82, 'context_recall': 0.74,
#  'faithfulness': 0.91, 'answer_relevancy': 0.88}

Read the numbers as a diagnostic, not a grade. Low context recall points at your retriever, chunking, or embeddings. High recall but low faithfulness points at your prompt or generation model. This separation is exactly why splitting retrieval from generation pays off.

Common RAG failure modes and which metric catches them

Most RAG problems fall into a handful of buckets, and each has a metric that exposes it. Missing context, where the answer is not in the knowledge base or the retriever missed it, shows up as low context recall. Noisy context, where too many irrelevant chunks crowd out the useful one, shows up as low context precision. Hallucination, where the model invents details, shows up as low faithfulness. Off-topic answers, where the model drifts from the question, show up as low answer relevance.

Putting RAG evaluation in production

Offline evaluation on a fixed set catches regressions before you ship, but real knowledge bases and query patterns drift. The mature move is to run reference-free faithfulness and relevance scoring on a sample of live traffic and alert when the aggregate drops. This connects RAG evaluation to the broader practice of LLM observability, where traces, scores, and monitoring come together. Frameworks like TruLens package these reference-free checks for live feedback.

Wire your evaluation set into continuous integration so every prompt change, model swap, or retriever tweak runs the metrics automatically and fails the build on a regression. The same discipline applies when your RAG lives inside a larger agent, where retrieval is one tool among many. Our guide to AI agent evaluation covers how retrieval scoring fits into evaluating tool calls and multi-step traces.

RAG evaluation is not a one-time gate. It is a feedback loop that turns a system you hope works into a system you can prove works. Start with faithfulness and context recall, build a small honest evaluation set, automate it, and expand from there.

For more engineering deep dives, browse the Levelop blog or learn what we are building at Levelop.