Evaluating your RAG pipeline

For a long time I never put evals on any RAG project I shipped. That worked fine when the corpus was small and the pipeline was a single retriever plus a prompt. It stopped working once I landed on systems with huge document sets and dozens of tools. Debugging those in production is painful. Tools fire out of order. You are not sure why the chunks ranked the way they did, and even when the right chunks come back you are not sure why the model ignored them or invented something that was never in context.

A lot of companies are building around RAG right now, so that failure mode is common. In this post I'll cover how to do error analysis on real traces, build a golden dataset, pick metrics that match how the system actually breaks, and run them in DeepEval, the framework I used for the evals.

Example

We'll keep coming back to this one across the post, so the metrics have something concrete to land on.

Take a RAG pipeline over company policy.

A user asks: How many annual leave days do I get, and can I carry them over?

The retriever returns these four chunks:

1. The office is closed on public holidays listed in Appendix C.

2. Full-time employees are entitled to 20 days of annual leave per calendar year.

3. Leave requests must be submitted through Workday at least two weeks in advance.

4. Unused annual leave of up to 5 days may be carried into the following year.

Only chunks 2 and 4 help answer the question. Chunks 1 and 3 look relevant. That's what makes retrieval hard: they're close in meaning, but they don't actually answer what was asked.

The model then answers: Full-time employees get 20 days of annual leave a year. You can carry up to 5 unused days into next year. Leave requests are usually approved within 24 hours.

The answer looks fine on a first read, until you hit the last sentence. The 24-hour approval line doesn't appear in any of the four chunks, so the model invented it. The second problem is harder to spot: the actual policy also says carried days expire on 31 March, but that fact never made it into the retrieved context, so the answer never mentions it. One is a generation failure and one is a retrieval failure, and fixing one does nothing for the other.

Evaluate retrieval and generation separately

Before looking at metrics, split the problem in half.

Every RAG pipeline has two jobs: find information, and generate an answer. Those fail for completely different reasons, so they need different metrics.

The retriever can bring back the wrong chunks, rank the right chunk too low, or miss a necessary fact entirely. The generator can ignore useful context, answer only half the question, or add claims the context never supported. If you only look at the final answer you know something went wrong but not where, and that's how you end up tuning prompts when the problem was retrieval.

The same bad answer, two different fixes.

Start with error analysis

Before you wire up any of the metrics below, look at real failures. The easiest mistake is to jump straight to off-the-shelf scores like faithfulness or toxicity and build a dashboard around them. Those numbers only help if they match how your product actually breaks.

Hamel Husain and Shreya Shankar walk through this properly on Lenny's Podcast and in their Lenny's Newsletter write-up. The same ideas show up across Hamel's blog and their Maven series. The short version: evals are not unit tests, and you should not start by writing tests. You start with error analysis on real traces.

Sample production traces and read them yourself. For each one, write down the first thing that went wrong. Don't create categories yet. Don't ask an LLM to do this. Just read failures. A model reading the trace will often say it looks fine, because it doesn't know your domain (for example, that a property doesn't actually offer virtual tours). Aim for around 100 traces, or stop earlier once new traces stop teaching you anything. Put one trusted domain expert on it (often the PM), not a committee.

Once you have the notes, then bring in AI. Dump them into Claude or ChatGPT, cluster them into failure categories, edit those categories by hand until they're specific enough to act on, re-label against that set, and count them. Only then decide what deserves an automated evaluator. Plenty of failures don't: a formatting bug is usually a missing line in the prompt, so fix it instead of instrumenting it. Prefer code-based checks where you can. Reach for an LLM-as-judge only for the subjective failures that keep showing up, keep each judge binary (pass/fail, not a 1–5 scale), and align it against your human labels before you trust the score.

The leave-policy example above is a simplified version of that diagnosis: one invented claim, one missing fact. Error analysis tells you which patterns dominate. The next step is turning that into something you can score.

The golden dataset

Once you've read the failures and know what breaks, you want a concrete way to test the pieces of the pipeline, not just the final answer. That means metrics on retrieval and generation separately. Those metrics need something to score against. That something is the golden dataset: a source of truth for what a correct reply should contain, so you can compare it to what your RAG pipeline actually produced.

You can't score contextual recall without that answer key. Nothing in the retrieved chunks tells you a fact is missing. So you build the golden dataset before you wire up the metrics.

Three fields. Two of them you have to write.

Each row has the question, a reference answer, and the reference context that contains those facts. Contextual precision and contextual recall need those reference fields. Faithfulness, answer relevancy and contextual relevancy can run on live traffic without them, so the writing cost is really for the retrieval metrics.

Put that effort into hard rows: multi-part questions, cross-document answers, near-duplicates, and "I don't know" cases. I'd rather have 75 of those than 1,000 easy ones. An LLM can draft candidates, but don't trust the scores until someone who knows the domain has checked every row.

Retrieval metrics

Retrieval comes first, because the generator can't recover a fact it never saw. Walk the metrics on the example above using the golden row.

Contextual precision

Contextual precision means how early the useful chunks appear in the ranking. The same four chunks can produce very different answers depending on ordering.

Formally, it is the mean of precision@k over every rank k where the chunk at that rank is relevant:

Only the relevant ranks contribute to the average.

Order matters. If the right chunk sits at rank 4 of 4, it's competing with three irrelevant chunks for space and attention, and if you later trim context to save tokens it can drop out completely.

In this example the relevant chunks are at positions 2 and 4. At position 2 you've seen two chunks and one is relevant, so Precision@2 = 1/2 = 0.50. At position 4 you've seen four and two are relevant, so Precision@4 = 2/4 = 0.50. Average those and contextual precision is 0.50.

If a reranker had moved both useful chunks to positions 1 and 2, you'd get Precision@1 = 1/1 = 1.00 and Precision@2 = 2/2 = 1.00, so 1.00 off the same four chunks in a different order.

Reordering the same four chunks doubles the score.

That's the metric to watch when you're deciding whether a reranker earns the latency it adds.

Contextual relevancy

Contextual relevancy means what fraction of the retrieved context is actually useful for the query. Unlike contextual precision, it does not care about ranking order.

No ranking in it, just the share that helped.

In this example two of the four chunks are relevant, so the score is 0.50. Half the prompt budget went to noise.

If answers are still correct and this score is low, the usual causes are top-k set too high, chunks that are too large, or context you're paying for and don't need.

Contextual recall

Contextual recall means what fraction of the information needed to answer the question actually showed up in the retrieved context. This is the metric that needs the golden dataset. Take the reference answer from the row above and check what fraction of its claims are attributable to the retrieved context:

The denominator comes from the golden row, not retrieval.

The reference answer breaks into three claims: full-time employees get 20 days, up to 5 unused days can be carried over, and carried days expire on 31 March. Checking those against the retrieved context, the first is in chunk 2, the second is in chunk 4, and the third is nowhere. Retrieval found 2 of 3 required claims, so contextual recall is 0.67.

Only the golden row exposes the missing fact.

Without the golden row, there is no way to show that retrieval missed the 31 March rule.

Generation metrics

Once retrieval has done its job, the remaining question is what the model did with the context it got. In the leave-policy example the retriever missed the 31 March rule, but it also invented a 24-hour approval line that none of the chunks supported. Faithfulness and answer relevancy catch that half.

Faithfulness

Faithfulness means whether claims in the generated answer are consistent with the retrieved context. In DeepEval, the metric extracts atomic claims from the answer, then checks each one against retrieval_context. A claim fails only if it contradicts that context.

Contradiction is the bar, not support.

Split the generated answer into claims: employees get 20 days a year, you can carry up to 5 unused days into next year, and leave requests are usually approved within 24 hours.

The first two are supported by chunks 2 and 4. The third isn't supported by anything, but no chunk says anything about approval times either, so it doesn't contradict the context. All three pass, and faithfulness is 1.00.

This caught me off guard the first time I read the docs. Models don't usually contradict retrieved text outright. More often they add a plausible extra sentence about something the chunks never covered. DeepEval's faithfulness metric will not fail that case. If catching invented-but-non-contradictory claims is what you care about, you need a different check: DeepEval's hallucination metric (which expects a curated context you treat as ground truth), or a custom G-Eval / DAG criterion that requires every claim to be supported, not merely non-contradictory.

For live RAG traffic, faithfulness still uses the retrieval_context actually fetched at runtime, which is usually the right input. Pointing hallucination at whatever retrieval happened to return treats noisy retrieval as truth, so keep those two metrics separate.

Answer relevancy

Answer relevancy means what fraction of the statements in the generated answer actually address the question. DeepEval does this at the claim level.

Relevance to the question, not truth.

The question had two parts: how many leave days, and whether they can be carried over. The first statement answers part one, the second answers part two, and the third about approval times answers neither. So answer relevancy is 0.67.

It doesn't check truth. A completely fabricated answer can score well here as long as it responds to the question, so it's only worth reading next to faithfulness (and whatever support check you add for unsupported claims).

Reading the scores together

Read them as a set, not a headline.

Look at the scores as a set, not one by one. Contextual precision at 0.50 says the useful chunks ranked poorly. Contextual relevancy at 0.50 says half the context was noise. Contextual recall at 0.67 says retrieval missed a required fact. All three point upstream, so chunking, top-k and reranking are where I'd look before touching the prompt.

Faithfulness at 1.00 does not mean the answer is clean. The model still invented the 24-hour approval line. That claim just doesn't contradict the retrieved chunks, so the contradiction test lets it through. Pair faithfulness with answer relevancy (0.67 here, because that third sentence answers nothing) or with a stricter support check if invented claims are the failure mode you care about.

Each score names one thing to change.

If you collapse all of that into one headline score, you lose the diagnosis. You'd know something moved without knowing which half to fix.

DeepEval

I use DeepEval because it behaves like a normal test framework. If you're already writing unit tests, the workflow feels familiar. That's probably its biggest advantage. It's Apache-2.0, covers the metrics above, and uses an LLM as judge for most of them. You define test cases, call assert_test, and run deepeval test run as part of CI. Each metric has a default threshold of 0.5, so it drops straight into a pass/fail gate.

That fits once you know what you're measuring. While you're still doing error analysis and changing retrieval settings, scored samples you can sort and read matter more than a red build. Once the system is stable and you want to stop regressions landing, DeepEval's CI path is the useful one.

Two details are worth watching. The faithfulness definition above is one: a claim fails only when it contradicts the retrieved context, not when it is merely unsupported. The other is hallucination vs faithfulness. Hallucination expects a curated context you supply as trusted ground truth. Faithfulness uses the retrieval_context actually fetched at runtime. For live RAG, faithfulness is usually the right default input. Don't point hallucination at noisy retrieval and treat that as truth.

Check score direction before you gate on anything. DeepEval's hallucination metric is lower-is-better. Most of the others are higher-is-better.

Custom metrics

At some point you'll want to measure something specific to your product rather than generic RAG quality. In DeepEval the two options are G-Eval and DAG.

G-Eval is the flexible one. You describe a criterion in natural language, the model derives evaluation steps, then scores the case. That works for things like tone or response style, and it's also how I'd encode a stricter "every claim must be supported" check if DeepEval faithfulness is too permissive for your failure mode. The docs are clear that it isn't deterministic, so if you need more stable behaviour, pass explicit evaluation_steps rather than free-text criteria alone.

DAG is the controlled one. You define a decision tree of judgments and each path ends in a fixed score, so the same input takes the same path and gets the same score. That's the one I'd use when the rule can be written down clearly.

Limits of LLM-as-judge

These numbers are another LLM giving an opinion. Useful for catching regressions. Not ground truth.

On standard judge benchmarks (MT-Bench), strong models agree with human preference judgments more than 80% of the time, which is about the level of agreement humans reach with each other. Judge models still have documented biases. They can prefer the first answer they see, reward verbosity, and prefer outputs from their own model family. That last one matters if the same model family writes the answer and grades it. There's also a limit on how closely the scores track humans at all: the G-Eval paper reports a Spearman correlation of 0.514 with human judgments on summarization.

So I'd treat these as a regression system rather than a report card. What matters is whether a score moved after a change, not the absolute value.

Where I'd start

If there's one thing I'd recommend, don't start by adding metrics. Start by reading failures. After about fifty bad traces you'll know exactly which metrics matter. Everything else becomes much easier after that.

The order I'd go in:

• Do error analysis on real traces first. Read ~50 to 100 of them, write freeform notes, cluster the failure modes, then decide what is worth automating.

• Build the golden dataset next: 50 to 100 hard questions with reference answers and reference context, covering the awkward edges error analysis surfaced.

• Wire the metrics into DeepEval on that set. Start with contextual recall and contextual precision, because those are the ones that need the golden rows.

• Add faithfulness and answer relevancy, including on live production traffic where you don't need a reference.

• Keep retrieval and generation metrics separate, so you know which half to fix.

• Read the exact DeepEval definition of every metric before you set CI thresholds. Faithfulness is contradiction-based; don't assume it means "fully supported".

• Use a different model for judging than the one generating, where you can.

That's enough for a useful eval loop. You don't need a complete benchmark before you start.

References I used

• Lenny's Podcast: Why AI evals are the hottest new skill for product builders (Hamel Husain & Shreya Shankar): https://youtu.be/BsWxPI9UM4c

• Building eval systems that improve your AI product (Lenny's Newsletter): https://www.lennysnewsletter.com/p/building-eval-systems-that-improve

• AI Product Engineering (Maven series, Hamel Husain & Shreya Shankar): https://maven.com/lls/21f487

• Hamel Husain's Blog: https://hamel.dev/

• Beyond Basic RAG: Similarity ≠ Relevance (Rodrigo Nader): https://medium.com/logspace/beyond-basic-rag-similarity-relevance-fdbd16373d4a

• DeepEval docs: https://deepeval.com/docs/metrics-introduction

• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment: https://arxiv.org/abs/2303.16634

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685

Originally published in Depth First, my LinkedIn newsletter. Read it on LinkedIn.