RAG Evaluation

RAG evaluation has to measure two separate stages — retrieval quality (did you fetch the right chunks?) and generation quality (did the model produce a good answer from them?) — a bad final answer can stem from either, and conflating them makes debugging much harder.

Two Distinct Failure Points

A "wrong answer" alone doesn't tell you which stage failed — you need to inspect both independently.

Retrieval Metrics

• Context relevance — are the retrieved chunks actually relevant to the query? (see Context Relevance)
• Recall — did retrieval include the chunk(s) that actually contain the answer, out of everything that could have been retrieved?

Generation Metrics

• Faithfulness — is the answer actually supported by the retrieved context, or does it include unsupported claims? (see Faithfulness)
• Answer relevance — does the answer actually address the question asked? (see Answer Relevance)

A Simple Evaluation Set Structure

eval_set = [
    {
        "question": "What's the refund window for damaged items?",
        "expected_source_chunk_id": "policy-doc-chunk-14",
        "expected_answer_contains": ["14 days"]
    },
    # ... more real, representative questions
]

for item in eval_set:
    retrieved = retrieve(item["question"])
    retrieval_correct = item["expected_source_chunk_id"] in [c.id for c in retrieved]

    answer = generate_answer(item["question"], retrieved)
    faithful = check_faithfulness(answer, retrieved)
    relevant = check_contains_expected(answer, item["expected_answer_contains"])

Practical Use Case

Before shipping any change to a RAG system — new chunking strategy, different embedding model, updated prompt — running it against a maintained evaluation set (covering both retrieval and generation) catches regressions that manual spot-checking routinely misses.

Common Mistakes

• Only evaluating final answer quality, without ever checking whether retrieval itself is working correctly — makes it impossible to tell which stage to fix
• Relying entirely on a handful of manual tests instead of a maintained, representative evaluation set covering real query patterns and edge cases
• Never re-running evaluation after changing any part of the pipeline (chunking, model, prompt), missing regressions

Interview Relevance

"A RAG system gives a wrong answer. How do you determine if it's a retrieval problem or a generation problem?" — inspecting the actual retrieved chunks separately from the final answer is the core diagnostic step; if the right chunks were retrieved but the answer is still wrong, it's a generation/prompt issue, not retrieval.

Practice Question

Design 3 evaluation test cases for a RAG system, each specifying the question, expected source chunk, and a way to check the final answer.

Related Notes

• Faithfulness
• Context Relevance
• LLM Evaluation