RAG evaluation has to score retrieval and generation separately.
Retrieval is scored on whether the right chunks were found.
Generation is scored on faithfulness to those chunks and answer quality.
A wrong answer might be a retrieval failure or a generation failure.
Measuring only the final answer leaves you unable to tell which to fix.
A RAG system has two parts that fail separately and must be measured separately.
Retrieval decides what the model sees, and generation decides what it does with it.
A wrong answer might mean the right chunk was never found.
Or the right chunk was found and the model ignored or misread it.
Those need entirely different fixes, and the final answer alone cannot tell you which.
Score retrieval with recall at k against questions with known correct chunks.
That set has to be built by hand, and a hundred questions is enough to be useful.
Retrieval recall is a hard ceiling on the whole system.
If the right chunk is present only 70 per cent of the time, no prompt raises you above that.
Score generation on faithfulness, checking each claim against the retrieved text.
Score it on relevance too, since a faithful answer can still dodge the question.
Frameworks such as RAGAS package these metrics, usually using a model as judge.
Test the case where the answer is genuinely absent, and check the system says so.
That single test catches more real problems than any other.
Measure end to end as well, because the parts can each look fine while the whole disappoints.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.