A RAG answer can fail at retrieval or at generation — measure them separately. Score context precision, recall, and faithfulness so you fix the right stage.
Why: a wrong RAG answer is either bad retrieval (the right passage never showed up) or bad generation (it was retrieved but the model ignored or misused it) — one score cannot tell you which. When: always measure retrieval and generation separately so you know which stage to fix. Where: this split is the single most useful idea in RAG evaluation.
RETRIEVAL metrics did we fetch the right context?
context precision -> of retrieved chunks, how many were relevant
context recall -> of relevant chunks, how many we retrieved
GENERATION metrics did we answer well FROM that context?
faithfulness -> is the answer grounded, no hallucinations
answer relevance -> does it actually address the questionWhy: RAGAS packages these RAG-specific metrics so you get numbers instead of hunches — it uses an LLM under the hood but gives you standardized, comparable scores. When: run it over a dataset of question, retrieved contexts, answer, and ground truth. Where: the metrics pinpoint the stage — low faithfulness means fix the prompt; low context recall means fix retrieval.
# pip install ragas datasets
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset
data = Dataset.from_dict({
"question": ["How long do refunds take?"],
"answer": ["Refunds are processed within 30 days."],
"contexts": [["Refunds are available within 30 days of purchase."]],
"ground_truth": ["Within 30 days."],
})
report = evaluate(data, metrics=[faithfulness, answer_relevancy, context_recall])
print(report) # per-metric scores you can track over timeWhy: once you have per-stage scores, a failing case tells you exactly where to intervene instead of blindly tweaking the prompt. When: triage each failure into a retrieval fix or a generation fix using the metric that dropped. Where: this loop — measure, localize, fix, re-measure — is how RAG quality actually improves.
Answer is wrong. Which metric dropped?
low context recall/precision -> RETRIEVAL problem
fix: chunking, embedding model, top-k, reranking, hybrid search
high retrieval BUT low faithfulness -> GENERATION problem
fix: system prompt (ground harder), context format, model
Don't tweak the prompt to fix a retrieval bug. Localize first.