Measure and monitor LLM apps instead of guessing. Build a golden dataset, run deterministic and LLM-as-judge evals, score RAG with RAGAS, trace requests for observability, and gate every deploy on a regression suite. Every lesson is runnable Python.
Before you start
You will write evals in Python. You need Python 3.10+ and an API key from the Anthropic Console (used for the system under test and the LLM judge). Later lessons install ragas and langfuse as they go. The concepts are provider-agnostic.
Why Evaluate? Building a Golden Dataset
You cannot improve what you cannot measure — and “it looks good” is not a measurement. Build the golden dataset that turns prompt changes from guesswork into a number.
Deterministic Evals
Start with the evals that are fast, free, and never flaky — exact match, keyword checks, regex, and schema validation. Run these first, on every change.
LLM-as-Judge
For quality that code can’t measure — helpfulness, tone, faithfulness — use a model as the grader. Write a rubric, force a structured verdict, and manage the variance.
Evaluating RAG with RAGAS
A RAG answer can fail at retrieval or at generation — measure them separately. Score context precision, recall, and faithfulness so you fix the right stage.
Tracing & Observability
You can’t debug what you can’t see. Trace every LLM call — inputs, outputs, tokens, latency, cost — and monitor production so quality and spend regressions surface fast.
Regression Testing in CI
Turn your evals into a gate. Run them automatically on every prompt or model change, fail the build below a threshold, and show the quality delta in the PR.