Turn your evals into a gate. Run them automatically on every prompt or model change, fail the build below a threshold, and show the quality delta in the PR.
Why: evals only prevent regressions if they run automatically — a suite you have to remember to run is a suite you will forget. When: trigger the eval suite on every change to prompts, models, or retrieval, exactly like unit tests. Where: run cheap deterministic evals on every commit; run the pricier judge/RAGAS evals on PRs and releases.
# tests/test_quality.py — runs in CI like any other test.
from evals import run_evals
def test_quality_above_threshold():
score, failures = run_evals()
assert score >= 0.90, f"Quality {score:.2%} below 90% baseline: {failures}"Why: gating the deploy on the eval suite means a quality regression blocks the merge instead of reaching users. When: fail the build if the score drops below the baseline, and surface the before/after delta in the PR so reviewers see the impact. Where: keep an API key in CI secrets; budget the run so evals do not cost more than they save.
# .github/workflows/evals.yml (sketch)
# on: pull_request
# - run: pip install -r requirements.txt
# - run: pytest tests/test_quality.py # fails below the threshold
# - run: python scripts/eval_diff.py # comment the delta on the PR
pytest tests/test_quality.py -qWhy: LLM-as-judge evals vary run to run, so a single dip is not necessarily a regression — treat deterministic failures as hard stops and judge variance as a signal to investigate, not to block blindly. When: average judge scores over a few runs and flag high variance separately. Where: a noisy metric usually means the rubric needs tightening, not that the feature regressed.
DETERMINISTIC eval fails -> hard stop, block the merge (it's real)
LLM-JUDGE score dips -> could be variance
- average over N runs before deciding
- flag high variance separately from a true drop
- persistent drop across runs = real regression
Track prompt version in every trace so a drop maps to a specific change.