Start with the evals that are fast, free, and never flaky — exact match, keyword checks, regex, and schema validation. Run these first, on every change.
Why: deterministic evals score output with plain code — no second model, no cost, no variance — so they catch obvious regressions instantly. When: run them first and on every change; only escalate to slower judges for what code cannot check. Where: they answer "does it contain the fact / match the format / satisfy the constraint", not "is it well written".
def contains_all(output: str, needles: list[str]) -> bool:
return all(n.lower() in output.lower() for n in needles)
def exact_match(output: str, expected: str) -> bool:
return output.strip().lower() == expected.strip().lower()
# Cheap, instant, deterministic — the first line of defense.
assert contains_all("Refunds are available within 30 days.", ["30 days"])Why: an eval run is just calling your system on every dataset case and scoring the result — the aggregate pass rate is your quality number. When: run this in seconds on every prompt or model change. Where: log which cases failed, not just the total, so you can see exactly what broke.
import anthropic
from eval_dataset import DATASET
client = anthropic.Anthropic()
def run_feature(question: str) -> str:
r = client.messages.create(model="claude-opus-4-8", max_tokens=256,
messages=[{"role": "user", "content": question}])
return r.content[0].text
def run_evals():
passed, failures = 0, []
for case in DATASET:
out = run_feature(case["input"])
if contains_all(out, case["must_include"]):
passed += 1
else:
failures.append((case["input"], out))
print(f"{passed}/{len(DATASET)} passed")
return passed / len(DATASET), failuresWhy: when a feature must return JSON or obey a limit, a schema/constraint check catches malformed output that a keyword match would miss. When: use these for structured-output and tool-calling features. Where: schema validity is deterministic; pair it with the value checks from the structured-outputs lessons.
import json
from pydantic import BaseModel, ValidationError
class Answer(BaseModel):
answer: str
confidence: float
def valid_schema(output: str) -> bool:
try:
Answer(**json.loads(output))
return True
except (ValidationError, json.JSONDecodeError):
return False
def within_length(output: str, max_words=100) -> bool:
return len(output.split()) <= max_words