For quality that code can’t measure — helpfulness, tone, faithfulness — use a model as the grader. Write a rubric, force a structured verdict, and manage the variance.
Why: "is this answer helpful, correct, and on-tone?" has no regex — an LLM judge reads the output and scores it against criteria you define. When: use it for open-ended quality after deterministic checks pass; it is slower, costs tokens, and varies run to run. Where: the judge is only as good as its rubric — vague criteria give vague, unreliable scores.
DETERMINISTIC "contains '30 days'?" -> code, instant, exact
LLM-AS-JUDGE "is it helpful and correct?" -> a model grades it
Use the judge ONLY for what code can't measure. It's slower,
costs tokens, and is noisier — so give it a strict rubric.Why: "rate this 1-10" yields meaningless, inconsistent numbers — an explicit rubric with defined levels makes the judge’s scores reproducible and interpretable. When: define each score level concretely and force a structured verdict so you can aggregate. Where: ask for a reason too; it exposes when the judge is confused and helps you debug failures.
JUDGE_RUBRIC = """You are grading a support answer against the ground truth.
Score one of:
PASS - factually correct AND grounded in the reference, no invented details
WARN - correct but missing detail, or slightly off-tone
FAIL - incorrect, contradicts the reference, or hallucinates
Reference: {reference}
Answer: {answer}
Return JSON: {{"verdict": "PASS|WARN|FAIL", "reason": "<one sentence>"}}"""Why: forcing structured output from the judge lets you aggregate verdicts across the dataset; setting temperature to 0 and averaging repeated judgments cuts the noise. When: use a strong model as judge, judge at temperature 0, and re-run flaky cases. Where: track judge disagreement separately — high variance means the rubric, not the feature, needs work.
import json, anthropic
client = anthropic.Anthropic()
def judge(answer: str, reference: str) -> dict:
prompt = JUDGE_RUBRIC.format(reference=reference, answer=answer)
r = client.messages.create(
model="claude-opus-4-8", max_tokens=256,
temperature=0, # low variance
messages=[{"role": "user", "content": prompt}],
)
return json.loads(r.content[0].text)
verdict = judge("Refunds take about 30 days.", "Refunds are within 30 days.")
print(verdict) # {'verdict': 'PASS', 'reason': '...'}