LLM apps fail in ways normal apps don’t — hallucination spikes, cost blowouts, provider outages. Gate releases on evals and keep a runbook for when things go wrong.
Why: LLM behavior changes with a prompt tweak, a model update, or a provider-side change you did not make — gating deploys on the eval suite is what stops a silent quality regression from reaching users. When: block the release if evals drop below baseline, and re-run them when the provider ships a new model version. Where: this ties the whole course together — evals, versioning, and traces make releases safe.
Release checklist for an LLM feature:
[ ] Eval suite passes above baseline (block deploy otherwise)
[ ] Prompts versioned; the active version is recorded
[ ] Retries, rate limits, and a fallback chain are in place
[ ] Tracing on: cost, latency, tokens, prompt version per call
[ ] Alerts wired for error rate, p95 latency, and spend
[ ] Rollback is one step (flip the prompt/model version)Why: the failure modes of LLM apps are specific — a hallucination spike, a cost blowout, a provider outage — and having a predefined response turns a scramble into a procedure. When: write the runbook before the incident; during one, follow it. Where: each failure has a fast mitigation (revert, cap, fail over) you can trigger without redeploying.
SYMPTOM FIRST MOVE
-------------------- ----------------------------------------
quality/hallucination revert to last-good prompt/model version
cost spike check for a loop; lower max_tokens; cap/kill
latency spike shed load, drop to a faster model
provider outage fail over to the fallback chain / provider
429 storm tighten rate limits; back off; queue
Predefine these. Trigger them without a redeploy.Why: the same models you deploy also help you build and operate — tools like Claude Code and Cursor write, review, and debug the retry logic, evals, and runbooks in this course. When: use them to scaffold instrumentation and tests fast, but review their output with the same evals and gates you apply to everything else. Where: they are an accelerator, not a substitute for the observability and release discipline above.
AI coding tools (Claude Code, Cursor) fit LLMOps well:
- scaffold retry/backoff, rate limiters, and eval harnesses quickly
- draft incident runbooks and CI eval workflows
- explain a confusing trace or a provider error
Still gate their output on your evals and reviews. Same rules apply.
(See the Claude Code course for a deep dive.)