Run LLM apps in production — retries and backoff, rate limiting, fallback model chains, cost and latency optimization, prompt versioning, and release and incident practices. Every lesson is runnable Python you can drop into a real service.
Before you start
You will write production-grade Python around a model API. You need Python 3.10+ and an API key from the Anthropic Console. A couple of lessons touch Redis for rate limiting. The patterns — retries, fallbacks, versioning — are provider-agnostic and apply to any model API.
Retries & Backoff
Model APIs fail transiently — rate limits, timeouts, 5xx. Retry the failures worth retrying, with exponential backoff and jitter, and give up on the ones that won’t recover.
Rate Limiting
You have limits from the provider and limits you impose on users. Stay under the provider’s token-per-minute ceiling and cap per-user usage so one caller can’t drain your quota.
Fallback Model Chains
When your primary model is down, overloaded, or over a context limit, don’t fail — fall back. Build a chain that degrades gracefully across models and providers.
Cost & Latency Optimization
Most LLM bills and slow responses are avoidable. Route to the cheapest model that’s good enough, cache repeated work, and stream so the app feels fast regardless.
Prompt Versioning
A prompt is production logic — treat it like code, not a hardcoded string. Version prompts, record which version served each request, and roll changes out safely.
Shipping & Incident Response
LLM apps fail in ways normal apps don’t — hallucination spikes, cost blowouts, provider outages. Gate releases on evals and keep a runbook for when things go wrong.