Model APIs fail transiently — rate limits, timeouts, 5xx. Retry the failures worth retrying, with exponential backoff and jitter, and give up on the ones that won’t recover.
Why: some errors are transient (429 rate limit, 500/503, network timeout) and succeed on a second try; others are permanent (400 bad request, 401 auth) and retrying just wastes time and money. When: retry only the recoverable classes; fail fast on the rest. Where: retrying a 400 in a loop is a common, expensive mistake — it will never succeed.
RETRY (transient) DON'T RETRY (permanent)
------------------------ ------------------------------
429 rate limit 400 bad request (fix the call)
500 / 502 / 503 server 401 / 403 auth (fix the key)
connection timeout 422 validation (fix the input)
network reset context length exceeded (trim first)Why: retrying immediately hammers an already-struggling API — backing off exponentially (1s, 2s, 4s…) gives it room, and random jitter stops many clients retrying in lockstep. When: cap the number of attempts and the maximum delay so a request cannot hang forever. Where: jitter matters most under load, exactly when everyone is retrying at once.
import time, random, anthropic
client = anthropic.Anthropic()
def call_with_retry(messages, max_attempts=5):
for attempt in range(max_attempts):
try:
return client.messages.create(
model="claude-opus-4-8", max_tokens=512, messages=messages)
except (anthropic.RateLimitError, anthropic.InternalServerError,
anthropic.APITimeoutError) as e:
if attempt == max_attempts - 1:
raise
delay = min(2 ** attempt, 30) + random.uniform(0, 1) # backoff + jitter
print(f"transient error ({e}); retrying in {delay:.1f}s")
time.sleep(delay)Why: a 429 often includes a Retry-After header telling you exactly when to try again — honoring it beats guessing. When: use the header if present, otherwise fall back to backoff; bound total retry time so a request cannot block a user indefinitely. Where: the official SDKs already retry a few times, so tune max_retries rather than double-wrapping them.
# The SDK has built-in retries — configure rather than duplicate them.
client = anthropic.Anthropic(max_retries=4, timeout=30.0)
# It reads Retry-After on 429s and backs off automatically. Add your OWN
# retry layer only for logic the SDK doesn't cover (e.g. fallback models,
# below) — and keep a total time budget so requests can't hang forever.