When your primary model is down, overloaded, or over a context limit, don’t fail — fall back. Build a chain that degrades gracefully across models and providers.
Why: any single model will occasionally be unavailable, overloaded, or reject a request that is too long — a fallback chain keeps the feature working by trying the next option instead of returning an error. When: use fallbacks for user-facing features where availability matters more than always using the best model. Where: fall back across models AND providers, since an outage often hits a whole provider.
PRIMARY (best/most expensive)
| fails (outage / overload / context limit)
v
SECONDARY (cheaper or different provider)
| fails
v
LAST RESORT (smallest model, or a cached/canned response)
Degrade gracefully. A slightly worse answer beats an error page.Why: iterating a list of (provider, model) candidates and returning the first that succeeds turns a single point of failure into a resilient path. When: order candidates by preference (quality, then cost, then a tiny reliable model) and log which one served the request. Where: keep each candidate behind its own retry layer so a transient blip does not skip a good model.
def generate_with_fallback(messages):
chain = [
("anthropic", "claude-opus-4-8"), # preferred
("anthropic", "claude-haiku-4-5-20251001"), # cheaper, faster
("openai", "gpt-4o-mini"), # different provider = outage-proof
]
last_error = None
for provider, model in chain:
try:
return call_provider(provider, model, messages) # your adapter
except Exception as e:
last_error = e
print(f"{provider}/{model} failed: {e}; trying next")
raise RuntimeError(f"All models failed. Last error: {last_error}")Why: "input too long" is not a transient error a retry fixes — the right fallback is to shrink the input (summarize or trim) or route to a model with a larger context window. When: detect the context-length error specifically and take a different action than for outages. Where: this is why fallback logic lives above the SDK’s retries — retrying an over-limit call never helps.
def safe_generate(messages):
try:
return generate_with_fallback(messages)
except anthropic.BadRequestError as e:
if "context" in str(e).lower() or "too long" in str(e).lower():
trimmed = summarize_old_turns(messages) # shrink, don't retry
return generate_with_fallback(trimmed)
raise