Most LLM bills and slow responses are avoidable. Route to the cheapest model that’s good enough, cache repeated work, and stream so the app feels fast regardless.
Why: cost and latency come down to using a smaller model when you can, not re-doing work you have already done, and hiding unavoidable latency with streaming. When: apply all three; they compound. Where: the single most common overspend is sending every request to the biggest model when most would be handled fine by a small one.
1. RIGHT-SIZE the model cheapest one that passes your evals per task
2. CACHE skip repeated/identical work (prompt + response)
3. STREAM same total time, feels far faster to the user
Measure with evals + traces first. Optimize the calls that actually cost.Why: not every request needs the frontier model — routing simple tasks to a small, cheap model and reserving the big one for hard cases can cut spend dramatically without hurting quality. When: classify the request (cheaply) and pick the model to match. Where: validate the routing with your eval suite so "good enough" is measured, not assumed.
def choose_model(task: str) -> str:
# A cheap heuristic (or a tiny classifier) decides the tier.
if task in ("classify", "extract", "short_answer"):
return "claude-haiku-4-5-20251001" # small, fast, cheap
return "claude-opus-4-8" # reserve the big model
def generate(task: str, messages):
return client.messages.create(
model=choose_model(task), max_tokens=512, messages=messages)Why: identical or near-identical prompts recur constantly — an exact-match cache skips the call entirely, and provider prompt caching cuts the cost of a big reused prefix. When: cache deterministic results (temperature 0) by a hash of the request; use prompt caching for stable system prompts and documents. Where: an exact cache is your cheapest optimization — a cache hit costs nothing and returns instantly.
import hashlib, json
cache: dict[str, str] = {}
def cache_key(model, messages):
return hashlib.sha256(json.dumps([model, messages]).encode()).hexdigest()
def generate_cached(model, messages):
key = cache_key(model, messages)
if key in cache: # identical request seen before
return cache[key] # zero cost, zero latency
out = client.messages.create(model=model, max_tokens=512,
temperature=0, messages=messages).content[0].text
cache[key] = out
return out