You can’t debug what you can’t see. Trace every LLM call — inputs, outputs, tokens, latency, cost — and monitor production so quality and spend regressions surface fast.
Why: LLM bugs (a bad answer, a cost spike, a slow request) are invisible without a record of what was sent and returned — a trace captures the full context to reproduce and diagnose them. When: instrument every model call from day one; retrofitting observability after an incident is painful. Where: capture prompt version and model version too, so you can pin a regression to a specific change.
Log on EVERY LLM call:
input (prompt + context) output (full response)
model + version prompt version
input/output tokens latency (ms)
cost ($) tool calls made (for agents)
request id / user id timestamp
This is the difference between "it's broken" and "here's the exact call".Why: a tracing tool like Langfuse or LangSmith records each call as a searchable trace so you can inspect, filter, and replay production traffic. When: wrap your model calls with the tracer’s decorator/observer; it captures inputs, outputs, tokens, and latency automatically. Where: traces become the input to production evals — you sample real traffic and score it.
# pip install langfuse
from langfuse.decorators import observe, langfuse_context
import anthropic
client = anthropic.Anthropic()
@observe() # captures this call as a trace
def answer(question: str) -> str:
r = client.messages.create(model="claude-opus-4-8", max_tokens=256,
messages=[{"role": "user", "content": question}])
langfuse_context.update_current_observation(
usage={"input": r.usage.input_tokens, "output": r.usage.output_tokens})
return r.content[0].textWhy: cost and p95 latency drift silently until a bill or a slow page forces attention — dashboards and alerts catch spikes the moment they happen. When: alert on error rate, p95 latency, and spend per feature; watch quality via sampled evals on production traces. Where: attribute cost per user and per feature so you know what to optimize.
Production dashboards + alerts:
error rate alert if it jumps (provider outage, bad deploy)
p95 latency alert on slow tail — users feel p95, not the mean
cost / request alert on spikes (runaway loops, bigger prompts)
quality sampled evals on real traces; alert if score drops
Attribute cost per user + per feature so you optimize the right thing.