Distributed systems fail partially and constantly. Design so failure is contained, not catastrophic — and so you can actually see what’s happening in production.
Why: in a distributed system something is always failing — a node, a network link, a dependency — so resilience means designing so a partial failure stays contained instead of cascading into an outage. When: assume every remote call can fail or hang, and build in redundancy (no single point of failure), timeouts, and graceful degradation. Where: the mindset shift is from "prevent failure" (impossible) to "survive failure" (achievable).
You can't prevent failure in a distributed system — design to SURVIVE it:
[ ] no single point of failure (redundancy at every tier)
[ ] timeouts on every remote call (never wait forever)
[ ] graceful degradation (serve reduced function, not an error page)
[ ] bulkheads (isolate failures so one bad dependency can't sink everything)
[ ] tested backups + a disaster-recovery plan (RTO / RPO defined)
"Everything fails all the time" — architect for it.Why: a single slow dependency can drag down the whole system as callers pile up waiting — so you contain it with retries + backoff (for transient blips), a circuit breaker (stop calling a failing dependency and fail fast, giving it time to recover), and backpressure (shed or queue load rather than collapse). When: wrap every cross-service call in these patterns. Where: naive retries without a circuit breaker actually amplify an outage, so they go together.
Contain a failing dependency so it can't take everything down:
RETRY + BACKOFF retry transient failures, with jitter (not instantly,
not forever) — but ONLY behind a circuit breaker
CIRCUIT BREAKER after N failures, "open" -> fail fast without calling the
dead dependency; probe periodically to "close" again
BACKPRESSURE shed or queue excess load instead of collapsing
Retries WITHOUT a breaker amplify an outage (a thundering herd). Use both.Why: a distributed system is opaque without observability — the three pillars of metrics (what is happening, aggregated), logs (discrete events for detail), and traces (one request’s path across services) — because when something breaks at 3am, this telemetry is the difference between a fast fix and a blind guess. When: build observability in from the start, not after the first incident. Where: distributed tracing is especially critical in microservices, where one user request touches many services.
The three pillars of observability (build them in from day one):
METRICS aggregated numbers over time (latency, error rate, throughput)
-> dashboards + alerts (see the Monitoring / Prometheus course)
LOGS discrete, detailed events -> the "why" behind a metric
TRACES one request's path ACROSS services -> find the slow/failing hop
In microservices, distributed TRACING is essential — a single request fans
out across many services. You can't operate what you can't see.