The moment your system spans machines, physics gets a vote. The CAP theorem, consistency choices, and how data is partitioned and replicated across nodes.
Why: in a distributed system a network partition (nodes unable to talk) will happen, and the CAP theorem says that when it does you must choose between consistency (every read sees the latest write) and availability (every request gets a response) — you cannot have both during a partition. When: decide per system whether stale-but-available or consistent-but-unavailable is correct (a bank balance vs. a social feed differ). Where: this is the fundamental trade-off behind every distributed data decision.
CAP: during a network Partition, you must choose Consistency OR Availability.
(Partition tolerance isn't optional in a distributed system — networks fail.)
CP stay CONSISTENT, refuse requests you can't serve correctly
-> e.g. a bank ledger: better unavailable than wrong
AP stay AVAILABLE, serve possibly-stale data, reconcile later
-> e.g. a social feed: better slightly stale than down
Not "which is better" — which is CORRECT for this data. Choose deliberately.Why: "consistency" is not binary — it ranges from strong (every read reflects the latest write, simple to reason about but costly and lower-availability) to eventual (replicas converge over time, high availability and scale, but reads can be stale) — and most large systems pick eventual where they can. When: use strong consistency where correctness demands it (money, inventory counts), eventual where availability and scale matter more. Where: this is the same trade-off as CQRS’s eventual consistency, now at the data-store level.
STRONG ◄─────────────────────────────────────────────► EVENTUAL
every read = latest write replicas converge "eventually"
easy to reason about highly available + scalable
costly, lower availability reads can be briefly stale
Money, stock counts -> lean strong
Feeds, view counts, caches -> eventual is fine (and much cheaper to scale)
Push toward eventual wherever correctness allows — it's what scales.Why: to scale data beyond one machine you partition (shard) it — split it across nodes by a key so each holds a subset — and to survive failures you replicate it — keep copies on multiple nodes. When: shard for scale (choose a key that spreads load evenly and avoids hot spots), replicate for availability and read throughput. Where: these two mechanisms, and the consistency choices around keeping replicas in sync, underlie every distributed database.
PARTITIONING (sharding) — split data for SCALE:
users A-H ─► node 1 users I-P ─► node 2 users Q-Z ─► node 3
(choose a shard key that spreads load evenly; beware "hot" keys)
REPLICATION — copy data for AVAILABILITY + read throughput:
node 1 (primary) ──replicates──► replica ──► replica
(a replica takes over if the primary dies; reads can fan out)
Keeping replicas in sync is where the consistency choice bites.