Distributed Systems What I Wish I Knew Earlier
Key concepts every engineer should understand before building distributed systems: CAP theorem, consensus, and failure modes.
The Fallacies
Every distributed system eventually teaches you the eight fallacies:
- The network is reliable
- Latency is zero
- Bandwidth is infinite
- The network is secure
- Topology doesn’t change
- There is one administrator
- Transport cost is zero
- The network is homogeneous
CAP Theorem in Practice
You can’t have all three: Consistency, Availability, Partition tolerance. In practice, partitions happen so you’re choosing between CP and AP.
- CP systems: ZooKeeper, etcd, Consul strong consistency, may reject writes during partitions
- AP systems: Cassandra, DynamoDB always available, eventually consistent
Consensus is Hard
Paxos is correct but incomprehensible. Raft is understandable but still tricky to implement. Use existing implementations (etcd, CockroachDB) unless you have a very good reason not to.
Key Takeaway
Design for failure. Every network call can fail, timeout, or return stale data. Build idempotent operations. Use circuit breakers. Log everything.