System Design · Core

Replication

Copy data across nodes for reads and durability: leader/follower, sync vs async, lag, failover, and how to avoid split brain.

1. Why replicate

High availability: survive node or AZ loss.
Read scale: serve reads from followers.
Locality: keep a copy near a region (with lag caveats).
Backup / analytics: offline workloads without touching primary.

2. Leader / follower (primary / replica)

Clients writes --> LEADER --replication stream--> FOLLOWER 1 --replication stream--> FOLLOWER 2 reads --> leader and/or followers

Single-leader is the interview default. Multi-leader and leaderless (Dynamo-style) exist; only open that door if the problem needs multi- region active-active writes.

3. Synchronous vs asynchronous vs semi-sync

Mode	Commit when	Pros	Cons
Async	Leader local durable	Low latency, high throughput	Failover may lose last writes
Sync	All (or quorum) replicas ACK	Strong durability	Latency = slowest replica; availability coupling
Semi-sync	At least one replica ACK	Middle ground	Still can lose data if that replica later fails badly

Async: Client <-ACK-- Leader (replica catches up later) Sync: Client <-ACK-- Leader only after Follower ACK Semi-sync: Client <-ACK-- after ≥1 Follower ACK

4. Replication lag

Followers apply changes behind the leader. Lag can be milliseconds or seconds under load. Symptoms: user refreshes and does not see their post; secondary reads disagree.

Monitor lag in seconds and in log bytes.
Read-your-writes: route that user’s reads to leader briefly after write.
Monotonic reads: sticky a user to one replica to avoid time-travel.
Do not run strongly consistent inventory checks on a lagged replica.

5. Failover

1) Detect leader death (leases, heartbeats, health checks) 2) Choose new leader (highest LSN / election) 3) Fence old leader (STONITH / fencing token) so it cannot accept writes 4) Redirect clients (DNS, proxy, service discovery) 5) Rebuild old node as follower

Automated failover is powerful and dangerous. The hard part is fencing: ensuring the old leader cannot write after a network partition.

6. Split brain

Two nodes both believe they are leader and accept writes. Divergent datasets; painful merge. Causes: network partition + weak fencing.

Prevention: - Quorum election (need majority of voters) - Fencing tokens / epoch numbers on every write - STONITH: power off old leader before promoting - Single writer via external lock (careful with lock expiry)

7. Multi-leader and leaderless (brief)

Multi-leader: write locally in each region; conflict resolution (last-write-wins, merge, CRDTs). Higher availability, harder correctness.
Leaderless (quorum R+W>N): client writes to W nodes, reads R; hinted handoff, read repair. Classic Dynamo/Cassandra story.

8. What to draw in an interview

One leader, two followers across AZs.
Async replication with lag callout.
Reads mostly to followers; critical reads to leader.
Failover + fencing one-liner.

Quick revision

Replication: HA, read scale, locality — not a free consistency lunch.
Single-leader is the default mental model.
Async = fast, possible data loss; sync = safe, slower and coupled.
Lag causes stale reads; use read-your-writes when needed.
Failover needs detection, election, fencing, client redirect.
Split brain = two writers; prevent with quorum + fencing.
Multi-leader / leaderless only when multi-region writes demand it.