HLD · Traffic

Load Balancing

Distribute incoming traffic across multiple servers for scale and availability.

1. Why load balancing exists

One server handles limited concurrent connections and CPU. A load balancer (LB) sits in front of a pool of identical app servers and routes each request to a healthy backend.

Example: An e-commerce site during a sale gets 50k RPS. One VM caps at 5k RPS. Ten VMs behind an LB share the load; if one dies, traffic shifts to the rest.

2. Layer 4 vs Layer 7

Interview tip: Use L7 when you need path-based routing (/api → API fleet, /static → CDN). Use L4 for raw TCP, WebSockets at scale, or lowest latency.

3. Load balancing algorithms

• Round robin: Rotate across servers in order. Simple; ignores server load.
• Weighted round robin: Bigger machines get more requests.
• Least connections: Send to server with fewest active connections. Good for long-lived requests.
• IP hash: Same client IP → same server. Sticky sessions without cookies.
• Consistent hash: Minimal reshuffle when nodes added/removed (see Consistent Hashing note).

4. Health checks

LBs probe backends (GET /health) every few seconds. Unhealthy nodes are removed from rotation automatically.

• Active checks: LB sends probe; knows real availability.
• Passive checks: LB marks node bad after N consecutive 5xx/timeouts from real traffic.

Always implement a cheap health endpoint that verifies critical dependencies (DB ping optional — sometimes you want the node up but draining).

5. Session stickiness (affinity)

Stateful apps may require the same user on the same server (in-memory cart). Options:

• LB cookie-based sticky sessions
• IP hash (brittle behind NAT)
• Better: Store session in Redis — any server can serve any user (stateless app tier)

6. Scaling pattern (interview walkthrough)

1. DNS → LB (single entry point)
2. LB → N stateless app servers (auto-scaling group)
3. Shared DB + cache behind apps
4. Scale horizontally until DB becomes bottleneck → read replicas, sharding

7. Failure modes

• LB itself is SPOF: Use managed LB (AWS ALB) or active-passive LB pair (keepalived + floating IP).
• Thundering herd: All traffic hits one node after others fail — use slow start / ramp-up on new nodes.
• Cold nodes: New instances accept traffic before JVM warmup — use readiness probes that fail until warm.