System Design · Core

Load Balancing

Spread traffic across healthy backends: L4 vs L7, algorithms, health checks, sticky sessions, and where TLS terminates.

1. What a load balancer does

A load balancer (LB) is a reverse proxy that accepts client connections and forwards them to a pool of backend instances. Goals: even utilization, hide node failures, and (at L7) route by path, host, or header.

2. Layer 4 vs Layer 7

Interview default: L7 at the edge for HTTP APIs; L4 when you need raw TCP (databases, some streaming, gaming sockets).

3. Algorithms

Round robin (RR)

Send request N to backend N % pool_size. Fair when backends are equal and requests cost similar CPU.

Weighted round robin

Larger machines get higher weight. Useful during canaries (send 5% to the new build).

Least connections

Prefer the backend with fewest open connections. Better for long-lived or uneven work (uploads, WebSockets, slow APIs).

IP hash / consistent hash

Hash client IP (or a key) onto a backend. Improves cache locality and can approximate stickiness without cookies. Prefer consistent hashing when the pool membership changes often.

Least response time

Prefer the backend with the lowest recent latency (sometimes combined with fewest connections). Good when backends are heterogeneous or some are temporarily slow. Needs the LB to track response-time samples; can oscillate if metrics are noisy — use a short moving average.

Random

Pick a backend uniformly at random (or weighted random). Extremely simple, no shared counter state, and statistically fair at high QPS. Worse than least-conn for long-lived uneven work; fine for short, similar-cost HTTP requests.

4. Health checks

• Active: LB probes backends on an interval.
• Passive: count 5xx / connection errors; eject after a threshold.
• Deep vs shallow: shallow = process up; deep = can reach DB/cache (risk: cascading fail if a dependency blips).

Prefer shallow liveness plus a separate readiness signal. On deploy, mark the instance not-ready, drain connections, then stop.

5. Sticky sessions (session affinity)

Route the same client to the same backend. Use only when you must (legacy in-memory sessions, pinning a WebSocket to a node). Prefer externalizing state so any node works. Stickiness hurts failover and creates hot nodes when one user is heavy.

6. SSL / TLS termination

• Terminate at LB: simpler cert management; LB can inspect HTTP.
• Pass-through: end-to-end encryption; LB stays L4.
• Re-encrypt: terminate then mTLS to backends for zero-trust.

Offload CPU-heavy crypto to the LB fleet; rotate certs with ACME or a cloud certificate manager.

7. Placement patterns

1. Edge LB / API gateway: public entry, auth, rate limit, routing.
2. Internal LB: service-to-service inside the VPC.
3. DNS LB: multiple A records; coarse, client-cached, no instant health.
4. Global LB / anycast: steer users to the nearest region.

8. HA for load balancers

Managed cloud LBs are usually active-active behind the scenes. If you run your own (HAProxy/NGINX), plan VIP failover (Keepalived/VRRP) or put them behind a cloud NLB.

9. LB technologies (interview map)

10. Global load balancing (GSLB)

GSLB steers users to a region or PoP before the regional LB picks an instance. Mechanisms: geo-DNS, latency-based DNS, anycast IP, or a global HTTP front door.

• Pair with multi-region data strategy (active-active vs primary-secondary).
• Say “global entry + regional LB + local pool” in multi-region designs.

11. Failure modes to mention

• LB itself is a SPOF → run active-active pairs or a managed cloud LB.
• Thundering herd when a backend returns → use slow start.
• Health check that depends on a shared DB → all nodes go unhealthy together.
• Connection draining too short → in-flight requests reset.

12. Interview talking points

Draw the LB in front of a pool, name L4 vs L7, pick an algorithm for the workload (least-conn for WebSockets), describe health + drain, and say where TLS ends. If asked about multi-region, add DNS or global LB on top.