System Design · Core

Scaling Fundamentals

Walk a product from one box to millions of users: when to split tiers, add cache, load-balance, replicate, and push static traffic to the edge.

1. Why stages matter in interviews

Interviewers rarely want you to jump straight to Kafka and fifty microservices. They want a growth story: start simple, name the bottleneck, then add the next component. Memorize the six-stage ladder below and narrate it with a concrete product (photo app, marketplace, SaaS API).

2. Stage 1 — Single server

Everything shares one CPU, RAM, disk, and network NIC. Fine for demos and early MVPs. Typical failure modes: disk fills with uploads, DB locks under concurrent writes, or a deploy takes the whole product offline.

• When it breaks: CPU pegged on request handling, or disk I/O on queries.
• Interview tip: Still put the app behind a reverse proxy and use migrations from day one.

3. Stage 2 — Separate database

Web work is often CPU/memory bound; databases are I/O and lock bound. Splitting tiers lets you size and restart them independently. Put the DB on faster disks, tune connection limits, and keep pools on the app side.

4. Stage 3 — Cache

Cache hot, mostly-read objects (profiles, product pages, session blobs) where a few seconds of staleness is acceptable. Prefer cache-aside first; document TTL and invalidation on writes. Full deep dive: Caching.

Cache patterns (quick recap)

• Cache-aside: app reads cache → miss → DB → fill; on write update DB then invalidate.
• Write-through: write hits cache and DB together; reads stay simple.
• Write-behind: write cache first, flush DB async — fast, durability risk.

Cache locations

5. Stage 4 — Multiple web servers + load balancer

Horizontal scale of the compute tier. Health checks remove dead nodes. Critical rule: keep apps stateless. Sticky sessions hide bad design and break when a node dies. Store sessions in Redis or signed tokens so any instance can serve any user.

6. Stage 5 — Database replication

Replicas multiply read QPS and give you a failover candidate. Trade-off: replica lag means a user may not see their own write if you read from a replica immediately. Mitigations: read-your-writes via primary for a short window, or route post-write reads to primary.

7. Stage 6 — CDN

Images, JS, CSS, video segments, and downloads leave your origin. Lower latency for global users and fewer origin bytes. Pair with good Cache-Control and a purge story for broken assets.

8. Vertical vs horizontal scaling

In practice you do both: vertically size each role to a sweet spot, then scale out when one machine cannot hold the write rate or dataset.

9. Sharding strategies

Replicas help reads. Sharding helps when writes or data size exceed one primary.

• Cross-shard joins and multi-shard transactions are expensive — design APIs to avoid them.
• Pick a shard key that is on every hot query path and has high cardinality.
• Plan resharding early: dual-write, migrate, cut over, or use a proxy that remaps.

10. Stateless application tier

1. No local disk as source of truth for user state.
2. Uploads go to object storage, not the app box.
3. Feature flags and config from a central store.
4. Idempotent handlers so retries after failover are safe.

11. Sixty-second interview pitch

Start monolith on one box. Split DB. Cache hot reads. Add LB and stateless apps. Replicate DB for read scale and HA. CDN for static. Shard only when the primary cannot absorb writes or storage. Every step names a bottleneck and a trade-off.

12. Reference latency numbers

Memorize orders of magnitude, not exact nanos. Use them to justify cache, CDN, and “why not sync cross-region on every click.”

13. Reference throughput numbers

Always re-estimate for the problem’s payload size. See also Estimation.