HLD · Full design

News Feed / Timeline — End-to-End Design

Classic interview problem: design Twitter/Instagram home feed from scratch.

1. Requirements clarification

Functional

Users post tweets (text, optional media)
Users follow other users
Home timeline: recent posts from people I follow, reverse chronological
User profile shows their posts

Non-functional (estimate scale)

300M DAU, 500M posts/day (~6k writes/sec avg, higher peaks)
Read:write ≈ 100:1 → heavy read optimization
Latency: feed load < 200ms p99
Eventual consistency OK (few seconds delay for new post in feed)

2. High-level architecture

Clients → API Gateway / LB
Post Service — create post, store metadata
Social Graph Service — follow/unfollow relationships
Feed Service — assemble timeline
Media Service — upload to object storage (S3), CDN serve
Notification Service — async fan-out on new post (optional)

3. Data model (core tables)

users(user_id, name, ...) posts(post_id, user_id, content, created_at, ...) follows(follower_id, followee_id, created_at) # fan-out table (see below) feed(user_id, post_id, rank/score, created_at)

4. Fan-out: push vs pull vs hybrid

Pull (read-time merge)

On feed request: get followee list → fetch recent posts from each → merge sort.

Pro: Simple writes. Con: Slow for users following 5000 accounts (celebrity read problem).

Push (write-time fan-out)

On new post: write post_id into every follower's precomputed feed cache (Redis/Cassandra).

Pro: Fast reads. Con: Slow write for celebrity with 50M followers (fan-out storm).

Hybrid (production approach)

Normal users (<10k followers): Push to followers' feed cache on post
Celebrities: Skip push; merge celebrity posts at read time for followers

5. Feed generation flow (push path)

User posts → Post Service writes to posts DB
Publish PostCreated event to Kafka
Fan-out workers consume event, fetch follower list (cached)
Batch insert post_id into each follower's feed store (Redis sorted set by timestamp)
Follower opens app → Feed Service reads top N from Redis → hydrate post details (batch get from posts DB)

6. Storage choices

Data	Store	Why
Posts metadata	SQL or Cassandra (sharded by post_id)	Durability, range queries for profile
Social graph	Graph DB or adjacency lists in Cassandra	Fast follower/followee lookups
Precomputed feed	Redis sorted sets	Sub-ms reads, trim to last 1000 items
Media	S3 + CDN	Cheap blob storage, edge delivery

7. Ranking extension (pro level)

Beyond chronological: score = f(recency, engagement, affinity). Offline ML pipeline computes scores; feed sorted set uses score not just timestamp.

8. Caching & pagination

Cursor-based pagination: ?cursor=post_id:timestamp&limit=20. Cache first page of hot users in CDN edge (usually no — too personalized).

9. Failure & scale tactics

Fan-out workers scale with Kafka consumer groups
Rate limit posts per user (anti-spam)
Celebrity list maintained in config — hybrid routing
Degrade: show cached stale feed if feed service slow

10. Back-of-envelope

500M posts/day × 200 followers avg push = 100B feed writes/day — too many. Hence hybrid + only active followers + async batch fan-out. Real systems use probabilistic fan-out, ranking filters, and "close friends" priority.

Quick revision

Clarify: DAU, read:write ratio, latency, consistency.
Push: Fast read, bad for celebrities.
Pull: Simple write, slow read for heavy followees.
Hybrid: Push normal users; pull/merge celebrities at read.
Stack: Kafka fan-out, Redis feed cache, SQL/Cassandra posts, S3+CDN media.
Pagination: Cursor-based, not offset.