System Design · Design Question
Design Twitter Search
Search recent tweets at scale: ingest, inverted index, ranking, and fan-in query serving.
1. Requirements
- Full-text search over tweets (keywords, hashtags, from:user).
- Freshness: new tweets searchable within seconds.
- Rank by relevance + recency + engagement.
- High query QPS; huge write stream of tweets.
Scope: text search on tweets; not the whole Twitter product
Ask: only last 7–30 days early? archive later?
2. API
GET /v1/search?q=world%20cup&type=latest|top&cursor=
→ { results: [{ tweetId, user, text, ts, score }], nextCursor }
3. Data model
Source of truth: tweet store (already exists in Twitter design)
Search index (inverted):
term → postings list of (tweet_id, tf, ts, ...)
Hashtags / mentions as special terms
User filter: secondary index or posting metadata
4. High-level design
Tweet Write Path → Kafka
|
Indexer workers → Lucene/ES shards (inverted index)
|
optionally early "realtime" segment + later merge
Query Path:
Client → Search API → Query Planner
→ fan-in to relevant shards
→ merge top-K → hydrate tweet objects from cache/DB
5. Inverted index refresher
tweet1: "cats are great"
tweet2: "cats and dogs"
cats → (t1, t2)
great → (t1)
dogs → (t2)
AND query: intersect postings; OR: union
6. Freshness strategy
- Realtime index: small in-memory/recent segments for last N minutes.
- Batch index: larger compacted segments for older tweets.
- Query both and merge; promote realtime → batch via merge jobs.
7. Ranking
latest: sort by ts DESC after text match
top: BM25 / learned ranker + engagement + recency decay
Retrieve a candidate set (e.g. few thousand) then re-rank; do not score the entire corpus deeply.
8. Sharding
- Shard by time (recent vs archive) and/or by term hash / tweet_id.
- Queries may fan-in to many shards — cap and prefer recent shards.
- Hot terms (“a”, “the”) need skip lists / careful query rewriting.
9. Scaling & ops
- Index replicas for query HA.
- Cache hot queries briefly (seconds) carefully — freshness trade-off.
- Backpressure indexers on Kafka lag; never block tweet create path.
10. Edge cases
- Deleted tweets → tombstones in index; filter on hydrate.
- Spam → quality filters in ranker / index exclusion.
- Unicode / tokenization for i18n.
11. Query planning examples
q = "cats filter:media since:2024-01-01"
1) tokenize "cats"
2) constrain postings to docs with media flag
3) time bound → only recent shards
4) retrieve → re-rank → hydrate
q = "from:alice dogs"
use author index / filter early to cut postings
12. Abuse and quality
- Rate-limit search scrapers.
- Downrank spam clusters and bot-like engagement.
- Sensitive queries may need safety filters before results render.
Quick revision
- Async index from tweet stream; do not block write path.
- Inverted index + realtime and batch segments for freshness.
- Query fan-in across shards; merge top-K; hydrate tweets.
- Latest vs Top ranking modes.
- Candidate retrieve then re-rank.
- Tombstones for deletes; lag monitoring on indexers.
- Time sharding helps bound “recent search.”