System Design · Design Question

Design a Web Crawler

Discover and fetch pages at scale: URL frontier, politeness, dedupe, parsers, and storage.

1. Requirements

Start from seed URLs; discover links; download HTML.
Respect robots.txt and per-host politeness (crawl-delay).
Dedupe URLs and near-duplicate content.
Freshness: recrawl important pages more often.
Scale to billions of pages (interview: design toward that).

2. API / control plane

POST /seeds { "urls": ["https://example.com"] } GET /status { queueDepth, fetchQps, hostsBlocked } POST /hosts/{h}/pause

3. Main components

URL Frontier (priority queues per host) | Fetcher workers --HTTP GET--> Web | DNS cache / robots cache | Parser (extract links, canonical URL) | +--> Content Store (S3/HDFS) + metadata DB +--> Link Filter → enqueue new URLs to Frontier +--> Dedupe (URL seen, content fingerprint)

4. URL frontier

The frontier holds URLs to visit. Needs priority (PageRank/seed distance) and politeness (do not hammer one host).

Per-host queue + global scheduler: - Never fetch same host more than once per politeness interval - Prefer high-priority hosts/URLs - Persist frontier for crash recovery (Kafka / DB backed)

5. Data model

urls(url_hash PK, url, host, priority, next_fetch_at, last_status) robots(host, rules_blob, fetched_at) documents(doc_id, url_hash, storage_path, content_hash, fetched_at) links(from_doc, to_url_hash)

6. Deduplication

URL canonicalization: scheme/host case, strip fragments, sort query params carefully.
Seen URL set: Bloom filter + durable key store.
Content: hash / simhash to skip near-duplicates.

7. Politeness and robots

Fetch robots.txt → cache If Disallow matches path → skip Else wait until host's next_allowed_ts Identify crawler via User-Agent

8. Deep dive — scaling fetchers

Workers pull from frontier; horizontal scale on bandwidth/CPU.
DNS caching and connection pools matter as much as raw download.
Separate queues for high-priority continuous recrawl.
Handle redirects, soft 404s, infinite calendar traps (URL filters).

9. Failure modes

Spider traps → max depth, URL regex denylist, budget per host.
Fetcher crash → frontier item not ACKed → retry with backoff.
Storage full → backpressure frontier.

10. Interview flow

Seeds → frontier → fetch → parse → store → enqueue links.
Call out robots + politeness.
Dedupe URL/content.
Priority + recrawl policy.
Scale workers and persist frontier.

11. Freshness and priority

score(url) = f(pagerank, change_rate_estimate, product_importance) next_fetch_at = now + interval(score) News sites: short interval; archive pages: long interval

Change-rate can be learned from past Last-Modified / content-hash diffs. Do not recrawl the whole web at one frequency.

12. Distributed frontier

Partition host queues by host hash across frontier servers.
Kafka topics per priority tier work well as durable frontier backing.
Exactly-once crawl is unnecessary; dedupe saves you from repeats.

Quick revision

Frontier schedules URLs with priority and per-host politeness.
Always honor robots.txt; cache it.
Canonicalize + Bloom/seen-set to avoid re-fetch loops.
Parser extracts links; content goes to blob store + metadata.
Recrawl important pages more often.
Guard against spider traps with budgets and filters.
Persist frontier; ACK fetches for reliability.