System Design · Design Question
Design a Metrics System
Ingest, aggregate, query, and alert on time-series metrics (mini Prometheus / Datadog mental model).
1. Requirements
- Apps emit metrics: counters, gauges, histograms (latency).
- High ingest write rate; store efficiently with downsampling.
- Query by metric name + labels over a time range.
- Alert when threshold / anomaly rules fire.
- Retention tiers: raw short, rollups long.
Cardinality warning: user_id as a label can explode series count
Ask: scrape pull vs push agents? multi-tenant?
2. API
Push:
POST /v1/write
{ "series": [{ "name": "http_requests", "labels": {"path":"/x","code":"200"},
"points": [[ts, value], ...] }] }
Query:
POST /v1/query_range
{ "query": "sum(rate(http_requests[5m])) by (path)", "start", "end", "step" }
Alerts CRUD:
POST /v1/rules { expr, threshold, for, notify }
3. Data model
metric_names + label dictionaries
time series identity = fingerprint(name + labels)
points: (series_id, ts, value) in TSDB segments / LSM / columnar blocks
rollups: 1m, 5m, 1h aggregates (sum/count/min/max for histograms sketches)
4. High-level design
Apps → Agent / SDK --push--> Ingest Gateway
|
Kafka (buffer)
|
Aggregators / writers → TSDB shards
|
Query API → fan-out to shards → merge
Alert evaluator → rules → Notification system
5. Ingest deep dive
- Batch points; compress; authenticate tenants.
- Kafka absorbs spikes; protects TSDB.
- Reject insane cardinality (too many unique label sets).
- Idempotent write optional via client batch ids (usually at-least-once OK).
6. Storage & downsampling
Hot: raw resolution 15s for 24–72h
Warm: 1m rollups for 30d
Cold: 1h rollups for 1y+
Compaction merges blocks; drop raw after retention
Histograms need careful aggregation (store buckets or use t-digest / DDSketch — mention sketches for percentiles).
7. Query path
- Parse PromQL-like expression.
- Resolve series matching label selectors.
- Fetch chunks from owning shards.
- Evaluate rate/sum/quantile; return aligned steps.
Cache recent query results carefully; prefer caching parsed series lists.
8. Alerting
Evaluator periodically runs rules against TSDB
Pending --for 5m--> Firing → notify (PagerDuty/Slack)
Resolve when expr false
Dedup and group alerts to avoid storms
9. Scaling
- Shard by series fingerprint hash.
- Separate ingest and query compute.
- Control cardinality at the gate — #1 production lesson.
- Multi-region: usually regional TSDB + global query optional.
10. Failure modes
- Kafka lag → delayed dashboards/alerts; SLO on lag.
- Hot series (one metric dominates) → isolate shard.
- Alert flapping → hysteresis /
forduration.
Quick revision
- Pipeline: agents → ingest → Kafka → TSDB → query/alert.
- Series = name + labels; guard cardinality.
- Downsample/retain: raw short, rollups long.
- Histograms need mergeable sketches/buckets.
- Query fans out to shards and merges.
- Alert rules with pending→firing and dedup.
- At-least-once ingest; dashboards tolerate small loss better than payments.