System Design · Core

Message Queues

Buffer and decouple work: why queues exist, ACKs, delivery guarantees, Kafka vs SQS vs RabbitMQ, and dead-letter queues.

1. Why queues

A queue sits between producers and consumers so they do not have to be up, fast, or scaled together. Classic wins: absorb spikes, retry safely, and fan out work to worker pools.

Sync pain: Upload API waits for resize + virus scan + notify → slow, fragile With queue: Upload API writes object + enqueues "process:photoId" Workers pull jobs at their own pace API returns 202/200 quickly

Decoupling: producers unaware of consumer count or language.
Smoothing: backlog grows instead of dropping requests.
Reliability: durable messages survive consumer crashes (with ACKs).

2. Core actors

Producer → [ Queue / Topic ] → Consumer(s) | optional DLQ

Producer: enqueues a message (command or event).
Broker: stores and delivers.
Consumer: processes; scales horizontally.

3. Acknowledgements (ACKs)

1) Consumer receives message (not deleted yet) 2) Processes work 3) Sends ACK → broker deletes / advances offset If crash before ACK → redelivery after timeout / rebalance

Visibility timeout (SQS) or unacked redelivery (Rabbit) or offset commit (Kafka) are variants of the same idea: do not drop work on failure.

4. Delivery guarantees

Guarantee	Meaning	Practice
At-most-once	May lose; no dup	Metrics where loss is OK
At-least-once	No loss; may dup	Default; need idempotent consumers
Exactly-once	Process once end-to-end	Hard; often “effectively once” via idempotency + dedupe

Non-idempotent: balance += 10 Idempotent: apply txn_id T once; store processed_ids

5. Ordering

Single queue / single partition → total order (limits throughput).
Partition by key (user_id) → order per key, parallel across keys (Kafka).
Competing consumers on one Rabbit queue → no global order.

6. Kafka vs SQS vs RabbitMQ

	Kafka	SQS	RabbitMQ
Model	Log / topics / partitions	Managed queue	Smart broker, exchanges
Replay	Yes (retain by time/size)	No (delete on ACK)	Limited
Fan-out	Consumer groups	SNS+SQS or multiple queues	Exchanges → queues
Ops	Heavier (or managed MSK)	Serverless-ish	Moderate
Best for	Event streams, high throughput	Simple async jobs on AWS	Routing, protocols, classic MQ

Interview default: Kafka for high-volume event streams and replay; SQS for straightforward task queues on AWS; Rabbit when you need flexible routing patterns.

7. Dead-letter queues (DLQ)

Main queue --fail N times--> DLQ Ops inspects / replays / alerts Poison message does not block the partition forever

Set max receive count; alert on DLQ depth.
Fix consumer bugs; replay when safe.
Do not silently drop business-critical messages.

8. Anti-patterns

Using a queue for sub-100ms synchronous UX without a clear async story.
Huge messages — store blob in S3, queue the pointer.
No idempotency while relying on at-least-once.
Unbounded retry without DLQ or backoff.

9. Sketch for designs

Draw producer → topic/queue → consumer group → DB side effects, plus DLQ. Call out partition key, ACK/idempotency, and how you scale consumers.

Quick revision

Queues decouple, buffer spikes, and enable retries.
ACK after success; crash before ACK → redelivery.
At-least-once + idempotent consumers is the practical combo.
Kafka = log/replay/partitions; SQS = managed tasks; Rabbit = routing MQ.
Order per partition key, not globally across the cluster.
DLQ for poison messages; alert and replay.
Queue pointers to big payloads, not the payloads themselves.