Dev Skills
Building the storage itself

Design a distributed message queue

Design a distributed message queue: the broker itself, not a service that uses one. Producers publish messages to named topics, several independent consumer groups read them, messages survive machine failures, and the system carries hundreds of thousands of messages a second.

hardMessage queues and backpressurePartitioning (sharding)ReplicationWrite-ahead logsConsensus, at concept levelIdempotency, and why exactly-once is a mythBack-of-envelope estimationBatch and stream processingObservability — metrics, logs, traces and alerts

About this problem

“Design a distributed message queue” is a hard-level problem in the building the storage itself family. You are not using a database. You are being asked to design one.

The key challenge: A queue exists to create slack for everyone else, so its own publish path has very little. The one place it can trade time is durability: confirm a message before or after the replicas have it.

Time budgets: Publish: a few milliseconds to tens of milliseconds. Consumption: seconds behind is normal. Rebalancing, retention and repair: hours.

The worked solution for Message queue goes through why those budgets shape the design, stage by stage.

Before you start

You will work this in five timed stages. Here is what each one asks you to think about. None of it is the answer; the answers unlock in the workspace below once you have written your own.

  1. Stage 1 · Requirements · 5 min
    • Does every consumer need to see every message, or does each message go to one worker?
    • Ordering: across the whole topic, or only for messages with the same key?
    • What may be lost, and what may be duplicated?
  2. Stage 2 · Actors and lifecycle · 6 min
    • When exactly is a message "safe"? When one machine has it, or more?
    • What happens if a consumer processes a message and crashes before saying so?
    • What happens to a consumer that falls further behind than the retention period?
  3. Stage 3 · Numbers · 6 min
    • Every byte is written more than once. How many times?
    • Seven days of retention: how many terabytes is that, with replicas?
    • How many consumers can one group run in parallel, and what limits it?
  4. Stage 4 · High-level design · 12 min
    • What does one partition look like on disk?
    • Who decides which broker leads a partition, and where is that decision stored?
    • Where does a consumer group keep its position?
  5. Stage 5 · Deep dive and tradeoffs · 14 min
    • A leader dies. Which messages are guaranteed to survive, and who decides the new leader?
    • How do you get exactly-once, and where does that promise stop?
    • What happens to consumption while a group rebalances?
Prefer to read first? The worked solution, end to end →

Work it through

Five stages, gated. You write your answer for each stage before its model answer unlocks. That is the core of the method.

Loading your progress…

Done all five? The diagrams, numbers, trade-offs and follow-ups for this problem are on the worked solution page, to check your own design against.

Before you leave
Try this problem again in a week without reading anything. Comparing what you write then with what you wrote today shows you what stuck.
Stuck, or new to this?Read the full worked solutionAll five stages end to end, with the diagrams and the costs, written out the way you would say it. Best read after you have had a go.Read it →

How it is really built

Write your own answer first. Then read how teams who run this at scale describe it. Their constraints were real, and the write-ups often say what their choices cost.