Message queues and backpressure — in depth
The summary said "write the job down now, do it soon". This page is about everything that goes wrong between now and soon. It covers work queues against logs and when each is right, delivery guarantees stated precisely, ordering per key rather than globally, retries and poison messages, dead letter queues that someone reads, backpressure and what saying no means in practice, and the metrics that tell you a backlog is a problem before users do.
Work queue or log: two different things
People say "queue" for two quite different systems, and choosing the wrong one shows up as a redesign six months later.
A work queue hands each message to exactly one consumer, and the message is gone once it is acknowledged. You scale by adding consumers. Think jobs, tasks, sending an email.
A log keeps an ordered, retained record. Many independent consumers read it at their own position, and any of them can go back and re-read from the beginning. Think Kafka, and think events several teams care about.
- If a second team will ever want these events, you want a log. You cannot retrofit replay onto a work queue.
- If you need to reprocess after fixing a bug, you want a log. This is the reason that decides it most often.
- Log consumers are bounded by partition count: 12 partitions means at most 12 parallel consumers in a group, so the partition count is a capacity decision made at creation time.
- Using a log for simple background jobs is over-building. A work queue is simpler to operate.
| Work queue | Log | |
|---|---|---|
| Consumers per message | Exactly one | Many, independently |
| After processing | Gone | Retained for days |
| Replay | Not possible | Core feature |
| Scaling | Add consumers freely | Bounded by partition count |
| Ordering | Usually none | Per partition |
| Right for | Jobs and tasks | Events many teams consume |
Delivery guarantees, stated precisely
There are three, only two of them are achievable, and interviewers listen for whether you state them precisely.
At-most-once: send and forget. Messages can be lost. Almost never what you want, though it is legitimate for high-volume metrics where losing a sample is cheaper than the machinery to avoid it.
At-least-once: retry until acknowledged. Messages can be duplicated. This is what most brokers give you in practice.
Exactly-once delivery is not achievable across a network that can drop messages, because you cannot distinguish a lost request from a lost response. What is achievable is at-least-once delivery plus processing that is safe to repeat, and that produces an exactly-once *effect*.
- Say "exactly-once effect, achieved with at-least-once delivery plus idempotent consumers". That phrasing shows you know where the guarantee comes from.
- Duplicates arrive for mundane reasons: a consumer crashed after doing the work and before acknowledging, or a rebalance reassigned a partition mid-flight.
- Some brokers offer "exactly-once semantics" within their own boundary, such as a Kafka transaction that reads from one topic and writes to another. That does not extend to an external side effect like charging a card.
- So the consumer contract stays the same: process, record that you processed it, and commit both together.
Ordering: per key, not globally
Total ordering across an entire topic is expensive and rarely required. What you usually need is that events which relate to each other arrive in order, and events that do not can proceed in parallel.
Partitioning by key gives you this: all events for one order, one user, one conversation go to one partition and are processed in sequence, while different keys proceed independently. Say "ordered per user, not globally" and you have answered the question properly.
- Choosing the partition key is the same decision as choosing a shard key, and it has the same failure: one hot key becomes one hot partition.
- Ordering only holds if a single consumer handles a partition at a time. Parallelising within a partition throws the guarantee away, which people do accidentally by processing messages concurrently inside a worker.
- Consumers should still be prepared for out-of-order arrival across partitions, because there is no ordering between them at all.
- If you do need global order, you need a single partition, which means a single consumer and a hard throughput ceiling. Say that ceiling out loud.
Retries, poison messages and the dead letter queue
A message that always fails will retry forever unless you stop it, and while it does it burns capacity and hides real work behind it. This is a poison message, and every queue design needs an answer for it.
- Retry with growing delays, and add randomness. A thousand consumers that failed at the same moment will otherwise retry at the same moment.
- Cap the attempts, typically three to five. Infinite retries are a decision to be down for longer.
- After the cap, move the message to a dead letter queue rather than dropping it, so a human can look at it.
- A dead letter queue nobody monitors is a silent data-loss machine. Alert on arrival rate, not on depth.
- Have a way to fix and replay from the dead letter queue. If the only option is to read it, it is a graveyard rather than a tool.
- Distinguish retryable from non-retryable failures: a timeout is worth retrying, malformed data never will be, and retrying it three times just wastes capacity before the inevitable.
Backpressure: what saying no looks like
If producers stay faster than consumers, the queue grows without bound. It does not fail gracefully at the end of that. It accepts work it can never do, then runs out of memory or disk having promised everybody something.
There are four responses. The first costs money; the other three mean refusing someone, and choosing which someone is the design.
- Scale consumers on backlog, not on CPU. And check what they are hitting: often the ceiling is a downstream database, and adding workers just moves the queue somewhere less visible.
- Slow the producer: reject or throttle at the edge, so the caller learns immediately rather than being told "accepted" for work that will never happen.
- Shed low-value work on purpose: drop the digest emails and keep the password resets. This requires having separated them, which is why priority queues exist.
- Bound the queue: set a maximum depth and reject when full. Harsh, but it converts an eventual collapse into an immediate, visible error.
- The thing to never say is that the queue will absorb it. An unbounded queue is not capacity, it is failure postponed and made harder to diagnose.
A 20% shortfall is invisible for the first few minutes, and within an hour every new message waits 15 minutes. So the alert goes on the growth rate and on oldest-message age, not on depth crossing a threshold.
What to monitor, and the incident it prevents
Queue metrics are unusual in that the obvious one is the least useful. Depth crossing a threshold tells you about a problem that started an hour ago.
- Oldest message age is the direct measure of user pain. Depth of 50,000 means nothing on its own; an oldest message of 40 minutes means somebody has been waiting 40 minutes.
- Rate of change of backlog is the leading indicator. Growing is an incident whether the depth is 100 or 100,000.
- Dead letter arrival rate, alerted from the first message.
- Consumer error rate versus throughput: the two failure shapes look identical on a depth graph and need opposite fixes. Healthy but outnumbered means add consumers. Failing and retrying means adding consumers makes it worse.
- Separate queues per priority, so you can see and alert on them independently. One queue means one number, and one number hides which work is being delayed.
The digest run that broke login
At 10:10 on Tuesday, support reports that password reset emails requested since about nine are not arriving. Nothing is down. The email service is healthy, the provider is healthy, and the queue consumers are all running and processing messages steadily. It has happened once before, three weeks ago, and resolved itself after about ninety minutes.
- 1Confirm what is late
Consumers are healthy and throughput is normal, which rules out a failure. Queue depth is 250,000 and the oldest message is 74 minutes old. So nothing is broken. Messages are being processed a long time after they were enqueued. The system is working as designed, and the design is wrong.
- 2Find what filled it
At 08:55 the weekly digest job enqueued 2 million messages. Password resets go into the same queue. A work queue is first-in-first-out, so every reset submitted after 08:55 sits behind up to 2 million digests. Consumers drain at about 400 a second, so a reset enqueued at 09:00 waits behind roughly 1.9 million digests and goes out around 10:20, long after the user has given up and contacted support.
- 3Mitigate before diagnosing further
Right now, get the resets out. Temporarily scale consumers up, which helps but does not reorder anything. Better and faster: publish new password resets to a separate temporary queue with its own consumers, so they bypass the backlog entirely. Ninety seconds of work, and the user-visible incident ends immediately while the digest backlog continues draining harmlessly.
- 4Fix the structure, not the incident
Split the queues permanently by priority: urgent, transactional, bulk, each with its own consumers. Bulk consumers are capped so a campaign can never consume the workers that transactional messages need. This comes straight from the slack analysis. A password reset has a minute and a digest has hours, and while work with a hundredfold difference in urgency shares one queue, this incident will keep coming back.
- 5Rate-shape the producer as well
Enqueuing 2 million messages in one burst is itself the problem. The digest job now expands its recipient list gradually, at a rate below what consumers drain, so the bulk queue never accumulates a large backlog in the first place. The last digests go out a couple of hours after the first, which is fine because nobody was promised anything faster.
- 6Change the alert so this is caught in minutes
The alert had been on queue depth above 5 million, set that high so the weekly digest spike would not page anyone. It never fired. Replace it with two alerts: oldest message age above five minutes on the transactional queue, and backlog growth sustained over ten minutes on any queue. The first would have fired within minutes of 09:00 and named which work was being delayed.