Message queues and backpressure
Write the job down now, do it soon, and survive the gap in between.
What problem does message queues solve?
When one part of a system is fast and the next is slow, calling directly means the fast part waits, and users feel it. A queue lets the fast part hand the work over and return straight away.
It also soaks up spikes. Ten times the traffic for an hour becomes a longer queue instead of a broken service. And it separates the two sides: the producer does not need the consumer to be up right now.
What message queues costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for message queues, and when not to
Use it when
- The work does not have to finish before you answer the user: emails, thumbnails, indexing, reports.
- You need to smooth spiky load onto something with fixed capacity.
- You are sending one event to several consumers who should not know about each other.
- The thing downstream is unreliable and you want retries handled in one place.
Do not, when
- The user is waiting for the result right now. A queue in that path adds delay and buys nothing.
- You need a definite yes or no before responding, like whether a seat was reserved.
- The work is tiny. A direct call is simpler, and simpler is better.
How message queues actually works
There are two shapes. A work queue hands each message to one consumer, and you scale by adding consumers. Use it for jobs, tasks, sending email. A log, like Kafka, keeps an ordered record that many independent consumers read at their own position, and can be read again from the start. Use a log when several teams need the same events, or when replay matters.
Ordering. Putting a whole topic in one global order is expensive and usually unnecessary. What you almost always want is ordering per key: all events for one order in sequence, while different orders move in parallel. That is what partitioning by key gives you. Say "ordered per user, not globally" and you have answered the question properly.
Retries need a limit and a delay. Retry immediately and forever, and a message that always fails, from bad data or a bug, spins forever, burns capacity and hides real work behind it. So use exponential backoff with jitter, a maximum number of attempts, and then move it aside.
Where it moves to is the dead letter queue, where poison messages wait for a human. The part people forget: a dead letter queue nobody watches loses data without anyone noticing. Alert when messages arrive in it, and have a way to fix and replay them.
Backpressure is what separates a design from a diagram. If producers stay faster than consumers, the queue grows without limit, and eventually the whole system falls over, having accepted work it can never do. The options are: scale consumers automatically based on the backlog. Slow the producer down by refusing or throttling at the edge. Drop low-value work on purpose. Or set a maximum queue size and reject when it is full. Every one of them means saying no to someone. Choosing which someone is the design.
What to monitor: how far consumers are behind and whether that is growing, the age of the oldest message (the best measure of user pain), and how fast messages arrive in the dead letter queue.
- Adding a queue and never mentioning idempotency. Duplicate delivery is guaranteed.
- No dead letter queue, or one nobody watches.
- Assuming global ordering. You almost certainly want ordering per key.
- Treating a queue with no size limit as infinite capacity. It is failure, postponed.
The message queues follow-up an interviewer will ask
"The queue is backed up by an hour. What do you do?"
First, work out which of two things it is, because the fixes are opposite. If consumers are healthy and just outnumbered, add more. And check what they are hitting, because often the limit is a database or a downstream API, and adding workers just moves the queue there. If consumers are failing and retrying, adding more makes it worse. Find the poison message or the broken dependency, and stop the retry storm. Second, sort the backlog instead of draining it in order. If password reset emails and weekly digests share a queue, the digests are now delaying something urgent. Split the queues by priority, which is a lesson worth taking into the design rather than the incident. Third, decide what to throw away. An hour-old "your driver is arriving" notification is worse than useless, and I would drop it rather than deliver it. Fourth, tell users. A status message beats silence. And afterwards, alert on how fast the backlog is growing, not just its size, because size crossing a threshold tells you an hour too late.
Your turn
Name one thing you should never put behind a queue in a checkout flow, and why.
The digest run that broke login
At 10:10 on Tuesday, support reports that password reset emails requested since about nine are not arriving. Nothing is down. The email service is healthy, the provider is healthy, and the queue consumers are all running and processing messages steadily. It has happened once before, three weeks ago, and resolved itself after about ninety minutes.
- 1Confirm what is late
Consumers are healthy and throughput is normal, which rules out a failure. Queue depth is 250,000 and the oldest message is 74 minutes old. So nothing is broken. Messages are being processed a long time after they were enqueued. The system is working as designed, and the design is wrong.
- 2Find what filled it
At 08:55 the weekly digest job enqueued 2 million messages. Password resets go into the same queue. A work queue is first-in-first-out, so every reset submitted after 08:55 sits behind up to 2 million digests. Consumers drain at about 400 a second, so a reset enqueued at 09:00 waits behind roughly 1.9 million digests and goes out around 10:20, long after the user has given up and contacted support.
- 3Mitigate before diagnosing further
Right now, get the resets out. Temporarily scale consumers up, which helps but does not reorder anything. Better and faster: publish new password resets to a separate temporary queue with its own consumers, so they bypass the backlog entirely. Ninety seconds of work, and the user-visible incident ends immediately while the digest backlog continues draining harmlessly.
- 4Fix the structure, not the incident
Split the queues permanently by priority: urgent, transactional, bulk, each with its own consumers. Bulk consumers are capped so a campaign can never consume the workers that transactional messages need. This comes straight from the slack analysis. A password reset has a minute and a digest has hours, and while work with a hundredfold difference in urgency shares one queue, this incident will keep coming back.
- 5Rate-shape the producer as well
Enqueuing 2 million messages in one burst is itself the problem. The digest job now expands its recipient list gradually, at a rate below what consumers drain, so the bulk queue never accumulates a large backlog in the first place. The last digests go out a couple of hours after the first, which is fine because nobody was promised anything faster.
- 6Change the alert so this is caught in minutes
The alert had been on queue depth above 5 million, set that high so the weekly digest spike would not page anyone. It never fired. Replace it with two alerts: oldest message age above five minutes on the transactional queue, and backlog growth sustained over ten minutes on any queue. The first would have fired within minutes of 09:00 and named which work was being delayed.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Apache Kafka — Design ↗The log-based design, partitions and consumer groups, from the source.