Dev Learning
Write-heavy / fan-out

Notification system

Design a service that other teams call to notify users. It should support push, email and SMS, respect user preferences and quiet hours, never send the same thing twice, and keep working when a provider goes down.

starterMessage queues and backpressureIdempotency, and why exactly-once is a mythCircuit breakers and retries with jitterRate limitingFan-out: push vs pullObservability — metrics, logs, traces and alertsBatch and stream processing

Where’s the slack?

Urgent: seconds. Transactional: a minute. Marketing: hours.

Almost every notification has slack. This system's value is that it takes the work off other teams' request paths and spends that slack for them.

A password reset email has maybe a minute before someone tries again. A "your driver has arrived" push has seconds, and it is worthless late. A weekly digest has hours; going out at 09:04 instead of 09:00 makes no difference to anyone.

That range drives the design. The slack varies by more than two orders of magnitude within one system, so you must not put all of it in the same queue. If you do, the digest run will sit in front of the password resets, and the outage will be reported as "login is broken" rather than "marketing is slow".

The calling service has no slack at all, though. When checkout calls the notification service, the call must return immediately: it accepts the request, it does not deliver it. If sending an email is inside the checkout request, a slow SMTP server makes checkout slow. Removing that coupling is why this service exists.

So: Accept and acknowledge in milliseconds, then do the work from priority-separated queues. Providers are wrapped in timeouts and breakers so a dead vendor degrades one channel instead of the whole service.

Work it through

Five stages, gated. You write your answer for each stage before its model answer unlocks. That is the core of the method.

Loading your progress…

Reference material for this problem. The diagrams below are here to check your own against once you have drawn it, not to read first.

The lifecycle

Requested by a calleranother serviceAccepted + deduplicatedsystemPreferences evaluatedsystemQueued by prioritysystemHanded to providersystemDeliveredproviderCaller retried after a timeoutidempotency key returns the original result — no second receiptUser unsubscribed or in quiet hourssuppressed or scheduled for 8am local, recorded as a queryable outcomeProvider downtimeout, jittered retries, breaker opens, fail over to the secondary vendorStuck three hourstime-to-live expires it — a late "driver arriving" is worse than noneSent but our status write failedprovider-side idempotency key, so the retry does not send twiceCaller has a bug and loopsper-user frequency cap stops it at the platform, not at the caller
Your responsibility ends at "provider accepted". Everything after that is a report you receive, not an outcome you control. Saying so is part of the answer.

The numbers

Baseline notifications~2,500 / sec
One campaign, if sent in 10 min~33,000 / sec — a 13x spike
SMS vendor contract limit~300 / sec — not yours to change

So the hard part is burst absorption against a provider ceiling you do not control. Baseline throughput is trivial; a marketing button is what breaks this system.

The average is the least interesting number here. The campaign drives the design.

The architecture

allowed?breaker openCalling servicesIngest APIdedupe, 202Preferencesopt-out, quiet hours, capsUrgent queueBulk queuerate-cappedChannel workersown pool + breakerPrimary providerSecondary providerStatus storewhy it was not sent
Ingest is trivial and fast. Priority queues keep the digest away from the password reset. Each channel is isolated behind its own breaker.

Follow-ups you should expect

Many interviews are decided in the follow-ups, not the first diagram.

A third-party API you depend on starts taking 30 seconds to respond. What happens?The write worked but the response was lost and the client retried. What happens?30x traffic for four hours on sale day. Same design?One customer's traffic is degrading everyone else. What do you do?Now a partner needs API access to this data. What changes?Which alert fires first when this system starts failing?What does one user cost you per month, and does that work?Why a queue instead of just calling the service directly?

Same shape, other products

One action by one person creates work for thousands of other people.

The question is always the same: do you do that work when they write, or when everyone else reads? And what do you do about the one user with 200 million followers?

Others with this shape: Twitter timeline · Chat delivery · Notification system · Activity feed

Before you leave
Try this problem again in a week without reading anything. Comparing what you write then with what you wrote today shows you what stuck.
Stuck, or new to this?Read the full worked solutionAll five stages end to end, with the diagrams and the costs, written out the way you would say it. Best read after you have had a go.Read it →

How it is really built

Write your own answer first. Then read how teams who run this at scale describe it. Their constraints were real, and the write-ups often say what their choices cost.