Notification system — a full worked answer
The whole thing, start to finish, in the order you would actually say it. Read it to learn the shape of a strong answer — then go and produce one yourself, because reading is not the same skill as producing.
The question
Design a service that other teams call to notify users. It should support push, email and SMS, respect user preferences and quiet hours, never send the same thing twice, and keep working when a provider goes down.
Write-heavy / fan-out. One action by one person creates work for thousands of other people.
The question is always the same: do you do that work when they write, or when everyone else reads? And what do you do about the one user with 200 million followers?
Before anything: where is the slack?
How long the system has before a person notices. Ask this before you draw a single box — it decides more of the design than scale does.
Almost every notification has slack. This system's value is that it takes the work off other teams' request paths and spends that slack for them.
A password reset email has maybe a minute before someone tries again. A "your driver has arrived" push has seconds, and it is worthless late. A weekly digest has hours; going out at 09:04 instead of 09:00 makes no difference to anyone.
That range drives the design. The slack varies by more than two orders of magnitude within one system, so you must not put all of it in the same queue. If you do, the digest run will sit in front of the password resets, and the outage will be reported as "login is broken" rather than "marketing is slow".
The calling service has no slack at all, though. When checkout calls the notification service, the call must return immediately: it accepts the request, it does not deliver it. If sending an email is inside the checkout request, a slow SMTP server makes checkout slow. Removing that coupling is why this service exists.
So: Accept and acknowledge in milliseconds, then do the work from priority-separated queues. Providers are wrapped in timeouts and breakers so a dead vendor degrades one channel instead of the whole service.
Requirements
about 5 minWhy this stage exists: Because the question is deliberately vague, and every box you draw before this is a guess.
Assumptions: this is an internal platform service used by many teams, with three channels to start (push, email, SMS) and more later, tens of millions of users, and templates owned by the calling teams rather than by me.
The question that changes a box: is duplicate sending unacceptable, or merely annoying? "Never send twice" needs a deduplication store on the write path with a retention window, and "rarely send twice" does not. For a marketing email a duplicate is embarrassing. For a two-factor code or a payment receipt it is a support ticket. I will assume it must not duplicate, which means every request carries an idempotency key.
Second question: do callers need to know whether it was delivered, or only that it was accepted? Delivery status means storing per-notification state, ingesting webhooks from every provider, and exposing a status API. That is a much bigger system. I will assume accepted-plus-best-effort-status, with delivery events recorded for analytics but not promised.
Scope I propose: a send API that accepts and queues, user preferences and quiet hours, per-channel provider integration with failover, deduplication, and rate limits per user so no one gets spammed. Out of scope: template authoring and localisation, in-app notification inboxes, and delivery guarantees stronger than best effort. Each can be built separately.
What an interviewer is ticking off in this stage
- Established this is a platform service with many internal callers
- Asked whether duplicates are unacceptable, and tied that to an idempotency key
- Asked whether delivery status must be tracked, and named what it would cost
- Proposed a scope, and named something you left out on purpose
- Mentioned per-user rate limits as a product requirement, not just an abuse one
Actors and lifecycle
about 6 minWhy this stage exists: Because the happy path is the easy half. The failure branches are where the design actually lives.
Actors: the calling service, the recipient, each provider (Apple, Google, an email vendor, an SMS aggregator) and the system itself. The system is the interesting actor. It decides whether the user wants this at all, whether it is a quiet hour where they live, whether they have had too many already today, and which provider to use right now.
The chain: requested by a caller → accepted and deduplicated → user preferences and quiet hours evaluated → queued by priority → rendered → handed to a provider → provider accepts → delivered to the device → opened. My responsibility ends at "provider accepts". Everything after that is a report I receive, not an outcome I control.
Failure branches. At accepted: the caller retried after a timeout, so the idempotency key catches it and returns the original result. Without this, a network timeout during a payment can send two receipts. At preferences: the user has unsubscribed, so it is dropped. Dropped is a normal outcome that must be recorded and countable, not an error. At queued: the provider is down, so it retries with growing delays, and after a few failures the breaker opens and sends to that provider stop rather than piling up. At handed to provider: the send succeeded but my write recording it failed, so a retry would send twice. That is why the provider call itself carries an idempotency key where the vendor supports one. Where it does not, a rare duplicate is possible, and I would say so.
The branch the system owns, and the one that is easy to miss: a notification that has been stuck for three hours. A "your driver is arriving" push delivered three hours late is worse than not delivering it. So messages carry a time-to-live, and expired ones are dropped and counted rather than delivered.
What an interviewer is ticking off in this stage
- Named providers as actors outside your control
- Named the system's silent decisions: preferences, quiet hours, frequency caps
- Ended your responsibility at "provider accepted" and said so
- Idempotency key on the inbound request, and on the provider call where supported
- Suppressed and unsubscribed are recorded outcomes, not errors
- Time-to-live, so a stale notification is dropped rather than delivered late
The lifecycle, drawn
Numbers
about 6 minWhy this stage exists: Because one part of this is hard and the rest is easy, and they look identical until you do the arithmetic.
Assume 50 million users receiving about 5 notifications a day. That is 250 million a day, divided by 100,000 seconds (a day is 86,400, rounded up), so roughly 2,500 per second on average. That is small.
The average is misleading here. A marketing campaign to 20 million users is submitted as one API call and becomes 20 million sends. If I try to do that in ten minutes, that is 33,000 per second on top of the baseline: a thirteen-fold spike, self-inflicted, and likely more than my email vendor accepts. So the peak is driven by one person clicking a button, not by user behaviour.
Storage: a notification record with status is maybe 500 bytes. 250 million a day is 125 GB a day, so I would keep detailed records for 30 days — about 4 TB — and roll older data into aggregates. Keeping every notification forever is a growing table nobody prunes.
Provider limits are the binding constraint, and they are not mine to change. An SMS aggregator might accept a few hundred per second on my contract. I cannot design that away. It is a ceiling I have to shape traffic to fit.
So the hard part here is burst absorption and provider limits, not throughput. The baseline is trivial. The design is about surviving a campaign without delaying a password reset, while staying inside a vendor rate limit I do not control.
What an interviewer is ticking off in this stage
- Produced a baseline rate and called it small
- Identified that peak is driven by campaigns, not by users
- Quantified the campaign spike as a multiple of baseline
- Named provider rate limits as an external ceiling
- Finished the sentence: burst absorption and provider limits, not throughput
The numbers, drawn
So the hard part is burst absorption against a provider ceiling you do not control. Baseline throughput is trivial; a marketing button is what breaks this system.
High-level design
about 12 minWhy this stage exists: Because now — and only now — you have a reason for every box you are about to draw.
The ingest API does the smallest possible amount of work: validate, claim the idempotency key, write the request, put it on a queue, return 202. That path must stay in single-digit milliseconds, because every calling team's latency depends on it. Nothing about preferences, rendering or providers happens here.
Priority-separated queues (urgent, transactional, bulk), each with its own consumers. This is the most important box on the board, and it comes straight from the slack analysis: the digest run must be unable to sit in front of a password reset. Bulk consumers are also capped, so a campaign cannot consume all the workers.
A preference and policy service evaluates each notification before it is rendered: has the user opted out of this category, is it a quiet hour in their timezone, have they already had too many today. For anything non-urgent, quiet hours mean scheduling rather than dropping: hold it until 8am local. Frequency caps are a per-user rate limiter, the same mechanism as abuse protection, applied to the product.
Channel workers, one pool per channel, render the message and call the provider. Each has its own connection pool, timeout and circuit breaker, so a dead SMS vendor cannot consume the workers that send push. That isolation is a bulkhead: one provider outage degrades one channel rather than the service.
Provider abstraction with failover: two vendors per channel where it matters, so a breaker opening on the primary routes to the secondary. Cost is a second contract and reconciling two sets of delivery reports.
Finally a status store and event stream: every state change is recorded, provider webhooks are ingested to update delivery status, and aggregates roll up for the dashboard. This is what makes the system debuggable when a team asks why their user did not get an email. Often the answer is "she unsubscribed in March", and you can only give it if suppression is recorded as data.
What an interviewer is ticking off in this stage
- Ingest does almost nothing and returns 202 immediately
- Separate queues by priority, justified from the slack budgets
- Preferences, quiet hours and frequency caps evaluated in one place
- Per-channel worker pools with their own breakers — a bulkhead
- Two providers per channel with automatic failover
- Suppression and status recorded as queryable data, not just logs
The architecture, drawn
Deep dive and tradeoffs
about 14 minWhy this stage exists: Because naming what a decision costs is the single clearest signal of seniority there is.
Hard part one: a provider goes down. Minute one, sends start timing out. Without protection, workers block on a dead vendor, the queue backs up, and because retries pile on, the vendor gets more traffic while it is trying to recover. So: a tight timeout on every provider call, retries with growing delays and randomness so a thousand workers do not retry in unison, and a breaker that opens on failure rate and stops calls to that provider. Then failover to the secondary provider for that channel, which is why you have one. Notifications keep queueing, and because they carry a time-to-live, anything that becomes worthless while we wait is dropped and counted rather than delivered stale. Cost: during the outage I am sending through a vendor with different deliverability and pricing, and my delivery reports come from two sources, so the analytics are messier for that period.
Hard part two: the campaign. 20 million messages submitted as one call. First, it is accepted as a *campaign* rather than 20 million individual requests, so the API call is fast and the expansion happens in the background. Otherwise the caller times out. Second, the expansion writes into the bulk queue only, at a controlled rate, so it can never starve the other two. Third, the rate at which bulk workers drain is capped below what the provider accepts, so I am shaping traffic rather than getting throttled. Fourth, preferences are evaluated per recipient during expansion, not at submission, because the list can be an hour old by the time it is sent, and some recipients will have unsubscribed in between. Cost: marketing waits an hour rather than ten minutes, and I have to explain why that is on purpose.
Runaway protection. A bug in a calling service that sends the same user a notification in a loop is a realistic incident. Per-user frequency caps stop it at the platform boundary rather than relying on every caller being correct. I would also cap per-caller volume, and alert on any sudden change in a caller's rate. Cost: a legitimate burst can be throttled, so the cap needs to be configurable per caller rather than a constant.
Debuggability. A question this service will get often is "why did my user not get this?". Every plausible answer (unsubscribed, quiet hours, frequency cap, provider rejected it, still queued) has to be a recorded, queryable outcome. If suppression is only a log line, this service becomes a support burden. That observability decision has to be made at design time.
What an interviewer is ticking off in this stage
- Timeout, jittered retries, breaker and failover, in that order
- Time-to-live so stale notifications are dropped rather than delivered late
- Campaigns accepted as one object and expanded in the background
- Bulk drain rate capped below the provider limit — shaping, not being throttled
- Preferences evaluated at send time, not at submission
- Per-user and per-caller caps to contain a buggy caller
- Every suppression reason recorded as queryable data
And then they push back
A complete answer is where the interview starts, not where it ends. These are the questions this design invites — the interview is decided here far more often than in the diagram.
A third-party API you depend on starts taking 30 seconds to respond. What happens?
"We would retry." Retrying against something already overloaded makes it worse, and retrying a 30-second call means holding your resources for a minute or more. This answer usually turns one slow dependency into your own outage.
"Slow is more dangerous than down, because down fails fast and slow holds resources. Without a timeout, every request calling that API holds a thread and a connection for 30 seconds, my pools exhaust, and endpoints that never touch that API start failing. One slow dependency has become a total outage. So: a timeout set from their real latency distribution, not a round number; a circuit breaker that opens on failure rate so I stop waiting entirely once it is clearly bad; a bulkhead giving that dependency its own limited pool so it cannot consume capacity the checkout path needs; and a defined fallback — cached data, a degraded response, or dropping an optional feature. The fallback has to be decided before the incident, per dependency, or I fail fast into a blank page. And retries need backoff with jitter and a cap, or I contribute to keeping them down."
The write worked but the response was lost and the client retried. What happens?
"The client would get an error and could try again." They already tried again. That is the premise. If the retry does the work a second time, you have charged the card twice.
"With idempotency keys, nothing bad: the retry carries the same key, the server finds it already claimed, and returns the original stored response, so the client sees success and the card is charged once. Without them, two charges and a customer complaint. The implementation detail that makes it airtight is inserting the key with a unique constraint in the same transaction as the work, because two separate steps leave a crash window between them. Two things people get wrong: the client must reuse the key across retries rather than generating a fresh one each attempt, which defeats the scheme without raising any error; and if the retry arrives while the first request is still running, I return an in-progress status so the client backs off, rather than doing the work again or returning a wrong answer. Keys expire after a day or so, or the table grows forever."
30x traffic for four hours on sale day. Same design?
"We would autoscale." Autoscaling takes minutes and this spike arrives in seconds. Also, the thing that breaks is usually the part that cannot autoscale, which is why naming autoscaling as the whole answer is a tell.
"Mostly the same design, but three things change. First, this spike is predictable (the date is known), so I pre-scale rather than relying on autoscaling, which reacts over minutes while the spike arrives in seconds. Second, the stateless tiers scale out, but the database and any shared counter do not, so I would protect them: a queue or waiting room in front to convert a burst into a rate, and aggressive caching of everything that does not need to be exact. Third, I would decide in advance what to shed (recommendations, analytics, non-essential enrichment) so degradation is a choice rather than a surprise. And I would load test at 30x before the day, because the failure I have not thought of is the one that matters."
One customer's traffic is degrading everyone else. What do you do?
"We would rate limit them." Eventually, yes — but first you need to be able to see that it is them, and then you need a way to act without a deploy.
"First I need to be able to attribute load per customer, which means metrics tagged by tenant, and that has to be decided in advance, not during the incident. Then immediate mitigation: tighten their rate limit, which must be per-key configuration so it is a config change rather than a deploy. Then the structural fix, which is isolation: separate connection pools or queues per tenant class so one customer cannot consume shared resources, which is the bulkhead pattern applied to tenancy. For a very large customer, dedicated capacity is often the right answer commercially as well as technically. What I would also check is whether they are abusing the system or using it as intended in a way I did not anticipate. If it is the latter, rate limiting them is treating a product problem as an abuse problem, and the better fix might be a bulk API that does in one call what they are doing in ten thousand."
Now a partner needs API access to this data. What changes?
"We would expose an API endpoint." Now you have an external contract you cannot change, an authentication system, and traffic patterns nothing was designed for.
"More changes than it sounds. Authentication and authorisation for a party that is not a user, which means API keys or OAuth and a permission model expressing what a partner may see. Rate limiting per partner, because they will write a script that pages through everything at 3am. Versioning, because the moment an external party depends on the shape of my response I cannot change it freely — so I would version from day one rather than retrofit it. Pagination with cursors rather than offsets, because a partner pulling a large dataset with offsets creates the deep-pagination problem that gets expensive. And a read model separate from my internal one, so my schema stays free to evolve behind a stable contract. The traffic pattern is also different from user traffic: bulk, bursty, and often at off-peak hours, which is good for capacity and bad for any cache tuned around user access patterns. If they need large volumes regularly, a bulk export to object storage beats an API — cheaper for both sides and far kinder to my database."
Which alert fires first when this system starts failing?
"We would alert on CPU and memory." Those are symptoms of the machine, not of the user experience, and they fire either far too late or constantly for no reason.
"I would alert on user-visible symptoms first and causes second. The first alert should be error rate or latency at p99 on the main user path, because that is what maps to someone being unhappy. Then a small number of leading indicators that fire before users notice: queue age (not depth), because age maps directly to how late the work is; replication lag; cache hit rate dropping, which precedes database saturation; and connection pool wait time, which is the earliest sign of a slow dependency. For this system specifically, the earliest signal is usually cache hit rate, because everything downstream is sized assuming the cache absorbs most reads. I would not page on CPU. It is a dashboard metric, not an alert, because a machine at 90% CPU serving requests fine is not an incident, and a machine at 20% CPU timing out is."
What does one user cost you per month, and does that work?
"I have not calculated that." It is derivable from your own estimates, and not having connected the design to unit economics is a gap the Cost Auditor will keep pressing.
"I can derive it from my estimates. Each user generates a known amount of storage and bandwidth — for this design, roughly the bytes per action times actions per day times retention. The number that matters is whether the marginal cost per user is well below the revenue per user, and whether it is flat or growing. Flat is fine. Growing is the dangerous case, and it happens when storage is retained forever while revenue is monthly — every user gets more expensive every month they stay, which eventually inverts. That is a strong argument for retention limits and tiering, framed as a business requirement rather than a technical preference. The other thing I would check is whether the heaviest 1% of users cost far more than the median, because that usually decides whether you need usage limits in the product."
Why a queue instead of just calling the service directly?
"Queues decouple services and are more scalable." True in the abstract and it does not explain why this particular call needs one. Queues also add duplicate delivery, ordering questions and debugging difficulty, and not mentioning those means you added one by reflex.
"Because the user does not need to wait for this work to finish, and because the downstream cannot absorb my spikes. Concretely: sending the receipt email takes 800 ms and can fail, and there is no reason to make the customer wait for it or to fail their order because a mail provider is down. The queue absorbs a 10x spike into a longer backlog instead of a broken service. What it costs: the customer gets 'received' rather than 'done', so the product has to be honest about that; every consumer must be idempotent because delivery is at-least-once; and a failure now happens in a worker minutes later, which is harder to debug. I would not queue the stock check or the payment, because the user needs a real yes-or-no answer before they leave the page. A queue there adds latency and buys nothing."
The topics this answer used
You have read a good answer. That is worth very little until you can produce one on a blank page — so go and write this problem without looking, and compare afterwards.