Dev Learning
Limits and abuse

Distributed API rate limiter

Design a rate limiter for a public API served from many machines across several regions. Each customer has a plan with a request quota. The limiter must not become the reason the API is slow or down.

coreRate limitingDistributed countersCachingCircuit breakers and retries with jitterConsistency modelsReverse proxies and gatewaysAPI design — pagination, versioning, errorsObservability — metrics, logs, traces and alerts

Where’s the slack?

Decision: ~1 ms. Accuracy: generous.

No time slack at all, because the decision sits in front of every request. But a lot of accuracy slack, and the design is built on that trade.

There is no latency to spend. The limiter runs before the work it protects, on every request, so it must cost a fraction of what it saves. A limiter that adds 20 ms to protect a 30 ms endpoint has made things worse. A realistic budget is about a millisecond, which rules out anything that needs coordination between regions.

But the accuracy budget is wide, and seeing that is what makes the problem tractable. If a customer's limit is 1,000 requests a minute and they occasionally get 1,050 through, the backend barely notices. The limit exists to protect capacity and prevent abuse, and both goals survive a few percent of error, as long as capacity was planned with that margin.

That asymmetry is unusual. Most systems in this app trade latency for accuracy in small amounts; here you can trade a lot of accuracy for a little latency, and you should say so out loud.

The one place accuracy tightens is security limits: login attempts and password resets. There, being generous means letting an attacker have more guesses, so those get exact counting and a slower path, because they are rare enough to afford it.

So: A millisecond budget plus generous accuracy means local decisions with asynchronous reconciliation instead of a globally consistent counter. It also means capacity limits must fail open, because a limiter outage must not become an API outage.

Work it through

Five stages, gated. You write your answer for each stage before its model answer unlocks. That is the core of the method.

Loading your progress…

Reference material for this problem. The diagrams below are here to check your own against once you have drawn it, not to read first.

The lifecycle

Request arrivesclientKey resolvedgatewayLimit looked uplocal cacheBucket checkedlocal memoryAllowed / rejectedsystemNo API keyfall back to a looser IP-based limitConfig store unreachableserve from the local cache; never fetch per requestCounter store downfail open to per-server local limits — and alert loudlyCounter store slowhard 2 ms timeout, then the same fallbackClient retries instantlyreturn 429 with Retry-After and quota headersRegion partitionedcounts drift; over-admission bounded and stated
A request through the limiter. The counter-store-down branch is the one that decides whether this component protects your API or takes it down.

The numbers

Decisions required~500,000 / sec
Total counter state~100 MB — one machine
Budget per decision~1 ms
Cross-region round trip150 ms — 150x the budget

A cross-region hop is 150x the budget, so global consistency on the request path is ruled out. So the hard part is deciding without coordinating: trade accuracy, which you have plenty of, for latency, which you have none of.

All the state fits in memory. The difficulty is the latency budget.

The architecture

allowedasyncasynclogin onlyAPI clientGatewaylocal bucketsGatewaylocal bucketsAPI servicesShared counterssync ~100 msLimit configcached locallySecurity countersexact, fail closed
No network call on the request path. Decisions are local; agreement is asynchronous and approximate. That is what a 1 ms budget allows.

The mechanism

CallerBuckettokens left: 1API
a token was there — request passesbucket empty — rejected at the door, cheaply
The bucket refills steadily; a request takes a token or is rejected at the door. Rejection has to be cheaper than acceptance, because during an attack most responses are rejections.

The decision this turns on

A
Local buckets, async reconciliation
  • —Zero network calls on the request path, so comfortably inside 1 ms.
  • —Survives the counter store being down: degrades to per-server limits.
  • —Over-admission bounded by servers × sync interval, and you can state the bound.
  • —A customer spreading across regions gets somewhat more than their quota.
B
Shared counter, checked per request
  • —Accurate within a region, and easy to explain to a customer.
  • —Adds a network hop to the most-executed code path in the system.
  • —The store becomes a hard dependency of every request.
  • —Still cannot be globally accurate: a cross-region hop is 150x the budget.
Which one: Local with async reconciliation. You have a lot of accuracy slack and none at all on latency, so spend the accuracy. The deciding argument is failure behaviour. With option B the limiter's outage is the API's outage, and a component built to protect availability must not reduce it.
Where the counter lives. The latency budget makes this decision for you.

Follow-ups you should expect

Many interviews are decided in the follow-ups, not the first diagram.

The cache just died. What happens?One key gets 90% of the traffic. What breaks, and what do you do?Two users click at the same millisecond. What happens?Why a queue instead of just calling the service directly?Which alert fires first when this system starts failing?A whole region is down. What do users see?There is an option that is 40% cheaper but 10% slower. Do you take it?10x traffic tomorrow. What fails first?

Same shape, other products

Deciding whether to say no, cheaply, on every request.

The decision has to be faster than the work it protects, which rules out most of the accurate answers.

Others with this shape: API rate limiter · Distributed counter · Fraud checks

Before you leave
Try this problem again in a week without reading anything. Comparing what you write then with what you wrote today shows you what stuck.
Stuck, or new to this?Read the full worked solutionAll five stages end to end, with the diagrams and the costs, written out the way you would say it. Best read after you have had a go.Read it →

How it is really built

Write your own answer first. Then read how teams who run this at scale describe it. Their constraints were real, and the write-ups often say what their choices cost.