Latency numbers worth memorising
Four numbers that tell you where the time goes.
What problem does the latency numbers solve?
When work moves from one machine to another, you pay for the distance. Most slow designs are slow for one reason. Something crosses a boundary far more often than anyone noticed.
You do not need a table of thirty numbers. You need four. And you need to feel the gaps between them, because the gaps are what change your design.
What the latency numbers costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for the latency numbers, and when not to
Use it when
- You are deciding whether to make a call inside a loop, or fetch everything in one go.
- You are choosing between a cache, a disk read, and a network call.
- Someone asks "is this fast enough?" and you want to answer with maths instead of a feeling.
- You need to explain a decision about serving users in another region.
Do not, when
- You have a real profiler and real traffic. Then measure. These numbers stand in for measurement, they do not replace it.
- Someone wants a number for an SLA. Do not quote these as a promise.
How the latency numbers actually works
The jump that matters is the last one. Anything that crosses an ocean, you get to do once per user action. Not once per item in a list.
Four numbers, all in nanoseconds so they compare cleanly. Reading from memory: about 100 ns. Reading from an SSD: about 100 µs, so a thousand times slower. A round trip between two machines in the same datacenter: about 0.5 ms, so five thousand times slower than memory. A round trip between continents: about 150 ms, because light in fibre only moves so fast and cables rarely run in a straight line.
That last number has a floor you cannot remove. London to Sydney and back is about 17,000 km each way. Light in glass covers about 200,000 km per second. So that is already 170 ms of pure physics, before any machine does any work. No amount of money removes it. You can only move the data closer, or stop needing the round trip at all.
This gives you a simple rule. One cross-region hop per user action is a design. Ten is a bug. If you are making a remote call inside a loop over 50 items, that loop is your whole time budget.
- Quoting numbers with fake precision. "About half a millisecond" is right. "0.47 ms" is a claim you cannot back up.
- Forgetting that a round trip crosses the distance twice. People quote the one-way number and end up half wrong.
The the latency numbers follow-up an interviewer will ask
Your page makes 30 backend calls, and users on the other side of the world complain. Where do you look first?
Look at two things. Are those 30 calls one after another? And do any of them cross a region? 30 calls at 0.5 ms each inside one datacenter is 15 ms, which users will not notice. The same 30 calls, each waiting on a 150 ms hop across the world, is 4.5 seconds. So the fix is not "make the calls faster". The fix is "stop making them one after another, and stop making them travel". Group them into one call, run them at the same time, or put a read replica near the user. Name the cost too. A replica near the user can be out of date, and grouping calls means one slow item delays the whole group.
Your turn
Roughly how many times slower is a round trip across the world than a read from memory — 100x, 1,000x, or a million times? And what does that ratio stop you doing?
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Latency Numbers Every Programmer Should Know ↗Jonas Bonér's gist of the numbers popularised by Jeff Dean and Peter Norvig. Most copies of this list trace back to it.
- Interactive latency numbers by year ↗The same numbers on a slider by year. Worth it to see which ones stopped improving.