Latency vs throughput
How long one thing takes, versus how many things you finish per second.
What problem does the throughput argument solve?
Latency is how long a single request takes, measured in milliseconds. Throughput is how many requests you finish per second.
They sound like the same thing said two ways. They are not. Many engineering decisions improve one by making the other worse.
Confusing them means you improve the wrong number, then wonder why nobody is happier.
What the throughput argument costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for the throughput argument, and when not to
Use it when
- Deciding whether to batch. Batching raises throughput and usually adds latency to each item.
- Setting a target. "Fast" is not a target. "p99 under 200 ms at 5,000 requests a second" is.
- Reading a benchmark someone hands you. Check which of the two it measured.
Do not, when
- Do not chase throughput on a path where one person is waiting. They cannot feel your requests per second. They feel their one request.
How the throughput argument actually works
Adding lanes raises throughput and does nothing for your 30 minutes. Raising the speed limit cuts latency for everyone. They are separate levers.
Averages lie, and this is the part worth taking to heart. Nobody experiences your average latency. Say your average is 100 ms, but one request in a hundred takes four seconds, and a page makes twenty requests. Then about one page load in five (1 − 0.99²⁰ ≈ 18%) contains at least one slow request. So you quote percentiles instead. p50 is the middle. p99 is the request that is worse than 99% of the others. p99 is what your unhappiest users live in, and it is the number worth putting on a dashboard.
The trade in concrete terms. Batching a hundred database writes into one statement raises throughput a lot, because you pay the round trip once instead of a hundred times. It also raises latency for the first write in the batch, which now waits for the other ninety-nine to arrive. That is a good trade for analytics and a terrible one for a checkout button.
One more relationship worth knowing. As a system nears its throughput limit, latency does not get slowly worse. It stays flat, flat, flat, and then climbs almost straight up, because requests start queueing behind each other. That is why "we are at 85% capacity" is much closer to trouble than it sounds.
- Quoting averages. They hide the tail. Quote p50 and p99.
- Chasing throughput on a path where a person is waiting.
- Assuming latency gets slowly worse as you near capacity. It does not. It hits a wall.
The the throughput argument follow-up an interviewer will ask
"Your p50 is 20 ms and your p99 is 3 seconds. What is going on?"
A gap that big is not the same work running slower. It is a different path, or a different shape of data. The usual causes, in order. Skewed data, where a few users have far more rows than a typical user, so their queries run differently. Queueing, where requests wait on something that has run out, like a connection pool. That shows up as two clusters of response times rather than a smooth curve. A dependency with its own bad tail that I call and wait for. Or garbage collection pauses. I would first check whether the response times form two clusters, because that points straight at queueing, and then trace one slow request end to end. There is also a structural cause worth naming. If a request fans out to many services and waits for all of them, it is as slow as the slowest reply, so each backend's rare slow tail shows up in a much larger share of my requests.
Your turn
You batch 100 writes into one database call. What happened to latency, and what happened to throughput?
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Google SRE Book — Monitoring Distributed Systems ↗The four golden signals: latency, traffic, errors, saturation. Where the distinction earns its keep.
- Brendan Gregg — The USE Method ↗Utilisation, saturation, errors. The method for finding which resource is actually the bottleneck.