Load balancing — in depth

The first decision is how much of the request the balancer looks at, and it changes what it can do for you.

Layer 4 works at the TCP level. It sees IP addresses and ports, and does not open the request. It picks a server and copies bytes between the two connections. Because it never parses anything, it is very fast and works for any protocol: HTTP, database connections, game traffic.

Layer 7 reads the HTTP request itself: the path, the headers, the cookies. That means it can send /api to one pool and /images to another, terminate HTTPS, add headers, and retry a failed request on a different server because it still has the request in hand. It costs more CPU per request, because it has to parse and re-serialise, and it only works for protocols it understands.

The simplest possible rule: keep a list of servers and hand each new request to the next one in the list, wrapping around at the end. Server 1, 2, 3, 1, 2, 3.

It works well when two things are true: every server is the same size, and every request costs about the same. Then taking turns spreads load evenly and costs nothing to compute.

It falls apart when requests differ. Imagine one request in twenty is a report that takes ten seconds while the rest take 20 ms. Round robin cheerfully sends the next request to a server already grinding on a report, because it is that server's turn. It has no idea anyone is busy. It counts requests, not work.

Same rotation, but each server gets a weight and receives requests in proportion. A machine with weight 3 gets three requests for every one that a weight-1 machine gets.

This exists because real fleets are rarely uniform. You add newer, bigger instances over time, or you run a mix of instance types to control cost. Without weights, your biggest machine sits at 20% while your smallest is on fire.

It is also the mechanism behind gradual rollouts: give the new version weight 1 against the old version's weight 99, watch the error rate, then shift the weights. That is a canary deploy, implemented entirely in the load balancer.

Instead of taking turns, send each request to whichever server currently has the fewest open connections. It is a live measure of how busy each server is, rather than an assumption that they are all equal.

This handles the case round robin cannot. When a server picks up a slow request, its connection count stays high, so the balancer naturally routes around it until it catches up. Nobody had to configure anything; the metric corrects itself.

For most web traffic, where most requests are fast and a few are much slower, this is the sensible default. Least response time is a refinement that also weighs how quickly each server has been answering, which catches a server that is degraded but still accepting connections.

Hash something about the request (the client IP, a user id, a cache key) and use the hash to pick the server. The same input always produces the same server.

The reason to want this is cache locality. If each server keeps an in-memory cache, sending the same user to the same server means their data is probably already there. Without it, every server ends up caching everything, which wastes memory and lowers hit rates.

Plain modulo hashing has a serious flaw: hash(key) % N changes for almost every key when N changes. Add or remove one server and nearly every user moves, so every in-memory cache is suddenly wrong at the moment you were trying to add capacity.

Consistent hashing fixes this. Servers are placed around a circle, a key belongs to the first server clockwise from it, and adding a server only reassigns the slice behind it: about 1/N of keys, not all of them.

A balancer is only useful if it stops sending traffic to broken servers, and health checks are how it knows. There are two kinds.

Passive checks watch real traffic: if a server starts returning errors or timing out, mark it down. Free, since it uses requests you were sending anyway, but it only notices after real users have been affected.

Active checks call a dedicated endpoint every couple of seconds. They catch a sick server before users do, at the cost of constant background traffic.

Now the failure that turns a small problem into a total outage. Suppose your health endpoint queries the database to prove the server "really works". The database gets slow. Every health check on every server times out at once. The balancer marks the entire fleet unhealthy and removes all of it. Now nothing is serving, including all the requests that never needed the database.

The fix is to separate two questions. Liveness: is this process alive and able to answer? Cheap, no dependencies, and the balancer uses this one. Readiness: is this instance ready for traffic, including its dependencies? Useful at startup and for orchestration, but dangerous as the sole routing signal. And a good balancer can be told not to remove every instance. If everything looks unhealthy, it keeps serving anyway (some call this fail-open), because degraded beats dead.

A sticky session pins a user to one server, usually with a cookie the balancer sets. Every request from that user goes back to the same place.

People reach for it when servers keep session state in memory: logged-in user data, a shopping cart, a partly completed form. If request two lands on a different server, that state is not there, and the user gets logged out.

It works, and it costs you more than it looks. Load becomes uneven, because users are not evenly distributed and long-lived sessions accumulate on whichever servers have been up longest. Scaling out helps less, because existing users stay pinned to existing servers and only new users reach the new ones. And when a server restarts, which happens on every deploy, everyone pinned to it loses their state at once.

The better answer is nearly always to make servers stateless: put session data in a shared store like Redis, or in a signed token the client carries. Then any server can handle any request, restarts cost nothing, and the balancer is free to route on actual load.

This is a frequent omission in interview diagrams. Someone draws one load balancer in front of five servers and calls the design highly available. It is not: those five servers are now protected by one box, and if it dies, all of them are unreachable.

Real setups run at least two, and there are three ways to make that work. Active-passive with a floating IP: a standby takes over the address when the primary fails. DNS round robin across several balancers: crude, and DNS caching makes failover slow. Anycast: several locations announce the same IP address and the network routes to the nearest one still announcing it. Failover is as fast as routing converges, and large providers use it for their edge.

In practice, if you are on a cloud provider, the managed load balancer is already a redundant distributed system, and the right answer in an interview is to say that explicitly rather than to draw a second box.

The points that score.