Load balancing
One address in front, many machines behind, and a way to stop sending traffic to a broken one.
What problem does load balancing solve?
One server can only do so much, and one day it will restart or die. A load balancer lets you put several servers behind a single address, spread requests between them, and automatically stop using any server that stops answering.
It is also what makes deploys boring. Take one server out, update it, put it back, repeat.
What load balancing costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for load balancing, and when not to
Use it when
- You have more than one server. That is the whole condition.
- You want deploys and restarts with no downtime.
- You want broken machines removed automatically by health checks.
Do not, when
- You have one machine and downtime is acceptable. Then it is complexity for nothing.
- The thing behind it keeps important state per connection and cannot be changed. Fix that first, or accept sticky routing and what it costs.
How load balancing actually works
Layer 4 balancing looks only at IP and port. It does not read the request, so it is fast and works for any protocol. Layer 7 reads the HTTP request, so it can route by path or header, handle TLS, and retry a failed request on another server. Layer 7 is the default for web traffic. Layer 4 is for raw speed and for protocols that are not HTTP.
The algorithms, in the order you should reach for them. Round robin: fine when all requests are similar. Least connections: better when some requests take much longer than others, which is common in real systems. Consistent hashing on a key: use it when you want the same user or key to keep landing on the same server, usually so its cache stays useful.
Health checks do much of the useful work. A passive check notices failing responses. An active check calls an endpoint every couple of seconds. That endpoint should test something real, like whether it can reach the database. But not so real that one slow dependency makes every server mark itself broken at the same time and takes down the whole fleet. That failure is common and embarrassing.
Sticky sessions pin a user to one server. It solves in-memory session state, and it costs you: uneven load, a messy story when you add machines, and users losing their state when a server restarts. It is better to put session state in a shared store and keep every server interchangeable.
Above the load balancer sits DNS, which spreads traffic across regions or across several balancers. Clients cache DNS for minutes, so DNS on its own is a poor way to fail over. Say that if someone suggests it.
- Drawing one load balancer and calling the design highly available. That box is now the single point of failure.
- Forgetting connection draining, and dropping requests on every deploy.
- Reaching for sticky sessions instead of taking state off the servers.
The load balancing follow-up an interviewer will ask
"How do you deploy a new version with zero downtime?"
A rolling deploy through the load balancer. Take one instance out of the pool. Wait for the requests it is already handling to finish. That waiting step is called connection draining, and forgetting it is how you drop live traffic during a "zero downtime" deploy. Then update it, wait for its health check to pass, put it back, and move to the next one. Keep enough instances in the pool the whole time to carry the traffic. Two things make this work. First, the new version has to run happily beside the old one, which means database changes get split across releases: add the column, deploy, backfill, then remove the old column. Second, you need a fast way out, and rolling forward to a known-good build is safer than trying to undo a migration. Cost: every schema change now takes three deploys instead of one.
Your turn
Your health check endpoint queries the database. The database gets slow. What happens to your fleet, and what would you do differently?
Putting it together: the front door of a link shortener
Redirects have to answer in well under 200 ms, and the read path is about 4,000 requests a second with peaks near 12,000. Creating a link is rare, around 40 a second, and can afford to be slower. How should traffic reach the servers?
- 1Pick the layer
Layer 7, because the read path and the write path want different treatment:
/{code}should go to the redirect pool and/api/createto the creation pool. A Layer 4 balancer cannot tell those apart, because it never reads the path. - 2Pick the algorithm
Redirects are uniform: every one is a cache lookup and a 302, a millisecond or two. Round robin handles that case well. But creation involves a database write with a possible retry on collision, so those requests vary. Least connections across both pools is the safer single choice, and it costs almost nothing.
- 3Health checks
A
/healthzendpoint that returns 200 if the process is up. It deliberately does not check Redis or the database. If the cache is down, redirect servers can still serve from the database, and I do not want the balancer removing every server because one dependency is slow. - 4Keep servers stateless
Redirect servers hold only a small in-process cache of hot codes. Nothing user-specific lives in memory, so no sticky sessions are needed and any server can serve any request. Losing one costs a small cache warm-up, nothing more.
- 5Make the balancer redundant
Two balancers, or a managed one. Drawing a single box in front of the fleet would make the component I added for availability the thing most likely to cause an outage.
- 6Handle deploys
Remove an instance from the pool, drain in-flight requests, update, wait for the health check to pass, put it back. Without draining, every deploy drops the requests that were mid-flight, which is invisible in staging and obvious in production.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Google SRE Book — Load Balancing at the Frontend ↗How this works at a scale where the load balancer is itself a distributed system.
- nginx — HTTP Load Balancing ↗Round robin, least connections and hashing as real configuration directives.