Availability, failover and the nines
How much downtime you are promising, and what it costs to promise less.
What problem does failover solve?
Availability is the share of time your system answers. People say "highly available" as if it were a setting you turn on. It is a number, it has a price, and each extra nine usually costs far more than the one before it.
The other half is failover. When a machine dies, what takes over, how fast, and what gets lost in the handover?
What failover costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for failover, and when not to
Use it when
- Agreeing a target with the business, before you design anything.
- Deciding how much backup a specific path needs. Different paths should get different answers.
- Working out whether a dependency is allowed on your critical path at all.
Do not, when
- Do not promise nines you have not priced. "Five nines" is about five minutes of downtime a year, including deploys.
- Do not add backups everywhere equally. Some paths can be down for an hour.
How failover actually works
Five nines is five minutes a year in total, including every deploy and every dependency failure. Most products do not need it and cannot afford it.
There are two ways to arrange failover, and the difference matters. Active-passive: one machine serves, and a standby sits idle watching for a heartbeat. When the heartbeat stops, it takes over. It is simple, but you pay for hardware doing nothing, and the standby never sees real traffic. So you may only discover it does not work at the worst possible moment. Active-active: both machines serve traffic all the time, so both are exercised continuously. Better use of money, and no cold standby surprise. The cost is that you now have data being written in two places.
The maths of combining parts is worth knowing, because it is not obvious. Parts in sequence (a request must pass through all of them) multiply their availabilities. Two parts at 99.9% each give 99.8% together. So adding dependencies to a critical path always lowers availability. Parts in parallel (either one can serve) multiply their failure rates instead. Two 99.9% parts in parallel give 99.9999%, as long as they fail independently. Shared power, a shared network or a shared bad deploy breaks that assumption. This is the core argument for redundancy, and against long chains of dependencies.
Failover has a cost you cannot remove: noticing takes time. Something has to see that the machine is gone. If it decides too fast, it will fail over during a brief network hiccup and cause an outage. If it decides too slowly, you are down. That window is downtime you cannot design away. You can only make it shorter.
One thing teams often leave out of the availability budget: a large share of outages are not hardware. They are deploys and config changes. Redundancy protects you from a machine dying. It does nothing about shipping a bug to every machine at once.
- Promising nines without counting your dependencies. They cap you.
- An untested failover. If it has never been used, it is a guess.
- Forgetting that deploys and config changes cause a large share of outages, and redundancy helps with neither.
The failover follow-up an interviewer will ask
"You want four nines. What does that actually require?"
Fifty-two minutes of downtime a year, in total. To get there you need several things. No single point of failure anywhere on the critical path, which means at least two of everything, including the load balancer people forget about. Automatic failover that has actually been tested, because an untested failover is only a theory. Deploys that do not take the service down, so rolling releases and backwards-compatible migrations. And isolated dependencies, so a slow third party breaks one feature instead of the whole product. Then the uncomfortable part: my dependencies cap me. If I call a payment provider that offers 99.9%, my checkout path cannot beat 99.9%, no matter what I do. So either that call comes off the critical path, or four nines on checkout is not a promise I can make. I would also point out that much of my downtime is likely to come from deploys and config changes, not hardware. So the highest-value work is probably gradual rollouts and fast rollback, not more redundancy.
Your turn
Two services, each 99.9% available. What is the combined availability if a request needs both? What if either will do?
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Google SRE Book — Embracing Risk ↗Why 100% is the wrong availability target, and how error budgets replace it.
- PostgreSQL — High Availability ↗Failover and standby as concrete configuration rather than a diagram.