Consensus, at concept level
How a group of machines agrees on one answer when some of them cannot be reached.
What problem does consensus solve?
Some decisions must have exactly one answer across the whole cluster. Who is the leader. Which config is live. Who holds the lock. Ask each machine on its own and they will disagree, especially during the failure you built this for.
Consensus algorithms such as Raft and Paxos solve that. You will never implement one. You need to know what they give you, what they cost, and when to reach for something that already has one built in.
What consensus costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for consensus, and when not to
Use it when
- Electing a leader, and failing over.
- Distributed locks and leases, where two owners would be a disaster.
- Configuration and cluster membership that everyone must agree on.
- Small, critical metadata, like which shard lives where.
Do not, when
- High-volume application data. Do not push a million writes a second through a consensus group. It is the wrong tool and it will not work.
- Anything where a slightly wrong answer is fine. Use something cheaper.
- When a single database with a unique constraint already gives you the guarantee. It does, and it is far simpler.
How consensus actually works
What Raft does, in plain terms. One node is the leader. All decisions go through it, and it adds them to an ordered log. It sends each entry to the others. Once a majority has written it down, the entry is committed and can never be lost or reordered. If the leader stops sending heartbeats, the others wait a random moment, and one of them stands for election. Whoever gets votes from a majority becomes leader. Each node votes once per election term, and a majority is more than half, so two leaders cannot be elected in the same term.
What it gives you: one ordered list of decisions that survives any minority failing, and no split brain. What it does not do: work when more than half are down, go faster with more machines, or handle high write volume.
The practical version of all this is: do not build it. Use etcd, ZooKeeper, or the consensus already inside your database or your orchestrator. The right answer to "how do you elect a leader" is usually "I would keep the leadership lease in etcd", not a description of an algorithm.
Distributed locks deserve a warning. A lock held by a process that then pauses, for garbage collection or a slow disk, can expire while that process still believes it holds it. Now two workers are doing the same job. Leases with an expiry help, but they do not remove the problem. The solid pattern is a fencing token. Every time the lock is taken, it returns a number that only goes up. The process sends that number with each write, and the storage layer rejects any write carrying an old number. If you cannot fence, make the operation safe to repeat, so doing it twice is harmless. That is usually easier, and it is the better answer in most interviews.
- Offering to implement Raft. Nobody wants that. They want to know you would use something that already has it.
- Pushing application traffic through a consensus store.
- Trusting a lock with no fencing token and no idempotency behind it.
The consensus follow-up an interviewer will ask
"Why not just use a lock in Redis?"
For many jobs that is fine, and I would say so rather than over-build. A single Redis with SET NX and an expiry gives you cheap mutual exclusion, and if the worst case of two workers running is a duplicate email, take it. The reasons to reach for something stronger: A single Redis is a single point of failure, so if it dies the lock is gone. The multi-node version makes assumptions about clocks that people argue about. And expiry-based locks can be held by two processes at once, when one pauses long enough for its lease to run out without it noticing. So my rule is this. If two holders means a duplicate side effect I can make safe to repeat, Redis is fine and simpler. If two holders means corrupted data or a double charge, I want a real consensus store and a fencing token, and I accept the extra latency and the extra system to run.
Your turn
You have a 5-node consensus cluster. Two nodes fail. Does it still work? What if three fail?
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- The Raft Consensus Algorithm ↗The paper, plus a visualisation you can step through leader election with.