Replication — in depth
The summary said "keep copies and decide who is allowed to write". That is true and it hides every decision that matters. This page covers the three topologies and when each is right, the difference synchronous replication makes to your data loss window, what happens second by second during a failover, split brain and how fencing prevents it, replication lag and the four ways to hide it from users, and the failure modes that only appear at three in the morning.
The three topologies, and which one you want
Every replication scheme is one of three shapes, and the difference between them is where writes are allowed. That one question decides whether conflicts are possible at all.
Single leader allows writes in exactly one place. Multi-leader allows writes in several places, usually one per region. Leaderless allows writes to any node and uses quorums to keep them consistent. Most designs should use the first one, and the interesting part is being able to say why the other two are not needed.
- Single leader: no conflicts are possible by construction, because there is only one writer. This is the default and it is what a normal relational database gives you.
- Multi-leader: local writes are fast in every region, and now the same row can be edited in two places, so you own a conflict problem forever.
- Leaderless: no leader to fail over, tunable durability, and conflict handling is pushed onto the application.
- The write ceiling is the same for all three: replication never adds write capacity, because every replica does every write.
- If you cannot name a specific reason single leader fails you, single leader is the answer.
| Single leader | Multi-leader | Leaderless | |
|---|---|---|---|
| Write conflicts | Impossible | Your problem, forever | Your problem, tunable |
| Write latency | Remote users pay the trip | Local everywhere | Local-ish, quorum-bound |
| Failover | Needed, and risky | Not needed | Not needed |
| Operational load | Low | High | Medium |
| Right for | Almost everything | Offline-first, per-region data | Very large key-value stores |
Synchronous, asynchronous, and the middle ground
This is the most consequential knob in replication, because it decides how much confirmed data you lose when the leader dies.
Asynchronous: the leader confirms the write as soon as it is durable locally, and ships it to followers afterwards. Writes are fast. If the leader dies before shipping, whatever it had not sent is gone, and it was confirmed to a user. That is the uncomfortable part.
Synchronous: the leader does not confirm until at least one follower has the write too. Losing the leader alone no longer loses confirmed writes. But now every write waits for the follower, and if that follower is slow or down, writes stop entirely. A fully synchronous setup has made your availability worse, not better.
Semi-synchronous is the common production choice: one follower synchronous, the rest asynchronous. You get durability against losing the leader without depending on all of them, and if the synchronous follower fails, another is promoted into that role.
- Asynchronous replication means your data loss window equals your replication lag. Say that number out loud.
- Fully synchronous replication to all followers means the slowest follower sets your write latency and any follower being down stops writes.
- Semi-synchronous (one synchronous, the rest asynchronous) is the default worth proposing.
- Cross-region synchronous replication costs 80 ms or more per write, permanently. That is a product decision, not a configuration one.
- A write confirmed to a user and then lost in failover is the worst outcome in this topic, because there is no error anyone can see.
The third row is the one people propose without pricing. It is not a failure-time cost. You pay it on every write, on every good day, for the life of the system.
Failover, second by second
Failover is not a switch. It is a sequence, and every step in it has a cost that shows up as downtime or as lost data. Talking through it step by step shows you know where the downtime and the data loss come from.
The sequence: the leader stops responding. Something notices, after a detection timeout. A new leader is chosen. The rest are told. Clients discover the change. Writes resume. Then, later and much harder, the old leader comes back and has to be dealt with.
- Detection has a timeout, and that timeout is pure downtime. Too short and a network blip triggers a failover; too long and, after a real failure, you stay down for the whole timeout.
- Choosing a new leader must be done by majority agreement, or two nodes can both believe they won.
- Clients must be redirected, through a proxy, a virtual IP, or a service discovery layer. DNS is usually too slow, because clients and resolvers cache records.
- Anything the old leader accepted and had not replicated is gone. Someone has to decide whether that is acceptable, in advance.
- The old leader returning is the dangerous moment, not the failure itself.
Split brain, and how fencing stops it
Split brain is the failure that makes people afraid of automatic failover, and it is worth understanding precisely rather than as a scary phrase.
The setup: the leader is healthy but the network between it and the rest is broken. From the cluster's point of view the leader is dead, so it promotes a new one. From the leader's point of view everything is fine and it keeps accepting writes. Now two nodes are taking writes for the same data, and the histories diverge in ways that cannot be merged afterwards.
Two defences, and you need both. Majority: only a node that can reach more than half the cluster may be leader, so the isolated side stops accepting writes on its own. Fencing: every leadership term gets an increasing number, every write carries it, and the storage layer refuses any write with an old number. Majority prevents the promotion; fencing catches the case where the old leader was paused rather than partitioned and wakes up mid-write.
- Majority alone is not enough, because a paused process (a long garbage collection, a frozen VM) can resume believing it is still leader and still within its lease.
- A fencing token is a number that only goes up, presented with every write and checked by the thing being written to.
- If you cannot fence, making the operation safe to repeat limits the damage from a duplicate write. It is usually easier, though it does not stop two leaders writing different values.
- Even-numbered clusters are worse than odd ones: four nodes tolerate the same single failure as three, while being slower.
Replication lag and the four ways to hide it
Lag is not a bug you fix, it is a property you design around. Asynchronous replication means followers are always behind, usually by milliseconds and occasionally by minutes when something is wrong.
Users notice lag in three specific ways, and each has its own fix. They do not see their own write. They see time go backwards between two page loads. Or they see an effect before its cause, such as a reply above the comment it answers.
- Read your own writes: route a user to the leader for a short window after they write, or track the write position and read from a replica that has caught up to it.
- Monotonic reads: pin a user to one replica so they never see a value older than one they already saw.
- Consistent prefix: route causally related writes through one partition, so a reply cannot arrive before the comment.
- The cheapest fix of all: render what the client already has instead of re-fetching. No replica involvement at all.
- Alert on lag, because a lag of 30 seconds is usually a symptom: a long transaction, a bulk write, or a follower that cannot keep up.
- —Simple: mark the session for ten seconds and send its reads to the leader.
- —Works for every read that user makes, including ones you did not anticipate.
- —Leader read load rises, driven by your most active users.
- —The window is a guess. Too short and the bug returns; too long and the leader carries more.
- —The client carries the position it saw; reads wait for a replica at least that current.
- —Precise: no guessed window, and no unnecessary leader load.
- —Reads can now block, which is a new latency risk on a path that used to be fast.
- —Every client and every read path has to carry and honour the token.
The failure modes that get missed
These are the ones that produce three-in-the-morning incidents, and mentioning even one shows you have thought past the happy path.
- A follower falls so far behind that the leader has discarded the log it needs. Its only recovery is a full rebuild from a snapshot, which takes hours, not minutes.
- A long-running read on a replica can block replication from applying, so the replica falls further behind just while someone is running a report on it.
- Failover promotes a follower that was behind, so the new leader is missing writes the old one had confirmed. This is silent unless you are checking.
- The read replica everyone uses for analytics is also the semi-synchronous one, so a heavy query slows down every write in the system.
- Schema changes replicate too, and a change that is fast on the leader can lock a follower for minutes.
- Adding a replica puts real load on the leader while it streams the initial snapshot. Doing it during peak makes an existing problem worse.
A user posts a comment and it disappears
Support has 40 tickets this week saying "I posted a comment and it vanished, then came back later". Your database is one leader with three asynchronous read replicas, and all reads go to replicas through a load balancer. Nothing is down. Every dashboard is green.
- 1Reproduce the shape, not the bug
It is not reproducible on demand, which is itself the clue: it depends on timing rather than on data. The write goes to the leader; the refresh a moment later is served by whichever replica the load balancer picked. If that replica has not applied the write yet, the comment is not there. It appears "later" because a subsequent refresh happens to land on a caught-up replica.
- 2Check the lag, and check its shape
Median lag is 15 ms, which is fine. But p99 lag is 4 seconds, and there are spikes to 30. The average hid the problem entirely. The spikes line up with a nightly bulk import and with a long analytics query someone runs on one replica, which blocks that replica from applying replication while it runs.
- 3Fix the user-visible symptom first
Pin a user to the leader for ten seconds after they write. Two lines in the routing layer, and it removes this class of complaint immediately. Measure the extra leader read load before and after. It is small, because only actively-writing users are affected, and they are a minority at any moment.
- 4Then fix the cause of the spikes
Move the analytics query to a dedicated replica that no user traffic touches, so a heavy report can no longer delay replication for real readers. Batch the nightly import with pauses so it does not saturate the replication stream. Both are cheap and both remove the tail rather than hiding it.
- 5Make it impossible to miss next time
Alert on p99 replication lag, not on replica health. A replica that is up and 30 seconds behind looks perfectly healthy on every dashboard while serving wrong answers. That gap between "healthy" and "correct" is what let 40 tickets accumulate before anyone looked.