Replication
Keep the same data on several machines, and decide who is allowed to change it.
What problem does replication solve?
One copy of your data means one machine whose failure loses everything, and one machine that has to serve every read. Replication fixes both. Extra copies survive a failure, and they can answer reads.
All the difficulty is in writes. As soon as two copies can be written to on their own, they can disagree. So you have to decide in advance who wins.
What replication costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for replication, and when not to
Use it when
- You need to survive a machine, or a whole datacenter, dying.
- Reads outnumber writes and you want to spread them out.
- You want users on another continent to read from a copy near them.
Do not, when
- You are adding replicas to take more writes. Replication does not help writes. Every replica does every write. Partitioning is the tool for that.
- Your application cannot tolerate old data anywhere, and you have not decided which reads must go to the leader.
How replication actually works
Leader and follower is the default, and the one to reach for. All writes go to one leader. Followers copy from it and serve reads. It is simple, write conflicts cannot happen by design, and it is what a normal relational database does out of the box. The costs: the leader caps your write rate, and when it dies something has to notice and promote a follower. Writes fail while that happens.
Failover has one nasty failure worth naming: split brain. The old leader was not dead, only unreachable. Now there are two leaders taking writes, and they drift apart. The defence is a majority rule, where only a node that can talk to more than half the cluster may be leader. Add fencing, so the storage layer refuses the old leader when it comes back. Also, whatever the old leader accepted but had not yet copied is gone. Someone has to decide whether that loss is acceptable.
The main knob is synchronous versus asynchronous replication. Synchronous means the write is not confirmed until a follower has it. No data is lost on failover, but writes now wait for the slowest follower, and if that follower is down, writes stop. Asynchronous means fast writes and a window where data can be lost. The usual middle ground is semi-synchronous: one follower synchronous, the rest asynchronous. You get safety without depending on all of them.
Multi-leader means several leaders accept writes, usually one per region. Local writes are fast everywhere. But now the same row can be edited in two places at once, and you have to resolve that. Options: last-write-wins, which is simple but throws away someone's work without an error, and depends on clocks you cannot trust. Or a merge rule written by your application. Or a data type designed to merge cleanly. Use multi-leader when local write speed matters and conflicts are rare: collaborative editing, offline-first apps.
Leaderless means the client writes to several nodes and reads from several. With N copies, if writes go to W nodes and reads to R nodes, and W + R > N, then any read touches at least one node that saw the latest write. It is tunable and survives failures well, and it pushes conflict handling onto you. Repairs also happen lazily, so a node can serve old data until it is fixed.
- Assuming failover is automatic and instant. Somebody has to notice the failure, noticing has a timeout, and that timeout is downtime.
- Drawing async replication and then promising no data loss.
- Multi-leader with no plan for conflicts. That plan is the whole question.
The replication follow-up an interviewer will ask
"Replica lag is 30 seconds and a user reads their own write. What do they see?"
Old data, and they will report it as a bug. They just posted a comment and it vanished. This is the read-your-own-writes problem, and the general fix is to send certain reads to the leader. In practice: after a user writes, mark their session for a short window and send their reads to the leader during it. Or record the write position at commit time, and make the read wait until a replica has caught up to that position. That is more precise, and it means reads can block. A cheaper trick for lists: show the user their own item from what they just submitted, instead of fetching it again. The cost of all of this is more read load on the leader, and it comes from your most active users. And I would alarm on lag, because 30 seconds is a symptom, usually a long transaction or a bulk write hogging the leader.
Your turn
You add five read replicas. Your write throughput is still maxed out. Why did that not help?
A user posts a comment and it disappears
Support has 40 tickets this week saying "I posted a comment and it vanished, then came back later". Your database is one leader with three asynchronous read replicas, and all reads go to replicas through a load balancer. Nothing is down. Every dashboard is green.
- 1Reproduce the shape, not the bug
It is not reproducible on demand, which is itself the clue: it depends on timing rather than on data. The write goes to the leader; the refresh a moment later is served by whichever replica the load balancer picked. If that replica has not applied the write yet, the comment is not there. It appears "later" because a subsequent refresh happens to land on a caught-up replica.
- 2Check the lag, and check its shape
Median lag is 15 ms, which is fine. But p99 lag is 4 seconds, and there are spikes to 30. The average hid the problem entirely. The spikes line up with a nightly bulk import and with a long analytics query someone runs on one replica, which blocks that replica from applying replication while it runs.
- 3Fix the user-visible symptom first
Pin a user to the leader for ten seconds after they write. Two lines in the routing layer, and it removes this class of complaint immediately. Measure the extra leader read load before and after. It is small, because only actively-writing users are affected, and they are a minority at any moment.
- 4Then fix the cause of the spikes
Move the analytics query to a dedicated replica that no user traffic touches, so a heavy report can no longer delay replication for real readers. Batch the nightly import with pauses so it does not saturate the replication stream. Both are cheap and both remove the tail rather than hiding it.
- 5Make it impossible to miss next time
Alert on p99 replication lag, not on replica health. A replica that is up and 30 seconds behind looks perfectly healthy on every dashboard while serving wrong answers. That gap between "healthy" and "correct" is what let 40 tickets accumulate before anyone looked.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- PostgreSQL — High Availability and Replication ↗Synchronous versus asynchronous replication, and what you lose on failover.