Dev Learning

Replication

Keep the same data on several machines, and decide who is allowed to change it.

1

What problem does replication solve?

One copy of your data means one machine whose failure loses everything, and one machine that has to serve every read. Replication fixes both. Extra copies survive a failure, and they can answer reads.

All the difficulty is in writes. As soon as two copies can be written to on their own, they can disagree. So you have to decide in advance who wins.

2

What replication costs you

What it costs you
Copies take time to arrive, so a replica can serve data that is out of date. The user who just wrote something is the most likely person to notice. You also pay for the extra machines. And failover is hard to get right: promote too eagerly and you get two leaders, too slowly and you are down.

Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.

3

When to reach for replication, and when not to

Use it when

  • You need to survive a machine, or a whole datacenter, dying.
  • Reads outnumber writes and you want to spread them out.
  • You want users on another continent to read from a copy near them.

Do not, when

  • You are adding replicas to take more writes. Replication does not help writes. Every replica does every write. Partitioning is the tool for that.
  • Your application cannot tolerate old data anywhere, and you have not decided which reads must go to the leader.
4

How replication actually works

WriterLeaderFollowerbehindReader
write lands on the leader instantlycopy crawls to the followera read in that gap sees the old value
The write lands on the leader straight away. The copy takes time to reach the follower. A read in that gap sees the old value. This is replication lag, and it is behind a lot of "the app lost my change" reports.

Leader and follower is the default, and the one to reach for. All writes go to one leader. Followers copy from it and serve reads. It is simple, write conflicts cannot happen by design, and it is what a normal relational database does out of the box. The costs: the leader caps your write rate, and when it dies something has to notice and promote a follower. Writes fail while that happens.

Failover has one nasty failure worth naming: split brain. The old leader was not dead, only unreachable. Now there are two leaders taking writes, and they drift apart. The defence is a majority rule, where only a node that can talk to more than half the cluster may be leader. Add fencing, so the storage layer refuses the old leader when it comes back. Also, whatever the old leader accepted but had not yet copied is gone. Someone has to decide whether that loss is acceptable.

The main knob is synchronous versus asynchronous replication. Synchronous means the write is not confirmed until a follower has it. No data is lost on failover, but writes now wait for the slowest follower, and if that follower is down, writes stop. Asynchronous means fast writes and a window where data can be lost. The usual middle ground is semi-synchronous: one follower synchronous, the rest asynchronous. You get safety without depending on all of them.

Multi-leader means several leaders accept writes, usually one per region. Local writes are fast everywhere. But now the same row can be edited in two places at once, and you have to resolve that. Options: last-write-wins, which is simple but throws away someone's work without an error, and depends on clocks you cannot trust. Or a merge rule written by your application. Or a data type designed to merge cleanly. Use multi-leader when local write speed matters and conflicts are rare: collaborative editing, offline-first apps.

Leaderless means the client writes to several nodes and reads from several. With N copies, if writes go to W nodes and reads to R nodes, and W + R > N, then any read touches at least one node that saw the latest write. It is tunable and survives failures well, and it pushes conflict handling onto you. Repairs also happen lazily, so a node can serve old data until it is fixed.

Common trap
  • Assuming failover is automatic and instant. Somebody has to notice the failure, noticing has a timeout, and that timeout is downtime.
  • Drawing async replication and then promising no data loss.
  • Multi-leader with no plan for conflicts. That plan is the whole question.
Say this
"Single leader with two async followers, plus one semi-sync follower so a failover does not lose committed writes. Reads go to followers, except within ten seconds of a user's own write, which go to the leader. Cost: the leader is still my write ceiling, and failover means a few seconds of write errors."
Go deeper · 20 minView more — replication in depth6 sections, every sub-topic covered separately, with a worked example at the end.Read it →
5

The replication follow-up an interviewer will ask

"Replica lag is 30 seconds and a user reads their own write. What do they see?"

What a strong answer sounds like

Old data, and they will report it as a bug. They just posted a comment and it vanished. This is the read-your-own-writes problem, and the general fix is to send certain reads to the leader. In practice: after a user writes, mark their session for a short window and send their reads to the leader during it. Or record the write position at commit time, and make the read wait until a replica has caught up to that position. That is more precise, and it means reads can block. A cheaper trick for lists: show the user their own item from what they just submitted, instead of fetching it again. The cost of all of this is more read load on the leader, and it comes from your most active users. And I would alarm on lag, because 30 seconds is a symptom, usually a long transaction or a bulk write hogging the leader.

6

Your turn

60-second self-check

You add five read replicas. Your write throughput is still maxed out. Why did that not help?

Stuck? Take one hint at a time0 of 2 used
0 / 25 characters
Worked example

A user posts a comment and it disappears

Support has 40 tickets this week saying "I posted a comment and it vanished, then came back later". Your database is one leader with three asynchronous read replicas, and all reads go to replicas through a load balancer. Nothing is down. Every dashboard is green.

  1. 1
    Reproduce the shape, not the bug

    It is not reproducible on demand, which is itself the clue: it depends on timing rather than on data. The write goes to the leader; the refresh a moment later is served by whichever replica the load balancer picked. If that replica has not applied the write yet, the comment is not there. It appears "later" because a subsequent refresh happens to land on a caught-up replica.

  2. 2
    Check the lag, and check its shape

    Median lag is 15 ms, which is fine. But p99 lag is 4 seconds, and there are spikes to 30. The average hid the problem entirely. The spikes line up with a nightly bulk import and with a long analytics query someone runs on one replica, which blocks that replica from applying replication while it runs.

  3. 3
    Fix the user-visible symptom first

    Pin a user to the leader for ten seconds after they write. Two lines in the routing layer, and it removes this class of complaint immediately. Measure the extra leader read load before and after. It is small, because only actively-writing users are affected, and they are a minority at any moment.

  4. 4
    Then fix the cause of the spikes

    Move the analytics query to a dedicated replica that no user traffic touches, so a heavy report can no longer delay replication for real readers. Batch the nightly import with pauses so it does not saturate the replication stream. Both are cheap and both remove the tail rather than hiding it.

  5. 5
    Make it impossible to miss next time

    Alert on p99 replication lag, not on replica health. A replica that is up and 30 seconds behind looks perfectly healthy on every dashboard while serving wrong answers. That gap between "healthy" and "correct" is what let 40 tickets accumulate before anyone looked.

Result: The complaints stop within an hour of the leader-pinning change, and the underlying lag spikes are gone within a day. The lasting lesson is the monitoring one: replica health and replica correctness are different questions, and only one of them was on the dashboard.
7

Learn it from the source

This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.

Where you will use this