Backups, disaster recovery, RPO and RTO
Two numbers: how much data you can lose, and how long you can be gone.
What problem does backups solve?
Replication is not a backup. A replica faithfully copies your mistakes (a bad migration, a wrong DELETE, a bug that corrupts rows) to every copy, within seconds. Backups exist for the failures replication makes worse, not better.
And the question that matters is not "do you have backups". It is "have you restored one", because an untested backup is a belief, not a plan.
What backups costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for backups, and when not to
Use it when
- Any data whose loss would be a serious event. Which is most of it.
- Whenever you are asked what happens if something is deleted or corrupted rather than merely down.
- Before promising an availability number. Recovery time is part of it.
Do not, when
- Data you can rebuild from a source of truth: caches, search indexes, derived aggregates. Back up the recipe, not the result.
- Do not treat replicas or snapshots-only as a backup strategy. Neither survives a logical error.
How backups actually works
RPO is set by how often you capture. RPO of five minutes needs continuous log shipping, not a nightly snapshot. RTO is set by how fast you can restore and cut over, and it is usually longer than people guess, because they have not timed it.
Get the two terms right, because interviewers use them. RPO (recovery point objective) is how much data you are willing to lose, measured in time. A nightly backup means an RPO of up to 24 hours. RTO (recovery time objective) is how long you are willing to be down while you recover. They are separate decisions with separate costs, and both should come from the business rather than from you.
What replication does and does not cover. Replication protects against a machine or a datacenter failing. It does not protect against anything logical, because a replica applies whatever the primary did, faithfully and immediately. A DELETE without a WHERE clause reaches every replica in milliseconds. So the failures backups exist for are: someone ran the wrong statement, a deploy corrupted data, a bug wrote nonsense for six hours before anyone noticed, or an attacker deleted things on purpose.
That last case is why backups must be isolated. If the credentials that run your application can also delete your backups, then a compromise takes both. Backups belong in a separate account or project, ideally write-once for a retention period, and the restore path should not depend on the system that is down.
The layers, in increasing cost. Snapshots are a point-in-time copy: cheap, easy, and your RPO is the snapshot interval. Continuous log archiving ships the write-ahead log to storage as it is produced, which enables point-in-time recovery: restore last night's snapshot, then replay the log up to 14:29, one minute before the bad statement. That combination is what gets an RPO down to minutes, and it is the answer to "someone dropped a table at half past two".
Retention has a shape, not a single number. Frequent recent backups, thinning out with age: hourly for a day, daily for a month, monthly for a year. The reason to keep old ones is the failure that is discovered late: corruption that started six weeks ago, unnoticed, which every recent backup faithfully contains.
Testing is what makes a backup real. A backup that has never been restored is unverified. The test has to be a real restore into a real environment, timed, because the timing is your RTO and it is usually worse than the estimate. Restore regularly and automatically, verify row counts and spot-check content, and record how long it took. Saying "we restore to staging weekly and that is how I know the RTO is 90 minutes" is far stronger than any backup schedule.
Two more that come up. Deletes should be soft first for anything a user can trigger (mark it deleted, purge it later), so an accidental deletion is a database update rather than a restore. And derived data does not need backing up: a search index, a cache, an aggregate table can all be rebuilt from the source, so back up the source and make sure the rebuild path is something you have actually run.
- Calling replication a backup. It replicates the mistake too.
- Backups reachable with the same credentials as the application.
- Never having run a restore, so the RTO is a guess.
- A nightly snapshot with no log archiving, then claiming an RPO of minutes.
The backups follow-up an interviewer will ask
"At 2:30pm someone runs a DELETE without a WHERE clause on your main table. What happens?"
Replication does not help me here. Every replica has already applied it, within milliseconds. So this is a backup question, and specifically a point-in-time recovery question. What I want is last night's snapshot plus the write-ahead log archived continuously since then, so I can restore to 14:29 and lose one minute of writes rather than a day. The part that decides whether this is a bad afternoon or a catastrophe is what I do next, and it is not restoring straight over production. I restore into a separate instance, verify the data looks right, and then decide how to bring it back, usually by copying the affected table across rather than swapping the whole database, because everything else that happened between 14:30 and now is real work I do not want to throw away. That reconciliation is the hard part, and it is where the time goes. I would also give realistic numbers: a multi-terabyte restore takes hours, so my RTO here is hours, not minutes, and if the business needs better than that the answer is not a better backup, it is preventing the event: production access through tooling rather than a direct console, statements requiring a WHERE clause, soft deletes so this is an UPDATE I can undo, and permissions that make a full-table delete something most people cannot do at all.
Your turn
You have three replicas across two regions. Why do you still need backups? Give a specific failure.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Google SRE Book — Data Integrity ↗Why a backup you have never restored is not a backup, and how to build restore testing in.