Write-ahead logs
Write down what you are about to do, before you do it, so a crash cannot lose it.
What problem does write-ahead logs solve?
Changing data in place is not all-or-nothing. The machine can lose power halfway through, leaving something that is neither the old version nor the new one. That is corrupted, not merely out of date.
A write-ahead log fixes it. You record the change you intend to make into an append-only file, and only then apply it. After a crash you replay the log and end up in a sensible state.
What write-ahead logs costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for write-ahead logs, and when not to
Use it when
- Anything that must survive a crash without corruption. This is why every serious database has one.
- You want sequential writes instead of scattered ones. Appending is much faster on spinning disks, and still faster on SSDs.
- You need a stream of changes for replication or change data capture. The log already is that stream.
Do not, when
- Data you can afford to lose or rebuild: a cache, derived state, a scratch index. Paying twice for safety you do not need is waste.
- You are using a database. It already has one. You are not building this yourself.
How write-ahead logs actually works
The rule is in the name. The log entry must be safely on disk before the change it describes is applied. Get that order wrong and the log protects nothing.
Recovery works like this. On restart, find the last checkpoint, replay every log entry after it, and throw away any half-written entry at the end. That last part is why entries carry a checksum. Committed transactions are redone. Uncommitted ones are dropped.
Checkpointing is the knob you turn. A checkpoint flushes the current state to the data files and marks that point in the log, so everything before it can be deleted. Frequent checkpoints mean fast recovery and steady disk work. Rare ones mean fast writes day to day, and a restart that can take a long time. Name that trade, because "how long does recovery take" is an operational question that is easy to forget.
The word to be precise about is fsync. A normal write hands the data to the operating system, which may hold it in memory for a while. That is fast, and lost in a power cut. fsync forces it onto the physical device and is far slower. Group commit is the standard trick: batch many transactions into one fsync, so overall throughput rises and each individual transaction waits a little longer.
The log is useful beyond crash recovery, and this is the part to carry into other designs. It is an ordered record of every change. Ship it to another machine and that is replication. Read it and that is change data capture. Event sourcing is the same idea moved up to the application level: store the events, work out the state from them. LSM-tree storage engines go further. Writes go to the log and to an in-memory table, which is later flushed into sorted files. That makes writes sequential and fast, at the cost of reads having to check several files. Per-file bloom filters let a read skip most of them.
- Describing the log but applying the change first. The mechanism depends on that order.
- Confusing a write with a durable write. Without fsync, "committed" means "in memory somewhere".
- Forgetting checkpointing, then being surprised that recovery takes 40 minutes.
The write-ahead logs follow-up an interviewer will ask
"How much data can you lose if the machine loses power right now?"
Whatever was confirmed but not yet fsynced. So the answer depends on a setting, not on the architecture. If I fsync the log on every commit, I lose nothing that was confirmed, at the cost of a real disk sync per transaction, which limits write throughput. If I fsync on a timer, say every 100 ms, writes are much faster and I can lose up to that window of confirmed writes in a power cut. Group commit gets me most of both: batch the transactions happening at the same moment into one fsync, so throughput is high and each commit still waits for a real sync. The point I would make in an interview is that this is a product decision, not a database one. For payments I take the fsync-per-commit cost. For view counts I absolutely do not. And on a single machine, an fsynced log still does not survive the disk itself failing. That needs the log copied to another machine before confirming, which trades latency for surviving hardware loss instead of only power loss.
Your turn
Why write everything twice? Explain the benefit in one sentence.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- PostgreSQL — Write-Ahead Logging ↗WAL as actually implemented, including why it makes both durability and replication possible.