Change data capture
Read the database's own log to find out what changed, instead of asking the application to tell you.
What problem does change data capture solve?
Several systems usually need to know when data changes: a search index, a cache, a warehouse, another service. Making the application notify all of them means every write path has to remember, and one that forgets creates a mismatch that can go unnoticed for months.
CDC takes the changes from the database's replication log instead. Nothing can be missed, because a change that is not in the log did not happen.
What change data capture costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for change data capture, and when not to
Use it when
- Keeping a search index in step with a database.
- Clearing caches reliably, instead of hoping every write path remembers.
- Feeding a data warehouse without nightly full exports.
- Getting events out of a system you cannot change: an old application, a vendor database.
- Moving between databases, running both at once, with no downtime.
Do not, when
- The consumer needs business events rather than row changes. "Order shipped" is a decision. Three UPDATE statements are not, and working out the intent from row differences is fragile.
- You need the change applied immediately, in the same request.
- One application owns the write path and can emit clean events through an outbox. That is simpler, and it expresses intent properly.
How change data capture actually works
Log-based CDC is the good version. A connector pretends to be a replica and reads the write-ahead log, so it sees every change, including ones made by a script someone ran by hand. It puts little load on the database, and it is complete by design.
The alternatives are worse, and worth being able to dismiss. Polling a modified-at column misses deletes completely, sees only the last version of a row updated twice between polls, and puts load on the database. Triggers that write to an audit table run inside the transaction, so they slow every write and can fail a transaction that would otherwise have succeeded.
CDC against the outbox pattern is a common interview comparison. The outbox gives you planned business events, in a stable shape you control, and it needs the application to write them. CDC gives you every change with no application involvement, at the price of exposing your schema and losing intent. You can see status change from PAID to SHIPPED, but the reason lives in application code. My default: an outbox for events other services depend on, and CDC for keeping derived stores like search indexes and warehouses in step.
Two operational details come up. First, the initial load. A new consumer needs the existing data as well as the changes, so you take a snapshot of the table, note the log position at that moment, and stream from there. Getting that boundary wrong means missing or duplicating a window of changes. Second, retention. The log is finite, so a consumer that has been down for a day may find the data it needs has been deleted, and its only way back is a fresh snapshot. Alert on how far consumers are behind compared to retention, not just on how far behind they are.
Because consumers may see a change twice after a restart, they must be safe to repeat. For derived stores that is usually easy, because writing the current value of a row again changes nothing.
- Writing to the database and a second system separately. They will drift.
- Treating row changes as business events and guessing the intent from differences.
- No plan for a consumer that falls behind the log retention.
The change data capture follow-up an interviewer will ask
"How do you keep a search index in sync with your database?"
Not by writing to both from the application. That is the tempting answer, and it is wrong. The two writes are not all-or-nothing, so a crash between them leaves the index permanently wrong and nothing tells you. I would drive the index from CDC. The connector reads the database log, publishes changes to a stream, and an indexer consumes it. Since a change is in the log if and only if it committed, the index cannot miss anything, and the indexer is naturally safe to repeat, because it writes the whole current document. The costs I would name: the index lags the database by about a second, which is fine for search and not fine if a user expects to see their own edit immediately. For that case I would merge their pending change on the client. And schema changes now affect the indexer, so it has to cope with unknown and missing fields. I would also run a regular reconciliation job comparing counts and checksums, because long-running pipelines tend to drift somewhere, and finding it on purpose beats hearing about it from a user.
Your turn
Your application writes to the database and then updates the search index directly. Name what goes wrong and what you would do instead.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Debezium — Architecture ↗A widely used open-source CDC implementation, reading the database log directly.