Distributed transactions: 2PC, sagas, outbox
When one action must change two systems, and no transaction covers both.
What problem does distributed transactions solve?
Inside one database, changing several rows together is a solved problem. That is what a transaction is. Across two databases, or a database and a payment provider, there is no such thing. So you can succeed at one half and fail at the other.
The result is money taken with no order, or an order with no stock. It is a common source of data that is wrong with no error to show for it.
What distributed transactions costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for distributed transactions, and when not to
Use it when
- Any workflow crossing two services or two datastores, where finishing halfway is not acceptable.
- Placing an order that touches payment, stock and delivery.
- Anything that writes to a database and then has to publish an event about it.
Do not, when
- Both writes could live in one database. Then use a transaction. The best answer to a distributed transaction question is often "I would not have one".
- The second effect is harmless if it never happens, like an analytics ping.
How distributed transactions actually works
Two-phase commit. A coordinator asks everyone "can you commit?" If they all say yes, it tells them to commit. It gives you true all-or-nothing behaviour, and it blocks. The participants hold locks while they wait, and if the coordinator dies after asking, they are stuck holding those locks with no authority to decide. In practice it is used inside single-vendor systems and rarely across services. Saying "I would avoid 2PC across services, because a coordinator failure leaves participants blocked holding locks" is the answer interviewers are listening for.
Sagas. Break the workflow into local transactions, each with an undo action. Reserve stock, take payment, book the courier. If the courier step fails, refund the payment and release the stock. Each step commits locally, so nothing blocks. The costs: users can see the in-between states. The undo is not a true undo: you cannot un-send an email, and a refund is a new event, not an erased one. And you have to handle an undo itself failing. Order the steps so the most likely failure comes first, and the step you cannot take back comes last. Take payment after you have confirmed stock, not before.
An orchestrated saga has one component that knows the whole flow. It is easy to follow and to debug, and it is a component that must not lose its state. A choreographed saga has services reacting to each other's events. There is no central piece, and no single place that tells you what the process is, which gets painful around the fifth step.
The outbox pattern solves a narrower and very common version of this. You updated the database, and now you need to publish an event. Do both directly and you can commit the row then crash before publishing, which loses the event. Or publish first and then fail to commit, which creates an event about something that never happened. Instead, write the event into an outbox table in the same transaction as the data, so the two cannot disagree. Then let a relay read that table and publish. The relay may publish twice, so consumers must be safe to repeat, which they had to be anyway.
Change data capture is the same idea, driven from the database log instead of a table. It gives you the same guarantee with less application code.
- Proposing 2PC across microservices without naming the blocking problem.
- A saga with no undo actions written down. That is just a list of calls.
- Assuming an undo restores the previous state. It does not. It adds a correcting event.
- Doing the step you cannot take back first.
The distributed transactions follow-up an interviewer will ask
"The payment succeeded but the order write failed. Now what?"
First, note that ordering the steps this way was a choice, and a poor one. I would take payment last, after stock is reserved and the order row exists in a pending state, because undoing a payment is the most expensive undo available. Given it has happened: the payment provider holds the truth, so I reconcile against it, not against my own logs. The order row write should be retried, because I have the payment reference and can make the retry safe on that reference. Most of the time the write failed for a temporary reason, and the retry completes the order, which is the outcome the customer wants. If it cannot be completed, because the item is gone, issue a refund as the undo step and tell the customer what happened. Two things make this survivable rather than a scramble. An authorise-then-capture flow, so the money is only held until I know the order is real. And a regular reconciliation job comparing payments to orders, because a system doing this at volume will always have a few stuck cases, and you want to find them before the customer does.
Your turn
You write a row and then publish an event about it. Name the two ways that can go wrong, and the pattern that fixes both.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Saga pattern ↗Chris Richardson on sagas and compensating transactions, with the tradeoffs listed.