Dev Learning
Money and correctness

Payments and ledger

Design the payment system behind a marketplace. Customers pay, money is held, sellers are paid out, refunds happen, and the books must balance exactly. An external payment provider holds the real money.

hardTransactions, isolation and lockingIdempotency, and why exactly-once is a mythDistributed transactions: 2PC, sagas, outboxAuthentication, authorization and secretsObservability — metrics, logs, traces and alertsBackups, disaster recovery, RPO and RTODeploys, migrations and safe releasesMessage queues and backpressure

Where’s the slack?

Authorisation: seconds. Capture: minutes. Settlement and payout: hours to days.

A person is waiting to find out whether their payment worked, and that is the only part with no slack. Everything behind it (capture, ledgers, payouts, reconciliation) has hours, and the design uses those hours to be correct.

Authorisation has seconds of slack at most. A customer is on the checkout page and every second past about three feels broken, so this call is synchronous, it has a hard timeout, and it must give a definite yes or no. Here a queue is the wrong answer, because "we will let you know" is not an acceptable response to "did I buy it".

Capture, which takes the authorised money, usually has minutes or more, and often should not happen until the seller confirms the order. Payouts to sellers are on a schedule measured in days. Reconciliation against the provider runs nightly. All of that is background work, and it can retry patiently, because no user is waiting on any single step.

What has zero slack, and no way to buy any, is correctness. A payment recorded twice, or money moved with no matching ledger entry, does not fix itself with time. It stays wrong until someone finds it, perhaps weeks later during an audit, and the longer it exists the harder it is to explain.

So: A short synchronous authorisation path with a hard timeout, and everything after it asynchronous, idempotent and reconciled. The ledger is append-only so history cannot be rewritten, and a nightly job compares our books against the provider's.

Work it through

Five stages, gated. You write your answer for each stage before its model answer unlocks. That is the core of the method.

Loading your progress…

Reference material for this problem. The diagrams below are here to check your own against once you have drawn it, not to read first.

The lifecycle

InitiatedbuyerAuthorised (held)providerCapturedsystemRecorded in ledgersystemPaid out to sellersystemReconciledsystemProvider call times outunknown state — query by our reference, never blindly retryClient retried checkoutidempotency key claimed in the same transaction — one payment, not twoHold expires before captureauthorisation lost; the payment must be re-taken, and the order state says soMoney taken, our write failednightly reconciliation finds it; provider is the truth, we write the missing entriesRefund after payoutclawback is a business rule — surface it, never silently failChargeback 90 days laternew correcting ledger entries; history is never edited
A payment's life. The unknown branch and the chargeback arriving months after settlement are both states, not errors, and both must exist in the model.

The numbers

Transactions at peak~40 / sec — very small
Ledger rows per year~1.2 billion rows, ~240 GB
Acceptable error rateeffectively zero

So the hard part is exactly-once effects and reconciliation, not scale. At 40 a second I can afford a single strongly consistent primary, synchronous replication and a nightly full reconciliation. None of that would be affordable at a million a second.

The volume is trivial, which pays for expensive correctness everywhere else.

The architecture

one txnauthorisecapture, payoutsettlement filecompareBuyerPayment svcidempotency keyLedgerappend-only, double entryOutboxsame transactionRelay + workersProviderholds the moneyReconcilernightly, two-wayHuman queueunresolved mismatches
A short synchronous path to the provider, then everything asynchronous through an outbox. The ledger is append-only and reconciled nightly against the provider.

Follow-ups you should expect

Many interviews are decided in the follow-ups, not the first diagram.

A payment succeeded but your order write failed. What is the customer's experience, and what do you do?You write to the database and then publish an event. Name the two failure modes.A third-party API you depend on starts taking 30 seconds to respond. What happens?Now every change must be auditable — who changed what, when, and what it was before.You deployed something bad. How fast can you undo it, and what cannot be undone?You need to move this data to a new store with no downtime. How?You picked a relational database for something with 50,000 writes a second. Defend that.A worker dies halfway through processing a batch. What is the state of the world?

Same shape, other products

Being fast is worthless if the number is wrong, and every mistake is somebody's actual money.

The design is dominated by exactly-once effects, audit trails and reconciliation with a third party who holds the authoritative record. Availability is negotiable here; correctness is not.

Others with this shape: Payments · Digital wallet · Ledger · Billing and invoicing

Before you leave
Try this problem again in a week without reading anything. Comparing what you write then with what you wrote today shows you what stuck.
Stuck, or new to this?Read the full worked solutionAll five stages end to end, with the diagrams and the costs, written out the way you would say it. Best read after you have had a go.Read it →

How it is really built

Write your own answer first. Then read how teams who run this at scale describe it. Their constraints were real, and the write-ups often say what their choices cost.