Payments and ledger — a full worked answer
The whole thing, start to finish, in the order you would actually say it. Read it to learn the shape of a strong answer — then go and produce one yourself, because reading is not the same skill as producing.
The question
Design the payment system behind a marketplace. Customers pay, money is held, sellers are paid out, refunds happen, and the books must balance exactly. An external payment provider holds the real money.
Money and correctness. Being fast is worthless if the number is wrong, and every mistake is somebody's actual money.
The design is dominated by exactly-once effects, audit trails and reconciliation with a third party who holds the authoritative record. Availability is negotiable here; correctness is not.
Before anything: where is the slack?
How long the system has before a person notices. Ask this before you draw a single box — it decides more of the design than scale does.
A person is waiting to find out whether their payment worked, and that is the only part with no slack. Everything behind it (capture, ledgers, payouts, reconciliation) has hours, and the design uses those hours to be correct.
Authorisation has seconds of slack at most. A customer is on the checkout page and every second past about three feels broken, so this call is synchronous, it has a hard timeout, and it must give a definite yes or no. Here a queue is the wrong answer, because "we will let you know" is not an acceptable response to "did I buy it".
Capture, which takes the authorised money, usually has minutes or more, and often should not happen until the seller confirms the order. Payouts to sellers are on a schedule measured in days. Reconciliation against the provider runs nightly. All of that is background work, and it can retry patiently, because no user is waiting on any single step.
What has zero slack, and no way to buy any, is correctness. A payment recorded twice, or money moved with no matching ledger entry, does not fix itself with time. It stays wrong until someone finds it, perhaps weeks later during an audit, and the longer it exists the harder it is to explain.
So: A short synchronous authorisation path with a hard timeout, and everything after it asynchronous, idempotent and reconciled. The ledger is append-only so history cannot be rewritten, and a nightly job compares our books against the provider's.
Requirements
about 5 minWhy this stage exists: Because the question is deliberately vague, and every box you draw before this is a guess.
Assumptions: a marketplace with buyers and sellers, an external provider like Stripe or Adyen actually moving the money and holding the regulated relationship, one currency to start, and we hold funds briefly between the buyer paying and the seller being paid out.
The question that changes a box: are we a regulated money transmitter, or is the provider? If we hold customer funds ourselves, we need segregated accounts, regulatory reporting and a compliance effort far larger than the engineering. I will assume the provider is the regulated party and we are recording our view of what they did. I will also say clearly that our ledger is a mirror, not the source of truth. The provider's records are the truth, which changes how we resolve every disagreement.
Second question: is it acceptable to occasionally decline a valid payment, or to occasionally accept an invalid one? They are not symmetric. Declining a good payment costs a sale. Accepting a bad one costs the money plus a chargeback fee plus investigation time. I will assume we err toward declining, which means when in doubt we fail closed.
Scope I propose: authorising and capturing a payment, recording it in a double-entry ledger, refunds, seller payouts, and reconciliation against the provider. Out of scope: fraud scoring, tax calculation, and multi-currency conversion. Each is a separate system, and I would flag fraud in particular as something a production design would not omit.
What an interviewer is ticking off in this stage
- Established who is the regulated party, and that our ledger mirrors the provider
- Named the provider's records as the source of truth for disputes
- Asked which error direction is acceptable, and chose fail-closed
- Proposed a scope and named fraud as a flagged omission you chose
- Did not claim to be building a bank
Actors and lifecycle
about 6 minWhy this stage exists: Because the happy path is the easy half. The failure branches are where the design actually lives.
Actors: the buyer, the seller, the payment provider, the buyer's bank behind the provider, and the system itself. The system decides when to capture, when a hold expires, when a payout is due, and when a discrepancy needs a human. The buyer's bank matters because it can reverse a payment months later through a chargeback, so "settled" is not final.
The chain: initiated → authorised, meaning the money is held but not taken → captured, meaning it is taken → recorded in the ledger → funds held pending → paid out to the seller → reconciled. Branching off it: refunded, and separately disputed by chargeback, which can typically arrive up to 120 days later (card network rules vary) and reverses a payment we already paid out on.
The failure branches are where the design is. The timeout on the provider call is the worst case in the system: we sent an authorisation, got no response, and do not know whether it succeeded. We must never retry without protection, because that risks a double charge. So every request carries an idempotency key, and a retry with the same key returns the original result rather than creating a second charge. If we are still unsure, we query the provider by our own reference rather than sending anything new.
More branches. Authorised but never captured: holds expire after days (often about a week), so an order stuck in review loses its authorisation without any error, and the payment must be re-taken. Captured but our database write failed: the money moved and our books do not show it. Reconciliation must catch that, and it is why the provider is the source of truth. Refund requested for a payout already made: we are now trying to claw back money the seller has. That is a business rule, not a technical one, and the design must surface it rather than fail without telling anyone. Chargeback 90 days later: it reverses a settled payment, and the ledger must express that as a new correcting entry rather than editing history.
What an interviewer is ticking off in this stage
- Separated authorise from capture and said why they are different states
- The provider timeout named as the worst case, with idempotency keys as the answer
- Query rather than retry when the outcome is unknown
- Authorisation expiry as a real branch that loses the payment
- Chargebacks arriving months later, reversing something already paid out
- Corrections are new entries, never edits to history
The lifecycle, drawn
Numbers
about 6 minWhy this stage exists: Because one part of this is hard and the rest is easy, and they look identical until you do the arithmetic.
Assume a marketplace doing 10 million transactions a month. That is about 4 per second on average, maybe 40 at peak on a sale day. That is tiny. Saying it early matters, because it tells me throughput is not the problem and I should not spend a minute on sharding.
Ledger growth: double-entry means each event writes at least two rows, and a single payment produces several events: authorise, capture, fee, payout, and possibly refund. Call it 10 rows per transaction, so 100 million rows a month, about 1.2 billion a year. At 200 bytes each that is around 240 GB a year. One database handles this comfortably for years, and I would partition by time for archival rather than for load.
The cost of being wrong is the number that matters here, and it is not a technical one. One incorrect transaction costs a support contact, an investigation, possibly a refund plus a chargeback fee, and if it is systematic, an audit. Ten thousand wrong transactions is a serious incident. So the acceptable error rate is effectively zero, and I would spend throughput freely to get correctness.
That reframes the design. At 40 transactions a second I can afford a strongly consistent single-primary database, synchronous replication, full audit logging on every write, and a nightly reconciliation that reads every row. None of that would be affordable at a million a second.
So the hard part here is exactly-once effects and reconciliation, not scale. The volume is trivial. The job is making sure every movement of money appears once in our books and matches the provider's.
What an interviewer is ticking off in this stage
- Stated the transaction rate and called it small, explicitly
- Ruled out sharding based on the number, rather than by habit
- Estimated ledger rows per transaction, not just transactions
- Quantified the cost of an error as the real constraint
- Used the low volume to justify expensive correctness measures
- Finished the sentence: exactly-once and reconciliation, not scale
The numbers, drawn
So the hard part is exactly-once effects and reconciliation, not scale. At 40 a second I can afford a single strongly consistent primary, synchronous replication and a nightly full reconciliation. None of that would be affordable at a million a second.
High-level design
about 12 minWhy this stage exists: Because now — and only now — you have a reason for every box you are about to draw.
The payment service owns the synchronous path. It takes the checkout request, creates a payment record in pending with a generated idempotency key, calls the provider with that key, and records the outcome. The database write claiming the key happens in the same transaction as creating the payment record, so a retry cannot create a second payment.
The ledger is the core, and it is append-only double-entry. Every movement writes at least two rows that sum to zero: taking 100 from a buyer credits a buyer-funds account and debits a holding account. Nothing is ever updated or deleted; a refund is new rows that reverse the original, not an edit. That gives two properties worth stating: summing an account's rows up to any moment gives its balance at that moment, and the full history can be reconstructed for an audit. The one invariant to assert continuously is that every transaction's rows sum to zero.
Balances are derived from the ledger. A running balance is a materialised total updated in the same transaction as the entries, never a separate counter that could drift. A periodic job recomputes balances from the raw entries and compares, because a materialised value that is never checked can go wrong unnoticed.
Crossing the boundary to the provider is a distributed transaction problem with no distributed transaction available. So: a saga with an outbox. Reserving the order and writing the ledger entry happen in one local transaction, which also writes an outbox row. A relay publishes from the outbox, and a worker calls the provider. If the provider call fails permanently, a compensating entry reverses the ledger. The irreversible step, taking the money, is ordered last, after everything reversible has succeeded.
Reconciliation runs nightly against the provider's settlement report. Every payment they report is matched against our ledger, and both directions are checked: things they have that we do not, and things we have that they do not. Mismatches go to a queue for a human, and the count of unresolved mismatches is a metric with an alert, because a slow leak can go unnoticed far longer than a sudden break.
Payouts run on a schedule, aggregating a seller's earned balance minus fees and holds, and are themselves idempotent on a payout id so a retried run cannot pay twice. Every payout is a ledger entry like everything else.
What an interviewer is ticking off in this stage
- Idempotency key claimed in the same transaction as the payment record
- Append-only double-entry ledger — corrections are new rows, never edits
- Invariant that every transaction's entries sum to zero
- Balances derived, materialised in the same transaction, and periodically recomputed
- Outbox plus saga for crossing to the provider, irreversible step ordered last
- Nightly two-way reconciliation with unresolved mismatches as an alerting metric
- Payouts idempotent on a payout id
The architecture, drawn
Deep dive and tradeoffs
about 14 minWhy this stage exists: Because naming what a decision costs is the single clearest signal of seniority there is.
Hard part one: the unknown outcome. We called the provider and got nothing back. The money may or may not have moved. Retrying blindly risks a double charge; giving up risks having taken money we do not record. The design has to make this survivable rather than clever. Every request carries an idempotency key generated by us and stored before the call, so a retry with that same key returns the original outcome from the provider rather than creating a second charge. Major providers such as Stripe and Adyen support this. If we are still unsure, we do not send anything; we query by our own reference. Meanwhile the payment sits in an unknown state, which the UI shows as processing, and a background job resolves it by querying. Cost: a customer can see "processing" for a minute. That is worse UX than a fast answer and much better than a double charge. And unknown is a real state in the model, which is easy to leave out.
Hard part two: reconciliation mismatches. Fourteen this morning. First, categorise, because the categories have different urgency: in their report and not in ours means we took money without recording it, which is the most serious and usually points at a crash between the provider call and our write. In ours and not theirs is often a timing difference at the report's cut-off, which clears the next night; if it does not clear, it is an optimistic write we should never have made. Amount mismatches are usually fee handling or rounding. Second, resolve mechanically where possible. For the first category the provider is the truth, so we write the missing ledger entries with a correcting reference and record why. Third, anything not mechanically resolvable goes to a human with full context. The trend matters more than the number. A small, steady count may be normal for your volume, fourteen thousand is an incident, and a slow rise from two to fourteen over a month suggests a bug someone shipped. So the alert is on the rate of change and on unresolved age, not on a fixed threshold.
Deploying to this safely. A bad release costs more in money code than almost anywhere else, so: canary at 1% with automatic rollback on error rate, schema changes strictly expand-and-contract so nothing is renamed in place, and a rule that ledger schema changes are additive only. Every write carries the code version, so a discrepancy can be traced to a release. Cost: a change takes days instead of hours, which is correct for this system and would be excessive elsewhere.
Access and audit. No person has direct write access to the ledger in production, engineers included. Corrections go through a tool that writes correcting entries with an operator id and a reason. Every read of payment data is logged. Cost: fixing a data problem is slower and more bureaucratic, which is what you want when the data is money.
What an interviewer is ticking off in this stage
- An explicit unknown state, resolved by querying rather than retrying
- Idempotency keys stored before the provider call, not after
- Mismatches categorised, with the provider as the truth
- Alerting on the rate of change and unresolved age, not a fixed threshold
- Canary with automatic rollback, and additive-only ledger schema changes
- No direct write access in production; corrections through an audited tool
And then they push back
A complete answer is where the interview starts, not where it ends. These are the questions this design invites — the interview is decided here far more often than in the diagram.
A payment succeeded but your order write failed. What is the customer's experience, and what do you do?
"We would refund them." Eventually, maybe — but the customer currently has money missing and no order, and if nothing detects it, nobody will refund anything.
"First, I would say that ordering the steps that way was a choice and a poor one — I would authorise the payment, create the order, then capture, because voiding an authorisation is cheap and clean while refunding a capture is slow and visible. Given it has happened: the payment provider holds the truth, so I reconcile against them rather than my own logs. The order write should be retried using the payment reference as an idempotency key, since most of the time it failed transiently and the retry completes the order, which is what the customer wants. If it cannot complete (the item is gone), I refund as a compensating action and tell them what happened, proactively. The two things that make this survivable rather than a scramble: authorise-then-capture, and a reconciliation job comparing payments to orders. Any system doing this at volume will have stuck cases; the only choice is whether you find them or the customer does."
You write to the database and then publish an event. Name the two failure modes.
"We would use a transaction." A database transaction does not cover the message broker, and that is the problem. This answer suggests not seeing that the two systems have no shared transaction.
"One: the row commits and the process dies before publishing, so downstream never learns about it. That is a lost event, and it is the worst kind, because nothing looks broken and no error is raised. Two: I publish first and the transaction rolls back, so consumers act on something that does not exist: a phantom event. At enough volume both will happen, because no transaction spans both systems. The fix is the outbox pattern: write the event into a table in the same transaction as the data, so they are atomic by construction, and let a separate relay publish from that table. The remaining cost is that the relay can publish twice if it dies between publishing and marking sent, so consumers must be idempotent. With at-least-once delivery they had to be anyway. Change data capture off the database log gets the same guarantee with less application code, and I would use that when the consumer is derived state like a search index."
A third-party API you depend on starts taking 30 seconds to respond. What happens?
"We would retry." Retrying against something already overloaded makes it worse, and retrying a 30-second call means holding your resources for a minute or more. This answer usually turns one slow dependency into your own outage.
"Slow is more dangerous than down, because down fails fast and slow holds resources. Without a timeout, every request calling that API holds a thread and a connection for 30 seconds, my pools exhaust, and endpoints that never touch that API start failing. One slow dependency has become a total outage. So: a timeout set from their real latency distribution, not a round number; a circuit breaker that opens on failure rate so I stop waiting entirely once it is clearly bad; a bulkhead giving that dependency its own limited pool so it cannot consume capacity the checkout path needs; and a defined fallback — cached data, a degraded response, or dropping an optional feature. The fallback has to be decided before the incident, per dependency, or I fail fast into a blank page. And retries need backoff with jitter and a cap, or I contribute to keeping them down."
Now every change must be auditable — who changed what, when, and what it was before.
"We would add a log line whenever something changes." Log lines get lost, get filtered, are not queryable and can be written when the transaction rolls back. An audit trail has to be as reliable as the data itself.
"The audit record has to be written in the same transaction as the change, or it is not an audit trail, just a hopeful log. So an audit table written transactionally alongside, holding actor, timestamp, entity, and before and after values. If I want it without touching every write path, change data capture off the database log gives me every change including ones made by a script someone ran manually, which is the case an audit is for. Retention and immutability matter: the audit store should be append-only with restricted permissions, because an audit trail an attacker can edit is worthless. The costs are real and I would name them: every write now writes twice, storage grows with change volume rather than data volume, and before-and-after values can contain sensitive data, so the audit store inherits the same privacy obligations as the primary, including deletion requests. That is an awkward interaction with immutability that needs a planned policy."
You deployed something bad. How fast can you undo it, and what cannot be undone?
"We would roll back to the previous version." Some things cannot be rolled back, and not knowing which ones is how a five-minute incident becomes a data corruption incident.
"Code rolls back in the time it takes to redeploy: a few minutes if the pipeline is fast, which is why a fast pipeline is itself a mitigation. What does not roll back: a schema migration that dropped a column, data written in a new format that the old code cannot read, messages already published to a queue and consumed, emails sent, and payments captured. So the practical answer is that I design so the risky parts are reversible: migrations are expand-then-contract so the destructive step happens days later once the new version is proven; new data formats are written alongside old ones during the transition; and anything with an external side effect gets a feature flag so I can turn the behaviour off without deploying at all. Feature flags are the fastest undo available, seconds rather than minutes, and for anything risky I would rather ship it behind a flag than rely on rollback."
You need to move this data to a new store with no downtime. How?
"We would migrate over a weekend." That is downtime, and at any real data volume a weekend is not long enough anyway.
"Dual-write and backfill, in phases. Phase one: start writing to both old and new stores, with the old one still authoritative. This is the risky part, because a write that succeeds in one and fails in the other creates drift, so I make the new-store write non-blocking and reconcilable rather than letting it fail user requests. Phase two: backfill history into the new store, in batches, throttled so it does not affect production. Phase three: verify, by comparing reads from both for a sample of traffic and measuring the disagreement rate. This step is easy to skip, and it is the one that tells you whether the migration is correct. Phase four: switch reads over gradually, a percentage at a time, with a fast way back. Phase five: stop writing to the old store, and only then, after a safe period, delete it. Cost: weeks of running both, double writes, and a comparison harness. That is why the shard key and data model choices made on day one matter so much."
You picked a relational database for something with 50,000 writes a second. Defend that.
"Postgres is very fast these days." Not a defence. 50,000 writes a second on a single primary is beyond what one machine commits comfortably, and hand-waving it means you did not check.
"If those 50,000 writes are all on one logical table with no partitioning, I would not defend it — that is the wrong choice and I would change it. What I would defend is a version where the writes are partitioned across, say, sixteen shards by tenant id, giving about 3,000 writes a second each, which one primary handles well. Then I keep real transactions within a tenant, which is where all my invariants live, and I give up cross-tenant transactions, which I never needed. So the defence is not that relational scales to 50,000 on one box — it does not — it is that my access pattern shards cleanly and the guarantees I need are all within a shard. If the writes could not be partitioned that way, I would take a distributed store and redesign the invariants around what it can give me."
A worker dies halfway through processing a batch. What is the state of the world?
"The job would be retried." Retried from where? If the first half already had side effects, retrying repeats them, and now you have sent 5,000 emails twice.
"It depends on whether I made the work idempotent, which is a decision I have to have made beforehand. The message was not acknowledged, so it is redelivered. That is at-least-once delivery, and it is the only guarantee I can get. So a retry reprocesses the first half. If each item's effect is idempotent — an upsert by a natural key, setting a status rather than incrementing — the retry is harmless and this is a non-event. If the effects are external, like sending an email or charging a card, each one needs its own idempotency key carried to the provider, and I need to record progress per item rather than per batch so a retry skips what is done. The design lesson is to record completion in the same transaction as the work where possible, so there is no window where the work happened and the record of it did not."
The topics this answer used
You have read a good answer. That is worth very little until you can produce one on a blank page — so go and write this problem without looking, and compare afterwards.