Idempotency, and why exactly-once is a myth
Make doing it twice the same as doing it once, because it will happen twice.
What problem does idempotency solve?
A request succeeds and the response is lost on the way back. The client cannot tell that apart from a failure, so it retries. If your handler charges a card, you have now charged twice. This is not a rare edge case. At scale it happens every day.
You cannot stop duplicate delivery. That is the myth. What you can do is make duplicates harmless, and that is a solvable engineering problem.
What idempotency costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for idempotency, and when not to
Use it when
- Payments, orders, transfers. Anything that moves money or stock.
- Every consumer of a message queue, with no exceptions. At-least-once delivery is normal.
- Any endpoint a mobile client calls over a shaky network.
- Any operation with an outside effect: sending an email, calling a partner API.
Do not, when
- Operations that are already safe to repeat. Setting a value to X, or deleting by id, are already fine. Adding a key store there is pure cost.
- Plain reads.
How idempotency actually works
There are three delivery promises. At-most-once: send and forget. It may be lost, and it is almost never what you want. At-least-once: retry until acknowledged. It may duplicate, and this is what real systems give you. Exactly-once delivery is impossible on a network that can drop messages, because you cannot tell a lost request from a lost response. What you can achieve, and what people mean when they say exactly-once, is at-least-once delivery plus processing that is safe to repeat. That produces an exactly-once effect, and that is the precise way to say it.
How to build it. The client makes a key: one UUID per logical operation, created once and reused across every retry. Regenerating it on each attempt is a bug that defeats the scheme without any error. The server tries to insert that key into a table with a unique constraint, inside the same transaction as the work. If the insert fails, it is a duplicate, so return the stored response. Doing the key insert and the work in one transaction is what makes it airtight. Two separate steps leave a window where you crash in between.
Choosing the key. Client-generated UUIDs are best for user actions. For queue consumers, use the message id, or a hash of the meaningful content. Avoid keys containing a timestamp or an attempt number, because those change on each retry and defeat the point.
Retention: keep keys long enough to cover the longest retry you can imagine, usually 24 hours to a few days, then expire them. Keeping them forever gives you a growing table nobody prunes.
The consumer version: do the work, record the message id as done, and commit both together. If you cannot do it in one transaction, because the side effect is an external API, then make that external call carry its own idempotency key. Payment APIs commonly support this; Stripe's Idempotency-Key header is one example.
- Claiming a queue gives exactly-once delivery. It gives at-least-once, and you supply the rest.
- Generating a fresh key on each retry.
- Checking "have I seen this key" and doing the work as two separate steps, with a crash window in between.
The idempotency follow-up an interviewer will ask
"The write worked but the response was lost and the client retried. What happens?"
With idempotency keys, nothing bad. The retry carries the same key, the server sees it is already claimed, and returns the original result. The client gets a success and the card is charged once. Without them, you charge twice and hear about it from a customer. The detail I would add is what happens if the retry arrives while the first request is still running. The key is claimed but there is no result yet. I would return an "in progress, retry shortly" status, rather than doing the work again or returning a wrong answer, so the client backs off and asks again. And I would make sure the client reuses the key across retries instead of generating a fresh one, because a fresh key per retry is an easy way to get this wrong.
Your turn
A queue delivers the same message twice. Name two ways to make that harmless, and say which you would prefer.
The retry that charged 300 customers twice
Over one weekend, 300 customers were charged twice for the same order. The code has an idempotency key. The team is confident it was implemented correctly, because they tested it: sending the same request twice returns the original result. The double charges are all from mobile clients, and all during a period when the payment provider was slow.
- 1Confirm the mechanism works when tested
Send the same request with the same key twice: the second returns the stored result and does not charge again. So the server-side key store is correct. That narrows it: if the same key protects properly, then the two charges must have arrived under two different keys.
- 2Look at the client, not the server
The mobile app generates the idempotency key inside the network layer, at the moment of sending. Its retry logic sits above that layer. So when a request times out and the app retries, it calls the network layer again, which generates a fresh UUID. Two attempts, two keys, two charges, and the server behaved perfectly both times. The protection was defeated one layer above where it was implemented.
- 3Understand why it only happened that weekend
The provider was slow, so requests crossed the client timeout while succeeding on the server. That is the condition idempotency exists for, and the only one that exposes this bug. Every other weekend the responses came back in time, the retry never fired, and the bug sat there looking like working code.
- 4Fix the key lifetime
Generate the key once per logical operation, when the user taps Pay, and store it in the client's local database before the first send. Every retry of that operation reuses it, including retries after the app is killed and reopened. The key now belongs to the intent, not to the network attempt.
- 5Make the server able to catch it anyway
The server cannot detect a fresh key, but it can detect a suspicious pattern: the same customer, same amount, same order id, within a short window under two different keys. That should not create a second charge silently; it should be rejected or flagged. This is defence in depth, because a future client release can get this wrong again.
- 6Test the condition, not the mechanism
The original test proved the key store works. It did not prove the system is safe, because it never exercised a client timeout on a request that succeeds. Add a test that delays the response past the client timeout and asserts exactly one charge. That is the scenario the feature exists for, and it was the one thing never tested.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Making retries safe with idempotent APIs ↗The Amazon Builders' Library on idempotency tokens in production.