Idempotency: the check-then-act that charges twice
At-least-once delivery is the normal case, not the edge case. "Have I done this already?" is only a useful question if nobody can answer it at the same time as you.
Train thisThe problem
A queue delivers charge events. The handler guards against duplicates by looking the charge up first and returning early if it finds one.
The queue, like most queues, guarantees at-least-once delivery. Today it delivers the same event twice, and two workers pick up the copies within a few milliseconds of each other.
interface ChargeEvent { chargeId: string; amount: number }
async function handleCharge(event: ChargeEvent) {
const existing = await db.findCharge(event.chargeId);
if (existing) return; // already handled
await payments.capture(event.chargeId, event.amount);
await db.insertCharge(event.chargeId, event.amount);
}PREDICT FIRST
Two workers process the same event concurrently. What does the customer see?
Commit to an answer before you read on. Being wrong here is the part that sticks.
Why it behaves that way
This is check-then-act, the same shape as a TOCTOU bug. The read establishes a fact, the code acts on that fact, and nothing prevents the fact from changing in between. Adding a unique constraint helps, but the constraint fires on the insert — which happens after the side effect you were trying to protect.
The fix is to reverse the order: claim the work first, and only perform it if the claim succeeded. A conditional insert (INSERT ... ON CONFLICT DO NOTHING, or a unique index plus a caught violation) is atomic, so exactly one of the two workers wins. The loser learns it lost before spending any money.
The claim also has to survive a crash between claiming and capturing, which is why the row is a small state machine rather than a boolean. A row stuck in pending past its lease is recoverable work, and recovery is safe precisely because the capture carries an idempotency key: replaying it returns the original result instead of charging again.
Pushing the key all the way to the payment provider is the part people skip, and it is the part that matters. Your database can only make your own writes exactly-once. Only the provider can make the charge exactly-once, and it needs the key to do it.
async function handleCharge(event: ChargeEvent) {
// INSERT ... ON CONFLICT DO NOTHING — atomic, so exactly one worker wins.
const claimed = await db.claimCharge(event.chargeId, event.amount);
if (!claimed) return;
try {
// The provider deduplicates on the key, so a replay after a crash
// returns the original charge instead of creating a second one.
const receipt = await payments.capture(event.chargeId, event.amount, {
idempotencyKey: event.chargeId,
});
await db.markCaptured(event.chargeId, receipt.id);
} catch (error) {
// Release the claim so a retry can pick it up; leaving it pending is
// also correct if a sweeper reclaims expired leases.
await db.markFailed(event.chargeId, String(error));
throw error;
}
}What the change buys you
| Behaviour | Before | After |
|---|---|---|
| Duplicate delivered concurrently | Two captures | One capture; the loser returns early |
| Duplicate delivered an hour later | One capture | One capture |
| Crash between capture and insert | Charge with no record; redelivery charges again | Row stays pending; replay is deduplicated by the key |
| Provider retry after a timeout | Possible second charge | Original charge returned |
Variations worth trying
- Derive the idempotency key from the event payload instead of trusting a producer-supplied id, and decide what happens when the same id arrives with a different amount.
- Add a lease with an expiry so a worker that dies mid-capture does not park the row in
pendingforever. - Make the read path idempotent too: a client that retries the HTTP request should get the original receipt back, not a fresh 409.
- Try the version-number variant for updates —
WHERE version = $expected— and compare it with the claim-first pattern for inserts.
TRAIN IT
Process an event once
Handle duplicate delivery, concurrent workers and a crash between the side effect and the write, with tests that interleave the failures on purpose.
Open the workout