Engineering

Idempotency keys, and what they cannot save you from

Our error conventions carry a line that is easy to read past. On a 5xx, a read request can simply be sent again — but for a write, use an Idempotency-Key.

That sentence is doing a lot of work. It is the difference between a retry that is safe and a retry that pays someone twice, and it turns on a question the header itself does not answer: what counts as the same operation.

# a unique value per logical operation
Idempotency-Key: 7f3a1c9e-2b4d-4a6f-8e1c-9d2f5a7b3c4e

# same key, same operation → the first result,
# and no second record

"Per logical operation" is the whole specification

A UUID is the right shape for the value. Where integrations go wrong is when they generate it.

Mint the UUID at the call site, inside the function that builds the request, and it is fresh on every attempt. The retry after a timeout carries a different key, so it reads as a different operation, and the header protects nothing at exactly the moment you needed it. The code looks correct. It has a UUID in it. It is decoration.

The opposite mistake is deriving the key from the request body. Now two genuinely different payments collapse into one whenever the bodies match — pay the same recipient the same amount twice in a day and the second payment silently returns the first one's result.

The key has to be as stable as the intent and as unique as the intent. In practice that means generating it once, at the point where your system decides a payment should happen, and storing it on that row: the invoice line, the payroll entry, the withdrawal request. Every retry path that can ever reach the endpoint then reuses the same key — including the one you did not design for, where somebody opens the page the next morning and presses the button again.

The retention question nobody asks until it matters

Keys are usually retained for a day, because that is the convention card APIs established and cards settle quickly.

Payouts do not share that shape. A destination rail can be closed for a weekend or a public holiday that is not on your calendar, and a client can sit in a backoff loop across all of it. If the key expires before the slowest legitimate settlement window on the corridor you are sending to, the retry that arrives afterwards is indistinguishable from a new instruction — and it will be treated as one.

Retention is a function of the slowest thing downstream, not the fastest thing you tested against.

The failure the key cannot reach

Here is the case that matters, and it is the reason idempotency is a distributed systems problem rather than a header.

A write is dispatched. The connection drops before the response comes back. You do not know whether it arrived. From inside your process, a request that was never received and a request that was received, executed and lost on the way back look exactly the same.

The key handles the half you control: if your caller retries, it gets the first result and no second record is created. What it cannot tell you is whether the first attempt already moved money. No header resolves that, because the information you need is not in your process.

So a timeout has to produce a third state — not success, not failure, but unknown — and that state is resolved by reading rather than writing. For us that read is GET /v1/transactions, or for an orchestration GET /v1/orchestrate/{id}, and the timeline it returns tells you which stages actually completed. A retry into an unknown state is how one payout becomes two.

This is also why the reference field on a movement is worth setting even when nothing requires it. It gives you a way to find a transaction again by something other than a response you never received.

Concurrency is the part that gets tested last

Two requests with the same key arriving a millisecond apart is the normal case under load, not an edge case. If the handler reads "have I seen this key" and then writes, both requests read nothing and both write.

The fix is a uniqueness constraint taken at the storage layer before any work begins, so the database arbitrates rather than the application. The loser of that race should not fail — it should wait for the winner and return the winner's result. That distinction is what makes the behaviour correct rather than merely safe: a client that retried aggressively still gets an answer, and it is the same answer the first attempt got.

What it buys

Idempotency is usually framed defensively — something you add so a bad client cannot hurt you. That undersells it. A correct key contract is what lets a caller retry freely: without a reconciliation conversation, without a support ticket, without a person deciding whether it is safe to press the button again.

At a few hundred payouts a day you can afford to have someone check. At a few hundred thousand nobody is checking, which is the same reason count matters more than notional when you size any of this.

Idempotency keys are in development on our write endpoints and ship with collections and payouts. The conventions are in the docs; questions to hi@spendfigo.com.

Common questions

What is an idempotency key?

A client-generated identifier sent with a write request so the server can recognise a retry of that exact request and return the original result instead of performing the work twice. The key belongs to the request, not to the resource.

How long should an idempotency key be retained?

Long enough to outlive every retry path that can reach the endpoint, including a client that retries the next morning after an overnight outage. Twenty-four hours is a common floor; for payouts, retention should exceed the slowest settlement window on any rail you route to.

Does an idempotency key prevent double payment?

It prevents your API from accepting the same instruction twice. It cannot undo a downstream leg that already dispatched before your request timed out. That case is resolved by reconciliation against the rail, not by the key.

Should the idempotency key be a hash of the request body?

No. Derive it from the caller-side event that justified the payment, and validate that a reused key arrives with an unchanged body. A body hash silently lets two genuinely different payments collapse into one when the bodies happen to match.

← All engineering writing