Lab instrument / Reliability

What happens when a request is sent twice?

A client places one order. Break the network in one of four ways, add defences, and follow every message. The question is never whether the request arrives — it is how many orders exist afterwards, and whether the client knows.

A deterministic teaching model: the same setup always produces the same run. Nothing is sent anywhere.

Inject a faultNothing goes wrongThe request arrives, the order is saved and the reply returns.The request times out before it arrivesThe connection drops on the way in. The server never sees it.The reply is lost after the order is savedThe server saves the order, then the connection drops. The client sees the same timeout as before.The button is pressed twiceTwo identical requests are in flight at the same time.The retry carries an edited orderThe reply is lost; before retrying, the person changes the quantity. The retry reuses the same request.

Duplicate order

Orders in the store: 2
What the client believes: Sees order #2 confirmed and believes it is the only one.

6/6

6 steps. Drag the slider or replay to follow them one at a time.

The customer now has 2 orders.

A timeout does not mean the request failed. The retry was a second request for the same order, and the server had no way to know.

What this relies on

Retries need a limit, backoff and jitter. Without them, retries from many clients can turn a brief slowdown into an overload.

Add defencesRetry after a timeoutThe client sends the request again when no reply arrives.Idempotency keyThe client creates one key per order and sends it with every attempt. The server remembers the reply for each key.Create-only key recordThe key is claimed with a write that fails if it already exists, instead of checking first and writing afterwards.Payload fingerprintThe server stores a hash of the request with the key and refuses a different request under the same key.

A timeout is an unknown, not a failure

Two failures that look the same

When a request times out, the client cannot tell whether it was lost on the way in or whether the work was done and only the reply was lost. Run both faults with retries on and no key: the first is repaired by the retry, the second becomes a duplicate. Same symptom, opposite consequence.

Not retrying does not solve this. It hands the same unknown to a person, who presses the button again.

What a key actually guarantees

An idempotency key names the intent, not the attempt. The client creates it once per order, before the first attempt, and sends it with every retry. The server stores the result under the key and returns that result for every later attempt — so a retry becomes safe because it can no longer be a new request.

HTTP already defines some methods as idempotent. A POST that creates an order is not one of them; the key is how an API makes it behave as one.

Where keys still fail

Races. Checking whether a key exists and then writing it is two steps. Two concurrent attempts can both pass the check — try the double submit with a key but without the create-only claim. Claiming the key must be one atomic write that fails if the key already exists.

Reuse. If the person edits the order and the retry reuses the key, a plain replay answers a request that was not asked. Storing a fingerprint of the request turns that into an explicit conflict.

On this site

The contact form follows the same design. Each message carries a request ID; the server claims a receipt for it with a create-only write, keeps a keyed fingerprint of the message, replays the original receipt for a repeated submission, and returns a conflict if the same ID arrives with a different message. Receipts expire after a day.

The model leaves out retry storms, clock-based expiry and queues that redeliver messages; the reasoning is the same, the failure shapes differ.

Sources and further reading

Making retries safe with idempotent APIsAmazon Builders’ Library
Timeouts, retries, and backoff with jitterAmazon Builders’ Library
The Idempotency-Key HTTP Header FieldIETF HTTPAPI working group, Internet-Draft
RFC 9110 §9.2.2 — Idempotent methodsIETF, HTTP Semantics, 2022
Idempotent requestsStripe API reference
Implementing Stripe-like idempotency keys in PostgresBrandur Leach

Let’s make a connection.

Working on something in software architecture? The contact form will start with that topic selected.

Discuss this challenge