Idempotency Patterns Distributed Systems

Idempotency Is the Distributed Systems Concept Interviewers Actually Test For

Ask any staff engineer what separates a junior distributed-systems answer from a senior one, and idempotency comes up almost immediately. Networks fail, retries happen, and at-least-once delivery is the default guarantee in nearly every message queue and HTTP client library in production. The question isn’t whether duplicate requests will happen — they will — it’s whether your system handles them safely. In 2026, with async payment flows, webhook-driven architectures, and event-sourced systems more common than ever, idempotency questions show up in system design interviews as often as classic CAP theorem questions did five years ago.

This article covers the core patterns engineers actually implement in production, the tradeoffs between them, and why “exactly-once” delivery is a marketing term, not a real guarantee.

Pattern 1: Idempotency Keys

The most widely deployed pattern, popularized by Stripe’s API design, is the client-generated idempotency key. The client generates a unique key (typically a UUID) per logical operation and sends it in a request header. The server stores a record of (idempotency_key, response, status) the first time it processes the request. If the same key arrives again — because the client retried after a timeout, not because it received a definitive success/failure response — the server returns the cached response instead of reprocessing.

The critical design decision is where you store that key-response mapping and how long you retain it. Most production systems use a dedicated table (or Redis with TTL) keyed on idempotency_key, with a retention window of 24 hours, matching the realistic window in which a client might retry a stalled request.

Pattern 2: Deduplication Tables (Exactly-Once Processing Illusion)

For event-driven systems consuming from Kafka, SQS, or similar, the pattern shifts to a deduplication table keyed on a message ID or a natural business key (e.g., order_id + event_type). Before processing a message, the consumer checks whether that key already exists in the dedup store; if so, it skips processing and acknowledges the message. This is what most systems mean when they claim “exactly-once processing” — it’s actually at-least-once delivery plus idempotent consumption, which produces exactly-once effects, not exactly-once delivery. This distinction is a favorite trap in senior interviews: a candidate who says “we guarantee exactly-once delivery” without qualification is signaling a gap.

Pattern 3: Idempotent Database Operations

Sometimes the cleanest solution isn’t a dedup table at all — it’s making the operation itself naturally idempotent. Examples:

• Using UPSERT (INSERT ... ON CONFLICT DO UPDATE) instead of plain INSERT, so re-running the same insert doesn’t create duplicates.
• Using absolute-value writes instead of relative increments — SET balance = 500 instead of balance = balance + 100, when the target state is known.
• Using conditional writes with version numbers or ETags (optimistic concurrency control), so a write only succeeds if the resource is in the expected prior state.

This pattern eliminates the need for a separate tracking table entirely but only works when the operation’s semantics naturally support it — financial ledger entries, for instance, are rarely idempotent by nature and usually need pattern 1 or 2.

Pattern 4: Fencing Tokens for Distributed Locks

When idempotency must be enforced across a distributed lock (e.g., leader election, distributed cron), a monotonically increasing fencing token prevents a “zombie” process — one that thinks it still holds the lock after a GC pause or network partition — from applying a stale write. Every write is tagged with the token the writer believed it held; the storage layer rejects any write carrying a token lower than the highest one it has already seen. This is the classic Martin Kleppmann pattern from “Designing Data-Intensive Applications” and remains a standard follow-up question after any distributed lock discussion.

Comparison: Idempotency Pattern Selection Guide

Interview Framing: How to Answer “Design an Idempotent Payment API”

A strong answer sequences the reasoning: first identify that the client can’t know whether a timed-out request actually succeeded server-side, so retries are inevitable. Then introduce the idempotency key as the contract between client and server. Then discuss storage (a dedicated table, indexed on the key, with TTL cleanup), the response caching behavior, and finally the edge case of concurrent identical requests arriving before the first has finished processing — solved with a row-level lock or a status: in_progress state that causes the second request to either wait or return a 409 conflict.

This exact reasoning chain — problem, contract, storage, concurrency edge case — is the structure taught for distributed systems questions in The 0-to-1 SWE Interview Playbook (https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20), which treats idempotency as a first-class topic rather than a footnote, since it recurs across payments, messaging, and infrastructure interview tracks alike.

FAQ

Q: Is “exactly-once” delivery actually achievable in distributed systems? A: Not at the delivery layer — network and process failures make true exactly-once delivery provably impossible in an asynchronous system. What’s achievable is exactly-once effects, via idempotent consumption of an at-least-once delivery stream, which is what production systems (Kafka with consumer-side dedup, Stripe with idempotency keys) actually implement.

Q: How long should an idempotency key be retained? A: Long enough to cover realistic client retry windows, typically 24 hours, balanced against storage cost. Stripe’s API, the reference implementation most engineers cite, uses a 24-hour retention window for idempotency keys.

Q: What happens if two identical requests with the same idempotency key arrive truly simultaneously? A: The server must handle this as a concurrency problem, not just a lookup problem — typically via a unique constraint on the idempotency key column (causing the second insert to fail and retry as a lookup) or an explicit in-progress status that the second request detects and handles with a wait-and-poll or conflict response.