Retries, Idempotency, and Duplicate Writes

Framing the problem

Timeouts are ambiguous: a request may have failed before execution, failed during execution, or succeeded but lost its response. Retrying is rational, but every retry is extra work against an already unhealthy dependency. If writes are not idempotent, each replay can produce a new side effect.

This page focuses on practical controls for real production systems where network faults, failover events, and overloaded downstreams are normal operating conditions.

Why duplicate writes happen

Client ambiguity: client times out and retries because it cannot prove the first attempt committed.
At-least-once delivery: queues and workflow engines can redeliver on worker crash or visibility timeout.
Multi-layer retries: SDK, service mesh, and application code all retry the same call.
Failover replay: a partially acknowledged operation is replayed after leader/election transitions.

The outcome is familiar: duplicate charges, repeated provisioning, double email/send actions, or state machines that jump two steps instead of one.

Idempotency design that holds up in production

Define operation identity, not just request identity

Idempotency keys should represent business intent (for example, "create order X for tenant Y") instead of transport-level request IDs. Otherwise retries may bypass dedupe because metadata differs while intent is the same.

Persist first outcome durably

Store key + normalized request fingerprint + outcome status in a durable dedupe store. Duplicate keys should return the original outcome body and status code, not recompute side effects.

Align dedupe TTL with real retry windows

If keys expire too quickly, delayed retries from mobile clients, queues, or async workflows can still double-apply writes. TTL should cover realistic replay horizons, not only synchronous API retries.

Commit write + dedupe atomically

If the side effect commits but dedupe record does not, the next retry may execute again. Use one transaction boundary where possible, or a compensating recovery log when storage boundaries differ.

Retry policy: recover without causing storms

Policy element	Recommended default	Why it matters
Error classification	Retry transient transport + selected 5xx/429 only	Avoid retrying deterministic failures
Backoff	Exponential with full/decorrelated jitter	Prevents synchronized retry bursts
Retry budget	Cap retries as fraction of baseline traffic	Limits load amplification during incidents
Deadlines	Bound total request time, not only per-attempt timeout	Avoids zombie work after caller gave up
Cancellation	Propagate upstream cancellation downstream	Reduces useless queued retries

Example: create-and-notify workflow

Consider an API that creates a resource and emits a notification:

Client sends POST /resources with idempotency key tenant123:create:abc.
Server writes resource and idempotency record in one transaction.
Notification is emitted from an outbox consumer keyed to the same operation identity.
If client times out and retries, server returns prior response; outbox consumer dedupes notification by operation key.

This keeps API correctness and asynchronous side effects aligned. Without shared operation identity, one layer may dedupe while another still duplicates.

Common mistakes

Using idempotency keys only on public API, but not on internal workflow steps.
Allowing per-service retries plus client retries with no global budget.
Retrying after hard validation or authorization failures.
Treating "exactly once" as guaranteed instead of engineered behavior on top of at-least-once substrates.

When engineers should care most

Payment-like operations and quota/entitlement changes.
Control-plane mutating APIs and tenant provisioning workflows.
Any incident where timeout rate spikes and retry traffic rises faster than baseline load.

Retry Strategies and Idempotency — deeper treatment of retry budgets and overload interaction.
Queue Design Under Load — queue dynamics that amplify retries.
Backpressure in Distributed Systems — how to prevent retries from becoming cascading failure.
Caching + Eventual Consistency Pitfalls — stale-state interactions that compound duplicate processing risk.
What Is Idempotency? — conceptual baseline for teams new to the pattern.

Conclusion

Retries are a correctness feature only when paired with strong idempotency semantics and overload-aware budgets. Build for replay as a first-class reality: define operation identity, persist outcomes durably, and bound retry behavior so failure recovery does not create new incidents.