Distributed Systems: Idempotency
In this System Design lesson, we will study Idempotency through small executable cases. First we will make the normal result clear; then inspect data flow, ownership, failure behavior, and the caller contract before adding framework or infrastructure complexity.
Professional code is readable under change. A future developer should be able to identify which input is trusted, who may update state, where I/O happens, and how an expected error reaches a caller without reconstructing the whole application.
What You Will Build
10 focused System Design examples for Idempotency. Each has a visible result, an explicit contract, and a next change to test so the lesson becomes reusable knowledge rather than passive reading.
Example 1: State an API response contract
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
public record TopicResponse(
UUID id,
String title,
Instant createdAt
) {}Expected Result
Consumers receive a stable id, title, and timestamp
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 2: Paginate with a cursor
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
GET /api/topics?limit=20&after=eyJjcmVhdGVkQXQiOiI..."
200 OK
{ "items": [], "nextCursor": "..." }Expected Result
A bounded page and an opaque continuation cursor
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 3: Use cache-aside deliberately
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
TopicResponse topic(UUID id) {
String key = "topic:" + id;
TopicResponse cached = cache.get(key);
if (cached != null) return cached;
TopicResponse loaded = repository.find(id);
cache.put(key, loaded, Duration.ofMinutes(5));
return loaded;
}Expected Result
The first lookup loads storage; later lookups use the cache
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 4: Model an indexed read path
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
CREATE TABLE topic (
id uuid PRIMARY KEY,
category_id uuid NOT NULL,
created_at timestamptz NOT NULL
);
CREATE INDEX topic_category_created_idx
ON topic (category_id, created_at DESC);Expected Result
Category feeds can read newest rows using the indexed order
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 5: Route only to healthy upstreams
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
upstream catalog_api {
least_conn;
server catalog-1:8080 max_fails=3 fail_timeout=10s;
server catalog-2:8080 max_fails=3 fail_timeout=10s;
}Expected Result
The proxy prefers the least-busy healthy backend
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 6: Publish a versioned event
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
{
"type": "topic.created.v1",
"eventId": "01J...",
"occurredAt": "2026-09-13T10:00:00Z",
"data": { "topicId": "...", "title": "System Design" }
}Expected Result
Consumers receive a versioned event with a unique id
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 7: Make command handling idempotent
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
public Result create(CreateCommand command) {
return idempotencyStore.executeOnce(
command.idempotencyKey(),
() -> service.create(command)
);
}Expected Result
Repeated delivery of the same command returns one stored result
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 8: Use an outbox for reliable publication
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
BEGIN;
INSERT INTO topic (id, title) VALUES ($1, $2);
INSERT INTO outbox_event (id, type, payload)
VALUES ($3, 'topic.created.v1', $4);
COMMIT;Expected Result
The domain write and its event record commit atomically
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 9: Bound retries with backoff
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
RetryPolicy policy = RetryPolicy.builder()
.maxAttempts(3)
.backoff(Duration.ofMillis(100), 2.0)
.retryOn(TimeoutException.class)
.build();Expected Result
Only transient timeout failures receive at most three spaced attempts
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Example 10: Expose readiness separately from liveness
Let’s isolate one Distributed Systems rule together. Run it, trace the inputs and output, then change one boundary value before reusing the pattern. Small observable programs turn syntax into a reliable engineering habit.
Keep contracts explicit: validate data before use, make ownership and mutation visible, and return a named result or failure a caller can test. Static types and a successful build help, but runtime boundaries still need intentional checks.
System Design
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080Expected Result
Traffic waits for readiness; a stuck process is restarted by liveness
Try next: make the nearest empty, invalid, missing, or boundary case observable instead of leaving behavior to an unchecked assumption.
Design Review and Worked Trace
Review Distributed Systems: Idempotency as a full trace: input, validation, calculation or side effect, output, and cleanup. State the normal result, then state the boundary behavior. If either is hard to explain, split the work into a smaller function with a narrower contract.
Keep secret values and privileged decisions outside display code. Authentication establishes an identity; authorization decides what that identity may do. Check both close to protected operations, and use intentional failure responses rather than leaking internal details or crashing generically.
- Run the normal input and record the exact result.
- Test an empty, malformed, or boundary input deliberately.
- Keep state, ownership, and side effects visible to the caller.
- Return a result or named failure that can be tested.
- Never log credentials, tokens, or sensitive input.
This discipline makes System Design code easier to review, safer to change, and more useful in production than a collection of untested syntax fragments.
Common Mistakes
- Skipping input validation because a local example happened to work.
- Hiding mutation, ownership, or failure behavior inside a convenient shortcut.
- Using a client-side display rule as the only protection for sensitive work.
- Copying code without changing an input to prove its contract.
