At-Least-Once Is the Default: Designing for Duplicate Events Without Lying to Users

Every post in this series comes back to the same idea: correctness and trust are architectural properties, enforced at boundaries and contracts — not by hoping the happy path holds. So far that idea has shown up as an authorization problem (a call is not consent) and an admission problem (a confident model output is not permission to mutate state). This post moves one layer downstream, to the point most teams stop paying attention: what happens after a mutation is admitted and turns into an event on a bus.

The last post ended with a promise: once a mutation is admitted — by a human or an agent — the rest of the platform learns about it through events, and events arrive at least once. This post is about what that guarantee actually costs you, and who is really responsible for paying it — the backend engineer who owns the consumer, and the quality function that has to prove it’s correct before it ships.

The honest answer is uncomfortable for two different roles at once. Backend engineers want to believe the broker will sort it out. Quality engineering wants to believe duplicate-delivery is an infrastructure edge case, not something on the test plan. Neither is true. At-least-once is not a bug in your Kafka cluster. It is the default physics of any distributed system, and it is a correctness requirement that has to be verified, not assumed.

“Exactly-once” is a broker myth, not a system property

Vendors advertise exactly-once semantics. What they actually deliver is exactly-once delivery between specific hops, under specific configuration, if nothing crashes at the wrong instant. The moment a consumer commits a database write and then crashes before committing its offset, the broker will redeliver — correctly, by design. The moment a producer retries after a timeout that was really just a slow acknowledgment, two copies of the same event exist — correctly, by design.

None of that is a defect in Kafka, MSK, SQS, or any other broker. It is what “at least once” means. The system did its job. The question is whether your consumer did its job when the same event showed up twice.

That is not an infrastructure question. It is a design question, and increasingly it is a quality gate question: can you prove, before release, that reprocessing a booking-confirmed event, a payout-updated event, or a claim-approved event does not create a second booking, a second payout, or a second claim?

Where duplicates actually come from

• Producer retries. A publish call times out after the broker already accepted it; the client retries and the same logical event is written twice.
• Consumer crash between processing and committing. The side effect happened. The offset commit did not. Redelivery is now guaranteed.
• Rebalances. A partition moves between consumers mid-batch; both the old and new owner may process overlapping messages.
• Upstream admission retries. If the action boundary from the previous post is not idempotent, one client retry can itself emit two distinct admitted events, not just one event delivered twice.
• Agent-driven retries. An orchestrator or agent that treats a timeout as failure will resubmit the entire plan, fanning a single intended action into several duplicate downstream events.

Four of these five sources are unrelated to your business logic. They will happen regardless of how careful your team is. That is exactly why the fix cannot live in “be more careful” — it has to live in the consumer contract, and it has to be something you can test on purpose.

The pattern: dedupe key, durable decision, fail closed on ambiguity

This is the same shape as the action-boundary pattern from the previous post, applied to consumption instead of admission.

consumer.on('BookingConfirmed', async (event) => {
  const seen = await store.getProcessedRecord(event.dedupeKey);
  if (seen && seen.eventHash === hash(event)) {
    return ack();               // already applied, safe no-op
  }
  if (seen && seen.eventHash !== hash(event)) {
    await quarantine(event);    // same key, different payload — do not guess
    return ack();
  }

  await db.transaction(async (tx) => {
    await applySideEffect(tx, event);
    await store.recordProcessed(tx, event.dedupeKey, hash(event));
  });
  ack();
});

1. Dedupe key first. Use the event’s natural business key (booking ID, payout instruction ID) or an explicit event ID — never “whatever arrived on the topic just now.”
2. Check before you act, record inside the same transaction you act in. If the side effect and the dedupe record are not committed together, a crash between them recreates the exact race you were trying to close.
3. Same key, different payload is a conflict, not a coin flip. Quarantine it and alert. Silently picking one version is how audit trails stop matching reality.
4. Ack only after the decision is durable. An early ack that later fails to persist is how “processed” and “actually applied” drift apart.

The outbox pattern is the producer-side mirror of this: write the event to an outbox row in the same transaction as the state change, then have a separate relay publish it. That way the event you emit is never orphaned from the state it describes — you cannot commit the booking and lose the notification, or publish the notification and roll back the booking.

Consumer lag versus event age — the metric teams get backwards

Dashboards default to consumer lag: how many messages are sitting unread. That number is useful for capacity planning and nearly useless for correctness or customer impact. A healthy consumer processing 50,000 messages a minute can show “high lag” during a burst and be completely fine. A stalled consumer with only 40 messages of lag can be sitting on an event that is now six hours old and blocking a customer-visible confirmation.

Event age — how old is the oldest unprocessed message — is the metric that maps to what a customer actually experiences. Alert on age crossing an SLO threshold (“booking confirmations must reflect within 30 seconds”), not on lag count crossing an arbitrary number. This distinction is exactly the kind of thing that separates a dashboard that looks healthy from a system that is healthy — the same gap that shows up in a defect density report that counts issues without weighting them by what a customer would actually notice.

Why this is a quality engineering problem, not only an ops problem

Here is the part that gets skipped in most system-design writeups: duplicate-event handling is discovered in production far more often than it is caught in a test suite, because most test suites verify the happy path — one event, once, correctly processed — and stop there.

That is a coverage gap with a name: the test plan is testing delivery, not the contract. “This consumer processes a BookingConfirmed event correctly” is not the requirement. The requirement is “this consumer processes a BookingConfirmed event correctly the first time, and produces the identical outcome every time after that.” If duplicate delivery is not a journey-level test case, you have not tested the actual contract — you have tested a convenient subset of it.

Concretely, a shift-left quality gate for an event-driven mutation should require, before merge:

• Replay tests — deliver the same event twice (identical payload) and assert the side effect and downstream state are identical to a single delivery, not merely “no error thrown.”
• Conflict tests — deliver the same dedupe key with a different payload and assert it is quarantined, not silently overwritten.
• Out-of-order tests — deliver two related events in reverse order and assert the consumer does not assume ordering the broker never promised.
• Crash-window tests — kill the consumer between side effect and offset commit (or between side effect and dedupe record) and assert recovery reproduces the same outcome, not a partial one.

None of these require exotic tooling. They require treating duplication as a first-class scenario in the same automation-first pipeline that already gates schema validation and contract tests — the same severity-weighted discipline that catches a P1 defect before it reaches ten releases, applied one layer earlier, to the event contract instead of only the API contract.

Agentic pipelines inherit the same problem they are supposed to catch

This gets sharper, not softer, once AI enters the test pipeline itself. Agentic test generation — a model proposing test cases, executing them, retrying on flaky failures, and reporting results — is subject to the exact same at-least-once physics as the production system it is testing. An orchestrator that retries a flaky test step can re-trigger a side effect (a seeded event, a test-environment mutation) without the human or the dashboard noticing that “one test run” actually fired the event twice.

If your quality engineering pipeline uses agentic generation or execution and it is not itself idempotent, you have built a system that is unreliable in exactly the dimension it exists to verify. The fix is the same pattern again: give every generated test run a dedupe key, make the environment-seeding step idempotent, and treat “the agent retried” as an expected event, not an anomaly to investigate case by case. Trustworthy AI-native quality engineering is not “let the model run tests faster.” It is applying the same architectural discipline — durable decisions, dedupe keys, fail-closed on ambiguity — to the thing generating and running the tests, not only to the thing being tested.

Fail closed on ambiguity, here too

The previous post argued that sensitive mutations should fail closed when assurance cannot complete. The consumer side has the same requirement in a quieter form: when the dedupe store is unreachable, do not guess. Do not process-and-hope. Nack and retry, or route to a dead-letter queue for manual review. An event silently applied twice because the dedupe check timed out is the same category of failure as a mutation admitted because the risk service timed out — convenient in the moment, expensive in the incident review.

A dead-letter queue is not a failure state to be embarrassed about. It is the honest alternative to guessing, and it is one of the clearest signals in a design review that a team has actually thought about what happens when the happy path does not hold.

Design and quality review checklist

For any consumer of a business-critical event stream, I want clear answers to:

1. What is the dedupe key, and is it a natural business key or an invented one that can collide?
2. Is the dedupe check and the side effect committed in the same transaction?
3. What happens on same-key-different-payload — quarantine, or silent overwrite?
4. Is replay tested as a first-class scenario, or only implied by “the code looks idempotent”?
5. Does the alerting threshold use event age, or only raw consumer lag?
6. What does the consumer do when the dedupe store itself is unavailable?
7. If AI/agentic tooling generates or executes tests against this consumer, is that tooling’s own retry behavior idempotent?
8. Can an operator reconstruct, from logs alone, whether a given customer-visible outcome came from a first delivery or a deduped replay?

If the answer to more than one of these is “we assume the broker handles it,” the gap will surface as a support ticket about a duplicate charge or a double-booked slot, not as a finding in a code review.

What comes next

Idempotent consumption keeps one event from becoming two side effects. It does not, by itself, tell you whether the write path that produced the event was designed correctly for a Tier‑1, customer-facing marketplace — sync versus async boundaries, partial-failure semantics, and what “correct” even means when three services disagree about the current state. That end-to-end design is the next post in this series.

Further reading

• Prior post — Action boundaries for AI-driven backends (proposal versus admission)
• Wire fraud controls that survive deepfakes (authorization as design)
• The Quiet War on Asset Managers — identity as control plane

#SystemDesign #EventDrivenArchitecture #Idempotency #QualityEngineering #AgenticAI #DistributedSystems #Kafka #StaffEngineer #TestAutomation #ShiftLeft