When Three Services Disagree: What Correct Means on a Tier-1 Write Path

Every post in this series comes back to the same idea: correctness and trust are architectural properties, enforced at boundaries and contracts — not by hoping the happy path holds. The last three posts applied that at three specific points: action admission when the caller is human or agent; idempotent consumption when events arrive at least once; and a quality function that makes those patterns the default instead of one team’s habit. This post asks what happens when all three are in place on individual steps, but the write path itself still crosses three services — and they disagree about what is true right now.

That is the shape of a Tier‑1, customer-facing mutation: a load booked on a freight marketplace, a claim approved on an insurance platform, a payout initiated on a payment rail. Not a CRUD update on one table. A coordinated change where inventory, payment, notification, audit, and search may each live in a different service, and the customer expects one answer when they refresh the screen.

The failure mode is not always “the API returned 500.” It is often: the customer saw success, two services agree, and a third never caught up — or caught up twice. That is a design problem. It is also the problem staff backend engineers and quality leaders share: define what “correct” means before on-call defines it in an incident channel.

Here is what that looks like at 2 a.m. A broker gets a text: load booked, carrier assigned. The booking service is telling the truth — it admitted the mutation, wrote the row, returned confirmed. Forty minutes later the carrier calls, confused, because they were never dispatched. The executor that assigns carriers crashed after the booking’s outbox row was written but before the relay published it. Nobody lied. Nobody’s code has a bug in the classic sense. Two systems were each, individually, correct — and the gap between them became the incident. That gap is what this post is about, and it does not show up in a single service’s test suite, because no single service was wrong.

Correct is not the same as done

Teams conflate three different states:

Requested — the client or agent sent a mutation; admission may not have finished.
Admitted — the system of record accepted the change; durable decision exists.
Done (customer-visible) — every dependency the product promises is satisfied, or the product honestly shows a pending state.

On a Tier‑1 path, “done” is a product contract, not a HTTP status code. Returning 200 before admission is lying. Returning 200 after admission but before async fan-out completes is fine — if the UI says pending. Returning 200 and showing confirmed while a downstream service is still unknown is how duplicate bookings and stranded payouts happen.

The design job is to name, per mutation type, which state the sync response represents and which work is explicitly async. That boundary was introduced in the action-boundaries post as “sync decides, async fans out.” Here the question is sharper: what is the single source of truth for customer-visible truth, and who is allowed to disagree with it temporarily?

When three services disagree

Consider a simplified marketplace booking (freight load matched, slot reserved, broker notified) or an insurance payment path (claim approved, disbursement queued, policyholder notified). Three common participants:

System of record — owns the mutation decision (booking row, claim status).
Downstream executor — payment capture, carrier assignment, print-and-mail check.
Projection / notify — email, mobile push, search index, partner webhook.

After a successful sync admission, any of these can fail independently:

Record admitted; executor never invoked (crash between commit and outbox relay).
Record admitted; executor succeeded; notification failed (customer thinks nothing happened).
Executor succeeded twice (retry without idempotency); record shows one booking, ledger shows two holds.
Executor still running; UI polled too early and showed failure (customer retries, creating a second admission attempt).

None of these are fixed by “use Kafka” or “use sagas” as labels. They are fixed by deciding, for each failure, whether the customer should see confirmed, pending, or failed — and whether operators can reconstruct which service is authoritative from logs alone.

Draw the sync boundary first

Before partitioning services, draw one line: what must be true before the sync response returns.

For a Tier‑1 mutation, the sync path usually includes:

Authentication, authorization, and action admission (proposal ≠ permission).
Invariant checks in the system of record (capacity, balance, policy state, cooling period).
Durable write of the decision and the primary state the product will show as truth.
Outbox row (or equivalent) for every async side effect that must eventually happen.
A response that matches the durable state: confirmed, pending_review, challenge_required, or rejected — never ambiguous success.

Everything after that line is async by definition: notifications, analytics, search, partner feeds, secondary agent follow-ups. If the product needs the customer to see “confirmed” only after payment clears, then payment clearance belongs above the line — or the product must show “pending payment” until it does. Pretending payment is async while the UI says “booked” is a correctness bug dressed as latency optimization.

Partial failure is a state, not an exception

Most runbooks treat partial failure as rare. On distributed write paths it is normal. Design for explicit states:

State	Meaning	Customer should see
Admitted, fan-out pending	Decision durable; outbox not yet delivered	Confirmed or pending (per product contract)
Admitted, executor failed	Truth in SoR; downstream stuck or dead-lettered	Pending with honest copy, or failed with support path
Admitted, executor succeeded, notify failed	Business outcome achieved; customer not informed	Confirmed on refresh; retry notify from DLQ
Conflict	Two services disagree on outcome	Never silent; quarantine and alert

A state machine in the system of record beats implicit hope. If booking can be confirmed, pending_carrier, or failed_settlement, operators and UIs share vocabulary. If those states live only in scattered logs, every partial failure becomes a forensic exercise.

One writer, many readers

The rule that scales: one service owns the mutation decision and its lifecycle state. Other services react; they do not co-decide.

Payment service consumes BookingAdmitted; it does not also write booking status.
Notification service sends email; it does not flip claim status on SMTP success.
Search projection updates index; it is rebuildable from events, not a second source of truth.

When two services both write “the” status field, you will eventually have split brain — and split brain is not only a networking term anymore. It also describes the moment two of your own services each hold a confident, internally consistent, mutually exclusive answer about the same customer, and nobody wrote the tie-breaker. Reconciliation meetings are not an architecture pattern. If a downstream service must reject work (insufficient funds, carrier unavailable), it emits a compensating or failure event; the system of record transitions state. The customer sees one narrative because one place owns the narrative.

Availability, correctness, latency — pick per mutation

You cannot maximize all three on the same path. Tier‑1 mutations should default correctness over availability on admission: fail closed when assurance cannot complete, return retryable errors instead of ambiguous success. Latency yields to honest pending states and async fan-out.

Where teams get hurt is applying marketplace peak-load instincts to money or booking paths: returning 200 with a best-effort side effect because the dependency timed out. That optimizes p95 latency and externalizes correctness to support. For load-board search or analytics, degraded mode may be acceptable. For “this load is yours” or “this claim is paid,” it is not.

Staff-level design is naming the tradeoff per mutation class in the design doc, not per incident.

Orchestration vs choreography — without the buzzword bingo

Both patterns work. The choice is operational:

Choreography (events only) — system of record emits facts; consumers react idempotently. Scales well when consumers are simple and conflicts are rare. Harder to answer “where is this booking stuck?” without a central state view.
Orchestration (explicit workflow) — a workflow engine tracks steps. Easier visibility and compensation; another system to operate and version.

Either way, the non-negotiables from earlier posts still apply: idempotent admission at the API, outbox tied to the SoR transaction, idempotent consumers on the bus, dedupe keys on conflicts, DLQ instead of guess. Orchestration does not remove at-least-once physics; it concentrates visibility.

For a small staff team on a revenue path, I usually prefer: SoR state machine + outbox + idempotent consumers, add orchestration when step count, human approval gates, or compensation logic make event-only tracing painful in postmortems.

What the quality gate should prove

A multi-service write path is not tested by a happy-path integration test alone. Before merge on a Tier‑1 mutation, I want evidence of:

Sync boundary tests — response status and body match durable SoR state; no 200 with uncommitted decision.
Partial-failure tests — executor down, notify down, dedupe store down; customer-visible state stays honest.
Replay tests — same admission key and same event twice (from the idempotent-consumer post).
Stale-read tests — UI or API poll immediately after pending admission; no false failure that triggers duplicate client retries.
Conflict tests — downstream reports failure after admission; SoR transitions to defined failure state, not silent drift.

This is the same automation-first discipline as the quality-engineering post, applied to the journey instead of only the endpoint. A thousand unit tests do not substitute for one test that kills the notify worker mid-flight and asserts the customer still sees pending, not confirmed.

Why this is a Center of Excellence problem, not a per-team one

The partial-failure table earlier in this post — admitted/pending, executor failed, notify failed, conflict — is easy to write for one team’s booking service. The hard part, and the part that actually belongs to a quality organization rather than any single backend team, is making that vocabulary canonical across every Tier‑1 mutation in the company, so “pending” means the same thing in claims as it does in payouts as it does in bookings, and every team’s design review gets asked the same ten questions instead of whichever ones the reviewer happened to remember.

That is precisely the shape of the Center of Excellence argument from the previous post: not owning every test, but owning the standard that makes “we’ll figure it out in ops” an answer that fails a design review instead of one that quietly ships. A severity-weighted defect metric tells you a release is risky. A shared partial-failure taxonomy and a required journey-test suite are what stop the risky release from shipping in the first place, on a write path where the customer-visible cost of guessing wrong is a support ticket, a chargeback, or a regulator’s question about why a claim payment silently duplicated.

Agents make stuck states worse, faster

When the caller is an agent, partial failure patterns accelerate. An agent treats timeout as failure and resubmits admission. A tool chain invokes payment before booking admission durably exists. A copilot reads stale search projection and proposes a mutation against outdated capacity.

The fixes are the same architecture, tightened: proposal vs admission, idempotency keys on every tool call, sync boundary that returns explicit pending/challenge states, and projections labeled as non-authoritative in agent context. An agent should not be given tools that mutate Tier‑1 state without the same gates a human UI would enforce — and should not interpret async projection lag as “available to book.”

Design review checklist

For any Tier‑1 mutation that crosses services:

What is the system of record, and what lifecycle states does it expose?
What exactly does the sync response mean — admitted, done, or pending?
Which side effects are above the sync line vs in the outbox?
What happens when executor succeeds but notify fails? When notify succeeds but executor failed?
Who is allowed to write status besides the SoR?
How does a client or agent retry without creating a second admission?
How does an operator answer “where is this stuck?” from logs and metrics alone?
What does the UI show during each partial-failure state — honestly?
Are replay, partial-failure, and stale-read scenarios in the merge gate?
If an agent calls this path, are tools scoped and schemas versioned like public APIs?

If more than one answer is “we’ll figure it out in ops,” the gap will surface as a customer who was told they booked a load, a carrier who was never assigned, and a broker dashboard that shows green.

The core argument

Correct ≠ done — name requested, admitted, and customer-visible done as separate states.
One system of record — one writer for mutation lifecycle; others react idempotently.
Draw the sync boundary — everything the UI promises as true must be true above the line, or the UI must say pending.
Partial failure is normal — model it as states, not surprises.
Correctness over availability on Tier‑1 admission; latency yields to honest pending.
Quality gates test journeys — partial failure, replay, and stale reads, not only happy path.
Partial-failure vocabulary is a CoE artifact — canonical across teams, not reinvented per service.
Agents amplify stuck and duplicate states — same boundaries, tighter tool contracts.

What comes next

End-to-end design tells you what should be true. It does not, by itself, tell you whether the running system is meeting that contract under load — which SLOs to set, which dashboards distinguish healthy lag from customer-visible staleness, and which incidents should drive architectural change instead of runbook patches. That operational layer is the next post in this series.