When Lag Looks Healthy and the Booking Is Stale

When Lag Looks Healthy and the Booking Is Stale

A few posts back I wrote about Tier‑1 write paths where booking can show confirmed while carrier assignment is still pending and the notification never went out. Naming those states is design work. This piece is about running the system afterward: which metrics actually tell you something is wrong, and which ones just look fine until a customer calls.

Most teams find out the hard way. Consumer lag looks normal. The API keeps returning 200. Nothing is down. One booking has been stuck for six hours anyway.

Where this showed up

A broker books a load on a freight marketplace. The UI shows confirmed. The booking service wrote the row and returned success.

Six hours later the carrier calls. They were never dispatched. On-call opens the monitoring board. Kafka consumer lag is forty-two messages. The chart is green.

Nobody was watching how old the oldest assignment event was. That message had been sitting in the queue since before breakfast. Lag told you how many messages were waiting. It did not tell you that one confirmed booking was still waiting on work that should have finished in seconds.

Lag and event age

Consumer lag is a capacity metric. It helps you decide whether to add consumers or whether the group is keeping up during a burst. Event age is different: how long has the oldest unprocessed message been sitting there? That one is closer to what the customer experiences when something is overdue.

During heavy traffic, lag can jump into the thousands while every message is still fresh. During a stall, lag can stay at forty-two while a single six-hour-old message is blocking a booking the UI already called confirmed. We have all seen the second case look “healthy” on the wrong chart.

Tier‑1 paths make this worse because they have lifecycle states: admitted, pending, confirmed, failed. Queue depth alone will not catch the case where the database is honest and the screen is not.

On a booking or payout path, alert on something tied to the product promise. No assignment event older than thirty seconds is a real SLO. Lag over ten thousand is usually someone else’s threshold pasted into Grafana.

SLOs worth defining

You do not need twenty of these. Pick a few that map to how the mutation actually moves:

Requested: the client sent it.
Admitted: the system of record accepted it.
Done for the customer: the UI is telling the truth, or it says pending honestly.

Each step can drift on its own. The SLO should say which drift is bad enough to wake someone up.

Measure	Why bother	Common substitute
Time from admission to customer-visible truth	Tells you whether confirmed, pending, or failed on screen is allowed yet	API p99 latency
Oldest event age on the mutation stream	Catches one overdue promise even when the queue looks busy	Total consumer lag
How long the UI shows pending	Pending that sits too long becomes a support call	Ticket volume
Conflict or quarantine rate	Two versions of the same decision are showing up	HTTP 5xx rate
Oldest dead-letter message	Fail-closed is turning into a pile nobody owns	DLQ count alone

Spend error budget here, not on uptime alone. An admission API can be available 99.99% of the time and still mislead customers if confirmed on the app and admitted in the database are not the same thing.

Dashboards that help at 2 a.m.

Most boards I see are built for whoever runs the broker: partition count, bytes in, consumer group lag, CPU. Fine for capacity. Less useful when the question is “where is booking 88412?”

Four things worth having on one screen:

Lifecycle for a single ID. Admitted, fan-out pending, executor failed, notify failed, conflict. Same vocabulary as the design doc. If you need a custom log query to answer that, the states never made it into operations.
Whether you are busy or stuck. Lag climbing with flat age often means burst. Age climbing with flat lag often means something is hung.
Whether the customer already saw something the database does not support. That is not a yellow warning. That is the incident.
Whether you have seen this failure class before this week. Third time is usually a design hole with a runbook taped on top.

Runbooks vs redesign

Notification failed, booking is correct in the system of record, dead-letter queue has the email: retry the notify. That is Tuesday.

Redesign territory looks more like: the customer saw confirmed and the database still says pending; the same failure happened again after the postmortem; recovery needs someone in Slack to pick which service is right; the fix is another retry on a non-idempotent path.

Postmortems that end with “updated runbook” sometimes mean two services still disagree and operators just got faster at choosing a winner by hand. That is design debt, not improved operations.

Paging

Page too much and people mute the channel. Page too little and support becomes the sensor.

Age SLO breach on Tier‑1 mutations is worth a page. So is a conflict or quarantine event before a second version of truth hits a payout. So is customer-visible state diverging from the system of record, if you can detect it.

Dead-letter growth while age is still inside the SLO, lag during a known burst, one notify failure when the booking is already correct in the database: ticket those. Annoying work, not a 2 a.m. fire.

Ship a service without an age SLO and without a way to look up one booking’s lifecycle, and you are betting on on-call to find the gap. Design review is cheaper.

Shared vocabulary

One team’s pending is another team’s confirmed, notify later. One team pages on lag, another on 5xx. You cannot run a quality program or compare error budgets when every dashboard speaks a different dialect.

Pending, confirmed, and failed need to mean the same thing in claims, payouts, and bookings. Someone has to own that. Teams can pick their own Grafana colors. They should not each invent their own definition of done.

Agents

People who hit a timeout often wait or click once. Agents tend to resubmit the whole plan. Slow consumer becomes duplicate, then conflict, then burned budget.

If agents can hit Tier‑1 mutation APIs, give them the same SLOs as the human UI and return pending or challenge when the backend is not done yet. An agent that only knows success and failure will keep pushing.

Design review

Before calling a write path done, I still ask:

What is the age SLO in seconds from admission to customer-visible truth?
Can on-call find one booking’s lifecycle from the dashboard?
Do we detect UI/database disagreement, or does support find it first?
Does the error budget cover silent wrong success?
How old is the oldest dead-letter message right now?
Did last month’s incident change a contract or only a wiki page?

Blank answers on the first two mean you have a document, not something you can operate.

Schema changes and cutovers on a path that already has pending states are a separate problem. SLOs tell you the live system is drifting. They do not tell you how to migrate without lying to the next customer in line.