A chain of dominoes falling forward, with a second smaller chain of hands carefully standing each one back up in reverse order behind it Tech
AI-generated, Working Theory
Tech · ◉ Evergreen

You can't undo it, so plan the apology

by · ·4 min·Working Theory

Across services there's no transaction that spans them all. The saga pattern trades atomic rollback for a chain of compensating actions — everything either happens, or gets undone by hand.

Here’s a request that sounds trivial and isn’t: book the flight, charge the card, and reserve the hotel — but if any one of them fails, none of them should have happened. On a single database this is a solved problem. You wrap the three writes in a transaction, and the database gives you a word that means “all or nothing”: atomicity. If the third write fails, the first two vanish as if they were never attempted.

Now split those three steps across three services — flights, payments, hotels — each with its own database, its own deploy schedule, its own team. The word “atomicity” quietly stops meaning anything. There is no transaction that spans all three. There is no lock you can hold across the internet. The moment you charged the card, that charge is real, in another company’s ledger, and no ROLLBACK you type will reach it.

The instinct is to reach for a protocol that restores the old guarantee — a two-phase commit, a coordinator that makes everyone promise before anyone acts. It exists, and at service scale it’s mostly a trap: it holds locks across systems you don’t control, and when the coordinator dies mid-decision, everyone else is frozen holding a promise they can’t resolve. You’ve traded a consistency problem for an availability problem.

The saga pattern makes a different bargain. It gives up on “all or nothing happening” and replaces it with “everything either happens, or gets undone by hand.”

A saga is a chain of local transactions, each with an apology

You break the big cross-service transaction into a sequence of small local ones, each of which commits normally in its own database. That’s the forward path: reserve the flight, charge the card, book the hotel. The trick is that for each forward step you also write its inverse — a compensating action that semantically undoes it. Not a rollback; the flight reservation already committed and can’t un-commit. A compensation is a new, real action that means “cancel the reservation,” “refund the card,” “release the room.”

If every step succeeds, the saga is done. If step three fails, you don’t rewind — you can’t — you run the compensations for the steps that already succeeded, in reverse: refund the card, cancel the flight. The system arrives back at a consistent state not by pretending nothing happened, but by doing the honest work of walking it back.

reserve flight charge card book hotel ✕ fails refund card compensate cancel flight compensate unwind in reverse → back to a consistent state
A saga trades atomic rollback for compensating actions: when a later step fails, the earlier ones are undone by real, reverse operations rather than erased. Original diagram · Working Theory

What the pattern makes you confront

The saga is less an algorithm than a forcing function. It makes you answer a question the single-database transaction let you dodge: what does it mean to undo this? For some steps the answer is clean — a refund reverses a charge. For others it’s genuinely hard. You can cancel a reservation, but you can’t un-send the confirmation email, and you can’t un-ship a package that’s already on a truck. Those steps have no clean compensation, and the saga forces you to see that early, while you’re designing, instead of at 2 a.m. during an incident. The usual fix is ordering: put the irreversible steps last, so that by the time you take them, every reversible step before them has already succeeded.

Two more things fall out of the design. First, compensations have to be idempotent and safe to retry, because the relay that runs them will crash and re-run — the same discipline that makes “make it safe to try again” worth its own essay. A refund that fires twice must not pay the customer twice. Second, a saga is never atomic in the instant sense; there’s a window where the flight is booked and the card isn’t yet charged, a state a user or another service can observe. You either design the intermediate states to be tolerable or you hide them behind a “pending” status, but you don’t get to pretend they don’t exist.

You choose how to run the chain, too — orchestration, where one coordinator explicitly calls each step and knows how to compensate, or choreography, where each service listens for the previous step’s event and reacts. Orchestration is easier to reason about and debug; choreography couples services more loosely but scatters the logic. For anything with more than a few steps, the explicit coordinator usually earns its keep.

The deeper shift is philosophical. A local transaction promises the outside world never sees a half-done state. A saga promises only that a half-done state won’t last. That’s a weaker promise — and across systems you don’t own, it’s the strongest one honestly available.

The systems, to look up: the saga pattern originates in Garcia-Molina & Salem, “Sagas” (1987), and is developed for microservices in Chris Richardson’s “Microservices Patterns.” Contrast with two-phase commit (2PC) and with the transactional outbox, which is about reliably publishing one event, not unwinding many.

Sources

  • The saga pattern — Garcia-Molina & Salem (1987)
  • Chris Richardson, Microservices Patterns
  • contrast with two-phase commit (2PC) and the transactional outbox

Liked this? Get the next one in Working Theory.

Going weekly in August (it's in beta now). One genuinely interesting read on building, the brain, and the science most people missed.

Subscribe →
Got a reaction, a counter-example, or something I missed? Reply by email — I read everything.
◉ join in

Where have you hit this — in a product you use, or one you're building?

Threads open here soon. For now, the conversation lives two clicks away — discuss on GitHub, or just reply by email. I read and answer everything.