You can't save the row and send the email in the same breath

Here’s a bug that has humbled almost everyone who’s shipped a real backend. The order is placed. Your code does two things, right next to each other: it saves the order to the database, and it sends the confirmation to the payment service, or drops a “new order” event on the queue, or fires the welcome email. Two lines. What could go wrong.

What goes wrong is the space between the two lines. Your database is one system. The queue, the email provider, the payment API — each is a different system, on a different machine, reached over a network that can fail at any instant. There is no transaction that spans both. So imagine the process crashes in the gap: the order committed, but the event never went out. Now your database says the order exists and the rest of the world has never heard of it — no confirmation, no fulfillment, a customer charged into silence. Flip the order of the two lines and you get the mirror-image disaster: the email goes out, the process dies, the order never commits, and you’ve cheerfully confirmed something that doesn’t exist. There is no ordering of “write here, then tell over there” that closes the gap. This is the dual-write problem, and you cannot code your way out of it by being careful, because the failure lives in the network and the crash, not in your logic.

The fix is quietly brilliant, and it starts by refusing to write to two systems at once. Instead of publishing the event to the outside world, you write it into your own database — a plain row in an “outbox” table — in the very same transaction that saves the order. Now there’s only one write, to one system, and your database’s transaction guarantees do what they’re built to do: either both the order and its outbox row commit, or neither does. There is no in-between state to crash into. The order can never exist without a pending record that it needs to be announced.

Then a second, separate process — the relay — does nothing but watch that outbox table, pick up unsent rows, publish them to the queue or call the API, and mark them done. The genius is where the danger moved. If the relay crashes mid-publish, nothing is lost: the row is still sitting in the outbox marked unsent, and when the relay comes back it tries again. You’ve converted an impossible “did both systems agree?” problem into an ordinary “keep retrying until it sticks” problem, which is a problem computers are extremely good at.

That conversion comes with one string attached. Because the relay retries, a message can go out more than once — the crash might happen after publishing but before marking the row done, so the relay, blameless, sends it again. The outbox gives you at-least-once delivery, never exactly-once (exactly-once across systems is the fairy tale). Which means the thing on the receiving end has to be able to see the same event twice and shrug — it has to be idempotent. The outbox and idempotent consumers are two halves of one design: the outbox guarantees the event will get out, idempotency guarantees it’s safe that it might get out twice.

The lesson generalizes past queues and emails. Any time your code changes your own state and then has to tell someone else about it, you’re standing over that same gap, and “just do both” is a bet that the machine won’t die in the half-second between them. It will, eventually, at the worst possible moment — that’s what production is. Make the record of “I need to tell the world” part of the same thing you’re already saving, and let a patient retry loop do the telling.

The systems, to look up: the dual-write problem; the transactional outbox pattern (Chris Richardson, microservices.io / Microservices Patterns); change-data-capture and log-tailing as the relay (the Debezium approach); the polling-publisher variant; at-least-once delivery with idempotent consumers; and why exactly-once delivery across independent systems is effectively impossible (the Two Generals result).