Why not just catch the publish exception and retry it right there?

A retry in the same function still only runs if the process is alive to run it. A plain process crash between the commit and the publish call raises no exception at all for anything to catch — the problem isn't handling a specific exception better; it's that there are two independent steps at all, with an unprotected gap between them.

Isn't this just a distributed transaction problem — can't a 2-phase commit solve it?

In principle, if both the database and the message broker support the same distributed transaction protocol — but most message brokers don't, and even where 2PC is available, it's typically avoided in practice for its own well-known availability and latency costs. The outbox pattern (the next lesson) works with none of that by keeping the event inside the same database transaction as the data change.

How often does this actually happen in production?

Rarely per attempt in a real system — this lesson exaggerates the failure rate to make the bug visible in a short run — but 'rare per attempt' compounds at scale: a service placing thousands of orders a day will eventually hit the gap, and when it does, there's no error in any log pointing at the missing event, because nothing failed loudly.

Doesn't wrapping both calls in one try/except at least contain the damage?

It contains the *reporting* — the caller finds out something went wrong — but a 'rollback' attempt after the fact still can't undo a commit that's already durable if the process dies before the rollback code runs. The commit already happened; there's no way to retroactively make it not have happened.

The Dual-Write Problem: A DB Commit and a Publish Can Fail Independently

A database commit and a call to a message broker are two entirely separate operations — nothing forces them to both succeed, or both fail, together.

An order that exists, with an event that never went out

import sqlite3

class BrokerDown(Exception):
    pass

def publish_to_broker(event):
    raise BrokerDown("message broker unreachable")

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, total INTEGER)")
conn.commit()

def place_order(order_id, total):
    conn.execute("INSERT INTO orders VALUES (?, ?)", (order_id, total))
    conn.commit()
    print(f"  order {order_id} committed to the database")
    publish_to_broker({"type": "OrderPlaced", "order_id": order_id, "total": total})
    print(f"  event published")

try:
    place_order(1, 50)
except BrokerDown as e:
    print(f"Publish failed: {e}")

row = conn.execute("SELECT * FROM orders WHERE id = 1").fetchone()
print(f"Order 1 in the database: {row}")
print("Nothing anywhere records that order 1's event still needs publishing.")

$ python dual_write_problem.py
  order 1 committed to the database
Publish failed: message broker unreachable
Order 1 in the database: (1, 50)
Nothing anywhere records that order 1's event still needs publishing.

conn.commit() succeeds and is durable — order 1 is genuinely, permanently in the database. publish_to_broker then raises, and the exception is caught, but catching it doesn’t create a record anywhere that says “this order still needs its event published.” Whatever inventory, shipping, or notification system was supposed to react to OrderPlaced never will, and nothing in this program will ever know to tell it.

Practice

1. Publish first and commit the database second, and see the opposite failure mode.

import sqlite3

class DatabaseDown(Exception):
    pass

published = []

def publish_to_broker(event):
    published.append(event)

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, total INTEGER)")
conn.commit()

def commit_order_fails(order_id, total):
    raise DatabaseDown("database unreachable")

def place_order_publish_first(order_id, total):
    publish_to_broker({"type": "OrderPlaced", "order_id": order_id, "total": total})
    print(f"  event published for order {order_id}")
    commit_order_fails(order_id, total)
    print(f"  order {order_id} committed to the database")

try:
    place_order_publish_first(1, 50)
except DatabaseDown as e:
    print(f"Database write failed: {e}")

print(f"Published events: {published}")
row = conn.execute("SELECT * FROM orders WHERE id = 1").fetchone()
print(f"Order 1 in the database: {row}")
print("A consumer now believes order 1 exists, but the database disagrees.")

$ python publish_first.py
  event published for order 1
Database write failed: database unreachable
Published events: [{'type': 'OrderPlaced', 'order_id': 1, 'total': 50}]
Order 1 in the database: None
A consumer now believes order 1 exists, but the database disagrees.

Swapping the order of operations doesn’t fix anything — it just swaps which side ends up wrong. There’s no ordering of two independent calls that makes them atomic with each other.

2. Simulate a process crash between the commit and the publish call, with no exception at all.

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, total INTEGER)")
conn.commit()

def place_order_crashes_before_publish(order_id, total):
    conn.execute("INSERT INTO orders VALUES (?, ?)", (order_id, total))
    conn.commit()
    print(f"  order {order_id} committed to the database")
    raise SystemExit("process killed here, before publish_to_broker ever runs")

try:
    place_order_crashes_before_publish(1, 50)
except SystemExit as e:
    print(f"Process 'crashed': {e}")

row = conn.execute("SELECT * FROM orders WHERE id = 1").fetchone()
print(f"Order 1 in the database: {row}")
print("No exception was raised by publish_to_broker at all - it was simply never called.")
print("The bug isn't 'the broker call failed', it's that committing and publishing are two")
print("separate steps and anything can happen between them.")

$ python crash_before_publish.py
  order 1 committed to the database
Process 'crashed': process killed here, before publish_to_broker ever runs
Order 1 in the database: (1, 50)
No exception was raised by publish_to_broker at all - it was simply never called.
The bug isn't 'the broker call failed', it's that committing and publishing are two
separate steps and anything can happen between them.

There’s no BrokerDown to catch here, because the broker was never even called — the earlier example’s except BrokerDown only handles one specific way this can go wrong, not the general shape of the problem.

3. Try "rolling back" the order when publish fails, and see why that's not actually safe either.

import sqlite3

class BrokerDown(Exception):
    pass

def publish_to_broker(event):
    raise BrokerDown("message broker unreachable")

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, total INTEGER)")
conn.commit()

def place_order_with_rollback_attempt(order_id, total):
    conn.execute("INSERT INTO orders VALUES (?, ?)", (order_id, total))
    conn.commit()
    print(f"  order {order_id} committed")
    try:
        publish_to_broker({"type": "OrderPlaced", "order_id": order_id, "total": total})
    except BrokerDown as e:
        print(f"  publish failed ({e}), attempting to 'roll back' the order...")
        conn.execute("DELETE FROM orders WHERE id = ?", (order_id,))
        conn.commit()
        print(f"  order {order_id} deleted")
        raise

try:
    place_order_with_rollback_attempt(1, 50)
except BrokerDown:
    pass

row = conn.execute("SELECT * FROM orders WHERE id = 1").fetchone()
print(f"Order 1 in the database after 'rollback': {row}")
print("This only works because nothing crashed between the commit and the DELETE - a real")
print("crash right after the first commit leaves the same orphaned row this pattern was")
print("trying to avoid, just one step later.")

$ python rollback_attempt.py
  order 1 committed
  publish failed (message broker unreachable), attempting to 'roll back' the order...
  order 1 deleted
Order 1 in the database after 'rollback': None
This only works because nothing crashed between the commit and the DELETE - a real
crash right after the first commit leaves the same orphaned row this pattern was
trying to avoid, just one step later.

The rollback attempt succeeds here only because the program ran to completion without a real crash. It moves the gap one function call later instead of closing it.

4. Quantify the actual data-loss rate across several attempts with a flaky broker.

import sqlite3

class BrokerDown(Exception):
    pass

flaky_pattern = [True, False, True, True, False, True, True, True, False, True]

def publish_to_broker(order_id):
    ok = flaky_pattern[order_id % len(flaky_pattern)]
    if not ok:
        raise BrokerDown("broker unreachable")

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, total INTEGER)")
conn.commit()

missing_events = 0
for order_id in range(10):
    conn.execute("INSERT INTO orders VALUES (?, ?)", (order_id, 10))
    conn.commit()
    try:
        publish_to_broker(order_id)
    except BrokerDown:
        missing_events += 1

total_orders = conn.execute("SELECT COUNT(*) FROM orders").fetchone()[0]
print(f"Orders committed: {total_orders}")
print(f"Orders whose event was never published: {missing_events}")
print(f"Silent data loss rate: {missing_events / total_orders:.0%}")

$ python quantify_loss.py
Orders committed: 10
Orders whose event was never published: 3
Silent data loss rate: 30%

30% is exaggerated on purpose to make the pattern visible in ten iterations — a real broker fails far less often per call — but the mechanism doesn’t care about the failure rate: every commit-then-publish pair with any nonzero chance of the second step failing eventually loses an event, and this program has no way of even detecting that it happened, only counting it here because the simulation deliberately tracks the “true” outcome on the side.

An order that exists, with an event that never went out

Practice

Frequently Asked Questions