Where does the duplicate event in this lesson actually come from?

The transactional outbox pattern's relay: if it crashes after calling the broker but before marking its row published, the next run republishes the same row. That duplicate is a normal, expected outcome of at-least-once delivery, not a bug — the delivery is already duplicated by the time a consumer sees it, so the consumer is the last place left to handle it.

Why track processed_event_ids instead of just checking 'have I seen this order_id before'?

An order_id identifies the business entity, not the delivery — a customer legitimately placing two separate orders would have two different order_ids that should both be credited. The event_id identifies this specific delivery attempt, which is the thing that can legitimately repeat and needs deduplicating.

Does every event consumer need this check?

No — a consumer that just logs what it received is naturally idempotent already, since printing the same line twice does no real harm, while one that credits loyalty points is not, since crediting twice really does double-pay. Add the dedup check where a duplicate would cause an observable, incorrect side effect; skip it where duplicates are harmless noise.

What if two duplicate deliveries of the same event arrive at almost the same instant?

The set-based version in the main example has a check-then-act race: two concurrent calls could both pass the `in processed_event_ids` check before either adds to the set. A database UNIQUE constraint closes that gap — the database itself rejects the second INSERT atomically, which a Python-level set can't guarantee under real concurrency.

Idempotent Consumers: Surviving a Duplicate Delivery

An idempotent consumer doesn’t prevent a duplicate delivery — it makes processing the same event twice produce the same result as processing it once.

The same balance bug, with and without deduplication

deliveries = [
    {"event_id": "evt-1", "order_id": 1, "points": 10},
    {"event_id": "evt-2", "order_id": 2, "points": 5},
    {"event_id": "evt-1", "order_id": 1, "points": 10},
]

loyalty_balance_naive = 0

def credit_naive(event):
    global loyalty_balance_naive
    loyalty_balance_naive += event["points"]
    print(f"  credited {event['points']} points for {event['event_id']} (order {event['order_id']})")

for event in deliveries:
    credit_naive(event)

print(f"Naive consumer's final balance: {loyalty_balance_naive} (should be 15)")

print()

loyalty_balance_idempotent = 0
processed_event_ids = set()

def credit_idempotent(event):
    global loyalty_balance_idempotent
    if event["event_id"] in processed_event_ids:
        print(f"  skipping {event['event_id']}, already processed")
        return
    loyalty_balance_idempotent += event["points"]
    processed_event_ids.add(event["event_id"])
    print(f"  credited {event['points']} points for {event['event_id']} (order {event['order_id']})")

for event in deliveries:
    credit_idempotent(event)

print(f"Idempotent consumer's final balance: {loyalty_balance_idempotent}")

$ python idempotent_consumers.py
  credited 10 points for evt-1 (order 1)
  credited 5 points for evt-2 (order 2)
  credited 10 points for evt-1 (order 1)
Naive consumer's final balance: 25 (should be 15)

  credited 10 points for evt-1 (order 1)
  credited 5 points for evt-2 (order 2)
  skipping evt-1, already processed
Idempotent consumer's final balance: 15

deliveries has evt-1 twice, simulating exactly the redelivery the outbox relay can produce. The naive consumer has no memory of what it’s already done, so it credits evt-1 a second time and silently over-pays by 10 points. The idempotent version’s only real change is checking processed_event_ids before acting — the moment an event’s ID is recognized, the rest of the function never runs.

Practice

1. Persist processed event IDs in SQLite instead of a Python set, and confirm dedup survives a simulated restart.

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE processed_events (event_id TEXT PRIMARY KEY)")
conn.execute("CREATE TABLE loyalty (balance INTEGER)")
conn.execute("INSERT INTO loyalty VALUES (0)")
conn.commit()

def already_processed(event_id):
    row = conn.execute("SELECT 1 FROM processed_events WHERE event_id = ?", (event_id,)).fetchone()
    return row is not None

def credit(event):
    if already_processed(event["event_id"]):
        print(f"  skipping {event['event_id']}, already processed")
        return
    conn.execute("UPDATE loyalty SET balance = balance + ?", (event["points"],))
    conn.execute("INSERT INTO processed_events VALUES (?)", (event["event_id"],))
    conn.commit()
    print(f"  credited {event['points']} points for {event['event_id']}")

deliveries = [
    {"event_id": "evt-1", "points": 10},
    {"event_id": "evt-2", "points": 5},
]
for event in deliveries:
    credit(event)

print("Process 'restarts' (new consumer instance, same database)...")
credit({"event_id": "evt-1", "points": 10})

final_balance = conn.execute("SELECT balance FROM loyalty").fetchone()[0]
print(f"Final balance: {final_balance}")

$ python persisted_dedup.py
  credited 10 points for evt-1
  credited 5 points for evt-2
Process 'restarts' (new consumer instance, same database)...
  skipping evt-1, already processed
Final balance: 15

A Python set only lives as long as the process does — a real consumer that restarts (crashes, redeploys, scales to a new instance) needs processed_event_ids to be as durable as the balance it’s protecting, which is exactly what moving it into the same database gives you.

2. Use a UNIQUE constraint and catch the resulting error, instead of checking membership first.

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE processed_events (event_id TEXT PRIMARY KEY)")
conn.commit()

def credit_via_constraint(event_id):
    try:
        conn.execute("INSERT INTO processed_events VALUES (?)", (event_id,))
        conn.commit()
        print(f"  processed {event_id} (first time)")
    except sqlite3.IntegrityError:
        print(f"  {event_id} already processed (caught by UNIQUE constraint)")

credit_via_constraint("evt-1")
credit_via_constraint("evt-2")
credit_via_constraint("evt-1")

$ python unique_constraint_dedup.py
  processed evt-1 (first time)
  processed evt-2 (first time)
  evt-1 already processed (caught by UNIQUE constraint)

There’s no separate “check if it exists” step here — the INSERT itself is the check, enforced atomically by the database’s PRIMARY KEY constraint. A SELECT followed by an INSERT (as in the main example and Practice 1) has a gap between the two where a concurrent duplicate could slip through; this pattern has none.

3. Compare a consumer that doesn't need dedup against one that does, side by side.

deliveries = [
    {"event_id": "evt-1", "order_id": 1, "points": 10},
    {"event_id": "evt-1", "order_id": 1, "points": 10},
]

def log_received(event):
    print(f"  [log] received {event['event_id']} for order {event['order_id']}")

for event in deliveries:
    log_received(event)
print("Logging twice is harmless - it's naturally idempotent, no dedup needed.")

print()

loyalty_balance = 0
processed_event_ids = set()

def credit_points(event):
    global loyalty_balance
    if event["event_id"] in processed_event_ids:
        return
    loyalty_balance += event["points"]
    processed_event_ids.add(event["event_id"])

for event in deliveries:
    credit_points(event)
print(f"Crediting points twice would double-pay a customer - dedup is required here.")
print(f"Balance with dedup: {loyalty_balance}")

$ python compare_consumers.py
  [log] received evt-1 for order 1
  [log] received evt-1 for order 1
Logging twice is harmless - it's naturally idempotent, no dedup needed.

Crediting points twice would double-pay a customer - dedup is required here.
Balance with dedup: 10

Idempotency isn’t a property every consumer needs bolted on by default — it’s specifically for consumers whose side effect isn’t already safe to repeat.

4. Show that ID-based dedup can't catch a business-level duplicate carrying a different event ID.

loyalty_balance = 0
processed_event_ids = set()

def credit_points(event):
    global loyalty_balance
    if event["event_id"] in processed_event_ids:
        print(f"  skipping {event['event_id']}, already processed")
        return
    loyalty_balance += event["points"]
    processed_event_ids.add(event["event_id"])
    print(f"  credited {event['points']} points for {event['event_id']}")

deliveries = [
    {"event_id": "evt-1", "order_id": 1, "points": 10},
    {"event_id": "evt-2", "order_id": 1, "points": 10},
]

for event in deliveries:
    credit_points(event)

print(f"Final balance: {loyalty_balance}")
print("Both events had a different event_id, so ID-based dedup let both through - even")
print("though they represent the exact same order being credited twice under the hood.")

$ python business_level_duplicate.py
  credited 10 points for evt-1
  credited 10 points for evt-2
Final balance: 20
Both events had a different event_id, so ID-based dedup let both through - even
though they represent the exact same order being credited twice under the hood.

Event-ID deduplication only ever protects against the same delivery arriving twice — a genuinely separate event that happens to describe the same underlying business fact needs its own, business-specific check (like a unique constraint on order_id in a real loyalty table), which is a different problem from the one this lesson solves.

The same balance bug, with and without deduplication

Practice

Frequently Asked Questions