Event-Driven Architecture · lesson 4 of 10
Beginner
3Intermediate
6- 4 Event Sourcing: Snapshots So Replay Doesn't Get Slower Forever
- 5 CQRS: Separate Write and Read Models, and the Lag Between Them
- 6 The Dual-Write Problem: A DB Commit and a Publish Can Fail Independently
- 7 Transactional Outbox: Committing the Event With the Data
- 8 Idempotent Consumers: Surviving a Duplicate Delivery
- 9 Dead-Letter Queues: What Happens When Retries Keep Failing
Also part of Event-Driven Architecture
6- + Saga Pattern: Orchestrating a Multi-Step Transaction from Microservices
- + Saga Pattern: Real Compensation When a Step Fails from Microservices
- + At-Most-Once, At-Least-Once, and Exactly-Once from Kafka
- + Serialization and the Schema Registry from Kafka
- + Pub/Sub: Channels, Patterns, and the Message That Never Arrives from Redis
- + Incremental Loads and Change Data Capture from Data Engineering
Event Sourcing: Snapshots So Replay Doesn't Get Slower Forever
Replaying 200,000 events gets measurably slower as a log grows; a snapshot resumes from a known balance and replays only what changed since.
By SkillByExample · Published
Full replay of an event log is always correct, but a log with 200,000 events takes measurably longer to replay than one with a handful — a snapshot is a cached answer you’re always allowed to throw away.
Measuring the actual speedup, not assuming one
import time
def apply(events):
balance = 0
for event in events:
if event["type"] == "Deposited":
balance += event["amount"]
else:
balance -= event["amount"]
return balance
events = []
for i in range(200_000):
if i % 3 == 0:
events.append({"type": "Withdrawn", "amount": 1})
else:
events.append({"type": "Deposited", "amount": 1})
start = time.perf_counter()
full_replay_balance = apply(events)
full_replay_time = time.perf_counter() - start
print(f"Full replay ({len(events)} events): balance={full_replay_balance}, took {full_replay_time * 1000:.1f}ms")
snapshot_at = 150_000
snapshot_balance = apply(events[:snapshot_at])
snapshot = {"balance": snapshot_balance, "as_of_event": snapshot_at}
print(f"Snapshot taken at event {snapshot['as_of_event']}: balance={snapshot['balance']}")
start = time.perf_counter()
remaining = events[snapshot["as_of_event"]:]
resumed_balance = snapshot["balance"] + apply(remaining)
snapshot_replay_time = time.perf_counter() - start
print(f"Snapshot + replay ({len(remaining)} events): balance={resumed_balance}, took {snapshot_replay_time * 1000:.1f}ms")
print(f"Speedup: {full_replay_time / snapshot_replay_time:.1f}x")
$ python event_sourcing_snapshots.py
Full replay (200000 events): balance=66666, took 9.6ms
Snapshot taken at event 150000: balance=50000
Snapshot + replay (50000 events): balance=66666, took 2.3ms
Speedup: 4.2x
Both paths agree on the final balance — 66666 either way — which is the actual point of a
snapshot: it changes how much work replay does, never what answer it produces. The exact
speedup will vary run to run (measured here, not assumed), but it tracks the ratio of events
skipped to events replayed, which is 150,000 : 50,000 in this run.
Practice
1. Take a snapshot at several different points and confirm each one matches a direct balance-history calculation.
def apply(events):
balance = 0
for event in events:
if event["type"] == "Deposited":
balance += event["amount"]
else:
balance -= event["amount"]
return balance
events = [
{"type": "Deposited", "amount": 100},
{"type": "Withdrawn", "amount": 30},
{"type": "Deposited", "amount": 50},
{"type": "Withdrawn", "amount": 20},
]
snapshot_points = [1, 2, 3, 4]
for point in snapshot_points:
snapshot_balance = apply(events[:point])
print(f"Snapshot after {point} event(s): balance={snapshot_balance}")
full_history = [apply(events[:i + 1]) for i in range(len(events))]
print(f"Direct balance_history: {full_history}")
$ python multiple_snapshots.py
Snapshot after 1 event(s): balance=100
Snapshot after 2 event(s): balance=70
Snapshot after 3 event(s): balance=120
Snapshot after 4 event(s): balance=100
Direct balance_history: [100, 70, 120, 100]
A “snapshot” is nothing more than apply called on a prefix of the log — there’s no special
snapshot machinery here, just a convention of remembering which prefix you already computed.
2. Take a snapshot, then let new events arrive, and confirm resuming from the (now not-newest) snapshot still matches a full replay.
def apply(events):
balance = 0
for event in events:
if event["type"] == "Deposited":
balance += event["amount"]
else:
balance -= event["amount"]
return balance
events = [
{"type": "Deposited", "amount": 100},
{"type": "Withdrawn", "amount": 30},
]
snapshot = {"balance": apply(events), "as_of_event": len(events)}
print(f"Snapshot: {snapshot}")
events.append({"type": "Deposited", "amount": 40})
events.append({"type": "Withdrawn", "amount": 10})
print(f"New events arrived, log now has {len(events)} total events")
new_events = events[snapshot["as_of_event"]:]
resumed_balance = snapshot["balance"] + apply(new_events)
print(f"Replaying only the {len(new_events)} new event(s): balance={resumed_balance}")
print(f"Full replay for comparison: {apply(events)}")
$ python resume_after_new_events.py
Snapshot: {'balance': 70, 'as_of_event': 2}
New events arrived, log now has 4 total events
Replaying only the 2 new event(s): balance=100
Full replay for comparison: 100
The snapshot doesn’t need to be the newest thing available to be useful — as_of_event says
precisely where to pick replay back up, so any older snapshot plus everything after it is just
as correct as a full replay, only cheaper.
3. Persist both the events and the snapshot itself in SQLite, so the snapshot survives a restart.
import sqlite3
conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE events (type TEXT, amount INTEGER)")
conn.execute("CREATE TABLE snapshots (balance INTEGER, as_of_event INTEGER)")
for amount, kind in [(100, "Deposited"), (30, "Withdrawn"), (50, "Deposited")]:
conn.execute("INSERT INTO events VALUES (?, ?)", (kind, amount))
conn.commit()
def apply_range(conn, start_rowid=0):
balance = 0
for event_type, amount in conn.execute(
"SELECT type, amount FROM events WHERE rowid > ? ORDER BY rowid", (start_rowid,)
):
balance += amount if event_type == "Deposited" else -amount
return balance
current_count = conn.execute("SELECT COUNT(*) FROM events").fetchone()[0]
snapshot_balance = apply_range(conn, 0)
conn.execute("INSERT INTO snapshots VALUES (?, ?)", (snapshot_balance, current_count))
conn.commit()
print(f"Snapshot saved: balance={snapshot_balance}, as_of_event={current_count}")
conn.execute("INSERT INTO events VALUES ('Withdrawn', 20)")
conn.commit()
saved_balance, saved_as_of = conn.execute("SELECT balance, as_of_event FROM snapshots").fetchone()
resumed_balance = saved_balance + apply_range(conn, saved_as_of)
print(f"Resumed from saved snapshot: balance={resumed_balance}")
$ python persisted_snapshot.py
Snapshot saved: balance=120, as_of_event=3
Resumed from saved snapshot: balance=100
as_of_event here is a row count, which only lines up with SQLite’s rowid because rows are
never deleted and rowid is assigned sequentially starting at 1 — a real system storing this
long-term would use an explicit sequence column rather than lean on that assumption.
4. Deliberately introduce an off-by-one in as_of_event and see how it silently corrupts the resumed balance.
def apply(events):
balance = 0
for event in events:
if event["type"] == "Deposited":
balance += event["amount"]
else:
balance -= event["amount"]
return balance
events = [
{"type": "Deposited", "amount": 100},
{"type": "Withdrawn", "amount": 30},
{"type": "Deposited", "amount": 50},
]
correct_snapshot = {"balance": apply(events), "as_of_event": len(events)}
print(f"Correct snapshot: {correct_snapshot}")
off_by_one_snapshot = {"balance": apply(events), "as_of_event": len(events) - 1}
print(f"Off-by-one snapshot (as_of_event wrong by 1): {off_by_one_snapshot}")
events.append({"type": "Withdrawn", "amount": 10})
correct_result = correct_snapshot["balance"] + apply(events[correct_snapshot["as_of_event"]:])
buggy_result = off_by_one_snapshot["balance"] + apply(events[off_by_one_snapshot["as_of_event"]:])
print(f"Resumed from correct snapshot: {correct_result}")
print(f"Resumed from off-by-one snapshot: {buggy_result}")
print(f"Full replay (ground truth): {apply(events)}")
$ python off_by_one_snapshot.py
Correct snapshot: {'balance': 120, 'as_of_event': 3}
Off-by-one snapshot (as_of_event wrong by 1): {'balance': 120, 'as_of_event': 2}
Resumed from correct snapshot: 110
Resumed from off-by-one snapshot: 160
Full replay (ground truth): 110
The off-by-one snapshot replays the third event (Deposited 50) a second time on top of a
balance that already includes it — no exception is raised anywhere, the result is just wrong.
This is exactly why snapshot logic needs tests that compare against a full replay on the same
data, not tests that only check the snapshot path in isolation.