Skip to main content
Data Engineering · lesson 22 of 23
Open the Data Engineering page →
Data Engineering advanced Lesson 22 of 23

Disaster Recovery: RPO, RTO, and What a Backup Actually Loses

Back up a database, keep writing to it, lose it entirely, then restore - and measure the real RPO and RTO numbers instead of quoting them from a slide.

By SkillByExample · Published

data-engineering disaster-recovery backup rpo-rto

Back up a database, let more writes happen, lose the live copy entirely, then restore — and measure the real RPO and RTO instead of quoting them.

A database, and a backup taken at a point in time

import duckdb, os, shutil, time
from datetime import datetime

DB_PATH, BACKUP_PATH = "warehouse.duckdb", "warehouse_backup.duckdb"

con = duckdb.connect(DB_PATH)
con.execute("create table orders (order_id int, amount decimal(10,2), created_at timestamp)")
con.execute("insert into orders values (1, 25.5, '2026-01-04 09:00:00'), (2, 12.0, '2026-01-04 10:00:00')")
con.close()

backup_time = datetime(2026, 1, 4, 12, 0, 0)
t0 = time.perf_counter()
shutil.copy(DB_PATH, BACKUP_PATH)
backup_duration = time.perf_counter() - t0
backup_size = os.path.getsize(BACKUP_PATH)
print(f"backed up at {backup_time}, {backup_size} bytes, took {backup_duration*1000:.1f}ms")
$ python disaster_recovery.py
== backup: copy the live database file ==
backed up at 2026-01-04 12:00:00, 536576 bytes, took 1.9ms

Writes keep happening after the backup

con = duckdb.connect(DB_PATH)
con.execute("insert into orders values (3, 8.75, '2026-01-04 14:00:00'), (4, 40.0, '2026-01-04 15:30:00')")
con.close()
disaster_time = datetime(2026, 1, 4, 16, 0, 0)
rpo = disaster_time - backup_time
print(f"disaster strikes at {disaster_time} - the live database is lost")
print(f"RPO (data since last backup, now unrecoverable): {rpo}")
== more writes happen after the backup, before disaster strikes ==
disaster strikes at 2026-01-04 16:00:00 - the live database is lost
RPO (data since last backup, now unrecoverable): 4:00:00

Two more orders (3 and 4) landed after the backup ran. RPO is not a policy someone chose in a meeting — it is a direct consequence of the backup schedule: a backup at noon and a disaster at 4pm means exactly four hours of writes are structurally unrecoverable, no matter how the restore itself goes.

The disaster, and the restore

os.remove(DB_PATH)  # the disaster: the live file is gone
t0 = time.perf_counter()
shutil.copy(BACKUP_PATH, DB_PATH)
rto = time.perf_counter() - t0
print(f"RTO (time to restore service): {rto*1000:.1f}ms")

con = duckdb.connect(DB_PATH)
rows = con.execute("select * from orders order by order_id").df()
print(rows.to_string(index=False))
print(f"restored {len(rows)} of the 4 orders that existed before the disaster - "
      f"orders 3 and 4 (created after the backup) are gone for good")
== restore from backup, and measure how long it took ==
RTO (time to restore service): 1.9ms
 order_id  amount          created_at
        1    25.5 2026-01-04 09:00:00
        2    12.0 2026-01-04 10:00:00
restored 2 of the 4 orders that existed before the disaster - orders 3 and 4 (created after the backup) are gone for good

RTO here is essentially the file-copy time — sub-millisecond for a half-megabyte file. That is the honest number for this exact setup, and also the reason RTO estimates from a small test database are misleading: a real production restore involves provisioning replacement infrastructure and moving far more data, where the copy itself is a small fraction of total RTO. What the restore cannot do, at any speed, is bring back orders 3 and 4 — they were never in the backup. Faster restores shrink RTO; they do nothing for RPO.

Practice

1. Take a second backup right before the disaster and recompute RPO.
RPO (data since last backup, now unrecoverable): 0:00:00

A backup taken at the moment of disaster has zero data loss — which is the theoretical ideal continuous backup (or replication) aims for, at the cost of running a backup constantly instead of on a schedule.

2. What is the RPO if backups run every 6 hours and the disaster happens 1 hour after the most recent one?

RPO is bounded by the backup interval, not by when the disaster happens to land within it — in the worst case (disaster right before the next scheduled backup), RPO approaches the full 6-hour interval, even though this particular disaster only cost 1 hour. Sizing an RPO requirement means sizing for the worst case, not the average case.

3. Simulate a corrupted backup file (write garbage bytes into `warehouse_backup.duckdb`) and try to restore from it.
duckdb.IOException: IO Error: The file "warehouse_backup.duckdb" exists, but it is not a valid DuckDB database file!

A restore is only as good as the backup’s own integrity — untested backups are a common real failure mode: the backup ran successfully every night for months, and the first time anyone tried to restore from one was during a real incident, when it turned out to be unreadable.

4. Why can't a shorter RTO ever fix a bad RPO, or vice versa?

They answer different questions about different failure windows. RTO is about how long restoring takes once a decision to restore is made; RPO is about how much data existed but was never captured by any backup. A one-second RTO still loses exactly the same four hours of orders in this lesson’s scenario — restoring faster does not restore data that was never saved.

Next: data platform architecture — how storage, orchestration, governance, and observability fit together as one system.

Frequently Asked Questions

What is RPO, and how is it different from RTO?
RPO (Recovery Point Objective) is how much data you can afford to lose, measured as time — the gap between your last good backup and the moment of failure. RTO (Recovery Time Objective) is how long you can afford to be down — the time it takes to actually restore service. A platform can have a strict RTO (restore in 5 minutes) and a loose RPO (lose up to a day of data), or the reverse; they are independent numbers driven by different business needs.
Why is this different from the orchestration-patterns lesson's backfill recovery?
Backfills recover from a known, bounded gap in a single pipeline's output — rerun the DAG for the missing dates, using data that still exists upstream. Disaster recovery assumes the platform itself — the database, the storage, the whole system — was lost or corrupted, and recovery means restoring from a separate backup copy, not rerunning a pipeline against data that may no longer exist.
Does a shorter backup interval always mean a better RPO?
It means a smaller upper bound on data loss, yes — but backups have a real cost (storage, the write load of taking one, the time each one takes), so RPO is a genuine trade-off against backup frequency and cost, not something to minimize for free. A platform ingesting slowly-changing reference data can tolerate a much larger RPO than one processing financial transactions.
What determines RTO in a real system, if not just 'copy the backup file'?
For a single file like this lesson's DuckDB database, restore is close to a raw copy. For a distributed warehouse or a multi-terabyte dataset, RTO includes provisioning replacement infrastructure, restoring from often-compressed cloud storage, replaying any write-ahead log since the backup, and validating the restored system before traffic returns to it — each step adds real time that a backup-file-size estimate alone won't capture.