Microservices · lesson 8 of 8
Beginner
2Intermediate
5Also part of Microservices
9- + Idempotent Consumers: Surviving a Duplicate Delivery from Event-Driven Architecture
- + Transactional Outbox: Committing the Event With the Data from Event-Driven Architecture
- + Scaling & Reliability: Retries With Backoff and Jitter from System Design
- + Rate Limiting: Token Bucket and the Fixed-Window Boundary Bug from System Design
- + Docker Compose from Docker
- + Deploying FastAPI with Docker and Kubernetes: Images, Signals, and Probes from FastAPI
- + OpenTelemetry Buffers Spans — a SIGKILL Can Lose Them from Monitoring & Observability
- + Events vs Requests: Why the Order Service Doesn't Wait for Email from Event-Driven Architecture
- + Contract Testing with OpenAPI, and Running in CI from API Testing
Also part of: Event-Driven Architecture
Saga Pattern: Real Compensation When a Step Fails
The same orchestrator, now with a deliberate mid-sequence failure — real output showing exactly which completed steps get rolled back, and in what order.
By SkillByExample · Published
The forward path already works — the real test of a saga is what happens when the second step in a three-step transaction fails.
A step fails, everything before it rolls back
class SagaStep:
def __init__(self, name, action, compensation):
self.name = name
self.action = action
self.compensation = compensation
class SagaOrchestrator:
def __init__(self, steps):
self.steps = steps
def execute(self):
completed = []
for step in self.steps:
print(f"Executing: {step.name}")
try:
step.action()
except RuntimeError as e:
print(f"Step '{step.name}' failed: {e}")
self._compensate(completed)
return False
completed.append(step)
print("Saga completed successfully")
return True
def _compensate(self, completed):
print("Rolling back completed steps...")
for step in reversed(completed):
print(f"Compensating: {step.name}")
step.compensation()
def create_order():
print(" order #1002 created")
def cancel_order():
print(" order #1002 cancelled")
def charge_payment():
print(" charged $89.50")
def refund_payment():
print(" refunded $89.50")
def reserve_inventory_fails():
raise RuntimeError("widget out of stock")
def release_inventory():
print(" released 1x widget")
steps = [
SagaStep("create-order", create_order, cancel_order),
SagaStep("charge-payment", charge_payment, refund_payment),
SagaStep("reserve-inventory", reserve_inventory_fails, release_inventory),
]
SagaOrchestrator(steps).execute()
$ python saga_failure.py
Executing: create-order
order #1002 created
Executing: charge-payment
charged $89.50
Executing: reserve-inventory
Step 'reserve-inventory' failed: widget out of stock
Rolling back completed steps...
Compensating: charge-payment
refunded $89.50
Compensating: create-order
order #1002 cancelled
reserve-inventory never even reaches release_inventory — a step that never succeeded has
nothing to compensate, so it’s never in completed in the first place. The two steps that did
succeed get rolled back in reverse: payment first (matching how it was the most recent
success), then the order. This is “show the failure, then the fix” applied to a distributed
transaction — the failure output above isn’t a crash, it’s the saga doing exactly its job.
Practice
1. Fail at charge-payment instead of reserve-inventory, and confirm only create-order gets rolled back.
def charge_payment_fails():
raise RuntimeError("card declined")
def reserve_inventory():
print(" reserved 1x widget")
def release_inventory():
print(" released 1x widget")
steps = [
SagaStep("create-order", create_order, cancel_order),
SagaStep("charge-payment", charge_payment_fails, refund_payment),
SagaStep("reserve-inventory", reserve_inventory, release_inventory),
]
SagaOrchestrator(steps).execute()
$ python fail_at_payment.py
Executing: create-order
order #1002 created
Executing: charge-payment
Step 'charge-payment' failed: card declined
Rolling back completed steps...
Compensating: create-order
order #1002 cancelled
reserve-inventory never runs at all this time — the loop stops at the failed step, so
whatever comes after it in the list is simply never attempted, successfully or otherwise.
2. Fail on the very first step, where nothing has succeeded yet.
def create_order_fails():
raise RuntimeError("order service unavailable")
steps = [
SagaStep("create-order", create_order_fails, cancel_order),
]
SagaOrchestrator(steps).execute()
$ python fail_first_step.py
Executing: create-order
Step 'create-order' failed: order service unavailable
Rolling back completed steps...
“Rolling back completed steps…” still prints, but the for step in reversed(completed) loop
underneath it has nothing to iterate — completed is empty, so no compensation actually runs.
The message is honest about attempting a rollback, even when there’s nothing to roll back.
3. Add an audit log recording every executed, failed, and compensated step.
class AuditedSagaOrchestrator:
def __init__(self, steps):
self.steps = steps
self.audit_log = []
def execute(self):
completed = []
for step in self.steps:
self.audit_log.append(f"executing:{step.name}")
try:
step.action()
except RuntimeError:
self.audit_log.append(f"failed:{step.name}")
self._compensate(completed)
return False
completed.append(step)
return True
def _compensate(self, completed):
for step in reversed(completed):
self.audit_log.append(f"compensating:{step.name}")
step.compensation()
def charge_fails():
raise RuntimeError("gateway timeout")
steps = [
SagaStep("create-order", create_order, cancel_order),
SagaStep("charge-payment", charge_fails, refund_payment),
]
saga = AuditedSagaOrchestrator(steps)
saga.execute()
print("Audit log:", saga.audit_log)
$ python audited_saga.py
order #1002 created
order #1002 cancelled
Audit log: ['executing:create-order', 'executing:charge-payment', 'failed:charge-payment', 'compensating:create-order']
The audit log is a flat list of strings instead of the print statements from before — in a real system, this is what would get written to a durable log or event store so “what happened to order #1002” is answerable later without asking every service involved.
4. Make a compensation itself fail, and see what the orchestrator does about it.
def cancel_order_fails():
raise RuntimeError("order service unreachable during rollback")
def charge_fails():
raise RuntimeError("card declined")
steps = [
SagaStep("create-order", create_order, cancel_order_fails),
SagaStep("charge-payment", charge_fails, refund_payment),
]
try:
SagaOrchestrator(steps).execute()
except RuntimeError as e:
print(f"Uncaught: rollback itself failed: {e}")
$ python compensation_fails.py
Executing: create-order
order #1002 created
Executing: charge-payment
Step 'charge-payment' failed: card declined
Rolling back completed steps...
Compensating: create-order
Uncaught: rollback itself failed: order service unreachable during rollback
_compensate has no try/except of its own, so cancel_order_fails raising propagates
straight out of execute() — this orchestrator has an answer for a forward step failing, but
none at all for a rollback failing, exactly as the FAQ above warns. charge-payment failed on
its own action, so it was never in completed and refund_payment never even runs — the
inconsistency here is narrower than a payment left uncompensated: the order was created and
stays created, because the one compensation the saga did attempt (cancel_order_fails) is
exactly the one that failed.