Skip to main content
Microservices · lesson 8 of 8
Microservices advanced Lesson 8 of 8

Also part of: Event-Driven Architecture

Saga Pattern: Real Compensation When a Step Fails

The same orchestrator, now with a deliberate mid-sequence failure — real output showing exactly which completed steps get rolled back, and in what order.

By SkillByExample · Published

microservices saga distributed-transactions failure-handling

The forward path already works — the real test of a saga is what happens when the second step in a three-step transaction fails.

A step fails, everything before it rolls back

class SagaStep:
    def __init__(self, name, action, compensation):
        self.name = name
        self.action = action
        self.compensation = compensation

class SagaOrchestrator:
    def __init__(self, steps):
        self.steps = steps

    def execute(self):
        completed = []
        for step in self.steps:
            print(f"Executing: {step.name}")
            try:
                step.action()
            except RuntimeError as e:
                print(f"Step '{step.name}' failed: {e}")
                self._compensate(completed)
                return False
            completed.append(step)
        print("Saga completed successfully")
        return True

    def _compensate(self, completed):
        print("Rolling back completed steps...")
        for step in reversed(completed):
            print(f"Compensating: {step.name}")
            step.compensation()

def create_order():
    print("  order #1002 created")

def cancel_order():
    print("  order #1002 cancelled")

def charge_payment():
    print("  charged $89.50")

def refund_payment():
    print("  refunded $89.50")

def reserve_inventory_fails():
    raise RuntimeError("widget out of stock")

def release_inventory():
    print("  released 1x widget")

steps = [
    SagaStep("create-order", create_order, cancel_order),
    SagaStep("charge-payment", charge_payment, refund_payment),
    SagaStep("reserve-inventory", reserve_inventory_fails, release_inventory),
]

SagaOrchestrator(steps).execute()
$ python saga_failure.py
Executing: create-order
  order #1002 created
Executing: charge-payment
  charged $89.50
Executing: reserve-inventory
Step 'reserve-inventory' failed: widget out of stock
Rolling back completed steps...
Compensating: charge-payment
  refunded $89.50
Compensating: create-order
  order #1002 cancelled

reserve-inventory never even reaches release_inventory — a step that never succeeded has nothing to compensate, so it’s never in completed in the first place. The two steps that did succeed get rolled back in reverse: payment first (matching how it was the most recent success), then the order. This is “show the failure, then the fix” applied to a distributed transaction — the failure output above isn’t a crash, it’s the saga doing exactly its job.

Practice

1. Fail at charge-payment instead of reserve-inventory, and confirm only create-order gets rolled back.
def charge_payment_fails():
    raise RuntimeError("card declined")

def reserve_inventory():
    print("  reserved 1x widget")

def release_inventory():
    print("  released 1x widget")

steps = [
    SagaStep("create-order", create_order, cancel_order),
    SagaStep("charge-payment", charge_payment_fails, refund_payment),
    SagaStep("reserve-inventory", reserve_inventory, release_inventory),
]
SagaOrchestrator(steps).execute()
$ python fail_at_payment.py
Executing: create-order
  order #1002 created
Executing: charge-payment
Step 'charge-payment' failed: card declined
Rolling back completed steps...
Compensating: create-order
  order #1002 cancelled

reserve-inventory never runs at all this time — the loop stops at the failed step, so whatever comes after it in the list is simply never attempted, successfully or otherwise.

2. Fail on the very first step, where nothing has succeeded yet.
def create_order_fails():
    raise RuntimeError("order service unavailable")

steps = [
    SagaStep("create-order", create_order_fails, cancel_order),
]
SagaOrchestrator(steps).execute()
$ python fail_first_step.py
Executing: create-order
Step 'create-order' failed: order service unavailable
Rolling back completed steps...

“Rolling back completed steps…” still prints, but the for step in reversed(completed) loop underneath it has nothing to iterate — completed is empty, so no compensation actually runs. The message is honest about attempting a rollback, even when there’s nothing to roll back.

3. Add an audit log recording every executed, failed, and compensated step.
class AuditedSagaOrchestrator:
    def __init__(self, steps):
        self.steps = steps
        self.audit_log = []

    def execute(self):
        completed = []
        for step in self.steps:
            self.audit_log.append(f"executing:{step.name}")
            try:
                step.action()
            except RuntimeError:
                self.audit_log.append(f"failed:{step.name}")
                self._compensate(completed)
                return False
            completed.append(step)
        return True

    def _compensate(self, completed):
        for step in reversed(completed):
            self.audit_log.append(f"compensating:{step.name}")
            step.compensation()

def charge_fails():
    raise RuntimeError("gateway timeout")

steps = [
    SagaStep("create-order", create_order, cancel_order),
    SagaStep("charge-payment", charge_fails, refund_payment),
]
saga = AuditedSagaOrchestrator(steps)
saga.execute()
print("Audit log:", saga.audit_log)
$ python audited_saga.py
  order #1002 created
  order #1002 cancelled
Audit log: ['executing:create-order', 'executing:charge-payment', 'failed:charge-payment', 'compensating:create-order']

The audit log is a flat list of strings instead of the print statements from before — in a real system, this is what would get written to a durable log or event store so “what happened to order #1002” is answerable later without asking every service involved.

4. Make a compensation itself fail, and see what the orchestrator does about it.
def cancel_order_fails():
    raise RuntimeError("order service unreachable during rollback")

def charge_fails():
    raise RuntimeError("card declined")

steps = [
    SagaStep("create-order", create_order, cancel_order_fails),
    SagaStep("charge-payment", charge_fails, refund_payment),
]
try:
    SagaOrchestrator(steps).execute()
except RuntimeError as e:
    print(f"Uncaught: rollback itself failed: {e}")
$ python compensation_fails.py
Executing: create-order
  order #1002 created
Executing: charge-payment
Step 'charge-payment' failed: card declined
Rolling back completed steps...
Compensating: create-order
Uncaught: rollback itself failed: order service unreachable during rollback

_compensate has no try/except of its own, so cancel_order_fails raising propagates straight out of execute() — this orchestrator has an answer for a forward step failing, but none at all for a rollback failing, exactly as the FAQ above warns. charge-payment failed on its own action, so it was never in completed and refund_payment never even runs — the inconsistency here is narrower than a payment left uncompensated: the order was created and stays created, because the one compensation the saga did attempt (cancel_order_fails) is exactly the one that failed.

Frequently Asked Questions

Why does compensation run in reverse order?
Because later steps may depend on earlier ones having succeeded — inventory was reserved *because* payment went through, so undoing inventory first (while the charge still stands) would leave the system in a state that never legitimately existed. Reverse order unwinds the same dependency chain that built it.
What if reserve-inventory had actually succeeded before charge-payment failed?
It couldn't in this example's order (payment comes before inventory), but if the steps were reordered so inventory ran first, a payment failure afterward would compensate inventory too — `_compensate` rolls back everything in `completed`, not just the step immediately before the failure.
This lesson's compensations all print and return — what if a compensation itself fails?
Then the orchestrator has no answer for it, as Practice 4 shows directly: the exception from a failing compensation propagates straight out of `execute()`, uncaught. Production sagas typically make compensations idempotent and retry them independently of the forward steps, precisely because a failed rollback can't be treated as automatically likely to succeed just because it's a 'cleanup' operation.
Is a full audit log necessary for a saga this small?
For three steps in a script, no — but for a real saga running across services with independent logs, an audit trail (Practice 3) is often the only way to answer 'what actually happened to order #1001' after the fact, since no single service has the full picture on its own.