Saga pattern and compensating actions

Payment captured. Inventory reserved. Shipping label failed — and now the order sits in a state no single service owns. The saga pattern sequences distributed steps with compensating actions: explicit undo logic that must be idempotent, ordered, and safe to run more than once.

Engineering8 min read
Saga patternDistributed transactionsCompensating actionsEvent-drivenMicroservices
Share

The checkout flow touched three services. Payment authorized. Inventory decremented. Carrier API timed out on label creation. Payment was not captured yet — but inventory already showed forty units gone. Support found the order in PENDING_LABEL nowhere in the runbook. Manual SQL fixed one row; three duplicate retry webhooks created two more partial orders. The saga pattern with compensating actions exists because there is no distributed COMMIT across payment, inventory, and shipping microservices — only local transactions, asynchronous messages, and failure at any step. Sagas make multi-step business processes explicit: forward steps advance state, failures trigger compensating steps that semantically undo prior work, and every handler assumes at-least-once delivery.

Two-phase commit across heterogeneous services is operationally heavy — coordinators, lock timeouts, fragile failure recovery. Most product teams reject it. Naive choreography — "each service listens and reacts" — hides orchestration in implicit event chains nobody can trace when step four fails. Sagas occupy the middle: visible process state, defined compensation order, idempotent handlers, and acceptance that consistency is eventual within bounded time.

Choreography vs orchestration: two saga shapes

Both implement the same idea — a sequence of local transactions with compensations — but differ in control flow.

Choreography. Each service completes its step and publishes an event. Downstream services subscribe and act. Compensation propagates as compensating events in reverse dependency order. No central coordinator.

OrderCreated → PaymentReserved → InventoryReserved → LabelFailed
  → InventoryReleased (compensate) → PaymentReleased (compensate) → OrderCancelled

Pros: loose coupling, no single point of failure for orchestration logic. Cons: implicit graph hard to visualize, compensation order encoded in topic conventions, debugging requires distributed trace literacy.

Orchestration. A saga coordinator (state machine service) issues commands to participants, tracks current step, and dispatches compensations on failure. Participants report success or failure; coordinator decides next forward or backward step.

Coordinator: RESERVE_PAYMENT → ok → RESERVE_INVENTORY → ok → CREATE_LABEL → fail
  → COMPENSATE_INVENTORY → COMPENSATE_PAYMENT → SAGA_FAILED

Pros: explicit state, centralized compensation policy, easier timeouts and human escalation. Cons: coordinator availability matters; can become a god service if boundaries blur.

FactorChoreographyOrchestration
Step countFew, stableMany, branching
Team topologyAutonomous teams per servicePlatform owns cross-cutting flows
Visibility needsHigh trace toolingBuilt-in saga state table
Compensation complexitySimple reverse eventsConditional partial compensation
Timeout / SLA enforcementPer-service timersCoordinator deadlines

Neither is universally correct. Checkout with five steps and revenue impact usually earns an orchestrator. Two-service notification pipelines often survive choreography.

Compensating actions are not rollbacks

Database rollback undoes a transaction before commit. Compensating actions undo committed business effects in other services — semantically, not mechanically.

Forward stepNaive rollback (wrong)Compensating action (right)
Capture paymentDELETE payment rowIssue refund or release authorization hold
Reserve inventoryRevert transaction if same DBRelease reservation; restore available count
Send confirmation emailImpossible to unsendSend correction email; mark saga failed
Create shipmentDELETE if same serviceVoid label; cancel pickup request
Publish domain eventHope nobody consumedEmit compensating event; consumers reverse

Compensations are business operations with their own rules. Refunding a captured payment may take three days. Voiding a label may fail if carrier already scanned the package. Compensation logic must handle "undo not possible" — escalate to human, partial compensation, or accept residual state with reconciliation.

A compensating action that assumes the forward step never partially succeeded will corrupt state on retry.

Design compensations idempotent from day one — the same patterns as Postgres idempotent upserts: dedup key per compensation command, unique constraint on (saga_id, step, direction), upsert compensation status so duplicate messages do not double-refund.

Saga state and the outbox at each step

Each forward and compensating step should be one local transaction: update domain state, record saga progress, emit next command or event.

BEGIN;
  UPDATE inventory
  SET reserved = reserved - $qty, available = available + $qty
  WHERE sku = $1 AND order_id = $2;
 
  INSERT INTO saga_step_log (saga_id, step, direction, status, idempotency_key)
  VALUES ($saga_id, 'release_inventory', 'compensate', 'completed', $key)
  ON CONFLICT (idempotency_key) DO NOTHING;
 
  INSERT INTO outbox (aggregate_id, event_type, payload)
  VALUES ($saga_id, 'InventoryReleased', $payload);
COMMIT;

The transactional outbox pattern ensures compensation events survive process crash — critical because failure often triggers compensation during instability. Publishing compensation events outside the transaction recreates the dual-write gap at the worst moment.

Saga state storage options:

Coordinator table. saga_instances(id, type, current_step, status, context_json, updated_at) — orchestrator reads and writes.

Event log. Append-only saga events; projector builds current state — aligns with event sourcing at saga granularity.

Per-service participation log. Each service records its step for a saga_id — choreography model; reconciliation joins logs.

Status values need precision: RUNNING, COMPENSATING, COMPLETED, FAILED, COMPENSATION_FAILED (requires human). COMPENSATION_FAILED is not optional in mature systems — some undo paths fail.

Ordering compensation: reverse, partial, and pivot steps

When step N fails, compensate N-1, N-2, … down to completed forward steps. Order matters when downstream compensation depends on upstream release — release inventory before refunding payment if refund triggers automatic reorder that re-reserves stock.

Reverse order rule. Compensate in reverse completion order of forward steps. If payment and inventory both succeeded, inventory typically compensates first (restore stock) before payment (release funds) — domain dependent, document per saga type.

Partial forward progress. Step three fails; steps one and two committed. Compensate two, then one — never skip because step three never started.

Non-compensatable steps. Email sent, external API called without void endpoint. Saga marks COMPLETED_WITH_IRREVERSIBLE_SIDE_EFFECT, triggers human workflow, does not pretend full rollback. Designing forward steps to be compensatable — delay send until saga commits — is architectural choice.

Pivot / retry vs compensate. Transient carrier timeout might retry label creation three times before compensating inventory. Coordinator distinguishes retryable failures (same step, backoff) from terminal failures (compensate chain). Blurring the two causes premature refunds or infinite retry loops.

Forward:  S1 ──► S2 ──► S3 (fail terminal)
Compensate:      ◄── C2 ◄── C1

Timeouts, staleness, and saga garbage collection

Long-running sagas stall when a participant never responds. Without timeouts, inventory stays reserved indefinitely.

Per-step deadline. Coordinator schedules timeout event; on fire, treat as failure and compensate completed steps — or escalate if compensation risky.

Saga TTL. Sagas in RUNNING longer than business SLA (e.g. 24 hours for checkout) auto-fail and compensate. Alert before auto-action when possible.

Stale compensation. Compensation message arrives after forward retry succeeded — classic race. Version counters on saga instance: ignore compensation for step_version superseded by successful retry. Monotonic saga_epoch per instance.

Idempotent forward retry. Step three fails transiently; retry must not re-execute side effects from step three's partial attempt. Use idempotency keys on external APIs; check participation log before calling carrier create twice.

Testing and observability for saga correctness

Sagas fail in production in ways unit tests miss: duplicate delivery, compensation-before-forward race, partial external success.

Test matrix.

ScenarioExpected outcome
All steps succeedCOMPLETED, no compensation
Step k fails terminalSteps 1..k-1 compensated, FAILED
Duplicate forward commandSingle side effect
Duplicate compensationSingle undo
Crash after forward commit, before outboxRelay delivers; no double forward
Compensation failsCOMPENSATION_FAILED, alert

Tracing. Propagate saga_id and step on every log line, message header, and span attribute. Dashboards: sagas by status, mean completion time, compensation rate, stuck sagas past deadline.

Reconciliation jobs. Nightly compare payment captures to inventory reservations to shipped labels — sagas heal drift automation misses.

Saga boundaries and when not to use the pattern

Not every multi-service interaction is a saga. Read paths that aggregate data from three APIs — no saga required. Single forward write with async notification — outbox plus idempotent consumer suffices. Reserve saga complexity for workflows where multiple committed steps must be undone as a unit when later steps fail.

Signals that sagas earn their cost:

  • Revenue or inventory impact if left in partial state overnight
  • External partners (payment, shipping, ERP) each with independent failure modes
  • User-visible process spanning minutes to hours (onboarding, provisioning, fulfillment)
  • Regulatory requirement to prove audit trail of forward and compensating steps

Signals to simplify:

  • Two services, one event, compensation is "delete the row we just inserted"
  • Failure acceptable as manual cleanup queue reviewed weekly
  • All steps share one database — use a local transaction instead

Over-saga-ing creates coordinator bottlenecks and compensation chains for flows that a single BEGIN…COMMIT would solve. Under-saga-ing leaves partial state as tribal knowledge in support macros.

How should teams implement sagas with compensating actions?

Implementation choices that prevent saga-shaped incidents.

When is choreography enough?

Two to three services, linear flow, compensations are simple event reversals, team shares on-call rotation. Add orchestration when branching appears, SLAs tighten, or compensation requires conditional logic.

How granular should saga steps be?

One business-meaningful step per local transaction — not one per SQL statement, not entire checkout in one step. Finer steps increase compensation precision; coarser steps simplify coordination but widen partial-failure blast radius.

Should compensations call external APIs synchronously?

Prefer async compensation commands with idempotent handlers — same as forward steps. Synchronous compensation in the HTTP request thread couples user latency to carrier void API availability.

How do sagas relate to two-phase commit?

Sagas trade atomicity for availability and simplicity. Accept temporary inconsistency (inventory released, refund pending) with clear status and reconciliation. Do not simulate 2PC with locks across services.

A common argument runs the other way

The opposing view holds that sagas over-engineer what retries and idempotent handlers solve — "just make every handler idempotent and let the message broker redeliver."

Idempotency prevents duplicate forward effects; it does not define what happens when step three fails after steps one and two committed. Without compensating actions, failed sagas leave resources reserved, payments held, and orders orphaned — idempotent retry of step three does not release inventory from a terminal failure. Retries address transient errors; compensations address terminal failure and semantic undo. Both are required.

Key takeaways

  • Distributed business processes need explicit saga coordination — implicit event chains hide failure ownership.
  • Compensating actions are idempotent business undos, not database rollbacks — design for partial failure and irreversible steps.
  • Each step commits locally with outbox-emitted events — atomic state change and message intent.
  • Compensate in reverse order of completed forward steps; distinguish retryable from terminal failures.
  • Timeouts, saga TTL, and version guards prevent stuck and raced compensation.
  • Orchestration fits complex branching; choreography fits short linear flows with strong tracing.
  • Test duplicate delivery, crash timing, and compensation failure — not only happy path.

Conclusion

The saga pattern makes distributed failure a first-class design input. Forward steps advance the business process; compensating actions restore invariants when a step fails terminally; saga state makes progress visible instead of scattered across service logs.

Teams shipping multi-service workflows without documented compensation paths should map one critical flow — identify forward steps, write compensating action per step, add idempotency keys, wire outbox at each boundary. The first drill exposes how many "rollback" assumptions are actually manual support tickets.

Related articles

Command Palette

Search for a command to run...