The transactional outbox pattern for reliable event publishing

The order marked paid. The database committed. The message broker never received OrderPaid — the application crashed between COMMIT and publish(). Inventory still showed available stock. The warehouse oversold. The on-call engineer called it a transient network blip and manually replayed from logs. The transactional outbox pattern exists because dual writes are not atomic: state change and event publication are two systems with no shared transaction boundary unless the event is written inside the same commit as the business row.

Event-driven architectures promise loose coupling — services react to facts instead of calling each other synchronously. The promise breaks at the seam where facts leave the database. Without an outbox, teams choose between publishing before commit (events for uncommitted work), publishing after commit (lost events on crash), or distributed transactions (impractical across Postgres and Kafka). The outbox is the pragmatic third path: persist the intent to publish atomically with the state change, then relay in a separate process that can retry without lying about durability.

The dual-write problem and why it persists

Dual write: update domain table, then call message broker API. Two independent operations. Failure modes:

"Use a transaction" does not span Postgres and RabbitMQ. Two-phase commit across heterogeneous stores adds operational cost most product teams reject. Saga choreography without durable outbound events pushes complexity to consumers compensating for events that never arrived.

If the event is not in the same transaction as the state change, the event is not guaranteed.

The outbox collapses dual write into single write: one transaction, two inserts (or update + insert), one commit. Publication becomes a read-and-forward problem — solvable with retries, idempotency, and monitoring.

Outbox table design and the atomic write

Minimal outbox schema:

CREATE TABLE outbox (
  id            BIGSERIAL PRIMARY KEY,
  event_id      UUID NOT NULL UNIQUE DEFAULT gen_random_uuid(),
  aggregate_type TEXT NOT NULL,
  aggregate_id  TEXT NOT NULL,
  event_type    TEXT NOT NULL,
  payload       JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at  TIMESTAMPTZ,
  publish_attempts INT NOT NULL DEFAULT 0
);
 
CREATE INDEX outbox_unpublished_idx
  ON outbox (created_at)
  WHERE published_at IS NULL;

Application transaction:

BEGIN;
  UPDATE orders
  SET status = 'paid', paid_at = now()
  WHERE id = $1 AND status = 'pending';
 
  INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
  VALUES (
    'order',
    $1::text,
    'OrderPaid',
    jsonb_build_object(
      'order_id', $1,
      'amount_cents', $2,
      'paid_at', now()
    )
  );
COMMIT;

If the transaction commits, the event exists. If it rolls back, neither state nor event persists. Crash timing no longer creates commit-without-event — the event row is part of the commit.

Payload conventions: include schema version, correlation ID, and causation ID for traceability. Keep payloads small — reference large blobs by ID rather than embedding. Event type strings should be stable across deploys; renames require consumer compatibility strategy.

Relay patterns: polling, CDC, and dedicated publishers

Something must move rows from outbox to the broker. Three common approaches:

Polling publisher. A worker queries unpublished rows, publishes, marks published_at. Simple, works everywhere, introduces poll interval latency.

SELECT id, event_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

FOR UPDATE SKIP LOCKED lets multiple publisher instances partition work without double-publish from the same row in one pass — though at-least-once to the broker still requires consumer idempotency.

Change data capture (CDC). Debezium or logical replication streams insert events from the outbox table. Lower latency than polling; adds CDC infrastructure and operational surface. Fits teams already running Kafka Connect.

In-process relay after commit. Same application thread publishes post-commit — recreates the dual-write window unless carefully implemented with transactional callbacks. Generally discouraged as primary pattern; post-commit hooks still fail on crash.

Polling is underrated for moderate throughput. CDC wins at high volume and sub-second latency requirements. Both mark published_at only after broker ack — if mark fails after successful publish, duplicate delivery occurs; consumers must dedupe.

Relay failure handling: increment publish_attempts, exponential backoff, dead-letter after threshold, alert on unpublished rows older than SLO. Stale unpublished rows are the outbox equivalent of queue depth — dashboard them.

Delivery semantics: at-least-once end to end

The outbox guarantees the event will eventually leave the database if the relay runs. It does not guarantee exactly-once delivery to consumers. Broker retries, relay crashes after publish-before-mark, and at-least-once broker semantics all produce duplicates.

Complete reliability stack:

Outbox (atomic with state) → relay (at-least-once) → broker (at-least-once)
  → consumer (idempotent handler)

Consumer idempotency uses event ID dedup — the same patterns as Postgres idempotent upserts: insert event ID into processed table in same transaction as side effects, or upsert domain state with monotonic version guards.

Outbox event_id must propagate as message key or header. Consumers key dedup on event_id, not broker message ID — broker IDs differ on redelivery.

Ordering guarantees are per aggregate, not global. Partition by aggregate_id so OrderPaid for order 42 always lands in the same partition if order matters. Cross-aggregate ordering is not provided — design consumers accordingly.

Outbox vs full event sourcing: scope the pattern correctly

The outbox solves reliable publication from OLTP writes. It is not event sourcing — though the two compose.

event sourcing without ceremony describes append-only domain logs as audit and projection source. The outbox is an integration table: events leave the service boundary. A service can event-source internally and still use an outbox for cross-service notifications — or use outbox-only without replaying internal state from events.

Teams sometimes store all domain events in an event table and treat unpublished rows as outbox — workable at small scale. At larger scale, separating integration outbox from domain event log clarifies retention, indexing, and access patterns. Outbox rows delete or archive after publish; domain events stay.

Operational concerns: cleanup, backpressure, and migrations

Retention and cleanup. Published rows accumulate. Archive to cold storage or delete after seven to thirty days depending on replay and audit needs. Partition outbox by week for cheap drops.

Backpressure. If relay lag grows — broker down, consumer slow — unpublished rows pile up. Alert on row count and age. Consider pausing non-critical producers or shedding load before outbox table bloat affects OLTP performance.

Schema migrations. Outbox table migrations follow zero-downtime migration discipline — the outbox is on the hot path. Add columns nullable; backfill; contract later. Never block relay with exclusive locks during peak.

Multi-region. Outbox assumes single-primary writes. Global active-active with outbox requires conflict resolution on aggregate IDs — usually avoided. Outbox fits single-primary per aggregate with regional read replicas.

Testing. Integration tests assert: commit without relay → row unpublished; relay → broker receives exactly one logical event per event_id; crash after publish before mark → redelivery without duplicate consumer effect when idempotent.

How should teams implement the transactional outbox?

Practical choices for services shipping event-driven integrations.

Polling interval vs CDC: how to choose?

Polling every one to five seconds with SKIP LOCKED handles thousands of events per minute on modest hardware. CDC when sub-second latency, very high throughput, or existing Kafka infrastructure justifies operational cost. Start polling; migrate to CDC when metrics prove need.

Should the outbox share a database with domain tables?

Yes, for the atomic guarantee. Outbox in a separate database recreates dual write unless both participate in XA — which defeats simplicity. Colocate outbox with the service's primary OLTP database.

When can post-commit publish replace outbox?

When lost events are acceptable (analytics only), when a reconciliation batch heals gaps within hours, or when volume is low and manual replay is tolerable. Not when downstream inventory, billing, or fulfillment depends on timely events.

How does outbox interact with sagas?

Sagas coordinate multi-step distributed workflows. Each step's local transaction includes outbox events for the next step's triggers. Compensating steps emit compensating events through the same outbox — idempotent consumers apply forward and compensating events safely.

A common argument runs the other way

The opposing view holds that modern brokers and cloud-managed messaging are reliable enough to publish directly after commit, especially with publisher confirms and retry libraries.

Publisher confirms prove the broker received the message — not that the application sent it. The crash window between database commit and HTTP POST to the broker is microseconds to milliseconds on every request — rare individually, guaranteed at scale. Retry libraries help transient failures; they do not help process death between commit and first publish attempt. Outbox moves publication intent into the durable commit boundary; relay retries are decoupled from request lifecycle. Direct publish is acceptable for best-effort analytics; it is insufficient when downstream correctness depends on every state change producing an event.

Key takeaways

• Dual write between database and broker is not atomic — outbox writes events in the same transaction as business state.
• Relay processes poll or CDC unpublished rows; mark published only after broker ack.
• End-to-end reliability requires idempotent consumers keyed on event ID — at-least-once is the realistic guarantee.
• Outbox is integration infrastructure, not event sourcing — scope retention and schema separately from domain event logs.
• Monitor unpublished row age and count — relay lag is a first-class metric.
• Partition work with FOR UPDATE SKIP LOCKED; partition messages by aggregate ID for ordering.
• Test crash timing: commit without relay, publish without mark, duplicate delivery to consumer.

Conclusion

Reliable event publishing is a database problem dressed as a messaging problem. The transactional outbox pattern makes publication intent durable alongside business writes, then delegates delivery to retryable relay processes that tolerate broker and process failures without silent gaps.

Services emitting domain events without an outbox should audit the crash window between commit and publish — and measure how many state changes never produced a downstream message. The fix is one table, one extra insert per transaction, and a publisher loop that operations can monitor like any other queue depth metric.