The transactional outbox pattern

Distributed systems Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Pick the fix

An order service commits the order, then publishes OrderPlaced. Sometimes the publish fails and shipping never hears about it. Fix?

Options
Hint 1 of 1

How can the order and the event commit together?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A Insert OrderPlaced into an outbox table in the order's transaction; a relay publishes it

Correct. The event exists if and only if the order committed.

A database commit and a broker publish are two systems that can't commit together. Writing the event as a row in the same transaction makes it atomic with the order; a relay publishes it later, at least once.

Why not the others?
  • B Publish first, then commit. Now a failed commit leaves an event for an order that doesn't exist.
  • C Retry the publish three times. A crash between commit and publish still loses it.
  • D Use a bigger broker cluster. Doesn't address the crash between two systems.

30-second answer

Don't write to the database and then publish to the broker as two steps; a crash between them loses the event or publishes one for a rolled-back change. Instead insert the event into an outbox table in the same transaction as the business change. A relay then reads the outbox and publishes, marking rows as sent. Delivery is at-least-once, so consumers must be idempotent.

Practise saying it · 35 s. No recording, just a timer

Explain the outbox pattern.

Say your answer out loud, then check which of these points you covered:

  • Dual write problem
  • Event row in same transaction
  • Relay publishes
  • At-least-once, idempotent consumers

Deep dive

The relay can poll with SELECT ... FOR UPDATE SKIP LOCKED or read the WAL with change data capture (Debezium).

Keep the event id stable so consumers can deduplicate.

Order is per aggregate: publish with the aggregate id as the partition key.

Delete or archive sent rows so the table stays small.

Write the order and its event together, then relay Ran on PostgreSQL 17.11
SQL · PostgreSQL 17
CREATE TABLE orders (id bigint PRIMARY KEY, total numeric NOT NULL);
CREATE TABLE outbox (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  aggregate_id bigint NOT NULL,
  type text NOT NULL,
  payload jsonb NOT NULL,
  sent_at timestamptz
);
BEGIN;
INSERT INTO orders VALUES (42, 19.99);
INSERT INTO outbox (aggregate_id, type, payload)
VALUES (42, 'OrderPlaced', '{"total": 19.99}');
COMMIT;
-- relay: claim a batch without blocking other relays
WITH batch AS (
  SELECT id FROM outbox WHERE sent_at IS NULL
  ORDER BY id LIMIT 100
  FOR UPDATE SKIP LOCKED
)
UPDATE outbox o SET sent_at = now()
FROM batch WHERE o.id = batch.id
RETURNING o.type, o.aggregate_id;

Common mistakes

  • Publishing to the broker inside the database transaction (the broker can't roll back).
  • Publishing after commit with no retry, losing events on crash.
  • Assuming the relay publishes exactly once.

Senior take & follow-ups

Explain the dual-write problem, the outbox insert in the same transaction, the relay, and idempotent consumers.

Follow-up questions

Think of your answer first, then open the hint.

  1. Why not two-phase commit with the broker?

    Show hint

    Few brokers support it well, and it couples availability of both.

  2. Polling or CDC?

    Show hint

    Polling is simple; CDC has lower latency and no polling load.

Related questions

Suggest a fix if something here is wrong or out of date.