The Outbox Pattern

place_order writes the order and then publishes "order placed" to the jobs stream so that the worker renders the tickets and sends the email. Those are two systems, Postgres and Redis, and two systems cannot be written atomically. If the process dies between the commit and the publish, the order exists and no tickets are ever sent. If the publish goes first and the commit then fails, tickets are sent for an order that does not exist. Every ordering of the two has a window, and no retry closes it, because a retry cannot run in a process that has died.

The outbox is the answer, and it is small. The message is written into the database, as a row in an outbox table, in the same transaction as the order; a relay in the worker reads unpublished rows and moves them to the stream afterwards. "A row and a message" becomes "two rows," and two rows is a thing a transaction can do. This topic is the second half of the double-charge repair: Topic 39 made the buyer's request safe to repeat, and this one makes the order and its consequences one unit, so that the ticket job the worker runs cannot exist without the order and the order cannot exist without it.

The Dual-Write Problem

Take the two operations, commit the order and publish the message, and put them in either order. Commit first: the process is killed by the deploy of Chapter 11 between the two lines, and the order is durable and silent forever. Publish first: the message is on the stream, the worker claims it within 50 milliseconds, and the commit fails on the unique constraint from Topic 34 of Chapter 6, so the worker is rendering tickets for an order that rolled back. Add a retry to the publish in the first case and it changes nothing, because the retry is code in the process that died. Add a check to the worker in the second case, "does the order exist yet," and it races the commit, which may land 10 milliseconds after the check.

The four columns are the four outcomes of Topic 03 of Chapter 1 applied to a pair of writes, and the first two are the trouble. What makes them unfixable by ordering is that the failure is between the lines, and nothing that runs after the failure can see which line it was on. The only design that works is one where there is no second system at the moment of the commit: the message is stored where the order is stored, by the same statement set, under the same commit.

The Outbox Table

The outbox table has five columns: id, kind, payload, created_at and published_at. place_order, inside the transaction that Topic 32 of Chapter 6 shaped, inserts the order, inserts the tickets, and inserts one outbox row with kind render_tickets and a payload that names the order's public_id and carries the request id and trace from Topic 21 of Chapter 4. Then it commits. Three inserts and one commit, and the message now has exactly the atomicity the order has: both exist or neither does, and no crash at any line in between can separate them.

async with conn.transaction():
    order_id = await insert_order(conn, buyer, holds, total_cents)
    await insert_tickets(conn, order_id, holds)                # one row per seat, code generated here
    await conn.execute(
        "INSERT INTO outbox (kind, payload, created_at) VALUES (%s, %s, now())",
        ("render_tickets", Jsonb({
            "order": str(public_id),
            "request_id": ctx.request_id, "trace": ctx.trace.id,   # the context crosses here
        })),
    )
# commit: the order, its tickets and the promise to announce it are now one durable fact
# nothing here touched Redis; the relay will, within 100 ms

The handler no longer talks to Redis at all. It inserts three kinds of rows and commits, and the commit is the only thing that has to succeed. The payload carries the order's public id rather than the order itself, because the worker will read the order from the database when it runs and the row is the truth, not the message; it also carries the request id and the trace, which is how Chapter 13 finds the checkout, the job and the email from one search. The comment about 100 milliseconds is the relay's poll interval, and it is the latency the outbox adds to a message that a direct publish would have sent in 1.

The Relay

The relay is a loop in the worker: select up to 100 unpublished rows in id order, locking them and skipping any another relay has locked, publish each to the stream, and set published_at on the ones that went. The lock is held for the length of the batch, about 20 milliseconds for 100 rows, and released at the commit that marks them. If the relay dies after publishing a row and before marking it, the row is still unpublished when the next relay wakes, and the message goes to the stream a second time.

async with conn.transaction():
    rows = await conn.execute(
        "SELECT id, kind, payload FROM outbox WHERE published_at IS NULL "
        "ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED"
    )
    for row in await rows.fetchall():
        await redis.xadd("jobs", {"kind": row.kind, "outbox_id": row.id, "payload": row.payload})
        await conn.execute("UPDATE outbox SET published_at = now() WHERE id = %s", (row.id,))
# a crash between xadd and the UPDATE republishes this row next pass: at-least-once, on purpose

Publish, then mark. The order of those two lines is the decision the whole pattern rests on, and it is the same decision as Chapter 1's fourth outcome made deliberately. A crash between them produces a duplicate on the stream, which the consumer of Chapter 8 must tolerate, and which it can, because the outbox id travels in the message and a job that has already run for that id does nothing the second time. The reverse order, mark then publish, produces a message that was recorded as sent and never was, which no consumer can tolerate because there is nothing to tolerate: the message is simply gone. Between losing a message and duplicating one, the duplicate is the failure a handler can be written for, so the duplicate is the one the relay is allowed to produce.

Ordering and Batching

One relay at a time per kind preserves the order of the rows, because ORDER BY id with a single reader publishes them as they were committed. SKIP LOCKED exists so that several relays can share the table without blocking each other, and it costs the order: two relays each take a batch, and whichever publishes first wins. Stagedoor runs one relay, because order matters. A render_tickets followed by a refund_payment for the same order must reach the worker in that sequence, or the refund handler finds no tickets to cancel and the placement handler then sends tickets for a refunded order. One relay moving 100 rows every 100 milliseconds is 1,000 messages a second, and the on-sale peak produces a few hundred orders a second, so one is enough with a margin of 3.

The batch of 100 is the relay's economy. One query, one lock, one commit per 100 messages keeps the relay's cost on the primary to 10 transactions a second at full tilt, against a partial index over the unpublished rows only, which stays small however large the table grows. A relay that selected one row at a time would issue 1,000 queries a second to move the same messages, and a relay that selected 10,000 would hold its lock for a second and delay every row behind it.

Change Data Capture as the Alternative

Polling is one way to notice a new row. The other is to read the database's own write-ahead log through logical replication: Postgres publishes every committed change on a replication slot, and a connector such as Debezium turns each outbox insert into a stream message as it commits, with no poll, no SELECT and a latency of milliseconds rather than a 100-millisecond interval. The outbox table stays exactly as it is; what changes is who reads it. The cost is machinery: a replication slot that must be monitored, because an unread slot holds WAL on the primary until the disk is full, a connector process to run and upgrade, and a second delivery contract to understand. Stagedoor polls, because 100 milliseconds is fine for a ticket email and one loop in the worker is easier to own than a connector. The book names CDC as the next step when the interval starts to matter, and leaves it to the Middleware course, whose territory the connector is.

Cleanup and Size

Published rows are deleted by the scheduled job of Chapter 8 after 7 days. Seven days is long enough to answer "did the message for this order ever go out" during an investigation, and short enough that the table holds a week of orders rather than the history of the company. Without the job the outbox grows by every order ever placed, becomes the second-largest table in the database after idempotency_keys if that one was also forgotten, and the partial index over unpublished rows keeps working only because those stay few. Deletion runs in batches of 1,000 rows, so it never holds a lock the relay would wait on.

The relay's health is one number: the age of the oldest unpublished row. On a healthy night it is under 100 milliseconds. A relay that has crashed and not restarted shows as that number climbing by a second every second, and it is the alert Chapter 13 puts on it, because an outbox with no relay is the first column of the dual-write figure reproduced at scale: every order durable, every message waiting, and nothing else in the system able to tell. The metric is also what makes an outage of Redis safe in Topic 42. The stream is unreachable, the rows accumulate, the age climbs, and when Redis returns the relay drains them in order, with nothing lost.