Sagas — Multi-Step Work Without a Global Transaction

Checkout is a sequence across three systems. Hold the seats, in Postgres. Charge the card, at Payrail. Issue the tickets and convert the holds to sales, in Postgres again, then a job. No transaction spans them: Topic 32 of Chapter 6 forbade holding one open across the Payrail call, and Payrail would not join it anyway. So step two can succeed after step one, and step three can fail after both, and the buyer's card is charged for seats that have no tickets. A saga is the pattern that accepts this instead of pretending a transaction exists. Each step has a compensating action, the sequence has a state, and the failure of any step runs the compensations for the steps before it.

Stagedoor's checkout has been a saga since the spring, whether or not anyone called it one, because every checkout that touches a payment provider is. What changed after the spring is that the saga's state moved out of the code's control flow and into orders.status, where a crash cannot lose it, and that refund was written down as the compensation for charge before the first refund was ever needed. This topic is those two decisions and the four that follow from them.

The Steps and Their Compensations

Three steps, three compensations. Holding the seats is compensated by releasing the holds: delete the holds rows and set the seats back to available, the two statements expire_holds already runs every minute. Charging the card is compensated by refunding it: payrail.refund(provider_ref) with an idempotency key of refund-{public_id}, through the same client class as the charge. Issuing the tickets is compensated by voiding them: the order goes to refunded, the seats return to available, and the scanner's GET /tickets/{code} answers 410 for those codes from then on, as Topic 07 of Chapter 2 arranged.

A compensation is not an undo. The charge happened and the refund happened, and the buyer's statement shows both lines, one of them for a few days as a pending debit. The hold existed for 4 minutes and another buyer could not take the seat during them. What a compensation restores is an acceptable state, not the previous one: the buyer has her money, the seat is on sale, and Stagedoor's books show a charge and a refund that add to zero. The distinction matters in the reconciliation report of Topic 55, where an order that was charged and refunded is a different row from one that was never charged, and a job that treats the refund as if the charge never happened files it under the wrong cause.

The State Machine

The saga's state is one column. orders.status begins at pending when the row is inserted with its holds, moves to charging when the call to Payrail is about to be made, to paid when Payrail answers charged, and to complete when the tickets are issued and the seats converted. failed and refunded are terminal. Two of those six, charging and complete, are additions to the four values Topic 13 of Chapter 3 listed, made the way that topic allows: new values beside the old ones, nothing renamed and nothing removed. Every transition is one row update in its own transaction, as Topic 32 of Chapter 6 shaped it, and each is conditional on the state it leaves from, so the current state decides what the next step or the next event means. That is the same machine Topic 54 consulted when a settlement arrived for a refunded order, and the same one Topic 55 reads when it decides whether a disagreement is fixable.

The edges that do not exist are as important as the ones that do. There is no edge from refunded to paid, which is why the late settlement of Topic 54 changes nothing. There is no edge from failed to paid, which is why a settlement for a failed order is a disagreement for a human and not a fix. There is no edge from complete back to charging, so a replayed job cannot charge a completed order. A transition is UPDATE orders SET status = 'paid' WHERE public_id = %s AND status IN ('pending', 'charging'), and a row count of zero is the machine saying no.

Forward and Backward

When a step fails, the saga goes one of two ways. Backward: run the compensations for the steps already done and end in a terminal state. Forward: retry the failed step until it succeeds, and continue. The choice is per step, and the rule Stagedoor uses is whose failure it was. A declined card is Payrail's answer about the buyer, and there is nothing to retry: the order goes to failed, the holds are released, and the buyer sees a 402 with the decline code. That is backward, and it is the common case, a few hundred times a night.

A failure at issuance is different. The charge succeeded, Payrail has the money, and the step that failed is Stagedoor's own code on Stagedoor's own database: a bug, a replica read that timed out, a worker killed by a deploy. Refunding a buyer because the ticket renderer hit a bad seat label punishes her for a defect she did not cause, and she loses her seats to the next buyer while she tries again. So issuance retries forward: the job's policy from Topic 46 tries again, and if the attempts are exhausted the job dead-letters, the alert pages someone, and a person fixes the seat label and replays the job. The order sits in paid for the hour that takes, and the buyer's page says her tickets are on the way. Backward from paid, a refund and a release, is a decision a person makes when forward has been tried and cannot work, and it happens a few times a month rather than a few times a night.

The decision per step is written beside the step, not improvised in an exception handler. Charge fails: compensate. Issue fails: retry forward, dead-letter, human. Hold fails: nothing to compensate, because nothing happened yet. Three lines in a table, decided once, and the code for each step reads its line.

Orchestration

One place drives the saga. The place_order domain function inserts the order as pending, moves it to charging, makes the call, and moves it to paid or failed on the answer, all within the request's 10-second budget and usually in 600 milliseconds. The paid transition writes an outbox row, and the issue_tickets job on worker-01 continues the saga from there: reads the state, does its step, writes complete. Each participant reads the state, does its work and writes the next state, and the state is the only thing they share.

async def place_order(svc, buyer, hold_ids, source, ctx) -> Order:
    order = await svc.orders.create_pending(buyer, hold_ids)         # transaction 1: order + holds, commit
    if not await svc.orders.transition(order, "pending", "charging"):  # transaction 2: the state says a charge may exist
        return await svc.orders.get(order.public_id)                 # someone else moved it: return what it is

    result = await svc.payrail.charge(order, source, ctx)             # no connection held; 400 ms; key order-{public_id}

    match result.outcome:
        case "charged":
            async with svc.pool.connection() as conn, conn.transaction():   # transaction 3
                await svc.orders.transition(order, "charging", "paid", conn=conn)
                await svc.payments.record(conn, order, result.provider_ref)
                await svc.outbox.add(conn, "issue_tickets", order_id=order.id)   # the job continues the saga
        case "declined":
            await svc.orders.fail_and_release(order, result.decline_code)   # backward: compensate the hold
        case "unknown":
            await svc.holds.extend(order, minutes=10)                      # stays charging; Topic 55 resolves it
    return await svc.orders.get(order.public_id)

The function does the first two steps and starts the third. It creates the order and its holds in one transaction, moves the state to charging in a second, and only then calls Payrail, with no connection in hand. On a charge it moves the state to paid, records what Payrail said and queues the issuance job, in a third transaction. On a decline it runs the compensation for the hold. On an unknown outcome it does nothing to the state: the order stays charging, the holds are extended, and Topic 55 will ask Payrail tonight. The conditional transition that returns false is the machine refusing, and the function returns the order as it is rather than pressing on, which is what a second request with the same key or a replayed job hits.

The alternative is choreography: no driver, and each step reacting to the previous step's event. The charge job listens for order.placed, the issuance job listens for payment.charged, the refund job listens for issuance.failed, and no single place knows where any order is. It scales across teams, which is why the System Design course that Chapter 1 named as a neighbour discusses it, and it is wrong for a three-step flow inside one service: the question "where is order 4471" is answered by reading three handlers' logs instead of one row. Stagedoor keeps the single driver, because one order's progress should be one column in one row.

Timeouts in the Saga

A saga has no transaction to time out, so its steps time out individually and the state records where it stopped. An order in charging for more than 15 minutes is a charge whose outcome nobody received: the call timed out, or the process died during it, and the request that would have written paid or failed is gone. Nothing in the service will move that order on its own. The reconciliation of Topic 55 asks Payrail about every order still pending or charging after 15 minutes, and its answer is the transition. Fifteen minutes is longer than any request's budget and shorter than a buyer's patience, and it is the number that keeps a crashed checkout from sitting in charging until somebody notices.

The holds have their own clock, and it does not consult the saga. expire_holds runs every minute and releases any hold past its 10 minutes regardless of the order's state, because a hold that waited for the saga would be a hold that lasts forever when the saga crashes. That creates one race the design must face: an order that reaches paid after its holds expired, because Payrail was slow and the extension of Topic 42 was not enough, or because reconciliation settled it at 03:00. Its seats may already be held by somebody else. So issuance converts the holds to sales and inserts the tickets in one transaction, with the seat rows locked and their status checked, and an issuance that finds a seat no longer held by this order fails the step. That is the one case where backward from paid runs without a human: refund, because the seats are gone, and the buyer is told why, with the seats she can still get.

Idempotency at Every Step

A saga resumes after a crash from whatever state the column holds, and resuming is safe only if every step can run again from that state without doing its work twice. The transitions are conditional updates, so a step that already moved the state cannot move it again. The charge carries order-{public_id}, so a resume from charging that calls Payrail again gets the first charge back, not a second. The issuance job checks state first and relies on a unique constraint on tickets.seat_id, so two runs cannot issue two tickets for one seat. The refund carries refund-{public_id}, so a compensation that runs twice refunds once. The four techniques of Topic 45 and the key of Topic 39, one per step.

That is what makes "resume from charging" a safe operation instead of a second charge. Without the key, a process restarted after a crash mid-call would see charging, not know whether the call went out, and have to choose between never charging and charging again; with it, the resume calls Payrail with the same key, and Payrail's own table says which. The state machine names where the saga is, the idempotent steps make every step safe to repeat from there, and the two together are the whole difference between a checkout that survives a deploy and one that charges twice when the deploy lands at the wrong millisecond.