Reconciliation

After every safeguard in the two topics before this one, 14 orders were still marked pending at the end of the month for payments Payrail had settled. Fourteen buyers with money taken and no tickets, found because 14 of them wrote to support, in a month when Stagedoor placed 4.3 million orders. The idempotency key was in place. The webhook was verified, deduplicated and processed in a job. The client had its timeouts and its breaker. None of it was wrong, and 14 orders still fell through, because every one of those mechanisms assumes that some message eventually arrives, and the network fact of Chapter 1 says some fraction never does.

Reconciliation is the job that asks Payrail for the truth and compares it with Stagedoor's, on a schedule, and fixes or reports every disagreement. It is the last line of the spine: when two systems disagree, one of them asks. It runs at 03:00, walks every order that has been pending for more than 15 minutes, walks every charge Payrail settled in the window, and produces a morning report whose ideal length is zero lines. The month after Marek wrote it, the count was 11 on the first night, 2 the second, and 0 for the next 26.

Why It Is Necessary

Every one of the 14 began the same way: the charge call came back with no answer, a timeout on a slow night or a 503 from an open breaker, and the order was left pending exactly as Topic 53 prescribes, for the webhook to resolve. What differed was why the webhook path then failed, and Marek found four reasons by reading each order's trace. Six were settlements whose delivery was never attempted: Payrail had an outage on the 9th, their delivery queue lost 40 minutes of events, and the retries they promise cover a delivery that failed, not one that was never made. Four were received, recorded and then dead-lettered: the process_webhook job hit the seat-label bug of Topic 46 and sat in jobs:dead with a traceback nobody read over a weekend. Three were recorded and never enqueued, because the first version of the receipt committed the event row and then called Redis directly, and on the night redis-01 was restarted for the Chapter 9 upgrade the second write failed after the first had succeeded. One was rejected as stale: api-02's clock ran 6 minutes fast for a day after a rebuild, every delivery that landed there got a 403 from the timestamp window, and one order drew api-02 on every retry.

Read the four columns and notice what they have in common. Two are Payrail's fault and two are Stagedoor's, and the two that are Stagedoor's were fixed, the outbox row in the receipt and an NTP check in the readiness probe, the week they were found. What none of the fixes changes is the shape of the problem: in all four, a fact existed at Payrail and never reached the order row, and the design's unstated assumption was that every fact eventually does, by some path. The only repair for that assumption is a path that does not wait for a message at all: a job that goes and asks. It is the same reasoning that gave the outbox a relay and the stream a reclaim loop, applied to the one neighbour Stagedoor does not run.

The Job

reconcile_payments runs at 03:00 from the scheduler of Topic 46, with no retries, because the next night's run covers a failure and a retry of a job that walks every payment of the day is a second walk on the same night. For every order whose status is still pending or charging 15 minutes after its created_at, it asks Payrail one question: GET /v1/charges?idempotency_key=order-{public_id}. The key is the one Topic 53 sent with the charge, and it is why the question has an exact answer. Payrail's key table returns the charge it made under that key, or an empty list, and either one resolves the order.

async def reconcile_payments(job):
    cursor = await load_cursor("reconcile_payments")              # created_at of the last order finished
    while batch := await stale_pending_orders(after=cursor, limit=100, older_than_min=15):
        for order in batch:
            async with payrail_quota:                                  # 10 requests/s, their published limit
                charges = await payrail.find_charges(idempotency_key=f"order-{order.public_id}")
            match [c.status for c in charges]:
                case ["charged"]:
                    await apply_settlement(order, charges[0])          # the webhook job's code path, Topic 54
                    log.info("reconcile.fixed", order=order.public_id, cause="pending_but_settled")
                case [] | ["declined"]:
                    await mark_failed_release_holds(order)               # no charge exists: the buyer was never charged
                    log.info("reconcile.fixed", order=order.public_id, cause="pending_no_charge")
                case _:
                    await report(order, ours="pending", theirs=charges)   # two charges, or a state we do not know
            cursor = order.created_at
        await save_cursor("reconcile_payments", cursor)             # resumable: a kill at 03:40 restarts here

The loop reads stale unresolved orders, pending or charging, in batches of 100 from a cursor, asks Payrail about each one under a rate limit of 10 requests a second, and acts on the answer. One charge with a status of charged means the order is really paid, and the job applies the settlement through the same function the webhook job uses, so tickets are issued by the one code path that knows how. No charge, or one declined charge, means the buyer was never charged and the order is marked failed with its holds released, which is what the request would have done had Payrail's answer arrived. Anything else, two charges under one key or a status the model does not know, is not fixed; it is written to the report with both sides' view. After every batch the cursor is saved, which is the subject of a later section. At 10 requests a second, the few hundred stale orders of a normal night take under a minute, and the 3,000 of a night after a Payrail outage take 5.

The rate limit is Payrail's, not Stagedoor's choice. Their API quota is per key, and a reconciliation that fires 3,000 requests in 3 seconds gets a 429 on the 11th and the wrapper's Retry-After sleep on every request after, which turns a 5-minute job into a 50-minute one. The semaphore in the loop paces the job under the quota so that it never hits it, and it leaves room for the checkout path, which shares the key and must not find the quota spent by a batch job at 03:00 on a night with a late show.

Both Directions

The forward pass asks about Stagedoor's pending orders. The backward pass asks the opposite question: for every charge Payrail settled in the last 24 hours, is there a paid or complete order behind it? The job pages through GET /v1/charges?settled_after=… and looks up each charge's idempotency key in orders. A charge whose key matches a paid order is fine. A charge whose key matches a pending order is the forward pass's case seen from the other side, and it is fixed the same way. A charge whose key matches no order at all is money taken for nothing. Since Topic 32 of Chapter 6 commits the order as pending before the call is made, the code believes that case impossible, and the backward pass exists precisely to check what the code believes. In its first week it found two: a staging deployment that had been handed the production Payrail key in a copied .env, the leak Topic 59 of Chapter 11 tells in full, charging real cards for orders that existed only in the staging database.

A charge with no order is refunded automatically and reported. There is no order to make paid, no holds to convert, no buyer's page to update; there is only a charge that should not exist, and the compensation for a charge is a refund, which Topic 56 formalizes. The job calls payrail.refund(provider_ref) with an idempotency key derived from the charge, logs it with the cause charge_no_order, and puts it in the report so that a human sees the buyer was refunded before the buyer sees the charge on a statement. Without the backward pass, that buyer's first contact with the problem is a dispute filed with her bank, which costs Stagedoor a fee, a mark against its merchant account, and the trust of somebody who did nothing wrong. The forward pass alone can never find this case, because the forward pass starts from orders, and there is no order.

Fix or Report

Every disagreement the job finds is one of two kinds, and the line between them is whether the fix is unambiguous. Pending on Stagedoor's side and settled on Payrail's has one correct resolution: the buyer paid, so the order is paid. Pending here and no charge there has one: the buyer was not charged, so the order failed. Charge there and no order here has one: refund. Those are fixed, logged with a cause, and counted. The job does not ask a human to approve a fix whose only possible answer is yes.

The other kind has more than one possible story. An order that is paid here and refunded there might be a refund Stagedoor issued and failed to record, or a chargeback the buyer's bank forced, or Payrail's list being a page behind and showing a stale state. Each story has a different fix, and a job that picks one is a job that marks a paid order refunded because a list was stale, which happened once in the first week and is why the rule exists. Ambiguous disagreements are written to the report with the order id, the provider_ref, Stagedoor's state and Payrail's state, and nothing else is done. The report is the first thing on-call reads in the morning, and on most mornings it says 0 fixed, 0 reported, last run 03:04.

Idempotent and Resumable

The job will be interrupted. The deploy of Chapter 11 sends SIGTERM to the worker at 03:40 on the night the reconciliation is walking 3,000 orders after an outage, and the 30-second grace is not enough to finish. A job that must complete in one run does not complete on that night, or the next, if the deploy is a nightly one. So the job saves its cursor, the created_at of the last order it finished, after every batch of 100, and the next run starts from there instead of from the beginning. The cursor is a row in a small table keyed by job name, written in the same transaction as the batch's last fix, so that a kill between the fix and the cursor write cannot leave an order fixed twice or a cursor past an order not yet asked about.

Running twice must also be harmless, because the scheduler of Topic 46 can enqueue the job twice on a night when the lock expires early, and because Marek will run it by hand after an incident. Every fix it makes is a conditional update of the kind Topic 34 of Chapter 6 taught: SET status = 'paid' WHERE public_id = %s AND status = 'pending', so the second run finds no row to change. The settlement is applied through the same idempotent job path as the webhook's, with the same unique constraints on tickets and the same provider key on the email. The refund carries a key derived from the charge, so a second run cannot refund twice. A reconciliation is a repair tool, and a repair tool that can make things worse when run twice is one nobody dares to run at all.

The Metric

The job's output is a number per cause per night: reconcile_disagreements_total with a label for pending_but_settled, pending_no_charge, charge_no_order and reported. On a healthy night every one of them is 0 or 1. A rising count under one cause is a new bug somewhere in Chapters 7 through 10 that the reconciliation has found before support did: 20 pending_but_settled in one night means the webhook path is broken, 5 charge_no_order in a week means something is crashing between the charge and the commit far more often than a deploy explains. The job is the detector as much as the fixer, and the count by cause is what turns "14 support tickets" into "the webhook endpoint returned 500 on Tuesday."

Zero for a month is one of two things. Either the system is healthy, or the job has not run since the cron entry was edited and the count is zero because nobody counted. The job's own last-run timestamp distinguishes them, and it is a metric of its own, reconcile_last_success_timestamp, with an alert in Topic 69 of Chapter 13 when it is more than 26 hours old. The alert fired once, three weeks after the job was written, when a config change renamed the scheduler's job kind and the reconciliation silently stopped. The disagreements count was a perfect 0 for four nights, and the timestamp said Tuesday.