Receiving Webhooks

Payrail tells Stagedoor that a payment settled by calling POST /webhooks/payrail, and that endpoint is an inbound boundary with an untrusted caller. Anyone on the internet can send a body that looks like a settlement; the URL is not a secret and the shape of the payload is in Payrail's public documentation. The handler therefore verifies a signature before it parses anything, records the event id to detect the replay that Payrail promises will happen, answers 200 in about 50 milliseconds so that Payrail does not retry, and does the actual work in a job, where the order's state machine, not the order of arrival, decides what the event means.

It is the mirror image of the previous topic. There Stagedoor was the client and Payrail's answer could be lost; here Payrail is the client and its delivery is at least once, not in order, and on a schedule Payrail chooses. Every rule of Chapter 7 applies with the roles reversed, and the two tables this book already has, the idempotency keys of Topic 39 and the outbox of Topic 41, do most of the work under a new name.

The Signature

Payrail signs every delivery with HMAC-SHA256 under a secret shared with Stagedoor at setup, and sends two headers: Payrail-Timestamp, the Unix time of the delivery, and Payrail-Signature, the hex digest of the timestamp, a dot and the raw body. The handler recomputes the digest over the same bytes and compares. Three details decide whether that check is real. It runs over the raw bytes as they arrived, not over the parsed and re-serialized JSON. It compares with hmac.compare_digest, which takes the same time whether the first byte differs or the last, so that an attacker cannot learn the digest one byte at a time from response latency. And it rejects a timestamp more than 5 minutes from now, so that a captured valid delivery cannot be replayed a week later against an order that has since been refunded.

async def payrail_webhook(request, svc):
    raw = await request.body()                                  # the bytes Payrail signed, untouched
    ts = request.headers.get("Payrail-Timestamp", "")
    sig = request.headers.get("Payrail-Signature", "")
    if not ts.isdigit() or abs(svc.clock.now() - int(ts)) > 300:
        raise Forbidden("stale-or-missing-timestamp")          # 403, body never parsed
    expected = hmac.new(svc.cfg.payrail_webhook_secret.get_secret_value().encode(),
                        ts.encode() + b"." + raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):
        raise Forbidden("bad-signature")                        # same 403, same timing either way
    event = PayrailEvent.model_validate_json(raw)               # parsed only now: Topic 14
    return await svc.webhooks.receive(event, raw)

The handler reads the body as bytes, checks the timestamp window, computes the digest over the timestamp and those bytes, and compares in constant time; only after all three pass does it parse the body into a model. The reason the raw bytes matter is that JSON has no canonical form. Payrail's serializer wrote the keys in one order with one whitespace convention, and the digest is over those exact bytes. A handler that parses the body into a dictionary and serializes it again produces different bytes, different key order, different spacing, and a digest that never matches. The developer who hits that on the first test sees "signature invalid" on every delivery, concludes the documentation is wrong, and disables verification "for now," which is the state Stagedoor's endpoint was in for four months. The fix is one line: sign what arrived, not what you rebuilt.

The secret is one of the seven in Topic 59 of Chapter 11, distinct from the API key, rotated on its own schedule. A signature check that passes is the only thing that makes the payload trustworthy. Payrail's source IP addresses are published and they change; a firewall rule built on them blocks a real delivery after their next migration and admits a forged one from inside any network that shares an egress with them. The signature is the credential, and it is the same kind of credential as the API key in Topic 28 of Chapter 5, presented by a machine that is not a person.

Idempotent Receipt

Payrail retries a delivery on any non-2xx response and, sometimes, on a 2xx it did not see because the connection dropped after Stagedoor wrote it. That second case is Chapter 1's fourth outcome with Payrail as the caller, and it means the same event arrives twice on a healthy night. The webhook_events table is how the second arrival is recognized: provider_event_id as the primary key, received_at, processed_at and the payload, and an insert that does nothing on conflict.

async with conn.transaction():
    inserted = await conn.execute(
        "INSERT INTO webhook_events (provider_event_id, received_at, payload) "
        "VALUES (%s, now(), %s) ON CONFLICT (provider_event_id) DO NOTHING RETURNING provider_event_id",
        (event.id, Jsonb(raw_json)))
    if await inserted.fetchone() is None:
        return Ack()                                     # seen before: 200, nothing enqueued
    await outbox.add(conn, "process_webhook", event_id=event.id)   # the job, Topic 41's table
return Ack()                                             # 200 at about 50 ms; the work has not started

The insert either creates the row and returns its id, or finds the id already there and returns nothing. When it returns nothing the event was received before, and the handler answers 200 without enqueueing anything, because whatever the first receipt started is already running or already done. When it inserts, the same transaction adds an outbox row for a process_webhook job carrying the event id, and the commit makes the receipt and the promise to process it one durable fact. It is the idempotency table of Topic 39 with a different caller and a simpler contract: Payrail does not need its response replayed, only acknowledged, so the table stores the payload and not a response. The constraint decides the race exactly as it did for the buyer's key, so two deliveries of the same event arriving on api-01 and api-02 in the same 50 milliseconds produce one row and one job. The outbox row is not optional: a first version committed the event and then called Redis directly, which is Chapter 7's dual write in a new handler, and Topic 55 counts what it cost on the night Redis was restarted.

Respond Fast, Process Later

Payrail waits 5 seconds for a 2xx and then treats the delivery as failed and schedules a retry. The handler above has done three things by the time it answers, a digest, an insert and an outbox row, and on api-01 that takes about 50 milliseconds. The work the event implies, mark the order paid and issue the tickets, is the process_webhook job on worker-01, and it takes whatever it takes: a transaction on the order row, the ticket rows, the outbox row for the email, and on a bad night a wait behind organizer reports on the replica. None of that time is on Payrail's clock.

The first version of the endpoint did the work inline. On a quiet afternoon that was fine, 300 milliseconds and a 200. On the night of a hot on-sale, with the primary busy and the replica behind, the handler took 6 seconds, Payrail gave up at 5, retried, and the retry found the first attempt still running with no event row committed yet, so it started a second one. Two processes marking one order paid and issuing tickets for it, saved from a second email only by the idempotent job of Topic 45. That is the third line of defence doing the first line's job. The event-id insert before any work, and the work outside the request, is the first line.

Order Is Not Promised

Payrail sends several kinds of event for one charge, and it does not promise the order they arrive in. payment.settled can arrive before payment.authorized, because the two were delivered by different workers on Payrail's side and one of them retried. It can arrive twice, as above. It can arrive after payment.refunded, when the settlement's first delivery was lost for an hour and the buyer asked for a refund in between. A handler that applies each event as an instruction, "settled means set paid," ends up with a refunded order marked paid because the settlement's retry arrived last.

The job does not apply events as instructions. It reads the order's current status and asks whether the transition the event describes is legal from there, using the state machine that Topic 56 draws in full. A settlement for an order that is pending or charging moves it to paid. A settlement for an order already paid or complete is a duplicate that slipped past the event-id table, perhaps because Payrail sent the same fact under two event ids, and it changes nothing. A settlement for an order that is refunded is logged and reported for the morning's reconciliation review, and the order stays refunded, because the refund is the later fact and the state machine knows it. Arrival order stops mattering once the current state, not the event, decides.

Each transition is one conditional UPDATE of the kind Topic 34 of Chapter 6 wrote for the seat: SET status = 'paid' WHERE public_id = %s AND status IN ('pending', 'charging'), and the row count says whether this run made the change. Two workers processing two deliveries of the same settlement both attempt it; one updates a row and issues tickets, the other updates nothing and stops. The job is idempotent by the first technique of Topic 45, reading the state and doing only what is missing, and the ticket issuance behind it is idempotent by the others.

What the Webhook Is Not

It is not the source of truth. Payrail's API is, and Topic 55 reconciles against it, because a webhook is a message and messages are lost. A design in which the webhook is the only way an order becomes paid has 14 pending orders at the end of the month for payments that settled, which is the number that opened Topic 55. The webhook is the fast path, and the fast path is allowed to fail.

It is not a command. It reports what happened at Payrail; what that means for the order is Stagedoor's decision, made by the state machine. A payload that says settled does not set anything; it is an input to a transition that may or may not be legal. And it is not authenticated by anything but the signature: not the IP, not a token in the URL, not a header with a fixed value that a copy of the documentation would reveal. The one thing an attacker cannot produce without the secret is the digest.

Failure on Our Side

The job fails. A bug in ticket issuance raises on an order whose seat label an organizer typed as 14-C, the same poison Topic 46 met in the render job. The event is in webhook_events with processed_at still null, the job retries on the kind's policy and, when that is exhausted, goes to jobs:dead with its traceback, and the alert on that stream's length pages someone. The order is still pending, and the nightly reconciliation asks Payrail about it and finds it settled, so even a dead-lettered event that nobody reads before morning is resolved by 03:00.

What the endpoint returned, through all of that, was 200. The receipt succeeded: the event is durable in Stagedoor's database, and Payrail should not retry a delivery that Stagedoor already has. Returning 500 because the processing failed would tell Payrail to send the event again, and the retry would conflict on the event id and be acknowledged with no work, which is harmless and pointless. Worse, if the failure were a bug that raised before the insert, the 500 would be right, and a handler that cannot tell the two apart returns the wrong one half the time. The rule is the line the insert draws: a 5xx only when the receipt itself failed, a 200 the moment the row is committed, and everything after the commit is Stagedoor's problem to retry, dead-letter and reconcile.