Calling the Payment Provider

The outbound call to Payrail is where every rule of Chapter 7 meets a real API. One POST /v1/charges, 400 milliseconds at P50, 3 seconds at the ceiling, and behind it a client with three timeouts and a total, a retry policy that knows a charge is not safe to repeat without a key, an idempotency key Payrail understands, a breaker that opens at 50 percent of 20 calls, and a response parsed at the boundary like any other input. None of that is new. What is new is that it all has to be true of one call at once, and the call is the one that moves money.

The client lives in one module. It is the only code in Stagedoor that knows Payrail's URL, the only code that imports httpx to talk to it, and the class that the test suite of Chapter 12 replaces with a fake. Six call sites talking to Payrail directly would be six timeouts, six retry policies and no fake; one class is one policy, one place to read it, and one thing to swap. Marek wrote the class after the spring on-sale, when the double charge turned out to have been made by a client that three people had edited and none of them owned.

One Client Module

PayrailClient exposes one method that matters, charge(order, source, ctx), returning a ChargeResult, and later a refund that Topic 56 needs. It is constructed once, in build_service of Topic 20 of Chapter 4, from three values in the typed config of Topic 23: the base URL, the key as a SecretStr, and the timeout in milliseconds. The domain function place_order receives the client as a dependency and calls charge; it does not know the URL, the header the key goes in, or that there is a retry. Every policy decision this topic makes is inside the class, and the domain reads a result.

class PayrailClient:
    def __init__(self, base_url, key: SecretStr, timeout_ms: int):
        self._http = httpx.AsyncClient(
            base_url=str(base_url),
            headers={"Authorization": f"Bearer {key.get_secret_value()}"},   # read once, never logged
            timeout=httpx.Timeout(connect=1.0, read=timeout_ms / 1000, write=timeout_ms / 1000, pool=1.0),
            limits=httpx.Limits(max_connections=20),
        )
        self._breaker = Breaker("payrail", window=20, threshold=0.5, cooldown_s=30)   # Topic 40

    async def charge(self, order, source: str, ctx) -> ChargeResult:
        body = {"amount_cents": order.total_cents, "currency": "EUR", "source": source}
        headers = {"Idempotency-Key": f"order-{order.public_id}"}                # same on every attempt
        async with self._breaker:
            resp = await with_retry(                                                # Topic 38's wrapper
                lambda timeout: self._post("/v1/charges", body, headers, timeout), ctx)
        return parse_charge(resp)                        # strict model; unknown shape -> ChargeResult.unknown()

The constructor builds the HTTP client once with the four numbers Chapter 7 settled and the key in a header it will never print, and it builds the breaker beside it. The charge method assembles a body of three fields and one header, runs the call inside the breaker and inside the retry wrapper, and hands whatever came back to a parser. Nothing in it is specific to any handler, which is why there is exactly one of it. A second service that talked to Payrail from the worker, for refunds, would import this class and not httpx.

The fake in Topic 65 of Chapter 12 is a class with the same charge and refund methods and a dictionary instead of a socket. It exists because the real one has one interface; a codebase with httpx.post in four handlers has no interface to fake and ends up patching the library, which tests the patch.

The Request

The request carries amount_cents, currency and source, the card token the buyer's browser obtained from Payrail directly so that the card number never passes through Stagedoor. The header that matters is Idempotency-Key, and its value is order-{public_id}: the order's UUID, decided in Topic 39 of Chapter 7 as the buyer's intent translated into Payrail's vocabulary. It is derived, not generated, and everything depends on that. The first attempt, the retry after a read timeout, the retry after api-01 crashed and restarted, and the reconciliation job asking tonight whether the charge exists all present the same string, and Payrail's own key table answers all of them with the one charge it made.

POST /v1/charges HTTP/1.1
Host: api.payrail.example
Authorization: Bearer pr_live_…                                    # the secret, never in a log
Idempotency-Key: order-9b1e4d3a-7c2f-4a51-b8e0-3d6f1c9a2e47
Content-Type: application/json

{"amount_cents": 12800, "currency": "EUR", "source": "pm_4Yc…"}

HTTP/1.1 201 Created                                            # charged: 400 ms at P50
{"id": "ch_8f2a…", "status": "charged", "amount_cents": 12800, "currency": "EUR"}

HTTP/1.1 402 Payment Required                                   # declined: also an answer
{"id": "ch_8f2a…", "status": "declined", "decline_code": "insufficient_funds"}

--- 3,000 ms with no bytes: not an answer. The charge may exist. ---

The exchange shows the two responses the client can act on. A 201 with a status of charged and Payrail's own id for the charge, which becomes payments.provider_ref. A 402 with a status of declined and a code the buyer's page can show. Both are Payrail working as designed, both arrive in about 400 milliseconds, and both are answers: the order moves to paid or to failed, and the buyer sees which. The third case in the exchange is the one with no response at all after 3 seconds, and it is not an answer to anything. The rest of this topic is mostly about that case.

Timeouts and Retries

The numbers are the Payrail rows of the configuration table in Topic 37: 1 second to connect, 3 seconds between bytes when reading, 3 seconds to write, 1 second to wait for one of the 20 pooled connections, and a total of 3 seconds around the whole exchange, taken from asyncio.timeout with the smaller of that ceiling and the request's remaining budget. Payrail's P50 is 400 milliseconds; 3 seconds is a slow Payrail, and 4 seconds was the incident. A client that waits longer than the buyer's 10-second budget has left is talking to a closed socket, so the ceiling is never the timeout; the deadline is.

The retry policy is Topic 38's wrapper with the outcome table applied to this one call. A connection refused or a connect timeout means the request never arrived, and the wrapper retries it, up to 3 attempts with full jitter. A 503 or a 429 from Payrail is a "not now" with a Retry-After, and the wrapper sleeps the header's number of seconds and tries again. A read timeout is the unknown outcome, and here the key changes the answer: without Idempotency-Key a read timeout on a charge is never retried, because the second attempt is the double charge from the service's side; with the key it is retried once, and the second attempt with the same key gets Payrail's first answer if the first attempt went through. Once and not twice, because each read timeout costs the full 3 seconds and a third attempt would leave the 10-second budget with less than a second. A 402 is never retried, because the same card will be declined the same way, and a 400 or a 422 means the request itself is wrong.

The breaker of Topic 40 wraps all of it. Fifty percent of the last 20 calls failing, where a timeout and a 5xx count and a 402 does not, opens it for 30 seconds, and during those 30 seconds charge raises without touching the network. What the buyer sees then is Topic 42's answer: a 503 with a Retry-After that is the cooldown's remaining seconds, the order left pending, and the holds extended by 10 minutes so that a buyer who comes back when the header said does not find her seats sold. The retry wrapper sits inside the breaker, not outside it, so that 3 attempts count as 3 calls in the window and an open breaker stops the retries too.

Parsing the Response

Payrail's JSON is input. It arrived over a network from a process Stagedoor does not run, and Topic 14 of Chapter 3 parses input into a typed model at the boundary before any code reads a field. ChargeResponse is that model: id as a string, status as one of two known values, amount_cents as an integer, currency as a string, decline_code as an optional string. It is strict about types, so a status of 1 or an amount sent as "128.00" fails the parse rather than becoming something. It ignores fields it does not know, because Payrail adds fields the way Stagedoor does in Topic 17, and a client that fails on a new optional field has broken the contract from its own side. The provider's id is stored in payments.provider_ref; it is the string the reconciliation job and the refund will use to name this charge to Payrail.

A response that does not parse is not a failure and not a success. It is an unknown outcome. A 201 whose body has no status, a 200 with an HTML error page from a proxy in front of Payrail, a body whose amount_cents does not match what was sent: each one is logged at error level with the status, the first 2 kilobytes of the body and the order id, and mapped to ChargeResult.unknown(), which the caller treats exactly like a timeout. The alternative that Stagedoor shipped once was a client that checked the status code alone: any 2xx was a success, and one night a 200 from a maintenance page marked 30 orders paid for charges that Payrail had never seen. The code did not understand the response, and it should have said so instead of guessing yes.

The Unknown Outcome

A read timeout, a 5xx that is not a 503 with a header, a parse failure, a breaker that refused to call: in all four the charge may have happened. The handler does not know, and nothing the handler can do in the next 6 seconds of budget will tell it. So the order does not move. It stays pending, or charging once Topic 56 splits that state in two, with a payments row that records the attempt, its key and no provider_ref, and the buyer gets the response Chapter 7 chose for each case: 504 with the payment-unresolved type when the call timed out, 503 with a true Retry-After when the breaker refused to make it. Her page says the payment is being confirmed. Her holds are extended. Her idempotency key row in Topic 39's table stays without a response, so a retry from her browser gets a 409 rather than a second attempt at the charge.

The state table has three inputs and two of them are not the request. The webhook of Topic 54 tells Stagedoor that a payment settled, usually within a minute, and the job that processes it moves a pending order to paid. The reconciliation of Topic 55 asks Payrail at 03:00 about every order still pending after 15 minutes, using the same key, and moves it to paid or failed on Payrail's word. The request handler's only job in the unknown case is to leave the order in a state those two can act on, which means not guessing. A handler that marks the order failed on a timeout has told a buyer whose card was charged to charge it again; a handler that marks it paid has issued tickets for a charge that may not exist. Pending is the honest state, and the machinery for leaving it is the next two topics.

Secrets and Logs

The Payrail key is read from the secret store of Topic 59 of Chapter 11 into a SecretStr, whose repr is ten asterisks, so that the config dump at startup and any exception that prints the settings show nothing. It goes into the Authorization header once, at construction. It is not in the log line for a charge, not in the trace attributes of Topic 70 of Chapter 13, and not in the retry wrapper's warning when an attempt fails, because the logger's redaction processor strips Authorization and any field named key, secret or token before a line is written. The redaction lives at the logger and not at each call site, since a call site that forgets is the leak, and a logger that forgets is one bug with one fix.

The card token in source is never logged either. What the charge's log line carries is the order's public id, the idempotency key, the status Payrail returned or the word timeout, and the latency in milliseconds: payrail.charge order=9b1e4d3a key=order-9b1e4d3a status=charged ms=412, and nothing else. Four fields answer every question on-call asks about a charge, the request id and trace of Topic 21 of Chapter 4 join it to the checkout that made it, and a log aggregator that indexes every line for a year indexes no credential. The night Marek found the Payrail key in the CI logs, printed by a failed test that dumped the environment, was the night the SecretStr went in; the redaction at the logger came a week later, when a debug line printed the client's headers.