The Network Fails Between Any Two Lines
Every call that leaves the process can time out, be delivered twice, arrive late, or succeed while its answer is lost. That is not a rare event to be caught in an exception handler. It is the fact that the entire design of a service falls out of, and this topic states it once, carefully, so that every later chapter can say "this is the network fact again" instead of deriving it from scratch.
The fact has a precise shape. When the service makes one call, to Postgres, to Redis or to Payrail, or when a client makes one call to the service, there are four things that can happen, and the caller can distinguish only three of them.
The Four Outcomes of One Call
The request never arrived: the connection was refused, or the packets were dropped before the far side read them. The request arrived and failed: the far side ran it and returned an error. The request arrived, succeeded, and the answer came back: the good case. The request arrived, succeeded, and the answer was lost: the far side did the work, wrote the response, and the response never made it to the caller.
The first and the last both look like a timeout. The caller waited, nothing came back, and it has no way to know whether the far side is holding a committed transaction or never saw the request. Every retry policy, every idempotency key, every reconciliation job in this book is a way of living with that one ambiguity.
The Double Charge, Read Slowly
On the night of the spring on-sale a buyer's browser sent POST /orders. The service held her seats, called Payrail, received a successful charge, wrote the order, committed, and began writing the response. Her phone switched from Wi-Fi to cellular at that moment and the connection died. Her browser, doing what browsers do with a request that received no response, retried. The service saw a new POST /orders with the same seats, held them again (the first order's holds had converted to sold, but the retry carried the original hold ids, which the code accepted), called Payrail again, received a second successful charge, wrote a second order, and this time the response arrived.
No line of code was wrong. Payrail did its job twice because it was asked twice. The browser was being helpful. The service could not tell a retry from a new order because nothing in the request said "this is the same intent as before." Chapter 7 gives the request an idempotency key, and the retry returns the first order's response without touching Payrail. But the lesson here is earlier than the fix: the service must be designed on the assumption that the last outcome, succeeded with the answer lost, happens routinely, because on a busy night it does.
The Same Fact, Every Boundary
The client to the service is the obvious boundary. It is not the only one. The service to Postgres: a commit whose acknowledgement is lost is still a commit, and the service must treat the timeout as "unknown," not "no." The service to Redis: a SET that timed out may have set. The service to Payrail: the charge that timed out at three seconds may have gone through at four. Payrail to the service's webhook endpoint: Payrail will deliver the event at least once, which means sometimes twice, and not always in order. The API to the worker through the stream: a job claimed by a worker that then crashed will be delivered again to another.
The chapters of this book are that list. Chapter 6 is the database boundary. Chapter 7 is the general mechanics. Chapter 8 is the queue. Chapter 9 is the cache. Chapter 10 is the provider. Each one is the same fact wearing a different protocol.
What Falls Out
A timeout on every call, because a call with no deadline turns "lost" into "forever" and a process with twenty of those is a process serving nothing. Retries only on operations that are safe to repeat, and a way to make the unsafe ones safe. An idempotency key on every request that creates or charges, so that a repeat is recognized and answered from memory. At-least-once delivery accepted on every internal path, with handlers written to tolerate a duplicate. And a reconciliation job that asks the neighbour for the truth on a schedule, because some fraction of every message will be lost no matter how well the rest is built.
result = await payrail.charge( order, idempotency_key=f"order-{order.public_id}", # same key on every attempt timeout=ctx.remaining(default=3.0), # never infinite ) # a Timeout here means UNKNOWN, not failed: leave the order pending, # tell the buyer to retry, and let reconciliation resolve it tonight.
Three lines carry the whole discipline. The idempotency key is derived from the order, so every attempt, the first, the retry after a timeout, the retry after a crash and restart, presents the same key, and Payrail charges once. The timeout comes from the request's remaining budget, never from a library default. And the comment is not decoration: a timeout leaves the order in pending, and the nightly reconciliation of Chapter 10 asks Payrail whether that charge exists.
Exactly-Once Is a Property of the Handler
No transport delivers exactly once across a failure. A queue can promise at-most-once, which loses messages, or at-least-once, which duplicates them; the third option requires the sender's commit and the receiver's acknowledgement to be a single atomic operation across two systems, which they are not. The only exactly-once that exists is at-least-once delivery plus a receiver that detects the duplicate and does nothing the second time. This book says that sentence in four different chapters on purpose, because the temptation to believe a product's "exactly-once" checkbox is strong and the consequence of believing it is the double charge.
Marek's Rule
Before any line that leaves the process, ask: what if this succeeds and I never hear back? If the answer is "then we do it again," the line needs a key, a version, a constraint, or a reconciliation. The rest of the book is a catalogue of which one fits where. The seat hold gets a row lock and a constraint. The order gets an idempotency key. The email job gets a state check. The webhook gets an event-id table. The nightly reconciliation gets everything the others missed.
Handling errors is a try-and-except around the call that logs the failure and returns a 500. It treats failure as an exception, and it is necessary. It does nothing about the fourth outcome, because from inside the handler the fourth outcome looks like success.
Designing for the network assumes the answer will be lost some fraction of the time on every boundary, and makes the repeat safe. It changes the schema: a key table, a version column, an outbox. The buyer charged twice needed the second; her service had only the first.
- Treating a timeout as "it did not happen" — the charge went through, the hold was created, the email was sent; a timeout means unknown, and code that assumes "no" creates the duplicate on retry.
- Retrying anything that failed — a retried
POSTwithout a key is the double charge; a retriedDELETEis harmless; the difference is whether the operation is safe to repeat, not what the status code was. - Believing a transport promises exactly-once — the stream, the queue, the webhook and the HTTP client all deliver at least once under failure, and a handler that runs twice must produce the same result as one that ran once.
- Fixing the double charge with a "please do not click twice" notice — the browser's retry was automatic and so was the mobile app's; a network fact cannot be repaired in a user interface.
- Designing for the total outage and not the lost answer — a dead Payrail is easy to detect and handle; a Payrail that charged and then timed out is the case that costs money, and it happens on a healthy network.
- Put a timeout on every call that leaves the process, derived from the request's remaining budget, with no exceptions.
- Make every write that can be retried idempotent — by key, by version, or by constraint — before adding the retry, never after.
- Accept at-least-once delivery on every internal path and write every handler so that running it twice reaches the same state as running it once.
- Reconcile against the neighbour's truth on a schedule instead of trusting that every message arrived.
- Ask "what if this succeeds and I never hear back?" in code review of every outbound call, and refuse the change until the line has an answer.
Knowledge Check
A call to Payrail times out after three seconds. Which two outcomes can the service not distinguish between?
- The request never reached Payrail, and Payrail charged the card but the response was lost
- Payrail declined the card, and Payrail charged the card but took longer than three seconds
- The connection was refused at once, and the request was accepted but queued behind other traffic
- Payrail rejected the idempotency key, and Payrail returned a malformed response body
On the spring on-sale night the buyer was charged twice. Which line of code was wrong?
- The browser's retry logic, which should never repeat a POST without asking the user first
- Payrail's charge endpoint, which should have detected two identical charges arriving a few seconds apart and refused the second
- No single line: the design as a whole had no way to recognize a retry as the same intent as the original
- The commit, which happened before the response was written and should have been deferred
Where does exactly-once delivery actually live?
- In the message broker, when it is configured with the exactly-once delivery guarantee
- In the transport layer, because TCP already guarantees each segment is delivered exactly once
- In the receiving handler, as at-least-once delivery plus a duplicate check on the receiver's side
- In the database transaction, which makes the send and the acknowledgement a single atomic step
Marek's rule asks "what if this succeeds and I never hear back?" before every outbound line. For the seat-hold write, what is the answer that satisfies it?
- A retry policy with exponential backoff and jitter, so a lost answer is retried until the seat is confirmed held
- A row lock during the write and a unique constraint on the hold, so a repeat cannot create a second hold
- A log line at warning level whenever the response write fails, so the case can be investigated later
- A longer timeout on the response write, so the answer is far less likely to be lost in the first place
You got correct