Chapter Seven · Failure by Design
Failure by Design
The network can fail between any two lines of your code, and this is the chapter built on that fact. Six topics give every outbound call a deadline, decide what is safe to retry and how long to wait, make a repeated request return the first answer, stop calling a provider that is down, make a row and a message one unit, and write down what Stagedoor does when each dependency is gone. This is the chapter that closes the second wound: one buyer, charged twice.
On the night of the spring on-sale a buyer's checkout took 4 seconds because Payrail was slow, her connection dropped while the service was writing the response, her browser retried, and she was charged twice. Chapter 1 read that night slowly and found no wrong line of code. Marek found the cause was not Payrail and not the browser: POST /orders had no timeout, so nothing declared the first attempt over; no idempotency key, so nothing recognized the second request as the first intent; and no way to tell a retry from a new order, so the service did what it was asked, twice. Topic 39 closes the wound with a table of keys and a unique constraint. Topics 37 and 38 are what make that fix reachable, and Topics 40 to 42 are what keep the service standing on the night the fix is not enough.
The chapter is one idea worn six ways. A call that leaves the process can time out, arrive twice, arrive late, or succeed while its answer is lost, and the caller can tell only three of those apart. A timeout gives the fourth case a number. A retry policy decides whether a second attempt is safe, which depends on the operation more than on the error. An idempotency key makes the unsafe operation safe by giving the repeat a name. A circuit breaker and a bulkhead stop the service from spending its whole capacity discovering that Payrail is down. The outbox makes the order and the job that delivers its tickets one transaction, so a crash between two lines cannot separate them. And the dependency table says, in advance and in writing, what the buyer sees when Redis, the replica, Payrail, the primary or the stream is gone.
Every mechanism has a number, and they come from the same canon: a 3-second timeout to Payrail against a 10-second budget, 100 milliseconds for Redis, 3 attempts with full jitter under a 10 percent budget, keys that live 24 hours, a breaker at 50 percent of the last 20 calls with a 30-second cooldown, a relay moving 100 rows every 100 milliseconds. The chapters after this one are the same fact wearing other protocols, and they lean on this one by name: the worker in Chapter 8 tolerates the duplicate the outbox produces, the cache in Chapter 9 falls back the way the table says, and Chapter 10's webhook and reconciliation are the neighbour's side of the timeout.