Testing Failure

The dependency table of Topic 42 says what Stagedoor does when Redis is gone, when the replica is gone, when Payrail is down, slow or failing 5 percent of its calls, and when the primary is unreachable. A table that is never exercised is a hope. Failure tests inject each row's condition, a Redis that refuses connections, a Postgres that stops answering, a Payrail that returns 503 on 5 percent of calls, and assert the row: the seat map still loads, the checkout returns 503 with Retry-After, the order is pending, the breaker opened, the metric moved. They are integration tests with the failure scripted, and the ones that fail on the first run are the ones worth having written.

Marek wrote the table in Chapter 7 and the tests three weeks later, and 3 of the 7 failed on their first run. The Redis-down test found that the in-process fallback limiter was never wired to the seat-map handler. The Payrail-slow test found that the handler returned in 3.4 seconds with the right response and left the hold unextended. The primary-paused test found that readiness kept returning 200 for 40 seconds after the database stopped answering, because the pool check had no timeout of its own. None of the three was a bug the functional suite could see, because the functional suite never makes anything fail.

One Test per Row of the Table

Seven tests, one per row, and each one has the same shape: set the condition, run the requests the row says are affected, assert the client-visible result and the internal one. The internal assertion is the half that most teams skip, and it is the half that catches the quiet failures: the log line with the dependency's name, the metric that records the fallback, the breaker's state. A test that asserts only the status code proves that an error was returned, not that the right thing happened before it.

The demonstrated row is the first. With the Redis container stopped, the test sends 120 seat-map requests inside one second for event 8812 and asserts three things: the first 100 return 200 with the current map, the remaining 20 return 429 with a Retry-After of 1 second, and the primary's statement counter shows at most 100 seat-map queries, which is the whole point of the row. Then it asserts the internal half: the fallback counter reads 100, and a log line at warning level names redis-01 and the word fallback. That is the in-process limiter of Topic 49 being tested at its number, and it is the test that found the limiter unwired.

Injecting the Condition

The injection is always at the boundary, never by patching a function inside the service. For the fakes, the scripted controls of Topic 65: fake.next, fake.fail_rate, a fake that sleeps. For the real Postgres in its container, two choices with different meanings. Pausing the container, which Testcontainers exposes through the underlying Docker container's pause, freezes every process in it, so connections stay open and every statement hangs until the timeout: that is "the database stopped answering," the condition Topic 37's timeouts exist for, and the cleanest way to produce it. Setting statement_timeout low on the test's session produces "the query was too slow" instead, which is a different row with a different error. For Redis, a wrong port in the config produces a refused connection in a millisecond; stopping the container produces the same refusal with more realism and 2 seconds more setup; both are honest.

async def test_payrail_down_degrades_checkout(app_client, svc, fake_payrail, hold):
    fake_payrail.next_n(PayrailTimeout(), count=20)          # enough failures to open the breaker: 50% of 20
    for _ in range(10):                                    # 10 checkouts, 2 attempts each, all time out
        await app_client.post("/orders", json=checkout_for(hold), headers=idem_key())

    resp = await app_client.post("/orders", json=checkout_for(hold), headers=idem_key())

    assert resp.status_code == 503                          # 1. the client-visible half
    assert 0 < int(resp.headers["Retry-After"]) <= 30       # 2. a true cooldown, not a guess
    order = await svc.orders.get(resp.json()["order_id"])
    assert order.status == "pending"                        # 3. not failed, not paid
    assert (await svc.holds.get(hold.id)).expires_at == hold.expires_at + timedelta(minutes=10)   # 4. hold extended
    assert fake_payrail.charges == {}                       # 5a. nothing charged while the breaker was open
    assert metrics.gauge("breaker_open", dep="payrail") == 1  # 5b. the internal signal moved

The test scripts 20 consecutive timeouts on the fake, then sends 10 checkouts through the real HTTP client to the real app, each of which times out on both attempts, which is 20 failed calls and enough to open the breaker at 50 percent of a 20-call window. The eleventh checkout is the one under test. Five assertions follow, and only the first is the status code. The Retry-After is a number inside the breaker's 30-second cooldown, not a constant somebody typed. The order is pending. The hold's expiry moved by exactly 10 minutes. Nothing was charged, and the breaker's gauge reads open. Remove the hold extension from the handler and the fourth line fails; make the handler mark the order failed on a timeout and the third does. A test with only the first assertion would pass both of those bugs.

Asserting the Degrade, Not Just the Error

"Returns 503" is half an assertion, and it is the half that has been wrong in the least interesting way. The other half is the state: the order row, the hold, the payments table, the header, the breaker's gauge. On the night of the spring on-sale the checkout also returned an error when Payrail was slow. What it did before returning the error was charge the card, and no test that looked at the status code would have noticed. Every failure test in Stagedoor asserts the state after the error, because the state is what the buyer, the reconciliation job and the on-call engineer will find in the morning.

Timeouts Are Tested by Being Hit

A timeout that has never fired in a test is a number in a config file. The Payrail-slow test makes the fake sleep 4 seconds against the client's 3-second read timeout and asserts that the handler returned in under 3.5, with a 504 and the payment-unresolved type, the order pending and the hold extended. The request deadline of Topic 37 gets the same treatment one level up: a handler wired to a fake that would take 12 seconds, against the 10-second budget, and an assertion that the response arrived at 10 with the deadline's error rather than at 12 with the fake's answer. These tests are slow by construction, 4 and 10 seconds each, so they carry the slow marker, run on merge rather than on every save, and there are exactly as many of them as there are timeouts in the configuration table: 7.

The temptation is to test the timeout with a smaller number, 300 milliseconds in the test config against a 400-millisecond sleep, and the temptation is right for most of the suite. The two slow tests above exist because the production number is the thing under test, and a suite where every timeout is overridden to 300 milliseconds has never once checked that the number in production is the number the handler honours.

The Partial Case

A dependency is rarely fully gone, and Topic 42 says the table's rows must hold for the partial case too. Two tests together are the breaker's threshold, written down. The first sets fake.fail_rate = 0.05, sends 200 checkouts, and asserts that every one of them succeeded, the breaker stayed closed, and the log shows about 10 retries: 5 percent is under the 50 percent threshold, the retry of Topic 38 absorbs it, and the buyer never knows. The second sets the rate to 0.6, sends 200, and asserts that the breaker opened within the first 40 calls, that the calls after it failed fast with 503 and no network attempt, and that the breaker gauge and the failure counter both moved. One test says the threshold is not too low; the other says it is not too high.

The partial test is the one that found Marek's first breaker, which counted a 402 decline as a failure. At a 5 percent decline rate, normal for a Saturday, the breaker opened after a busy minute and every checkout failed for 30 seconds while every card was fine. The test with fail_rate at 0.05 and 10 declines mixed in fails on the assertion that the breaker stayed closed, and that is the assertion that would have prevented the incident.

Recovery

Every failure test has a second half, and it is the half most often skipped: the condition is removed and the test asserts that the service comes back. The Payrail-down test clears the fake's script, advances the frozen clock past the 30-second cooldown, sends one checkout and asserts that the breaker went half-open, let the call through, saw it succeed and closed; the next checkout is a 201. The primary-paused test unpauses the container and asserts that the pool's acquire succeeds again and /readyz returns 200 within 5 seconds, without a restart. The replica test repoints the URL and asserts the replica pool of Topic 31 reconnected and the reads-on-primary counter stopped moving.

A service that degrades correctly and never recovers is the next incident, scheduled for the minute after the dependency returns. The half-open trial of Topic 40 is the kind of code that is written once, never runs in development, and is wrong in a way nobody sees until a breaker stays open for an hour after Payrail recovered. The recovery assertion is a fifth of the test's lines and the reason those lines have run.