Degrading Gracefully
Each of Stagedoor's dependencies will be unreachable at some point. Redis will be restarted by a memory limit, the replica will fall behind and be taken out, Payrail will have the outage its status page later calls a "degradation," and one day the primary will fail over. What the service does in each case should be a decision made in advance and written down, not an exception discovered in production at 8 p.m. on an on-sale night. The breaker of Topic 40 detects the outage in 20 calls; this topic is the table of what happens next, one row per dependency, with the fallback, the endpoints it touches, what the buyer sees and what the on-call engineer is told.
The table has five rows because Stagedoor has five things it cannot run without, and for four of the five the answer is not "the service is down." Serve stale, fall back to a slower path, switch a feature off, or fail the one affected request cleanly with a code the client can act on: those are the four moves, and each row picks one. The row that picks "fail cleanly" is as much a design as the row that picks "serve stale"; what is not a design is a 500 and a stack trace.
The Dependency Table
The table is kept beside the breaker configuration, in the repository, and it is the document the load test of Chapter 12 is written against. Each row names the dependency, what breaks when it is gone, what the service does instead, which endpoints are affected, what the client receives, and what the alert says. The last column is the one most tables omit and the one the on-call engineer reads first, because "replica down, reports on primary" is a sentence that tells them what to do, and "error rate elevated" is not.
| Gone | What breaks | What the service does | Endpoints | Client sees | Alert |
|---|---|---|---|---|---|
redis-01 | seat-map cache, sessions, rate limits, the jobs stream | seat map from Postgres at a lower limit; browser sessions 401; outbox accumulates; breaker fails in 100 ms | seat map slower; browser users signed out; jobs delayed | 200 slower; 401 for browser sessions; orders confirmed, tickets later | "redis down: seat map on primary, sessions off, relay queueing" |
pg-replica-a | organizer reports, the public event list | those reads go to the primary; a counter increments | reports and the event list, unchanged from outside | 200, identical | "replica down: reads on primary, watch primary CPU" |
| Payrail | charging a card | 503 with Retry-After; order stays pending; hold extended by 10 minutes | POST /orders only | 503, "payments delayed, your seats are held" | "payrail breaker open on N instances: checkout degraded" |
pg-primary | every write | writes return 503; replica reads continue; instance alive, not ready | everything that writes; reads survive | 503 on writes; 200 on reads | "primary unreachable: writes failing, failover in progress" |
| the stream | the worker's supply of jobs | outbox rows accumulate; relay retries; oldest-unpublished age climbs | none directly; tickets and email delayed | orders confirmed, tickets later | "outbox age over 60 s: relay cannot publish" |
Five rows, and reading them across is the point. The Redis row is the longest because Redis holds four unrelated things, and each of the four has its own fallback: the seat map has a slower path, sessions do not, the stream has the outbox, and the rate limiter loses its counters, which is a decision Chapter 14 records in the same table. The replica row is the shortest because a replica is a copy and the original is still there. The Payrail row affects exactly one endpoint and says so, which is what lets the status page say "payments delayed" rather than "we are having issues." The primary row is the only one where the service genuinely cannot do its job, and even there the reads that were on the replica keep answering. The stream row is Redis again from the worker's side, and its alert is the outbox age from Topic 41, because that is the number that says the messages are safe and waiting.
Redis Gone
The seat map is read 2,600 times a second from seatmap:{event_id}, and when the breaker on Redis opens, those reads go to Postgres. That is the fallback, and it is also the danger, because 2,600 reads a second of a 2,000-row seat map is a load the primary was never sized for during checkout. Chapter 9 makes the fallback safe by putting it behind a lower rate limit, so that the primary sees a bounded number of seat-map queries a second rather than 2,600, and the buyers beyond the limit get a 429 with a Retry-After of a second. A fallback that is heavier than the path it replaces is not a fallback; it is a second outage scheduled to follow the first, and the seat map is the row where that mistake costs the most.
Sessions have no fallback. The browser session of Chapter 5 lives in Redis and nowhere else, so a browser user's next request fails authentication and gets 401, and she signs in again when Redis is back. The row records that as a decision, with the alternative it rejected: a fallback session store in Postgres was considered and refused, because sessions are written on every request and the primary is the last place to send that write during an outage. The mobile app, whose access token is self-contained for 15 minutes, notices nothing until it needs a refresh. The jobs stream is unreachable, so the outbox of Topic 41 accumulates, the relay retries every 100 milliseconds against its own breaker, and when Redis returns it drains the backlog in order. Each of these fails in 100 milliseconds rather than 3 seconds because the Redis timeout from Topic 37 is 100 milliseconds and the breaker makes the second call free.
The Replica Gone
pg-replica-a serves the organizer reports and the public event list, and Topic 36 of Chapter 6 built the routing so that those reads name the replica pool explicitly. When the replica breaker opens, the same queries run on the primary, a counter of reads-on-primary increments, and nothing changes from outside: the same 200, the same rows, arguably fresher. The cost is on the primary, which now serves reports it was spared, and the alert fires on the replica's absence, not on the primary's CPU, because by the time the CPU is the signal the reports have already slowed checkout. The row is the shortest in the table and the easiest to test, and it is the one whose fallback is most likely to be silently heavier than expected, because a report query that was fine on an idle replica is a different thing on a primary handling 3,000 requests a second.
Payrail Gone
Checkout returns 503 with Retry-After from the breaker's cooldown and the Problem Details type from Topic 40. The order stays pending, and the row adds one thing the breaker does not: the buyer's holds are extended by 10 minutes, so that a buyer who came back in a minute as the header asked does not find her seats released to somebody else. Everything else works. The seat map, the event list, the scanner at the door, sign-in: none of them calls Payrail and none of them knows it is gone. The status page, a static page on a host that shares nothing with Stagedoor, says payments are delayed, and the buyer's page says the same thing in the same words.
The alternative is the one Stagedoor shipped first: Payrail down meant checkout returned 500. The buyer's browser retried, the retries hit the same 500, the hold expired at 10 minutes while she retried, and the seat went to the next buyer whose browser happened to retry after Payrail recovered. A 503 with a true Retry-After, a pending order and an extended hold is the same outage with none of those consequences, and it costs one row in a table and one line in the checkout handler.
The Primary Gone
The service cannot write. Every write returns 503, the reads that were on the replica keep answering, and the reads that were on the primary, which is every read on the checkout path, fail with the writes. This is the one row where the answer is largely "down," and the design in it is about what the service does not do: it does not restart, and it does not pretend. The health check of Chapter 11 has two endpoints for exactly this. Liveness says the process is running and its loop is responsive, which it is; readiness says the process can serve traffic, which it cannot. The instance reports alive and not ready, the load balancer stops sending it requests, and the orchestrator leaves it alone.
The distinction is the difference between an outage and a restart loop. A health check that reports "down" when the database is down tells the orchestrator to restart a process that has nothing wrong with it, and the replacement comes up, fails the same check, and is restarted, every 30 seconds, until the primary is back and someone has to explain why the deploy dashboard shows 200 restarts. Readiness is the answer that names a dependency; liveness is the answer that names the process. The failover itself, promoting pg-replica-a and repointing the pool, is the engine's business and is Chapter 13 of the PostgreSQL course; this book's job ends at making sure the service is standing and quiet when the new primary appears.
Partial Is Normal
A dependency is rarely fully gone. It is slow, or 5 percent of its calls fail, or a slice of the charges time out while the rest answer in 400 milliseconds. The total outage is the case everyone tests and the partial one is the case that happens, and the table's rows have to hold for both. They hold because of two mechanisms from earlier in the chapter. The breaker's threshold is a rate, so 5 percent failures do not open it and 50 percent do, and the fallback is per call, so each failed call takes its row's fallback while the successful calls take the normal path. A 5 percent failure at Payrail is then 5 percent of checkouts seeing a 503 with a true Retry-After, a 5 percent degrade, and not a 100 percent outage produced by a breaker that opened on the first failure or a handler that raised on it.
The load test of Chapter 12 exercises each row on purpose. It runs the on-sale profile, 3,000 requests a second, and while it runs it kills Redis, then slows the replica, then makes the fake Payrail time out 5 percent of calls, then 50 percent, then all of them, and it asserts the table: the seat map stays under its limit on the primary, browser sessions fail with 401 and nothing else, checkout returns 503 with a Retry-After and no 500, and the outbox age climbs and then drains. A row that has not been exercised that way is a hypothesis, and the difference between a hypothesis and a design is the load test.
- No table — each dependency's outage is met for the first time as an unhandled exception, a 500 with a stack trace, and an on-call engineer reading code at 8 p.m. to find out what the service was supposed to do.
- Fallbacks that are heavier than the primary path — Redis goes down, every seat-map read goes to the primary at 2,600 a second with no limit, and the primary follows Redis down within a minute.
- A health check that reports "down" when a dependency is down — the orchestrator restarts a healthy process every 30 seconds for the length of the outage, and the dashboard shows 200 restarts and no cause.
- Payrail down means checkout returns 500 — the buyer's browser retries into the same 500, her hold expires at 10 minutes while it does, and her seats go to whoever retried after Payrail recovered.
- Testing only the total outage — the 5 percent failure case is the one that happens, and a breaker that opens on the first failure or a handler that raises on it turns it into a total outage of the service's own making.
- Write the dependency table before the first deploy, one row per dependency with the fallback, the endpoints, the client's view and the alert text, and keep it beside the breakers' configuration.
- Make every fallback cheaper or rate-limited relative to the path it replaces, and treat "reads go to the primary" as a load decision that needs a number.
- Separate liveness from readiness, so that a dependency outage takes the instance out of rotation without restarting a process that has nothing wrong with it.
- Answer a dependency outage with 503 and a true
Retry-After, leave the orderpending, and extend the hold, so the buyer loses time and not her seats. - Exercise every row of the table in the load test, including the 5 percent and 50 percent partial-failure cases, and assert the table's promises rather than the absence of errors.
Knowledge Check
Redis is gone and, separately, the primary is gone. How does the service's behaviour differ between the two?
- Both stop every request, because the seat map and the checkout path need both stores
- Redis gone degrades the seat map and sessions; the primary gone fails writes and keeps replica reads
- Redis gone fails writes, since the outbox needs it; the primary gone falls back to the replica for writes
- Both cause the instance to restart, because the health check reports down for either store
Why must the seat map's fallback to Postgres be rate-limited when Redis is down?
- Because the database's copy of the seat map may be stale and must not be served too often
- Because the breaker on Redis cannot close again until the fallback traffic drops below its threshold
- Because Postgres cannot serve a 2,000-row seat map at all without the cache in front of it
- Because a fallback heavier than the path it replaces turns one outage into two, and the primary is next
The primary is unreachable. What should the two health endpoints report, and why?
- Alive but not ready, so the balancer stops routing to it and nothing restarts a healthy process
- Not alive and not ready, so the orchestrator replaces the instance with a fresh one that may connect
- Alive and ready, so the balancer keeps routing to it and the 503s tell clients what is happening
- Not alive but ready, so the orchestrator restarts it while the balancer keeps it in its rotation
Payrail is timing out 5 percent of charges and answering the rest in 400 milliseconds. What should a buyer whose charge timed out see, and why is this case the important one?
- A 503 because the breaker opened on the first timeout, which is important because it protects Payrail
- A 402 and an invitation to try again, which is important because most retries will now succeed
- A 504 with her order left pending, which is important because partial failure is what actually happens on a real night
- A 500 like the other 5 percent, which is important because the error rate must reflect Payrail's
You got correct