Designing for Failure

Links get cut by backhoes, line cards die, providers have regional outages, and BGP sessions flap. Resilient design starts from the premise that every component will eventually fail and asks one question of each: when this dies, what else dies with it? The goal is never to prevent failure — that is impossible — but to ensure that no single failure turns into an outage. That demands redundant paths, components that do not share fate, and a system that degrades gracefully instead of falling over.

Redundancy is only worth what its independence is worth. Two links that follow the same conduit, two routers fed by the same power strip, two regions behind one DNS provider — these look redundant on the diagram and fail together in the field. This topic is the synthesis of the whole course's reliability thread: anycast (Topic 23), multi-region failure domains, health-checked load balancing, and the single-DNS trap all reduce to the same discipline of eliminating shared fate.

Eliminating Single Points of Failure

A single point of failure is any component whose loss takes down the whole path. Find them by walking the request end to end and asking, at each hop, "what is there exactly one of?" One uplink, one router, one firewall, one provider, one DNS zone — each is a SPOF until it is doubled. The subtle ones live where the diagram hides them: two fibers in the same trench fail to the same backhoe, two instances in the same rack fail to the same top-of-rack switch.

True redundancy requires independence at every layer the components share — physical path, power, provider, and control plane. Two links from the same carrier riding the same physical route are one link for failure purposes, no matter how the contract reads. The honest test is correlation: if a single event can take out both copies, you have one component drawn twice, not two.

Multipath and ECMP

Redundant paths held purely as standby waste half your capacity and hide a failover bug until the day you need it. Equal-cost multipath (ECMP) uses them all at once: when several routes to a destination share the same cost, the router hashes each flow's 5-tuple to one of them and load-balances across the set. A failed next hop simply drops out of the hash set, and the surviving paths absorb the traffic — no failover event, no standby to wake up.

The catch is per-flow hashing, not per-packet: ECMP pins each connection to one path to avoid reordering, so a single elephant flow rides one link and cannot exceed that link's capacity. Anycast (Topic 23) is the same idea at internet scale — many sites advertise one address, and BGP steers each client to the nearest, with a withdrawn route shifting traffic to the next site automatically.

# two equal-cost next hops to the same prefix — Linux hashes
# each flow to one, and a dead nexthop drops out of the set
ip route add 10.20.0.0/16 \
    nexthop via 10.0.0.1 dev eth0 weight 1 \
    nexthop via 10.0.0.2 dev eth1 weight 1
ip route get 10.20.5.7   # shows which nexthop this flow lands on

Failure Domains and Blast Radius

A failure domain is the set of things that fail together. A zone is a failure domain bounded by shared power and cooling; a region bounds shared zones; a provider bounds everything you run on it. Designing for failure means choosing how wide your blast radius is allowed to be and then spreading replicas across domains so no single one holds a majority of your capacity — three zones at 34% each survives the loss of one with room to spare; two zones at 50% does not.

The trap is the hidden shared dependency that collapses two domains into one. Replicas in two regions that both resolve through a single DNS provider share that provider's failure domain — when it goes down, both "independent" regions go dark at once. The single-DNS SPOF and the single-upstream SPOF are the same mistake: a global dependency silently spanning the domains you thought you had separated.

Graceful Degradation

Failover only counts if it has been exercised. A standby that has never taken live traffic is a hypothesis, not a backup — health checks must actively probe each member and pull failing ones out of rotation faster than clients notice, and connection draining must let in-flight requests finish before an instance is removed. Without draining, every deploy and every failover resets live connections, turning a routine event into a visible blip.

Graceful degradation means the system sheds load instead of toppling. When a backend pool loses half its members, the survivors should serve a reduced but correct service — shedding non-essential work, serving stale cache, returning a fast error rather than a slow hang — not pile every retry onto the remaining capacity until it too falls over. The standby must also be sized for full load: failover to a member that buckles the instant it inherits production traffic is no failover at all.