Algorithms and Health Checks

A load balancer makes two decisions on every request: which backends are eligible, and which eligible one gets this request. Health checks answer the first; the balancing algorithm answers the second. Get the algorithm wrong and you pile uneven load on one node while others idle. Get health checks wrong and you either route traffic to a dead backend or evict a healthy one over a momentary blip.

These two mechanisms are what turn a backend failure from an outage into a non-event. A pool of ten servers with working health checks loses one and shrugs; the same pool with checks hitting the wrong path keeps sending a tenth of all traffic into a black hole. The algorithm and the checks are not tuning details — they are the difference between a balancer that absorbs failure and one that amplifies it.

Balancing Algorithms

Round-robin hands each new request to the next backend in rotation. It is simple and spreads connection count evenly, but it is blind to load — if one request costs 10 ms and the next costs 10 seconds, round-robin still deals them out one-for-one and a slow backend keeps getting fed. Weighted round-robin biases the rotation toward bigger machines, useful when the pool is not uniform.

Least-connections sends the next request to whichever backend currently has the fewest open connections, which tracks actual load far better when request costs vary. Consistent hashing maps a key — client IP, a session ID, a cache key — to a backend so the same key lands on the same backend repeatedly, and crucially, adding or removing one backend remaps only 1/N of keys instead of reshuffling all of them. That stability is what makes it the right choice for cache affinity.

# nginx: least-connections plus a weighted node and checks
upstream app {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.13:8080 weight=2;   # bigger box
}

Session Affinity

Sticky sessions pin a given client to one backend for the duration of its session, usually keyed on a cookie the balancer sets or on the client IP via a hash. You need it when the backend holds per-user state in memory — an in-process session, an upload assembled across requests — and losing that backend means logging the user out. The cost is real: affinity defeats even balancing, so a few heavy users can hammer one node while the rest of the pool sits idle, and removing that node drops every session pinned to it.

The better answer is usually to remove the reason for stickiness — push session state into Redis or a database so any backend can serve any request — and reserve affinity for cases where you genuinely cannot, like a WebSocket already bound to one backend. Consistent hashing is the affinity mechanism with the least collateral damage, because adding a node reshuffles only a fraction of the keys instead of scrambling every session at once.

Health Checks

An active health check has the balancer probe each backend on a schedule — typically an HTTP GET to a readiness path every few seconds — and mark it down after a threshold of consecutive failures. A passive check infers health from real traffic: if a backend returns errors or times out on live requests, the balancer ejects it. Active checks catch a sick backend before a user hits it; passive checks catch failures the probe path misses. Production systems run both.

The thresholds matter as much as the probe. Check too aggressively — say, eject after one failure on a 1-second interval — and a single garbage-collection pause flaps a healthy node out and back, scrambling the pool. Check too leniently — 30-second intervals, five failures to eject — and a dead backend keeps taking traffic for over two minutes. The probe must hit a real readiness endpoint that exercises the app's dependencies, not a path that returns 200 while the database connection behind it is dead.

# a readiness path that actually checks dependencies
GET /healthz HTTP/1.1

# 200 only if the DB and cache are reachable:
#   interval 2s · timeout 1s · healthy after 2 · unhealthy after 3
# a backend that is "listening but broken" must return 503 here

Draining and Graceful Removal

When you deploy or scale down, you do not want to yank a backend mid-request. Connection draining stops sending the node new requests while letting in-flight ones finish, then removes it once they drain or a timeout expires — typically 30 to 300 seconds. Without it, a deploy kills live connections and users see resets and 502s for every request that happened to be in flight on a terminating node.

Draining is the deploy-time twin of health checking. A health check ejects a node that failed unexpectedly; draining removes one you are taking out on purpose, gracefully. The two together mean a rolling deploy across a ten-node pool is invisible to users — each node drains, exits, returns, and rejoins, with traffic shifting smoothly the whole time and not a single dropped request.