Caches That Lie

A cache adds a second place a fact can live, and every way the two places can disagree is a bug class with a name. Negative caching hides a newly published event for a minute. A cold start after a Redis restart is the stampede on every key at once. A cache that outlives a deploy serves bytes the new code cannot read. A fallback that is heavier than the cached path takes the primary down the moment Redis goes. A counter that was only ever incremented in Redis is zero after the restart and nobody knows what it was. Each of these happened to Stagedoor, or nearly did, and each has a shape that is recognizable from the dashboard once it has been seen once.

This topic is the catalogue. The fixes are mostly things earlier topics already built, applied in a place nobody thought to apply them: the versioned key, the rebuild lock, the rate limit from Chapter 7's dependency table, the rule that Redis holds copies and coordination and nothing else. What is new is the debugging question at the end, which turns "the page looks wrong" into a minute's work.

Negative Caching

A scraper walks GET /events/{id} with random ids, 200 a second, and every one misses the cache and queries the primary for a row that does not exist. Caching the absence fixes it: the read path stores a sentinel under the event's key, and the next request for the same missing id is answered 404 from Redis. The trap is the TTL. Stagedoor's first version gave the sentinel the positive value's 60 seconds, and then an organizer created event 9140, opened its page, and was told it did not exist. She tried again, was told again, filed a bug, and 50 seconds later the page worked. Someone, possibly her own browser's preview, had asked for 9140 five seconds before she saved it.

Two fixes, and both are needed. The negative TTL is short, 5 seconds, because the cost of a missed negative entry is 1 query per id per 5 seconds, which is 40 queries a second from a scraper at 200, against the cost of hiding a real event for a minute. And the create path deletes the key after its commit, exactly as the hold path deletes the seat map, because creating a row is a write to a key that may exist as a sentinel. The organizer's page is right within 5 seconds at worst and within milliseconds on the ordinary path.

ABSENT = b"__absent__"                                # the sentinel; never a valid event body

async def get_event(svc, event_id: int) -> Event:
    key = f"event:v1:{event_id}"
    raw = await svc.cache.get(key)
    if raw == ABSENT:
        raise NotFound("event", event_id)               # 404 from Redis, no query
    if raw is not None:
        return Event.from_json(raw)
    event = await svc.events.by_id(event_id)
    if event is None:
        await svc.cache.set(key, ABSENT, ex=5)       # 5 s, not the positive 60
        raise NotFound("event", event_id)
    await svc.cache.set(key, event.to_json(), ex=jittered(60, 5))
    return event

# in create_event, after the commit:
await svc.cache.delete(f"event:v1:{event.id}")   # the sentinel may be there; a delete of nothing is fine

The read checks for the sentinel first and answers 404 without a query when it finds it. On a real miss it queries, and when the row is absent it stores the sentinel for 5 seconds; when the row exists it stores the event for a jittered minute. The create path, after its commit, deletes the key for the new id, which removes a sentinel if one was cached and does nothing otherwise. The sentinel is a byte string no real event could serialize to, so the two kinds of value cannot be confused when the code changes.

Cold Start

Redis restarts empty, or fails over to a replacement that is, and every key is missing at once. The 2,000 event keys, the seat maps of the 40 events on sale, the sessions, the counters. Jitter does nothing here; there are no expiries to spread, and the first request for each key is a miss. Without the rebuild lock of Topic 50, that is the stampede multiplied by the number of hot keys: every seat map is rebuilt by every concurrent reader of it, and the primary that was at 10 percent is at 100 within the first second, for every event, at the same time. With the lock, each key is rebuilt once, and the cold start costs the primary one query per hot key spread over the seconds it takes the first readers to arrive: 40 seat-map queries and a few hundred event reads, a burst it can absorb.

For the hottest keys Stagedoor does not wait for the first reader. A warm-up job, run at instance start and again when the Redis breaker closes after an outage, rebuilds the seat maps of every event whose on-sale is within the hour and sets them. The readiness check of Chapter 11 can wait for it, so that an instance that has just started does not take traffic into an empty cache during an on-sale. Redis's own persistence, an RDB snapshot or the append-only file, is the other mitigation: a restart reloads what was on disk and the cache is only as cold as the snapshot is old. Stagedoor runs with it, and still keeps the lock and the warm-up, because the failover to a fresh instance is the case persistence does not cover.

Stale Across Deploys

A deploy adds a section field to every seat in the map and changes the reader to require it. The keys are still seatmap:v2:8812, so api-01, running the new code, reads 60 KB of last-deploy bytes with no section in them and raises on the first seat. The map that was cached before the deploy is wrong for every reader until it expires, and the rolling deploy makes it worse: for the minute when api-01 is new and api-02 is old, each writes bytes the other cannot read, and the seat map flaps between two shapes under the same key.

The versioned key from Topic 49 fixes it on its own. The new code reads and writes seatmap:v3:8812, the old code stays on v2, and for the minute of the rolling deploy both shapes exist in Redis under different keys, each read only by the code that wrote it. The old keys expire on their own once the last old instance is gone. What makes it happen is a line in the review checklist: a change to a cached type's shape is a change to its key version, and the pull request that changes one without the other is the incident. The parser's fallback, a default for the missing field, is a second line of defence for the day the checklist is skipped, and Stagedoor has both.

The Heavier Fallback

Redis goes, and the breaker of Topic 40 turns every seat-map read into a miss in under a millisecond. Each miss is a query, so the primary now sees 2,600 seat-map queries a second, and the primary was sized for 260 of them with room for checkout. That is the row in Chapter 7's dependency table where the fallback is heavier than the path it replaces: Redis's outage becomes Postgres's within a minute, and Postgres's outage takes the writes with it. A fallback that costs more than what it stands in for is a second outage scheduled to follow the first.

The dependency table says which of two moves each cached read makes. The seat map's fallback is rate-limited: a token bucket of 100 seat-map queries a second per instance, which cannot live in Redis because Redis is what is missing, so it lives in the process, and beyond it the buyer gets a 429 with a Retry-After of 1 second. Two instances, 200 queries a second, under the 260 the primary can carry alongside checkout. The public event list makes the other move: each instance keeps the last list it served in process memory, and while Redis is gone serves that copy, stale by at most the minutes of the outage, because a stale event list costs nothing and a rate-limited one would 429 the home page. Both are written in the table, and the load test of Chapter 12 kills Redis to check them.

Two Sources of Truth by Accident

The "seats remaining" badge on the event page was once a counter: INCR holds:8812 when a hold was placed, a decrement when it expired, never a row in Postgres, because a count in Redis is 1 millisecond and a count over 2,000 rows is 8. Then Redis restarted, the counter was zero, the badge said 2,000 seats remaining for an event that had sold 1,400, and the number stayed wrong until an organizer emailed. Nothing could rebuild it, because nothing had been written down that could. That is the failure the rule in Topic 48 exists for: every value in Redis is a copy of something Postgres has, or coordination state whose loss is a known and bounded event, and a business fact that lives only in Redis is neither.

The fix was to make the counter a copy. It is rebuilt from a count over the seat rows on a miss, cached for 30 seconds with the seat map's delete beside it, and a Redis restart costs one count query per event instead of a wrong number until someone notices. The same test sorts the rest of Redis. The seat-map version counter behind the ETag is a copy: on a miss it is re-derived from the sum of the event's seat versions, so a restart cannot hand a client a validator it has seen before. Sessions, the rebuild locks and the rate-limit counters are coordination state, and Chapter 7's table already says what each one's loss costs. Anything that fits neither description does not go in Redis.

The Debugging Question

"Is it stale?" is the first question for any read that looks wrong, and it should take a minute to answer. Three things make it so. The value carries its own cached_at from Topic 49, and the response carries the age it was served at, so the engineer looking at a wrong seat map sees at once that it was built 28 seconds ago and the suspect is a missed delete. The hit and miss counters per key family, on the dashboard, show whether the key was even in Redis at the time. And a nocache query parameter, honoured only for a principal with the admin role from Chapter 5 and logged every time it is used, bypasses Redis for one request and reads the primary, so that "what does the truth say right now?" is one request away instead of a psql session on the primary at the busiest hour.

The bypass is admin-only for the same reason the fallback is rate-limited: a parameter any client can send is a way for any client to turn the cache off, and a scraper that discovers it has the primary to itself. Logged, gated and counted, it is the difference between an afternoon of guessing whether the cache or the code is wrong and a minute of knowing.