Safe Rollouts and Rollbacks

Tests pass, staging looks fine, and the change goes live — and it still breaks. Not because anyone was careless, but because production has things no rehearsal fully copies: real traffic, real data, the odd reader doing something nobody imagined. A change can clear every earlier check and still misbehave the moment it meets real users.

So the last line of defense isn't another test before release — it's how you release. Teams roll a change out to a few users first and keep an instant way back. These two habits are what turn shipping often from a nightly gamble into something routine.

New versionready to release

→

Canarysmall slice of users

→

Widen or undohealthy → all · bad → roll back

Release to a few first

Instead of switching every reader to the new version at once, the team gives it to a small slice of users — say one in twenty — while everyone else stays on the version that already works. This is called a canary release: a small, deliberate test of the new version on real traffic, but only a little of it. The exact slice varies from team to team — it might be one in twenty, one percent, a single region, or just internal staff first.

The name comes from coal mining, where miners carried a canary underground because the bird reacted to dangerous gas before the people did. A canary release is the same idea: a small group meets the danger first, so if something's wrong, only they feel it — not the whole readership.

Watch it, then widen

A canary only helps if someone is watching it. While that small slice uses the new version, the team checks whether it's healthy — are those users seeing errors, is it slower for them than for everyone still on the old version? That watching is monitoring, the subject of the very next chapter.

If the canary stays healthy, the team widens the release — more users, then more, until everyone is on the new version. This is the same gradual-replacement idea as the rolling update from Chapter 6, now steered by what the canary group tells you, so confidence grows one careful step at a time.

The instant undo: rollback

Sometimes the canary looks bad. The fix for that is a rollback: switching the app back to the previous version that was working, undoing the release. Because the old version is kept ready, a rollback can happen in seconds — far faster than trying to diagnose and patch the broken one while users suffer.

One thing rollback does not mean: erasing users' data. It reverts the app's version — the code that's running — back to the last good one. The books a reader logged minutes ago are their data, handled separately and carefully; a rollback changes which version of Pageturn is serving, not what's stored in it. (That's the simplified picture — handling data across versions has real subtleties the deep courses cover.)

Why this beats the big bang

Put the two together and you get the opposite of the old, scary release. A canary keeps the blast radius small — if a change is bad, it hurts a handful of users, not all of them. And a rollback keeps recovery fast — the way back is one quick switch, not a frantic late-night repair.

Small blast radius and fast recovery are exactly what make frequent shipping safe. This is the feedback loop from Chapter 1 made concrete: ship a small change to a few, watch what happens, and either widen it or undo it — then do it again tomorrow, without dread.