Testing in Production
Staging cannot tell you what Saturday morning will do. It has synthetic data, a fraction of production's traffic, none of its cardinality, and no third-party payment processor having a bad day. The checkout path that passes every staging test at 40 requests per minute meets a different world behind Saturday's real customer traffic — and the differences are not noise around the edges, they are the substance of what breaks.
You don't know it works until prod says so. The honest response is not to test less before deploying but to make production observations the gate a release must pass. This topic is where the stack built since Chapter 3 becomes a deploy safety net: the RED metrics judge the canary, the burn rate triggers the rollback, and the annotation records what shipped.
The Honest Case
The differences staging cannot reproduce are exactly the ones that cause incidents: the real traffic mix, the real data distributions, the real behavior of dependencies you do not control. The August incident from Chapter 12 — the payments processor slowing down and retries amplifying the load — existed in no test environment Harborline could have built, because the processor's bad night was never Harborline's to simulate.
Progressive delivery accepts that gap instead of pretending it away. New code meets production, but on a leash: a small slice of traffic, a bounded observation window, and an automatic path back. The observability stack is what holds the leash — without trustworthy signals there is no verdict, only exposure.
Canary Releases Judged by RED
A canary release routes a small share of traffic — 5% is typical — to the new bookings version while the stable version keeps the rest. For 15 to 30 minutes the two versions run side by side under identical real traffic, and the question is purely comparative: is the canary's error ratio or p95 latency worse than the stable version's? The same harborline_ histograms from Chapter 5 answer it, split by a version label.
# p95 checkout latency, computed per running version
histogram_quantile(0.95, sum by (le, version) (
rate(harborline_checkout_duration_seconds_bucket{job="bookings"}[5m])
))
The query computes p95 checkout latency separately for each version over a five-minute window, and the release system compares the two results to reach its verdict. That is the load-bearing point of the whole pattern: the canary judgment is a PromQL comparison, so the deploy gate is exactly as good as the instrumentation underneath it. A service whose metrics carry no version label has no canary judgment, whatever the rollout tooling promises.
Feature Flags as the Cheaper Canary
A feature flag decouples deploy from release. The code ships dark, disabled for everyone; then the flag turns on for 1% of sessions, and the flagged cohort's metrics answer whether the change misbehaves. Disabling the flag is a sub-second rollback with no redeploy — cheaper and faster than shifting traffic between two running versions.
The flag state belongs in the telemetry: an info-style metric or a span attribute recording which flags each request saw. During the incident a flag may one day cause, "who has this flag" has to be a queryable question in Grafana, not an archaeology dig through the flag service's admin screens.
The Rollback Trigger
The multi-window burn-rate logic from Chapter 10 works unchanged as a release gate. If the checkout SLO's fast-window burn rate crosses its threshold while a canary is live, the release system rolls back before Alertmanager pages a human. The SLO stops being a monthly report and becomes an actuator — the budget's own math decides the release is spending too fast and pulls it.
Deploy Annotations Closing the Loop
Chapter 6's Grafana annotations mark every deploy on every dashboard — one API call from the pipeline. When latency steps up at 09:14, the annotation at 09:12 turns "did the deploy cause it" into a question answered by looking. Without it, release-caused incidents masquerade as mysteries, and the postmortem burns its first hour reconstructing a timeline that one HTTP request could have drawn in advance.
The Scope Boundary
The machinery that executes all of this — rollout controllers like Argo Rollouts or Flagger, traffic shifting, pipeline wiring — is delivery engineering, a discipline with its own books. This one owns the observability side of the contract: the signals, the queries, and the SLO math the machinery consumes. CI fundamentals live in the Git, GitHub & GitHub Actions course; what no delivery tool can supply on its own is the trustworthy RED metrics its analysis runs on, and those are Chapter 5's work.
- Canarying on 5% of traffic for 5 minutes and calling it judged — at Harborline's off-peak rate that is a few hundred requests, too few for a stable error ratio. Size the canary window in requests observed, not minutes elapsed.
- Comparing the canary against a fixed threshold instead of against the stable version running beside it — if the whole system is slow because
db-01is degraded, the canary fails the threshold for reasons that have nothing to do with the release. The concurrent stable version is the control group. - Shipping feature flags that never appear in telemetry — six months later 40 stale flags interact, an incident hits the 3% of users carrying one specific combination, and no query can even identify the cohort.
- Automating rollback on any firing alert rather than on SLO burn — a cause-based alert from topic 70's noisy channel fires mid-canary and rolls back a healthy release. After two false rollbacks the team disables the automation, and the gate is gone.
- Treating "testing in production" as permission to skip pre-production testing — the canary catches what staging structurally cannot, and unit tests catch in seconds what a canary needs 30 minutes and real users to find. The practices stack; they do not substitute.
- Emit the running version as a label on the Chapter 5 RED metrics from day one — every canary comparison, annotation, and "what changed" query keys off it, and retrofitting it mid-incident is too late.
- Gate every canary on two checks together: the SLO's fast-window burn rate and a canary-vs-stable RED comparison — and size the observation window in requests, thousands rather than hundreds, before promoting.
- Fire a Grafana annotation from the deploy pipeline for every release to every environment — it is one API call, and it converts the deploy-versus-symptom timeline from reconstructed to visible.
- Register every feature flag in telemetry when it is created, and delete it from code within 30 days of full rollout — flags are config-shaped deploys and deserve the same observability and the same cleanup discipline.
Knowledge Check
What can production observe that staging structurally cannot?
- Higher hardware specifications and faster disks
- The real traffic mix, data distributions, and dependency behavior
- Newer versions of Prometheus and Grafana
- Larger log volumes than any load generator in staging could ever hope to produce
Why is the canary compared against the concurrent stable version rather than a fixed latency threshold?
- The stable version is a control group that absorbs system-wide degradation
- PromQL simply cannot express a comparison of one series against a fixed constant value
- Fixed thresholds leak SLO internals to the delivery tooling
- The stable version produces less data and is cheaper to query
Rollback automation should key on SLO burn rate, not on any firing alert. What failure does this prevent?
- Prometheus overloads from evaluating far too many alert rules within a single scrape interval
- Slow rollbacks, since burn rates evaluate faster than alerts
- False rollbacks from unrelated noisy alerts, which destroy trust in the automation
- Alertmanager routing pages to the wrong on-call rotation
What do deploy annotations buy during an incident?
- Automatic reversion of whichever release first caused the symptom
- A visible deploy-versus-symptom timeline on every dashboard
- A line-by-line diff of the code that shipped
- Suppression of alerts for the duration of the rollout
You got correct