SRE Principles on GCP
Site Reliability Engineering is the discipline Google invented to run its own services and then published in two books that became the field's reference. SRE is not a job title — it is a way of operating production systems where reliability is measured, budgeted, and treated as a feature with cost. This topic translates the core SRE concepts onto the GCP toolbox the previous eleven chapters built up. Cloud Monitoring, Cloud Logging, and Cloud Trace are the instruments; the SRE practices are how an engineer reads and acts on them.
The shortest summary of SRE: 100% reliability is the wrong target. Every percentage point of availability past what users actually need costs disproportionately more to achieve and slows down feature delivery. The right target is the lowest number that still keeps users happy, and the disciplined practice of spending the difference — the error budget — on the work that creates value.
Service Level Indicators (SLIs)
An SLI is a number that measures one specific aspect of service behavior from the user's point of view. The two most common shapes are availability (proportion of requests that succeeded) and latency (proportion of requests served within a threshold). Saturation, throughput, and correctness join the list for specific workloads. The classic mistake is measuring what is easy — CPU utilization, queue depth — rather than what users experience.
SLIs must be measured from a vantage point close to the user. The HTTP 200 rate at the load balancer is closer to truth than the 200 rate inside the application. Cloud Monitoring automatically captures request counts and latency histograms for Cloud Load Balancing, Cloud Run, GKE, and most managed services — these are the SLI sources the platform gives you for free.
A good SLI is binary and clearly defined: was this request a success or a failure, was its latency below 200ms or not. Aggregating ambiguous outcomes into percentages obscures what is actually broken. If two engineers cannot agree on whether a given request was "good", the SLI definition is wrong.
Service Level Objectives (SLOs)
An SLO is a target for an SLI over a window of time. "99.9% of HTTP requests succeed over a rolling 28-day window" is an SLO. The number is chosen by the product owner and the engineering team together; it expresses the commitment to users and bounds the engineering investment.
Pick the smallest SLO that keeps users happy. 99.9% (three nines) is a common target for user-facing services; it permits about 43 minutes of downtime per month. 99.99% is dramatically more expensive — multi-region active-active, careful release controls, mature on-call — and is rarely justified for anything but payments, login, or core infrastructure. Internal-only services often run on 99.5% or even 99% with no user complaint.
GCP has first-class support for SLOs in Cloud Monitoring. Define a "Service" object, attach SLIs to it, declare SLO targets, and the platform tracks them with rolling-window dashboards. The configuration is portable across services because the SLI definitions live in the platform, not in the application.
Error Budgets
If the SLO is 99.9%, the error budget is the remaining 0.1% — the amount of failure the team is allowed to spend before reliability work takes priority over feature work. Error budgets turn reliability from a vague aspiration into a quantity you can spend, save, and run out of. This is the single most important concept in SRE.
The discipline is to make policies in advance. If the error budget for the month is consumed, all feature releases pause and the team works exclusively on reliability — fixing the proximate causes of the burn and the structural causes that made it possible. If the budget is healthy, the team is encouraged to ship faster, take more risk, do larger rollouts. The dial of release velocity is moved by the error budget, not by manager preference.
Error budgets put reliability work and feature work on the same axis. The team does not have to argue "should we fix this?" — the question becomes "do we have the budget to defer it, or not?". Engineering managers and product owners look at the same number. Conflicts about priorities get resolved without escalation.
Burn-Rate Alerting
A naive alert fires when an SLI dips below threshold for some duration. A burn-rate alert measures how fast the error budget is being consumed and fires when the rate would exhaust it before the SLO window closes. This is dramatically better for two reasons: alerts arrive earlier when the burn is fast, and alerts do not fire at all when the burn is slow enough to be absorbed.
The standard pattern is the multi-window multi-burn-rate alert. A "fast burn" alert with a short window (e.g., 1-hour and 5-minute windows, burn rate > 14x) catches outages immediately. A "slow burn" alert with a long window (e.g., 6-hour and 30-minute windows, burn rate > 6x) catches degraded conditions that would not trip a threshold alert. Cloud Monitoring implements the pattern natively — SLOs come with burn-rate alerts as the default.
The output of good alerting is pages that humans should respond to, and silence everywhere else. Page fatigue is the death of an on-call rotation. Every page that fires and turns out to be self-resolving was the alert's fault, not the engineer's; tune the alert.
Toil Reduction
Toil is operational work that is manual, repetitive, automatable, lacks enduring value, and scales linearly with service growth. Restarting a stuck job by hand, copy-pasting a runbook command, manually rotating a credential — all toil. SRE limits toil to a fraction of an engineer's time (Google's target is under 50%) and treats the surplus as a signal that automation work is overdue.
The cure for toil is engineering. A stuck-job restart becomes a health check and an automatic restart; a runbook becomes a script; manual rotation becomes a scheduled rotation in Secret Manager. Cloud Scheduler, Cloud Workflows, and Cloud Build are the GCP-native tools for converting toil into code.
Toil that cannot yet be automated should still be measured. A "toil ticket" per recurring manual operation tracks the cost; when the cost of the toil exceeds the cost of automating it, the engineering work moves up the priority queue. Without measurement, toil is invisible and grows forever.
Blameless Postmortems
Every significant incident gets a written postmortem. The postmortem describes what happened, how it was detected, how it was mitigated, why it happened, and what will change so it cannot happen the same way again. The defining word is blameless: the document names actions, not people, and treats the human in the incident as the second victim of the system that allowed the mistake.
Blamelessness is not politeness. It is the technical requirement that the postmortem identify the systemic root cause. "An engineer ran the wrong command" is not a root cause; "the command line accepts a destructive flag without confirmation, and the production environment is not visually distinguished from staging" is. The action items target the system, because the next engineer will make the same kind of mistake unless the system stops allowing it.
Postmortem action items become tracked work, not aspirational lists. Owners and due dates are required; review of completion is part of the next postmortem cycle. A team that writes excellent postmortems and then forgets about the action items learns nothing.
Production Readiness Reviews
Before a service goes to production, it should pass a production readiness review (PRR) — a checklist signed by an SRE that confirms the service is operable, observable, and recoverable. The PRR is what prevents the recurring pattern where engineering ships a service and operations finds out about it during the first incident.
A reasonable GCP-specific PRR covers: SLIs defined and dashboards visible in Cloud Monitoring; SLOs documented and burn-rate alerts wired to the on-call rotation; structured logs flowing to Cloud Logging with a sane retention policy; Cloud Trace enabled for any service that calls others; rollback procedure documented and tested through Cloud Deploy; capacity plan with load test evidence; runbook for the top five known failure modes; secrets in Secret Manager, not in code or configs; IAM least-privilege review complete; backup and restore procedure tested end-to-end.
The PRR is owned by SRE, not by the developing team. SRE has the authority to refuse to take on-call for a service that has not passed it. That authority is what makes the checklist real instead of theatrical.
SRE Anti-Patterns
Common failure modes in adopting SRE on GCP:
- SLO theatre — defining an SLO, missing it, and changing the SLO to match reality. The SLO exists to drive behavior; revising it down after a miss erases that signal. Miss it, work down the burn, then earn the right to revise it.
- Aspirational 100% — choosing 99.99% or higher without budgeting the engineering cost. Five-nines availability costs ten times more than four; the team will miss it, the error budget will always be exhausted, the release pipeline will always be paused, and morale will collapse.
- Vanity SLIs — measuring CPU, memory, queue length, error rates by host. These are platform metrics, not service-level indicators. They tell you nothing about whether users are happy.
- Alert per host — alerts on individual instance metrics that fire whenever one Cloud Run revision behaves oddly. Auto-scaling and managed services make per-instance health irrelevant; alert on aggregate user-visible SLIs.
- Blame in postmortems — "Engineer X should have known better." If the system allowed Engineer X to do that, Engineer Y will do it next month. Name the system, not the engineer.
- Postmortems with no follow-up — writing a beautiful document, filing it, never doing the action items. The postmortem is a checkpoint, not the work itself.
- SRE as ops outsource — treating SRE as the team that gets paged so the dev team does not have to. Dev teams own their services in production; SRE provides leverage, not labor.
- Define SLIs from the user's vantage point at the load balancer or API gateway, not from inside the service. The platform gives you these signals for free in Cloud Monitoring.
- Pick the smallest SLO that keeps users happy. The cost of each extra nine is roughly an order of magnitude in engineering effort.
- Wire multi-window multi-burn-rate alerts to the on-call rotation. Cloud Monitoring SLOs ship with this pattern as the default — use it.
- Make the error budget the public scoreboard. Both the engineering team and the product owner look at the same chart and make the same decision.
- Measure toil. Anything done by hand more than twice is a candidate for automation; the budget for that work is funded by the time saved.
- Write blameless postmortems that name systems, not engineers. Track action items as ordinary work with owners and due dates.
- Require a production readiness review before a service starts taking real traffic. SRE has the authority to say no until it passes.
Knowledge Check
A team is defining the availability SLI for a web service. Which measurement is most aligned with SRE principles?
- Average CPU utilization across all serving instances, sampled over a rolling 5-minute window
- The number of process restarts per day, summed across every instance in the serving fleet
- Proportion of HTTP requests at the load balancer returning a successful status over a rolling window — measured close to the user's vantage point
- Average latency reported by the internal application middleware just before the response payload is serialized and sent back to the load balancer
A product owner asks SRE to commit to 99.999% availability for a new internal admin tool used by 50 employees. What is the right response?
- Agree to the 99.999% target straight away, since five-nines has become the modern industry baseline that every new service, whether internal or external facing, is now generally expected to meet from its first day
- Refuse the request outright, on the grounds that five nines is simply impossible to reach on GCP
- Push back: each additional nine costs roughly an order of magnitude more engineering. For an internal tool with 50 users, 99% or 99.5% is almost certainly enough. Pick the smallest target that keeps users happy.
- Commit to the full 99.999% target but only during business hours, treating all off-hours time as out-of-scope
The error budget for the current month is 70% consumed at week two. What is the correct interpretation?
- The SLO is clearly too tight and should be loosened now so the team recovers the spent budget
- Reliability work should take priority over new feature work; deploy velocity should slow until the burn rate normalizes, and the team should fix the proximate and structural causes that drove the burn
- Nothing at all needs to change; the error budget resets cleanly at the start of next month anyway, so the team can simply keep shipping all new feature work at full deploy velocity exactly as it did before
- The underlying SLI definition is wrong; redefine it so that the monthly error budget reads healthier than it currently does
A team is choosing between a simple threshold alert ("fire if error rate > 1% for 5 minutes") and a multi-window multi-burn-rate alert. Why is the burn-rate alert better?
- Burn-rate alerts consume noticeably less storage in Cloud Monitoring than equivalent threshold alerts do
- Burn-rate alerts fire earlier on fast burns (the budget would be exhausted before the window closes) and do not fire at all on slow burns the SLO can absorb — better signal-to-noise than a single threshold
- Plain threshold alerts have been formally deprecated by Cloud Monitoring, so multi-window multi-burn-rate alerting is now the only alerting pattern the platform still officially supports for any new alerting policies
- Burn-rate alerts are designed to page only during normal business hours, which is what reduces on-call fatigue
A postmortem concludes with "Engineer A typed the wrong command into the production CLI." What is the most important thing to add?
- A note recording that Engineer A has been formally reprimanded by management for the production mistake and warned in writing that any repeat of this error will directly affect both their next scheduled performance review and their annual bonus
- A mandatory retraining requirement assigned individually to Engineer A as a follow-up
- A systemic action item: the CLI should require a confirmation step for destructive operations in production, and the production environment should be visually distinguished from staging so the next engineer cannot make the same mistake
- A note recording that Engineer A has been removed from the on-call rotation
A service is about to launch. Which of the following is a non-negotiable production readiness checkpoint?
- The service runs on Cloud Run as its compute platform rather than on a GKE cluster
- The service is implemented in the Go programming language rather than in Python
- SLIs are defined and visible in Cloud Monitoring, burn-rate alerts page the on-call rotation, structured logs flow to Cloud Logging, rollback is tested through Cloud Deploy, secrets live in Secret Manager, and the runbook documents the top failure modes
- The service has been successfully deployed to the staging environment at least once before launch and a quick manual smoke test confirmed the home page returned a 200 response that day, so the team is fully confident it will behave the same way in production
A team estimates that one engineer spends 60% of their week on manual operational work (restarts, manual rotations, runbook execution). What is the SRE-aligned response?
- Hire a second engineer to absorb the manual operational load alongside the first one
- Accept that toil is simply the unavoidable cost of running production systems and move on
- Treat the 60% as a signal that automation is overdue; measure each toil item, and fund engineering work to convert it into Cloud Scheduler, Cloud Workflows, or Cloud Build automation, prioritized by the cost of the toil
- Reduce the service's SLO target so that the team is left with more slack in the schedule to keep doing the manual restarts, manual rotations, and runbook execution operational work that it already handles every single week
You got correct