Cloud Design Patterns
Picking the right service is the first skill. Composing several of them into a working distributed system is the next one. The decision trees from Topic 52 answer "which brick"; this topic answers "which shape of wall". The patterns below are not GCP inventions — they predate the cloud — but each has a canonical realization on GCP that the rest of this course already gave you the pieces for. The job here is to recognize the pattern when a problem shows it, and to map it onto the right services without ceremony.
A pattern is durable; a service is not. Pub/Sub may be renamed in a decade, Cloud Run may absorb Cloud Functions, but the shape of a fan-out subscription tree or a circuit breaker around a flaky downstream is the same problem regardless of vendor. Internalize the shapes and the products fall out.
Microservices on GCP
OrderCloud RunOrderPlaced.InventoryCloud RunShipmentCloud RunEach service owns its own database — never a shared one (a shared DB is one service with two codebases). Services talk asynchronously through Pub/Sub, not direct calls, so one slow service does not stall the others.
The canonical microservices stack on GCP, the one that fits the majority of new projects, is this: Cloud Run for each service, Pub/Sub for asynchronous communication between them, Firestore or Cloud SQL per service for state, IAM service accounts as the identity boundary, Cloud Logging and Cloud Trace for cross-service visibility. Nothing exotic; nothing managed by the team. Each service owns its data; nothing shares a database.
Service boundaries are the hard part, not the wiring. The rule of thumb is to draw boundaries around data ownership and rate of change, not around technical layers. An "order service" that owns the order lifecycle is a boundary; an "auth layer" that wraps every other service is not. When two services share a database, they are one service with two deploy pipelines — the worst of both architectures. When a service can be redeployed without coordinating with any other team, the boundary is right.
Synchronous calls between services should be the exception, not the rule. Each synchronous hop multiplies failure probability and adds latency that compounds. Prefer asynchronous Pub/Sub messages where the business operation tolerates eventual consistency, which is most of the time. Reserve synchronous HTTP for cases where the caller genuinely cannot proceed without the response.
Service mesh — Cloud Service Mesh, the GKE-native managed Istio — solves real problems (mTLS between services, traffic splitting for canary releases, fine-grained authorization between workloads) but adds operational complexity. The honest position: small fleets on Cloud Run with Pub/Sub between them rarely need a mesh. Reach for one only when the count of services and the maturity of the platform team both justify it.
Event-Driven Architecture
An event-driven system reacts to facts that have already happened, instead of being told what to do. "OrderPlaced" is an event; "ShipOrder" is a command. Events are emitted by the service that knows the truth — usually after a state change is committed — and consumed by any number of independent consumers that each derive their own work from it.
On GCP the canonical realization uses Pub/Sub as the event bus, with each consumer subscribing to its own subscription (Pub/Sub fan-out semantics — every subscription sees every message). Topics become a contract; payload schemas should be versioned and additive. Eventarc extends this to GCP-emitted events (audit logs, Cloud Storage object creation, BigQuery job completion) so the same pattern applies to platform events, not just application events.
The shift in thinking that makes event-driven work is treating the event log itself as the source of truth, not the database. The database is a projection of the event stream; if you can rebuild it by replaying events, the architecture is self-recoverable. The cost is that consumers must be idempotent — the same event may be delivered more than once, and the system must converge to the same state either way.
Lambda Architecture
Pub/Subsingle event stream
Speed layerDataflow streaming → Bigtable, low latency, approximate
Batch layerCloud Storage archiveDataflow batchcomplete history
BigQueryaccurate, high latencyLambda architecture is the canonical pattern for systems that need both real-time and historical analytics over the same data. The same event stream flows into two paths: a batch layer that computes accurate but high-latency aggregates over the complete history, and a speed layer that computes approximate, low-latency aggregates over the recent window. A serving layer merges the two so that queries see batch results for the past and speed results for the present.
On GCP the shape is well established: Pub/Sub ingests the event stream. The speed layer is Dataflow running in streaming mode, writing to Bigtable or Memorystore for low-latency lookups. The batch layer is also Dataflow (the unified model is one of Dataflow's main reasons to exist), running in batch mode against the historical archive in Cloud Storage, writing into BigQuery. The serving layer is whatever read API the application exposes, joining the two stores by the timestamp of the watermark.
The modern simplification, when it fits, is "kappa architecture" — a single streaming path that reprocesses historical data by replaying the event stream from the beginning. It is simpler but only works when reprocessing time is acceptable. The choice between lambda and kappa is whether you can tolerate full-history replay; for most production systems with years of data, lambda still wins.
Saga Pattern
…so the saga walks backwards, running each compensating transaction:
When a business operation spans multiple services that each own their own database, you cannot wrap them in a distributed transaction — at least not without paying a high availability cost. The saga pattern is the standard alternative: decompose the operation into a sequence of local transactions, each in one service, where every step has a compensating action that undoes it. If any step fails, the saga walks backwards executing compensations until the system is consistent again.
The classic example is order placement: reserve inventory, charge the payment method, ship the order. If the shipment fails, the compensation is to refund the payment, which compensates by releasing the inventory reservation. Each compensation is an independent local transaction in its own service — no global lock, no two-phase commit.
On GCP there are two orchestration styles. Choreographed sagas use Pub/Sub events — each service reacts to the previous service's event and emits its own. No central coordinator, but the flow is implicit and hard to debug. Orchestrated sagas use Cloud Workflows as the explicit coordinator that calls each service in turn and invokes compensations on failure. Workflows are easier to reason about and observe; choreography scales to more services without a central bottleneck. Most teams should start orchestrated and stay there.
Sagas demand that every step and every compensation is idempotent. Without idempotency, a retry after a partial failure corrupts state. This is the same idempotency requirement events impose — it is the price of building distributed systems that recover automatically.
CQRS and Event Sourcing
CQRS — Command Query Responsibility Segregation — separates the model used to write data from the model used to read it. Writes go through a narrow command surface that mutates the write store; reads go through query-optimized projections built from the same event stream. The two sides can scale, model, and store data completely independently.
Event sourcing is the natural pair: the write store is an append-only log of events, not the current state. Current state is a projection — fold the event stream and you get the current snapshot. The audit trail is automatic because the event log is the audit trail. Time travel is automatic because you can replay to any point.
A GCP realization: writes append events to Pub/Sub after committing them to a write-side Firestore or Cloud SQL ledger. Subscribers update read-side projections — denormalized views in Firestore for low-latency reads, summary tables in BigQuery for analytics, search indexes elsewhere. The read side is eventually consistent with the write side; that latency is the price for scalability and flexibility.
CQRS is not a default. It is appropriate when read and write workloads diverge dramatically (orders of magnitude more reads than writes, or fundamentally different query shapes), or when audit-grade history is a hard requirement. For ordinary CRUD apps with similar read and write volumes, CQRS is over-engineering and adds latency between write-and-read that the team will spend the next year working around.
Strangler Fig
Migrating a monolith to microservices in one cutover is the surest way to ship a broken product. The strangler fig pattern — named for the tree that grows around its host and eventually replaces it — is the way to do it safely. Stand up a routing layer in front of the monolith. Each new feature, and each carved-out slice of existing functionality, lives in a new service behind that router. The router decides per request whether to call the new service or fall through to the monolith. Over time the new services strangle the monolith and it can be retired.
On GCP the router is typically Cloud Load Balancing with URL maps, or Apigee / API Gateway for finer routing rules. New services run on Cloud Run; the monolith stays on whatever compute it already runs on (often Compute Engine or GKE). Data migration usually lags behind code migration — start with read-through proxies and dual writes, finish with a one-way cutover when the new service owns the data.
The discipline that makes strangler fig actually work is committing to never adding new functionality to the monolith. Every new feature lives in a new service, even when it would be three lines in the monolith. Without that rule, the monolith grows faster than the strangler and the migration never finishes.
Resilience Patterns
Distributed systems fail constantly. Resilience patterns are the small, well-understood techniques that prevent a single failure from cascading into an outage.
- Circuit breaker — wrap calls to a flaky dependency. After N consecutive failures, "open" the circuit and fail fast for a cooldown period instead of piling requests on a downed service. After the cooldown, allow one probe; if it succeeds, close the circuit. The pattern is built into most service mesh implementations and most modern HTTP client libraries.
- Retry with exponential backoff and jitter — on transient failure, retry; double the wait between attempts; add randomness so retries from many clients do not stampede the recovering service at the same instant. Pub/Sub, Cloud Tasks, and most managed services implement this themselves; for outbound calls from your own code, use the cloud-client libraries' built-in retry rather than writing your own.
- Idempotency keys — every operation that may be retried must accept a key (caller-generated UUID) and refuse to repeat work for the same key. Cloud Tasks deduplicates by name. Pub/Sub-delivered messages may repeat; consumers must deduplicate by message ID or a domain-level key. Without idempotency, every retry is a potential data corruption.
- Bulkhead isolation — divide resources so that one failure mode cannot consume all of them. A separate thread pool or connection pool for each downstream; a separate Cloud Run service for each tenant when one tenant's traffic must not starve another's. The metaphor is a ship's hull — one flooded compartment does not sink the boat.
- Timeout budgets — every outbound call has a timeout; the timeouts down a call chain must add up to less than the caller's timeout, with headroom for retry. The default of "no timeout" is the worst possible setting: a hung downstream pins a worker forever and propagates the hang upstream.
- Graceful degradation — when a non-essential dependency is down, return a degraded response instead of failing the whole request. The product page loads without the personalization sidebar; the order completes even when the recommendation engine is down. Decide which dependencies are essential at design time, not during the incident.
Composition Anti-Patterns
Recurring shapes of bad distributed-system design:
- Distributed monolith — services that must be deployed together, share a database, and call each other synchronously on every request. All the operational cost of microservices, none of the independence. The fix is harder than it sounds: usually the boundaries were drawn wrong, and the work to redraw them is months.
- Synchronous chain of doom — request walks through five services synchronously. Each service is 99.9% available, the composite is 99.5%, and the p99 latency is the sum, not the max. Replace inner hops with async messages or denormalize the read path.
- Shared database — two services reading and writing the same tables. One team's schema change breaks the other. There is no service boundary here; there is one service with two codebases.
- Event spaghetti — choreographed events emitted and consumed by every service with no documentation of who emits or consumes what. The flow is impossible to trace, debugging an incident means grepping logs across ten repos. The fix is an event catalog and ownership rules, or a switch to orchestrated workflows.
- No idempotency — retry causes duplicate orders, double charges, duplicate emails. Every distributed operation that retries must dedupe by key. This is non-optional, not nice-to-have.
- Cascading retry storm — every layer retries on failure, no backoff, no jitter. A brief downstream blip becomes a thundering herd that prevents recovery. Retries are essential but must be bounded and jittered.
- Draw service boundaries around data ownership and rate of change, not around technical layers. If two services share a database, they are one service.
- Prefer asynchronous messages over synchronous calls. Pub/Sub between services is the GCP-native default; reserve synchronous HTTP for cases where the caller genuinely cannot proceed without the response.
- Treat the event log as the source of truth where the workload tolerates eventual consistency. The database becomes a projection that you can rebuild.
- Make every step idempotent. Every retry, every replay, every duplicate delivery must converge to the same state.
- Use orchestrated sagas (Cloud Workflows) over choreography until the team has the maturity to debug a hundred event handlers across ten services.
- Wrap every outbound call in a timeout. Default timeouts are usually too generous; cascading hangs are worse than failures.
- Test failure modes deliberately. The first time the team learns that the circuit breaker does not actually open should not be during an incident.
Knowledge Check
A new team is starting a microservices architecture. They are designing the canonical GCP stack. Which combination represents the right defaults?
- Cloud Run per service, Pub/Sub for async communication, Firestore or Cloud SQL per service for state, IAM service accounts for identity
- GKE Autopilot for every service with a service mesh, one shared Cloud SQL instance for all state, and a single service account shared across every workload
- Compute Engine managed instance groups with synchronous gRPC between every service on each hop and Memorystore as the single shared cache
- App Engine Standard for every service with one shared Datastore as the single source of truth for all of them
A team is composing an event-driven analytics pipeline that needs both real-time dashboards and accurate batch reports over the full history. Which pattern fits?
- Saga pattern — chain the real-time dashboard and the batch reporting steps as local transactions, each with a compensating action on failure
- Lambda architecture — Dataflow streaming into Bigtable for the real-time path, Dataflow batch over Cloud Storage into BigQuery for history
- CQRS — separate the read and write models for the dashboard service so each scales on its own path
- Strangler fig — stand up a router and slowly replace the existing reporting pipeline service by service
Order placement needs to reserve inventory, charge a payment, and create a shipment — each owned by a different service. The team wants the business transaction to span all three without sharing a database. What is the right pattern?
- Open one distributed transaction spanning all three service databases using a two-phase commit coordinator across them
- Move all three independently owned services onto a single shared Spanner database to obtain global ACID guarantees
- Saga pattern — a sequence of local transactions in each service, each with a compensating action; orchestrate with Cloud Workflows or choreograph with Pub/Sub events
- Synchronous chain of HTTP calls fired from a single coordinator service; on any failure, ignore the error and simply let the user retry the whole order from scratch later
A team is migrating a 10-year-old monolith to microservices. The business cannot stop shipping features during the migration. What is the right approach?
- Stop all feature work for six months and do a single big-bang rewrite of the entire monolith at once
- Add a separate database for every new microservice but keep all of them running behind the unchanged monolith
- Strangler fig — stand up a router in front of the monolith, route new features and carved-out slices to Cloud Run, commit to no new functionality in the monolith
- Wrap the existing monolith in a service mesh for traffic observability and mutual TLS, and then consider the microservices migration effectively complete and shippable
Why must consumers in a Pub/Sub-driven event architecture be idempotent?
- Because Pub/Sub guarantees exactly-once delivery only inside its limited beta tier; the generally available production service stays stuck at at-least-once delivery until that exactly-once feature finally graduates and ships
- Because Pub/Sub delivers at-least-once — duplicates happen on retries, redeliveries after a consumer crash, or rebalance — so the system must converge to the same state no matter how many times a message is processed
- Because Pub/Sub orders every message strictly and globally, and idempotency is the only mechanism that can preserve that ordering downstream
- Because Pub/Sub never retries a delivery on its own, so idempotency is what lets a consumer manually retry the failed deliveries safely
A service makes a critical outbound call that fails intermittently with transient errors. What is the right resilience pattern?
- Retry immediately on every failure without any backoff delay between attempts — the fastest possible path to eventual success
- Open a circuit breaker on the very first failure and never let it close again until an operator manually intervenes
- Retry with exponential backoff and jitter, bounded by a max attempt count; wrap in a circuit breaker to fail fast when the downstream is persistently down; add a per-call timeout so a hung downstream cannot pin a worker
- Replace the request-response call with a synchronous Pub/Sub publish that blocks waiting on the reply, so the message bus itself then transparently handles all every single retry, backoff, and timeout for you automatically
A team is considering CQRS for their new CRUD app. The read and write volumes are similar; queries match the write schema closely. Should they adopt it?
- Yes — CQRS is the modern standard for any production system and should be adopted by default on every new build
- No — CQRS fits when read and write workloads diverge dramatically or audit-grade history is a hard requirement; for ordinary CRUD it is over-engineering that adds eventual-consistency complexity for no benefit
- Yes — physically separating the read and write models always improves database throughput and latency, regardless of how symmetric or how small the actual production read and write workload shape happens to be in practice
- No — CQRS can only be implemented on Cloud Spanner and the team is unwilling to take on its higher running cost
You got correct