Architecture Case Studies

Five scenarios. Each one has a bad architecture that a real team shipped, a good architecture that the same team eventually arrived at, and the tradeoffs that distinguish them. The synthesis topic for the course: every concept from chapters 1 through 11 and every framework from topics 52 through 56 lands in one of these five stories. Read them as patterns rather than recipes — the products will change; the shape of the decisions is durable.

The format is deliberate. Architecture is rarely about choosing between a brilliant design and a terrible one; it is about choosing between a reasonable-looking design and one that turns out to be slightly better after another year of operating it. The "bad" architectures below are not strawmen — they are designs that competent engineers built and lived with for a while before learning the lessons that made the "good" version inevitable.

Case 1 — E-Commerce Platform

Context. Mid-sized online retailer, 200k daily active users, peaks at 5x average during sales, multi-region traffic, mix of catalog browsing, checkout, and order management. Existing team is comfortable with relational databases; product team wants real-time inventory and personalized recommendations.

Bad architecture. A single Cloud SQL MySQL instance hosts everything — products, inventory, orders, sessions, recommendations cache. The web tier is one large GKE cluster running a monolithic Spring application. Background work — emails, recommendation training, inventory sync to the warehouse — runs as cron jobs inside the same monolith. During the November sale the database CPU pegs at 100%, the cluster autoscales until quota runs out, and checkout fails for two hours. Postmortem reveals that recommendation queries were taking 80% of database capacity at peak.

Good architecture. Separate concerns by data shape and rate of change. The product catalog lives in Firestore with Cloud CDN in front of Cloud Load Balancing serving cached product pages from the edge. The checkout and orders service runs on Cloud Run against Cloud SQL sized for transactional workload only. Inventory updates flow through Pub/Sub; the saga pattern (orchestrated in Cloud Workflows) coordinates reserve-inventory, charge-payment, create-shipment with compensating actions. Recommendations are a separate Bigtable-backed service trained nightly by Vertex AI (rebranded Gemini Enterprise Agent Platform in 2026) against analytics in BigQuery. Cloud Armor rate-limits bot traffic at the edge.

Tradeoffs. The good architecture is six services instead of one. Operational surface is larger; the team must own observability, deploys, and SLOs per service. The payoff: each service scales independently, the recommendation workload cannot starve checkout, and the November sale survives autoscale on Cloud Run without touching SQL. The complexity is real but bounded by managed services that own most of the operations.

Case 2 — IoT Telemetry Pipeline

Context. Manufacturer of connected industrial sensors. 50,000 devices in the field, each emitting one record every 10 seconds. Customers need a real-time dashboard, alerting on anomalies, and historical reports for compliance. Data must be retained for seven years.

Bad architecture. Devices POST JSON over HTTPS to a Cloud Run service that inserts each row into Cloud SQL. A separate cron job aggregates the last 24 hours into a reporting table every hour. The dashboard queries the reporting table; the alerting service queries the raw table every minute. At 5,000 inserts per second the Cloud SQL instance falls behind; the reporting job times out; the alerting service runs slow queries that further degrade the database. Devices retry on failure, amplifying load. Within a quarter, customers are complaining about stale dashboards and missed alerts.

Good architecture. Lambda architecture, GCP-native. Devices publish to Pub/Sub (the right ingestion shape for high-fanout high-volume telemetry — millions of messages per second with no per-device tuning). A Dataflow streaming job consumes the topic, validates and enriches records, writes raw events to Bigtable for sub-second dashboard reads, and computes anomaly signals it publishes back to Pub/Sub. The alerting service subscribes to the anomaly topic — no polling. A second Dataflow batch job consumes the same source archived in Cloud Storage and lands aggregates in BigQuery for the compliance reports. Storage lifecycle rules move raw archives to Coldline after 90 days and Archive after a year. IAM Workload Identity Federation authenticates devices without long-lived keys.

Tradeoffs. The pipeline has more components than a simple database insert, but every component is managed and individually scales. Bigtable handles the write rate effortlessly; Dataflow handles the streaming compute; BigQuery handles the long-horizon analytics. Cost per record drops by an order of magnitude despite using more services because each service is sized to what it does. Eventual consistency between dashboard and report is acceptable for this workload.

Case 3 — ML Model Serving

Context. SaaS product offering real-time predictions to enterprise customers. Multiple models in flight, latency budget of 200ms p99, request volume varies from a hundred per minute on small accounts to thousands per second on large accounts. A new model release happens roughly weekly.

Bad architecture. A team of ML engineers ships a Flask app that loads the model into memory at startup and serves predictions. The app runs on three permanent Compute Engine VMs sized for peak. Model retraining is a Jupyter notebook that runs on a researcher's laptop; the resulting .pkl file is copied to the production VMs via SSH. There is no rollback path; the previous model is overwritten. The first time the new model has a regression in production, recovering takes half a day of finding the previous .pkl in someone's email. Quality metrics are not measured in production at all.

Good architecture. Build the lifecycle around Vertex AI. Training runs as Vertex AI Custom Training jobs on managed compute; the resulting models register to Vertex AI Model Registry with versioned IDs. Online prediction uses Vertex AI Endpoints with traffic splitting — the new model gets 5% traffic initially, then 25%, then 100%, with rollback by reverting the split ratio. Cloud Monitoring tracks per-version latency and prediction quality (drift detection in Vertex AI). The application calls the endpoint through IAM-authenticated service accounts. For batch scoring workloads, the same registered models run as Vertex AI Batch Prediction against data in BigQuery. Cloud Build automates the training-to-registry-to-endpoint pipeline; every model release leaves an audit trail.

Tradeoffs. The team gives up some control over the serving container in exchange for traffic splitting, autoscaling, and quality monitoring as managed features. Cost per prediction at low volume is higher than the cheap-VM baseline, but the operational benefit — production-quality release, drift visibility, no rollback drama — is worth it once the model matters to revenue.

Case 4 — Multi-Tenant SaaS

Context. B2B SaaS application serving 500 enterprise tenants of widely varying sizes. Largest tenants demand strict data isolation; smallest tenants are cost-sensitive. New tenants onboard weekly; some churn quarterly. Audit-ready data residency requirements for European tenants.

Bad architecture. Single shared database with a tenant_id column on every table. One large Cloud Run service handles all tenants. Authentication is a homegrown JWT system. A noisy small tenant once consumed enough database connections to slow the whole platform; an inadvertent missing tenant_id filter once exposed cross-tenant data in a report query (the bug was found by a customer). EU tenants are nervous because all data sits in us-central1 with no clear story for residency.

Good architecture. Tiered isolation matched to tenant tier. The compute layer is shared Cloud Run services with tenant context propagated through authenticated requests; IAP + Identity Platform handles end-user auth federated to each enterprise's IdP. Data isolation has three tiers: small tenants share a logical Cloud SQL database with row-level security and per-tenant schema; mid-tier tenants get a dedicated database within a shared instance; top-tier and EU tenants get a dedicated Cloud SQL instance in their preferred region, provisioned by a tenant-onboarding workflow in Cloud Workflows. Per-tenant Cloud KMS CMEK keys give individual tenants control over their own encryption. Cloud Load Balancing routes EU traffic to europe-west1 and US traffic to us-central1 based on the tenant's home region. Bulkhead isolation at the Cloud Run service level means one tenant's load spike cannot starve others.

Tradeoffs. Three isolation tiers mean three operational shapes to maintain. The good architecture is more complex to provision, but it lets the business price-differentiate (small tenants on shared infrastructure at low cost, top tenants on dedicated infrastructure at premium price). Cross-tenant data leakage is structurally prevented for the top tier because the data lives in a separate database. Data residency is no longer a manual reassurance — it is enforced at the routing layer.

Case 5 — Regulated Fintech Workload

Context. Fintech startup processing payment authorizations and ledger updates. PCI DSS scope, SOC 2 Type II in flight, regulator expects defensible audit trails. Latency budget on the authorization path is 150ms p99. Ledger must be immutable for seven years.

Bad architecture. Authorization service on Cloud Run writing to Cloud SQL. The ledger is a table in the same database with an UPDATE path for corrections (which means it is mutable). Secrets are environment variables baked into the container image. Logs go to Cloud Logging with default retention. The PCI assessor asks for evidence of access reviews; the team scrambles to assemble a CSV from manually-recorded data. The first attempted post-incident reconstruction reveals that audit logs do not cover data reads, so it is impossible to know who saw which transactions during the incident window.

Good architecture. The authorization service runs on Cloud Run sitting inside a VPC Service Controls perimeter; Cloud Armor filters edge traffic; IAP gates administrative endpoints. The transactional database is Cloud Spanner for global ACID at the latency budget; CMEK keys in Cloud KMS wrap encryption. The ledger is append-only — every change is a new event published to Pub/Sub and persisted in immutable Cloud Storage with bucket lock and retention policies. Secrets live in Secret Manager with scheduled rotation notifications driving a rotation service; workload identity replaces every key file. Admin Activity and Data Access audit logs route to BigQuery with a seven-year retention table. Security Command Center Premium continuously evaluates posture; findings have a weekly triage cycle. Privileged Access Manager grants engineering escalations with auto-expiry. The PCI assessor's evidence requests are answered by SQL queries against BigQuery rather than by manual CSV assembly.

Tradeoffs. Spanner is materially more expensive than Cloud SQL, the audit-log export to BigQuery adds storage cost, and the operational discipline of immutable ledger plus dedicated KMS keys is more work than the casual setup. The payoff is compliance posture that is verifiable rather than aspirational, latency that fits the authorization budget at global scale, and an incident-investigation surface that has actual answers.

Cross-Case Patterns

Five different domains, the same handful of patterns appearing in each:

• Separate concerns by data shape and rate of change. Every "good" architecture replaced one large shared resource with smaller resources matched to the workload. The corollary is that the right service for one part of the system is rarely the right service for the next part.
• Asynchronous between services, synchronous only when the caller cannot proceed. Pub/Sub is the workhorse of the chapter — it appeared in every "good" architecture and in none of the "bad" ones.
• Managed services where you can. Vertex AI for ML, Dataflow for streaming compute, Spanner for global ACID, BigQuery for analytics. The team owns the business logic; Google owns the operations of the substrate.
• Identity, not network, as the boundary. Workload Identity Federation, per-service service accounts, IAP-gated admin paths. The "bad" architectures all had some implicit trust in network position; the "good" ones did not.
• Observability and audit are wired in from day one. Cloud Logging, Cloud Monitoring, SLOs, error budgets, audit log exports to BigQuery, Security Command Center triage. The "bad" architectures discovered the need during incidents; the "good" architectures already had the data when the incident arrived.
• Cost is part of the architecture. CUDs on steady-state compute, Cloud Storage lifecycle for cold data, BigQuery partitioning, scheduled shutdown of non-prod, label everything. The "good" architectures cost less to run despite using more services because each service is sized to what it does.

The Closing Synthesis

Architecture on GCP comes down to a small number of disciplines exercised in every project. Pick services by walking the decision trees from Topic 52, not by reaching for what is familiar. Compose them through the patterns of Topic 53, preferring async messages, idempotency, and managed orchestration. Operate the system through SRE principles from Topic 54 — SLOs, error budgets, blameless postmortems, toil reduction. Bake security from Topic 55 into the defaults — zero trust, least privilege, secrets management, classification. Engineer cost continuously per Topic 56 — labels, CUDs, lifecycle, budget alerts. None of these are exotic; all of them require sustained effort more than any individual insight.

The case studies show that the gap between the "bad" and "good" architectures is rarely a moment of brilliant insight. It is a year of small decisions made with the right disciplines in place, in a team that revisits choices when the data changes. The fifty-one services in this course are bricks; the disciplines in this chapter are how to build with them.