Five services for moving operational data into the analytical world. From low-level pipeline engines to enterprise BI dashboards.
5 services
The decisions here are about where a workload should run and which paradigm fits the team. Dataflow versus Dataproc is a question of unified-model versus Spark-native. Composer versus Workflows is not old versus new — Composer is managed Apache Airflow for heavy, dependency-rich data orchestration, while Workflows does lightweight Google Cloud-native orchestration of services and APIs; they solve different problems. Looker versus Data Studio is, despite the name, two different products. Datastream replaces the periodic-dump habit with continuous change capture. Picking right matters more than mastering any one service.
From Operational Data to Insight
SourcesDatabases, apps, event streams
→
Dataflow / Dataprocprocess — Beam or Spark
→
BigQueryWarehouse & query
→
Looker / Data StudioModel & visualize
Source databasesCDC path
→
Datastreamstreams inserts / updates / deletes
→
BigQuerynear real time
Two ways in: batch/stream processing (Dataflow or Dataproc) and change-data capture (Datastream) — both land in BigQuery for analysis. Datastream replaces nightly dumps with a continuous stream of inserts, updates, and deletes.
Dataflow / Dataprocprocess — Beam or Spark
BigQueryWarehouse & query
Looker / Data StudioModel & visualize
Datastreamstreams inserts / updates / deletes