Service 28

Dataflow

AnalyticsPipelines

Dataflow is the managed runner for Apache Beam, a unified model for batch and streaming data processing. The same Beam code runs as a batch pipeline (bounded input, finishes when input is exhausted) or as a streaming pipeline (unbounded input, runs continuously with windowing). One mental model, two execution shapes, no rewrite to switch.

Dataflow's distinguishing trait among GCP analytics services: it is serverless. No clusters to provision, autoscaling out of the box, per-second worker billing. The main fork in this chapter is Dataflow versus Dataproc — Dataproc fits teams already deep in Spark, Dataflow fits unified-model pipelines and teams without Spark legacy.

Apache Beam — Unified Model

Beam is an open-source framework owned by the Apache Foundation, originally derived from Google's internal Flume system. A Beam pipeline expresses transformations on PCollection objects — collections that may be bounded (batch) or unbounded (streaming). Pipelines are written in Java, Python, or Go SDKs and run on a Beam runner; Dataflow is one runner among several (others include Apache Flink, Apache Spark, and a local DirectRunner). The Beam abstraction lets you switch runners without rewriting transformations.

Batch vs Streaming Pipelines

Batchbounded input

Finite dataset, then done. Workers spin up, process the whole input, and scale back to zero. Cost is per job, not continuous.

Streamingunbounded input

Never-ending event stream. Workers run continuously at steady state, processing records as they arrive. You pay for always-on capacity.

A batch pipeline reads a bounded source (a file in Cloud Storage, a BigQuery query result), runs transformations, writes the output, and exits. Dataflow scales workers up during the run and down to zero at the end. The right shape for daily ETL, weekly reports, model training data prep.

A streaming pipeline reads an unbounded source (Pub/Sub, an unending Kafka topic) and runs continuously. Workers stay up; the pipeline keeps consuming, transforming, and writing as events arrive. The right shape for real-time enrichment, anomaly detection, continuously-updated aggregates.

Windowing

Tumblingfixed

Non-overlapping, equal-length windows (e.g. every 1 min). Each event lands in exactly one. "Count per minute."

Slidingoverlapping

Fixed-length windows that overlap (e.g. 5 min, every 1 min). Each event lands in several. "5-minute moving average."

Sessiongap-based

Variable-length windows that close after a gap of inactivity. Groups bursts of activity. "One user session."

Streaming aggregations need windows — bounded groups of events to aggregate over. Three common shapes:

Tumbling (fixed) windows — non-overlapping intervals of equal size. "Count events per minute" uses tumbling windows of one minute.
Sliding windows — overlapping intervals at a fixed cadence. "Moving 5-minute average computed every minute" uses sliding windows of 5 minutes with a 1-minute slide.
Session windows — grouped by gaps in activity. "All clicks within 30 minutes of each other are one session" uses session windows with a 30-minute gap timeout. The right choice for irregular per-entity activity.

Watermarks and Late Data

Streaming data does not arrive in order. A watermark is Beam's estimate of "all events with timestamps up to here have probably been seen." When the watermark passes the end of a window, the pipeline closes the window and emits its result. Events that arrive after the watermark passes their window are late data. Configure allowed_lateness to accept late events and re-emit updated aggregates, or drop them silently. Without thinking about watermarks, streaming pipelines silently lose data or produce premature results.

Templates: Classic vs Flex

Dataflow templates are pre-packaged pipelines you can launch with parameters — useful for non-engineer triggers (e.g., a Cloud Function kicking off a Dataflow job). Classic templates are tied to a specific pipeline version baked at template creation time. Flex templates wrap the pipeline as a container image, support parameterization at runtime, and are the modern default. Beware: putting pipelines behind a template instead of in version-controlled code loses git history and makes review awkward. Use templates when the surface that consumes them genuinely needs declarative deploy; otherwise keep the pipeline in code.

Autoscaling and Dataflow Prime

Standard Dataflow autoscales worker count based on backlog and CPU. Dataflow Prime goes further: it auto-tunes machine types per step, applies vertical scaling, and uses a separate workload distribution model. Prime usually wins for variable workloads where standard autoscaling thrashes; it costs more per worker-hour but typically reduces total cost for those workloads. For steady-state batch jobs of known size, standard Dataflow remains the right choice.

Dataflow vs Dataproc

The biggest decision in this chapter. Same data, very different engines.

Dataflow vs Dataproc

Dataflow — unified batch and streaming via Beam, fully serverless, autoscaling. The right default for new pipelines and teams without Spark legacy. Choose when you want managed pipeline execution and the unified model matters.

Dataproc — managed Spark, Hadoop, Hive, Presto on configurable clusters. The right default for teams already running Spark or Hadoop workloads; migration to GCP without rewriting pipelines. Cheaper to operate when the team already knows Spark.

Common Mistakes

Picking Dataflow when the team already has a portfolio of Spark code — Dataproc lets you bring it to GCP without rewriting.
Streaming pipelines without explicit watermark and allowed-lateness configuration. Events drift past the window, results are wrong, no error appears.
Fixed (tumbling) windows on irregular per-entity data. Sparse entities show empty windows; bursty entities overload one window. Session windows are usually the right fit.
Templates as the primary deploy mechanism. The pipeline lives in Cloud Storage instead of git; review and rollback become awkward.
Manual worker count instead of autoscaling. The pipeline either runs out of capacity at peak or wastes money during quiet periods.
Ignoring Dataflow Prime for variable workloads where standard autoscaling thrashes. Prime's total cost is often lower despite the higher per-hour rate.
Streaming pipeline without a backfill plan. Reprocessing historical events requires a parallel batch job; if it was not designed in, retrofitting is painful.

Best Practices

Beam pipelines in version control, not Cloud Storage templates as the source of truth. Templates are an artifact, not the home.
Explicit watermark, trigger, and allowed_lateness on every streaming pipeline. Decide consciously whether late events update or drop.
Session windows for user-level activity, tumbling for regular metrics, sliding for moving averages. Match window shape to data shape.
Autoscaling on by default with a max-workers cap tied to budget.
Dataflow Prime when the workload is variable and standard autoscaling does not keep up.
Idempotent sinks (BigQuery streaming inserts with insertId, Pub/Sub at-least-once consumers) for streaming pipelines.
Backfill capability planned for every streaming pipeline — a parallel batch job that can reprocess from a historical timestamp.

Comparable services AWS Managed Service for Apache Flink + EMR Serverless Azure Stream Analytics + HDInsight

Knowledge Check

What does the Apache Beam unified model give you?

A single SDK that can read from any data source without per-connector configuration or driver setup of any kind
The same pipeline code can run as either a batch (bounded input) or streaming (unbounded input) job — no rewrite to switch shapes
Automatic conversion of existing Spark or Hadoop code into equivalent Beam transformations at job submit time, with no manual porting
Built-in machine learning operators for training and scoring that work without any external libraries

When are session windows the right choice over tumbling windows?

When you need windows that exactly match wall-clock boundaries such as every minute or every hour on the hour, with no overlap
When per-entity activity is irregular and you want windows defined by gaps in that activity rather than fixed time boundaries
When the streaming source delivers events strictly in timestamp order with no late or out-of-order arrivals
When you need overlapping windows for a continuous moving-average computation over the stream

What is a watermark in a Dataflow streaming pipeline?

A monitoring threshold for backlog size that the pipeline raises an alert against whenever the unprocessed queue grows past it
An estimate of "all events with timestamps up to here have probably been seen" — when it passes a window's end, the pipeline closes that window and emits its result
A cryptographic signature attached to each emitted record so that any downstream consumer can independently verify the record was not tampered with anywhere in transit
A throttling mechanism that pauses ingestion from the source whenever downstream sinks slow down or fall behind

When does Dataproc typically beat Dataflow as a choice?

For new streaming pipelines that want unified batch and streaming code written once and run on either input shape
When the team already has a portfolio of Spark or Hadoop code and migrating to a different model would require a rewrite
When autoscaling is unacceptable for the workload and only fixed, manually pinned worker counts are tolerated for the entire run
For all batch workloads regardless of language, framework, or existing codebase shape

What is the principal drawback of using Dataflow templates as the primary deploy mechanism?

Templates run measurably more slowly than direct pipeline launches because each run requires an extra parameter-binding and validation step
The pipeline source of truth lives in Cloud Storage rather than git, which makes review, rollback, and version history awkward
Templates do not support autoscaling at all; the worker count is frozen at template creation time and cannot change
Templates require Dataflow Prime to be enabled and simply cannot run on the standard Dataflow service

You got correct