Dataproc
Dataproc is the managed Hadoop ecosystem on Google Cloud — Spark, Hadoop, Hive, Presto, Flink, and their friends, running on configurable clusters Google operates. (In 2026 Google folded the whole offering under the umbrella name Managed Service for Apache Spark; the APIs, CLI, and IAM roles still say "dataproc", so this page keeps the classic name.) The reason to pick Dataproc is almost always team legacy: an existing portfolio of Spark jobs, Hive queries, or Hadoop tooling that would be expensive to rewrite. Without that legacy, the answer for new pipelines is usually Dataflow or BigQuery.
The most consequential modern shift in Dataproc is serverless Spark — now branded Google Cloud Serverless for Apache Spark (formerly Dataproc Serverless): Spark batch jobs without cluster management at all. For batch Spark workloads in 2026, Serverless is the right default — pay per execution, no cluster to size or shut down, comparable performance with far less operational surface.
The Managed Hadoop Stack
Dataproc ships pre-configured cluster images with the Hadoop ecosystem already installed and tuned for GCP: HDFS (or Cloud Storage as the durable layer), Spark (the workhorse), Hive (SQL on HDFS), Tez and MapReduce (older execution engines), Presto/Trino (interactive SQL), and Flink (streaming). A new cluster is up in 90 seconds, configured, ready. The escape hatch from running your own Hadoop bare metal — but only worth the escape if you actually need the stack.
Ephemeral vs Long-Running Clusters
Ephemeral clusters are created for one job, run it, then are deleted. The right pattern for almost all batch workloads: no cluster sitting idle between runs, no patching of OS, no drift over time. Each job gets a fresh tuned cluster. Long-running clusters stay up — appropriate for interactive use (notebooks, Hive SQL exploration, dashboard backends) where startup latency matters and consistent state is useful. For batch, ephemeral wins on cost and operational simplicity; the only argument for long-running is interactive workload.
Dataproc Serverless for Spark
Serverless removes the cluster from the picture entirely. You submit a Spark batch job; Dataproc allocates and runs workers; you pay for the duration of execution; nothing lingers afterward. The trade-off versus traditional Dataproc clusters: less control over node sizing, no custom init actions, only Spark workloads (no Hive, Presto, custom Hadoop). For the common case of submitting a Spark job and getting results, Serverless is dramatically simpler. Reserve traditional clusters for custom Hadoop ecosystems or interactive workloads.
Cluster Configuration and Autoscaling
Traditional Dataproc clusters have three node types: master (cluster coordinator, one or three for HA), workers (primary work executors, run YARN node managers), and secondary workers (no HDFS storage, often preemptible). Configure an autoscaling policy to adjust worker count based on YARN backlog. Preemptible secondary workers are dramatically cheaper but can be reclaimed with 30-second notice — use them only for restartable work where job failure on reclamation is acceptable.
Dataproc vs Dataflow
Restated from the Dataflow chapter for the inverse perspective. The choice is rarely about the workload's technical needs (both can do most batch and streaming work). It is about the team's existing skill and codebase.
- Team writes Spark jobs daily, has Hive queries in production, runs MapReduce → Dataproc, often Dataproc Serverless.
- Team is new to data processing, no Spark expertise yet, building fresh pipelines → Dataflow. The unified model is easier to learn than Spark plus the Hadoop ecosystem.
- Streaming requirements with windowing and watermarks → Dataflow. Dataproc streams via Spark Structured Streaming or Flink, but Dataflow's Beam model is more polished for streaming semantics.
- Long-running cluster for batch workloads that could be ephemeral. The cluster sits idle between runs, paying for capacity nobody uses.
- Traditional Dataproc cluster for a plain Spark batch job that Dataproc Serverless would run with no cluster overhead at all.
- Custom init actions for setup work that the Dataproc image catalog already provides. Slow startup, brittle clusters, drift over time.
- No autoscaling on bursty workloads. The cluster is sized for peak (waste at low load) or for average (poor performance at peak).
- Choosing Dataproc when the team has no Spark or Hadoop expertise. Learning Spark to use Dataproc is more cost than just learning Dataflow or BigQuery first.
- Preemptible secondary workers in the core data path of a critical batch job. One reclamation kills the run; retries hit the same problem.
- Dataproc Serverless as the default for batch Spark workloads. Reserve traditional clusters for Hive, Presto, custom Hadoop, or interactive use.
- Ephemeral clusters for batch jobs. Long-running only for interactive notebooks and dashboard backends.
- Autoscaling policies tied to YARN backlog with sensible max-worker caps.
- Preemptible secondaries only for restartable work, never for the critical path of a job that must complete.
- Dataproc image versions over custom init actions wherever possible. Lock the image version per job for reproducibility.
- Cloud Storage as the durable storage layer (Hadoop FS connector), not on-cluster HDFS. Cluster dies, data lives on.
Knowledge Check
When is Dataproc Serverless for Spark the right default over a traditional Dataproc cluster?
- When the workload runs on Hive or Presto rather than Spark and relies on cluster-side Hadoop ecosystem tooling, custom init actions, and a persistent metastore that Serverless does not provide
- For plain Spark batch jobs where you do not need custom init actions, custom Hadoop ecosystems, or interactive cluster state — Serverless removes the cluster entirely
- When the cluster must be sized at exactly 10 nodes with specific machine types and locked-down disk layouts
- When the team needs on-cluster HDFS as the durable storage layer rather than reading from Cloud Storage
Why prefer ephemeral clusters over long-running clusters for batch workloads?
- Long-running clusters cannot run Spark batch jobs at all; they are restricted to interactive notebook and shell use only
- Ephemeral clusters do not sit idle between runs, do not accumulate drift, and let each job have a fresh tuned cluster with no patching overhead
- Ephemeral clusters offer measurably faster startup times to first job than long-running clusters do
- Long-running clusters are limited to a fixed maximum node count set at creation time, while ephemeral clusters can scale freely without any such cap
When is choosing Dataproc the right call over Dataflow?
- For all new streaming pipelines that need event-time windowing, watermarks, and late-data handling out of the box
- When the team has an existing Spark, Hive, or Hadoop codebase, and migrating to a different framework would require a substantial rewrite
- When you need a fully serverless execution model with no cluster of any kind to provision, size, patch, or manage
- When the workload is small enough to run comfortably on a single VM but still needs full cluster autoscaling for occasional bursty traffic spikes
What is the risk of using preemptible secondary workers in the core data path of a critical batch job?
- Preemptible workers are limited to compute-only operations and cannot read inputs directly from Cloud Storage buckets
- One reclamation can kill the running job; if the workload is not designed to restart from the reclaim point, the entire run fails
- Preemptible workers run a deprecated YARN version that is incompatible with the current Spark and Hadoop releases installed on the cluster
- Preemptible workers do not count toward autoscaling decisions, leading to chronic over-provisioning of primary workers
Why use Cloud Storage as the durable storage layer instead of on-cluster HDFS in Dataproc?
- HDFS is not supported on Dataproc clusters at all; only the Cloud Storage connector works as a backing store for any Spark or Hive job you submit
- Cloud Storage outlives the cluster — ephemeral clusters can come and go without losing data, and multiple clusters can share the same data set
- HDFS on Dataproc is billed at a noticeably higher per-gigabyte rate than the equivalent amount of Cloud Storage capacity would cost
- Cloud Storage offers stronger data locality than on-cluster HDFS for repeated cluster-local reads
You got correct