Cloud Composer
Cloud Composer is managed Apache Airflow on Google Kubernetes Engine — in 2026 Google renamed it Managed Service for Apache Airflow, and current docs use the new name, so treat the two as identical. It is not a generic orchestrator — it is specifically Airflow (Airflow 3 is GA on Composer 3), with its DAGs, operators, scheduler, web UI, and ecosystem of community-maintained plugins. Composer is the right choice when the team already knows Airflow or has an existing portfolio of DAGs to migrate. For greenfield orchestration with no Airflow legacy, Cloud Workflows from Chapter 5 is almost always the better fit.
Composer's other defining trait is cost. Unlike Workflows or Cloud Run, a Composer environment is always on — there is no scale-to-zero. The GKE cluster runs continuously, the scheduler, the web server, the workers. A small environment runs roughly a few hundred dollars per month at minimum, before any DAG executes. This matters: Composer is not a thing you spin up for one workflow.
Managed Airflow on GKE
Composer 3 (the current generation, GA since 2025) manages the cluster, the Airflow upgrades, the database, and the metadata store, and decouples the environment from a customer-visible GKE cluster — faster environment creation and upgrades than earlier generations. You manage the DAGs and a small set of environment-level configuration (Python packages, environment variables, KubernetesPodOperator settings). Composer 1 and Composer 2 are older generations (Composer 2 ran Airflow on GKE Autopilot); Composer 1 and Composer 2 version 2.0.x reach end of life on September 15, 2026, while Composer 2 environments at version 2.1 or later are not affected.
DAGs and the Airflow Model
A DAG (Directed Acyclic Graph) is Airflow's unit of orchestration: a set of tasks with dependencies between them, scheduled to run on a cadence. Each task uses an operator — a unit of work that knows how to do a specific thing (run a Bash command, call a BigQuery query, trigger a Dataflow job). The Airflow scheduler reads DAGs from a directory, evaluates which tasks need to run, and dispatches them to workers. The web UI exposes the state of every DAG run, every task, every retry — a level of observability that is one of Airflow's main selling points.
GCP Operators and Hooks
Airflow ships with a deep catalog of GCP operators: BigQuery (run query, load, export), Cloud Storage (transfer, sense, delete), Dataflow (launch template, launch flex template), Dataproc (create cluster, submit job, delete cluster), Cloud Functions (invoke), Pub/Sub (publish, pull), and many more. Pre-built operators handle auth, retry, and result extraction so the DAG focuses on flow. When an operator does not yet exist for a GCP feature, the KubernetesPodOperator or a generic PythonOperator usually fills the gap.
Composer vs Cloud Workflows
The biggest decision around orchestration on GCP. Both can call services, run conditionally, retry on failure. The choice is about ecosystem and ongoing cost, not capability.
Cloud Composer — managed Airflow. Choose when the team already knows Airflow, you have a portfolio of DAGs to bring over, you need the depth of Airflow's plugin ecosystem, or you want the web UI for non-engineer visibility. Cost is fixed and significant.
Cloud Workflows — GCP-native YAML orchestration. Choose for greenfield work, for simple-to-medium orchestration, for serverless billing. Lower learning curve, lower running cost, smaller ecosystem.
Environment Sizing and Cost
Composer environments come in sizes (small, medium, large) plus customizable worker counts and machine types. The smallest environment is still always-on; right-sizing matters because Composer is one of the most expensive ongoing GCP services per workload. Review environment utilization quarterly. If DAGs are sparse and execution time is short, a smaller environment usually suffices. If scheduler latency or worker queue depth is bad, the environment is undersized. The middle ground — "environment is fine but expensive" — is where most teams sit; Workflows is sometimes a cheaper alternative path even for live workloads.
- Composer for simple workflows that Cloud Workflows would cover cheaply. The always-on environment cost is the first surprise on the next month's bill.
- Composer adopted in a team with no Airflow expertise. The learning curve is steep, ongoing cost is real, and most of the value is unused.
- Older generations on new projects. Composer 1 (already closed to new environments) and Composer 2 versions 2.0.x reach end of life on September 15, 2026; Composer 3 is the current generation for anything new.
- DAGs with heavy logic embedded in Python task functions. The DAG becomes a programming language pretending to be orchestration. Push logic into separate services (Cloud Run, Cloud Functions) that operators call.
- No environment sizing review. The cluster runs over-provisioned for years; no one notices because the cost is steady.
- Composer used for what is actually a periodic trigger of a single endpoint. Cloud Scheduler is a fraction of the cost and accomplishes the same thing.
- Composer only when the team has Airflow expertise or an existing DAG portfolio. Otherwise Cloud Workflows.
- Composer 3 for all new environments.
- Pre-built GCP operators over raw
PythonOperatororBashOperatorwherever they exist. The operators handle auth, retry, and result extraction correctly. - DAGs in git with CI/CD that syncs them to the environment's DAG folder. No editing the live DAG file in Cloud Storage.
- Environment sizing review at least quarterly; alert on scheduler lag and worker queue depth.
- Monitor task failure rate and DAG duration through Cloud Monitoring. Composer's web UI is for inspection, not alerting.
Knowledge Check
What is Cloud Composer at the platform level?
- A serverless YAML orchestrator built specifically for GCP, with steps defined inline and no cluster to manage
- Managed Apache Airflow running on GKE, with DAGs, operators, scheduler, and web UI
- A renamed version of Cloud Workflows that adds an ecosystem of connector plugins on top
- A streaming data orchestrator built directly on top of the Dataflow runner engine
When should you pick Cloud Composer over Cloud Workflows?
- Whenever the orchestration logic involves more than five sequential tasks chained together in a single run
- When the workflow needs to be triggered by GCP events delivered through Eventarc, such as a new object landing in a Cloud Storage bucket or a Pub/Sub message arriving
- When the team already has Airflow expertise or a portfolio of DAGs to migrate, or needs the depth of Airflow's plugin ecosystem and web UI
- When the workflow must complete in under 60 seconds end to end to satisfy a strict latency SLA
What is the principal recurring cost characteristic of Cloud Composer?
- Per-DAG-run billing, scaling exactly with the number of orchestration invocations across the month, so an idle environment costs nothing
- The environment is always on with no scale-to-zero — a small environment costs hundreds of dollars per month before any DAG executes
- Per-second worker billing similar to Cloud Run, with idle costs essentially zero between runs
- A one-time setup fee charged at environment creation, with day-to-day operation effectively free thereafter
Why prefer pre-built GCP operators over raw PythonOperator in a Composer DAG?
- PythonOperator is deprecated and slated for removal in the next Apache Airflow release, so any DAG that still relies on it will eventually stop scheduling its tasks altogether
- Pre-built operators handle auth, retry, and result extraction correctly for the target service; PythonOperator forces you to reimplement those concerns by hand
- PythonOperator cannot access secrets stored in Secret Manager through any Airflow hook
- Pre-built operators run at a lower per-execution price than a task wrapped in PythonOperator
A team without Airflow background adopts Composer because "we need orchestration on GCP". What is the most common outcome?
- The team quickly masters Airflow's abstractions within a sprint and finds that Composer works out cheaper than any other orchestration option on GCP
- The learning curve is steep, the always-on environment cost is felt, and most of Airflow's depth goes unused — Cloud Workflows would have served the actual need with less overhead
- Composer auto-tunes itself based on the observed workload pattern, hiding the operational complexity from the team entirely
- The team rapidly switches to Composer 1 once they discover it offers a noticeably gentler learning curve, a smaller always-on footprint, and meaningfully lower idle cost than the Composer 2 environment they started on
You got correct