Cloud Workflows
Cloud Workflows is serverless orchestration. You write a YAML definition that calls GCP services and HTTP endpoints in a defined order with conditions, error handling, retries, and parallel steps. The platform runs the executions, tracks every step, and persists state across invocations that can last up to a year.
When a workflow has more than three or four services strung together with conditional branches and retry logic, the alternative is hand-rolling orchestration with chained Pub/Sub messages — each step publishes the next, each consumer has to know what comes after. That approach falls apart fast. Workflows trades a YAML file for explicit flow, automatic state, and an execution log you can audit.
Orchestration vs Choreography
Two patterns for coordinating async work. Orchestration has a central controller that decides what runs next — Cloud Workflows is this pattern, and it shines when the flow is complex enough that you need to reason about it as a whole. Choreography has no central controller; each component reacts to events from others — Pub/Sub plus Eventarc is this pattern, and it shines when components evolve independently and the flow is naturally event-driven. Pick orchestration when you need to reason about end-to-end correctness; choreography when you need independent scaling and team autonomy.
Cloud Workflows — orchestrate a defined sequence of steps across services, with conditional logic, retries, and explicit error handling. Choose when you can describe the full flow ahead of time.
Eventarc — react to events as they happen. No central flow; each handler reacts and possibly triggers more events. Choose when the flow is naturally event-driven and components are decoupled.
Cloud Tasks — call one specific HTTP endpoint later, with retry and rate limits. Choose when the unit of work is "do this HTTP call eventually" rather than "coordinate several services".
YAML Workflow Definitions
A workflow is a YAML file: an ordered list of steps. Each step has a name, a call (which service or HTTP endpoint to invoke), arguments, and an optional result that captures the output for later steps. Conditional branches use the switch construct; assignments use assign. The language is purposefully small — enough for orchestration logic, not so much that the workflow becomes a programming language pretending to be YAML. When logic gets complex, push it into the called services instead.
Native GCP Connectors
Workflows ships with native connectors for most GCP services: Cloud Storage, BigQuery, Pub/Sub, Cloud Run, Cloud Functions, Cloud Build, Vertex AI, and dozens more. A connector is a typed wrapper that handles auth, request shape, retry, and long-running operation polling for that service. Use connectors instead of raw HTTP calls wherever they exist — they remove enormous amounts of boilerplate and handle the awkward edges (LRO polling, pagination) correctly.
Error Handling and Retries
Every step can be wrapped in a try / except / retry block. The retry policy supports max attempts, backoff strategy, and predicates that match specific error codes. The except branch catches errors that exhausted retries and either recovers (alternate flow) or fails the workflow explicitly. Without this scaffolding, one transient 503 from a downstream service kills the entire execution — which is fine for non-critical paths but unacceptable for anything that needs to be reliable.
Parallel Steps and Subworkflows
The parallel construct runs branches concurrently and joins them when all complete (or one fails, depending on the shared config). Use it for genuinely independent work — fanning out to multiple regions, processing items from a list — so the wall-clock time tracks the slowest branch rather than the sum. Subworkflows let you factor out reusable sequences; the main workflow calls a subworkflow the same way it calls a connector.
Long-Running Executions
A Workflows execution can run for up to a year, persisting state across pauses. This makes Workflows the right tool for processes that span human time — multi-step approval flows, scheduled data pipelines with overnight waits, anything that involves an await on a real-world event. The execution log shows every step and its result, making it trivial to debug what happened weeks after the fact.
- Hand-rolling orchestration with chained Pub/Sub messages when Workflows would express the same flow declaratively in a fraction of the code.
- Using Workflows for purely event-driven scenarios where Eventarc is a more direct fit — no central flow, just reactions.
- No try/except around external calls. A single transient 503 kills the execution; the workflow looks fragile.
- Polling loops inside the workflow waiting for an external event. Use Eventarc or a callback URL pattern instead of burning execution time on poll.
- Pushing complex logic into the YAML — the workflow language is intentionally small. When you need real programming, call out to a Cloud Function and put the logic there.
- Long-running operations called without using the connector's LRO polling. Calling start once, then never checking completion, leaves the workflow assuming success when the LRO is still in flight.
- Choose Workflows over ad-hoc Pub/Sub chains whenever the flow has conditional logic or more than a few steps.
- Use native GCP connectors instead of raw HTTP calls. They handle auth, LRO polling, and pagination correctly.
- Wrap every external call in
try / except / retrywith explicit backoff and max attempts. - Parallel steps for any independent work — the wall-clock time tracks the slowest branch instead of summing.
- Subworkflows for sequences used in more than one place. Treat them like functions.
- Keep the YAML focused on orchestration. Push real logic into Cloud Functions or Cloud Run services that the workflow calls.
- Cloud Logging on every workflow execution; alert on failed executions.
Knowledge Check
When is Cloud Workflows the right choice over ad-hoc Pub/Sub chains?
- Always — Workflows is the modern replacement for Pub/Sub in messaging architectures and supersedes topic-based fan-out for every new design
- When the flow has conditional logic or more than a few steps and you can describe the end-to-end sequence ahead of time
- When you need a global anycast endpoint for receiving events from multiple regions
- When the workload publishes more than 10,000 messages per second
What is the principal benefit of using a native GCP connector in a workflow versus a raw HTTP call?
- Connectors are billed at a lower per-call rate than raw HTTP calls
- Connectors handle auth, request shape, retry, and long-running operation polling correctly without per-call boilerplate
- Connectors bypass the target GCP service's IAM rules entirely, so the workflow can call any API without being granted the matching permission
- Connectors allow workflow YAML to embed arbitrary JavaScript expressions
Which pattern best describes Cloud Workflows?
- Orchestration — a central controller decides what runs next, with explicit flow described upfront
- Choreography — each component independently reacts to events emitted by others, with no central controller deciding the order of steps
- Pub-sub — one publish reaches every subscription
- Task-queue — one task targets one endpoint with retry policy
A workflow step calls an external service that occasionally returns 503. Without explicit error handling, what happens?
- Workflows retries the step indefinitely until it succeeds, slowing the execution
- The error propagates up and the entire execution fails — try/except/retry must be added to recover from transient failures
- The step is skipped and execution continues to the next step
- Workflows automatically buffers the failed call, pauses the execution, and replays the step on its own once the downstream service starts returning 200s again
What is the longest a single Cloud Workflows execution can run?
- 10 minutes — any work that runs longer must be split into separate chained executions, each handing its state to the next over Pub/Sub
- 24 hours, matching the default Cloud Run job timeout
- 7 days, matching the Pub/Sub message retention window
- Up to one year — state persists across the entire execution, enabling multi-step approval flows and overnight data pipelines
You got correct