Chapter 10: Orchestration & Scale
Topic 60

Async Tasks and Polling

Workflow

Ansible holds the SSH connection open while a task runs, so a thirty-minute database reindex or a large package build will hit a connection timeout long before it finishes. async detaches the task: Ansible kicks it off, gets a job id back, and either polls for completion or moves on entirely. The async/poll pair is how long-running operations survive on a push-based tool that would otherwise sit blocked on a socket.

There are two modes here, polled and fire-and-forget, and they are the same mechanism with the dial turned to different ends. This topic walks the problem, both modes, the module that reaps a fire-and-forget result, and where each belongs in a Larkspur deploy.

The Problem async Solves

A task that runs for many minutes risks an SSH or task timeout, and it blocks the control node's slot for that host the entire time. Ansible is waiting on a live connection, and connections do not stay healthy forever — a long-running migration that would have completed fine gets reported as a failure when the socket drops underneath it. async: 1800 tells Ansible the task may take up to 1800 seconds before it gives up, which lets the work run to completion instead of being cut off at the default connection timeout.

async with poll — Background but Wait

async: 1800 paired with poll: 15 runs the task in the background on the managed node and checks on it every fifteen seconds. Because the work is detached and running on the host rather than inline over a single command invocation, the task itself is no longer bound by the connection timeout, but the play still waits for the result before continuing. This is the common case: you have a long task, you want to wait for it, and you want to survive the wait without a timeout.

async with polling — a long reindex that survives the SSH timeout
- name: Rebuild the search index
  ansible.builtin.command:
    cmd: /opt/larkspur/bin/reindex --full
  async: 1800            # allow up to 30 minutes
  poll: 15               # check every 15s; play waits for the result

Fire-and-Forget — poll: 0

poll: 0 changes the contract: Ansible starts the task and immediately moves on without waiting. The job keeps running on the host after Ansible disconnects. This is how you launch several long jobs in parallel — kick each one off and come back for them later — or run a reboot-and-continue step where waiting in line makes no sense. The play does not block; it fires the task and proceeds to the next one.

async_status — Collecting a Fire-and-Forget Result

Fire-and-forget returns before the work is done, so on its own it tells you nothing about whether the job actually succeeded. After poll: 0 you hold the returned ansible_job_id and check it later with the async_status module, typically in a retries/until loop that waits for the job to report finished. That reap step is what turns a launched job into a confirmed one — without it, a silent failure in the background work is invisible to the run that started it.

Fire-and-forget, then reap the result
launch task with async
poll: 0, register job id
check async_status
done, no SSH timeout
poll: 0 then async_status — launch, then reap the result
- name: Kick off the cache rebuild
  ansible.builtin.command:
    cmd: /opt/larkspur/bin/warm-cache
  async: 600
  poll: 0                # fire and forget — do not wait
  register: cache_job

- name: Wait for the cache rebuild to finish
  ansible.builtin.async_status:
    jid: "{{ cache_job.ansible_job_id }}"
  register: job_result
  until: job_result.finished
  retries: 30
  delay: 20

Where async Belongs in a Deploy

In a Larkspur deploy the two modes split by intent. A long migration or a warm-up step on the db tier runs async with polling, because the play genuinely needs to wait for it before the dependent steps proceed and the only goal is surviving the SSH timeout. A fleet-wide "kick off the cache rebuild" runs poll: 0 across the hosts, because there is no reason to wait on each host serially — you launch them all and reap them afterward with async_status.

The rule that prevents the worst bug: never make the next task depend on a poll: 0 job without reaping it first. Fire-and-forget returns before the work is done, so a dependent step that runs immediately is running against an unfinished result. Either poll and wait, or fire-and-forget and explicitly reap before depending on the outcome.

Common Mistakes
  • Running a thirty-minute task with no async and hitting the SSH or task timeout — Ansible reports a failure on a job that was actually still running fine on the host.
  • Setting poll: 0 and never checking async_status — the job is launched and forgotten, so a silent failure in that background work is invisible to the run that started it.
  • Setting async shorter than the task actually needs (async: 60 on a five-minute job) — Ansible marks it timed out and failed even though the work would have completed on its own.
  • Using poll: 0 for something the next task depends on — fire-and-forget returns before the work is done, so a dependent step runs against an unfinished result.
  • Assuming poll: 0 survives when the managed node has no way to keep the process alive — it relies on the job continuing on the host after disconnect, which a misconfigured target can break.
Best Practices
  • Wrap any task that can exceed an SSH timeout in async with a polling interval, sizing async to the realistic worst-case duration with a little headroom.
  • Use poll: 0 only when you will reap the job — pair every fire-and-forget with an async_status check in a retries/until loop before depending on its result.
  • Run independent long jobs across hosts as poll: 0 and collect them afterward, instead of waiting on each serially and multiplying the wall-clock time.
  • Keep the async timeout slightly above the measured task duration, not generously huge, so a genuinely stuck job is caught rather than masked by a long ceiling.
  • Register the launched job and carry its ansible_job_id explicitly, so the reap step has a concrete id to check rather than guessing which job to wait on.
Comparable tools No direct equivalent async exists because Ansible pushes over a live SSH connection Puppet · Chef agents run locally on the node, with no remote-connection timeout to dodge nohup · systemd-run the raw "detach a long job" the async mechanism formalizes

Knowledge Check

Why does a long-running task need async?

  • Ansible holds the SSH connection open while a task runs, so a long job risks a timeout and a false failure unless detached
  • Without async the task runs locally on the control node itself, instead of executing remotely on the managed host the play actually targets
  • async is the only way to escalate to root for a task that runs longer than a minute or two
  • Long tasks are skipped entirely at runtime unless async with a ceiling is set on them

What is the difference between poll: 15 and poll: 0?

  • poll: 15 runs the task in the background and waits, re-checking every 15s; poll: 0 starts it and moves on
  • poll: 15 retries the failed task up to 15 times before giving up; poll: 0 never retries at all
  • poll: 15 runs the task synchronously on the managed host while poll: 0 shifts it onto the control node instead
  • They are equivalent; the number only affects how verbose the polling output is in the log

What is async_status for after a fire-and-forget task?

  • It checks the returned ansible_job_id in a retries/until loop, confirming the background job finished and succeeded
  • It cancels the still-running background job by the saved ansible_job_id and kills its remote process so the play can continue cleanly
  • It converts a fired-and-forgotten poll: 0 task back into a synchronous one after the fact
  • It is required only when the task's async ceiling has been set to exactly zero seconds

What happens if the async ceiling is shorter than the task actually needs?

  • Ansible marks the task timed out and failed even though the work would have completed
  • The task automatically extends the ceiling in small increments until the work finishes
  • The task keeps running to completion but its result is silently discarded by the control node
  • Ansible quietly falls back to running the same task synchronously over the live connection

You got correct