Multi-tier Orchestration

This is the worked example the whole chapter builds toward: deploy a new larkspur release to the six-host prod web tier behind HAProxy on lb1.larkspur.io without dropping a single request. The play runs serial: 1 over web, and for each host it delegates a drain to lb1, deploys and health-checks the host, then delegates a re-add — one host out of the pool at a time, never the whole tier.

It is the clearest demonstration in the book that an ordered, gated, cross-host rollout is what a push model does and a pull model cannot. Each primitive from the earlier topics has a job here: serial for the wave, delegate_to for the load-balancer actions, the health gate for correctness, max_fail_percentage for the stop. The sections assemble them into one play.

The Play Shape

The play is hosts: web, serial: 1, become: true — one prod web host per wave, escalating to root for the deploy. The structure that makes it safe is the split between pre_tasks, the main tasks, and post_tasks: the drain runs before the main work and the re-add runs after, so the load-balancer actions cleanly bracket the deploy. Each wave is therefore a tidy unit — drain this host, deploy it, return it — repeated six times down the tier.

pre_tasks — Drain From HAProxy

For the current web host, a pre_tasks step with delegate_to: lb1.larkspur.io disables that server in the HAProxy backend and waits for in-flight connections to drain. The drain runs on the load balancer but names the web host the play is on, so HAProxy stops sending it new traffic and lets existing requests finish. Only once the host is out of rotation does the deploy touch it — no live traffic ever hits a host mid-deploy.

- hosts: web
  serial: 1
  max_fail_percentage: 0     # first failure halts the whole rollout
  become: true
  pre_tasks:
    - name: Drain {{ inventory_hostname }} from HAProxy
      community.general.haproxy:
        state: disabled
        backend: web_backend
        host: "{{ inventory_hostname }}"
        wait: true            # wait for in-flight connections to drain
      delegate_to: lb1.larkspur.io

The Deploy and Health Check

With the host drained, the main tasks pull the new release into /opt/larkspur, restart gunicorn, and reload nginx through handlers. Then comes the gate: a uri task polls the host's own health endpoint in a retries/until loop until it returns healthy. This is what refuses to proceed on a host that did not come back up — a deploy that reports ok is not proof the app is serving; only a passing health check is.

  tasks:
    - name: Pull the new release
      ansible.builtin.git:
        repo: https://github.com/larkspur-io/app.git
        dest: /opt/larkspur
        version: "{{ release_tag }}"
      notify: Reload nginx

    - name: Restart gunicorn
      ansible.builtin.service:
        name: gunicorn
        state: restarted

    - name: Wait for the host to report healthy
      ansible.builtin.uri:
        url: "http://{{ inventory_hostname }}:8000/healthz"
        status_code: 200
      register: health
      until: health.status == 200
      retries: 12
      delay: 5

post_tasks — Re-add to the Pool

Only after the health check passes does a post_tasks step, again delegate_to: lb1.larkspur.io, re-enable the server in HAProxy. The host rejoins the pool, takes traffic again, and the next wave begins on the next host. The ordering is the safety property: a host is returned to rotation strictly after it has proven it is healthy, never before. Drain, deploy, gate, re-add — and only then advance.

Guaranteeing Re-add on Failure

The dangerous edge case is a deploy that fails after the drain. If the re-add lives only in the happy path, a failed host stays drained out of the pool indefinitely. The fix is to make the re-add unconditional — wrap the drain-deploy-readd in a block with the re-add (or an explicit abort) in an always or rescue, so the host is either returned to service or the rollout stops loudly. Combined with max_fail_percentage: 0, the first broken host halts everything instead of the play draining a second one behind it.

Ordering Across Tiers

The web rollout does not stand alone. In a full release the db tier migrates first in an earlier play, the web tier rolls behind the load balancer second, and lb1 itself is configured before either. That ordering is enforced by running the tiers as separate plays in sequence and by the linear strategy's per-task barrier, which keeps "db before web" honest — a later tier never runs against an unfinished earlier one. The push model sequences the whole fleet from one place; a pull model, where each node converges alone on its own timer, cannot coordinate "drain at the LB, then touch this host," which is exactly why this play is the chapter's closing argument.