Topic 42

Failure Control

Reliability

Ansible decides a task failed or changed from the module's own report, and for idempotent modules that report is honest. command and shell can't self-report against state — they always run, always claim changed, and fail only on a non-zero exit — so changed_when and failed_when exist precisely to teach Ansible what success and change actually mean for that command.

The fleet-wide switches, any_errors_fatal and max_fail_percentage, then decide whether one host's failure stops everything or the rollout tolerates losing a few. Together these four knobs are how you make a playbook honest about change and deliberate about blast radius.

Why command and shell Need Overrides

With no state file, an idempotent module checks reality before it acts and reports truthfully — apt knows whether the package was already installed, so it reports ok on a no-op and changed only when it installed something. command and shell cannot do this: they run a black-box program, have no idea what it touched, and so report changed on every single run while conflating "the command ran" with "something actually changed." changed_when and failed_when restore honest reporting by letting you define those two outcomes yourself. This is the no-state-file spine of the whole book made concrete — the absence of stored state is exactly why a raw command can't self-report.

changed_when

Set changed_when to a Jinja expression over the registered result so a shell that changed nothing reports ok instead of changed. changed_when: "'updated' in cmd_out.stdout" reads the command's own output and decides change from it, which keeps your change counts meaningful and — critically — keeps a notify handler from firing on a no-op. A task that reports changed every run notifies every run, and a service that restarts every run is the opposite of idempotent.

failed_when

failed_when redefines failure beyond the exit code. failed_when: "'ERROR' in result.stderr" fails a task whose tool exits 0 but logs an error to stderr — the common case of a command that lies about its own success. The reverse form, failed_when: false, forces a task to never fail, the controlled way to say "non-zero is expected here." Both override the crude exit-code default with something that matches what the command actually means.

Overriding what counts as failed or changed

changed_when

Redefines change from the command's output, so a shell no-op reports ok and no handler fires.

failed_when

Redefines failure beyond the exit code — fail a tool that exits 0 but logs an error to stderr.

ignore_errors

Tolerates a failure — it still shows red and registers, it just doesn't stop the play.

Teach a shell task what change and failure really mean

- name: Refresh the search index
  ansible.builtin.command: /usr/local/bin/reindex --quiet
  register: reindex
  changed_when: "'documents added' in reindex.stdout"
  failed_when: reindex.rc != 0 or 'ERROR' in reindex.stderr

Now the task reports changed only when the reindex actually added documents, and fails on a logged error even when the tool exits 0 — two things the default exit-code-only behavior gets wrong for any non-trivial command.

ignore_errors versus failed_when

ignore_errors: true lets a task fail and keeps going — the failure still shows red and still registers, it just is not fatal. failed_when: false goes further: it redefines the task as never-failed, so it shows green. The choice is about intent. Reach for failed_when when non-zero is genuinely fine here and you want it reported as success; reach for ignore_errors when you want the failure recorded and visible but not allowed to stop the play. One reclassifies the outcome, the other tolerates it.

Fleet-Level Control

any_errors_fatal: true aborts the entire play the instant any host fails — the right choice for a coordinated rollout where a single broken node means the whole batch should stop and be looked at. max_fail_percentage: 30 is the opposite posture: it tolerates losing up to that share of the batch before aborting, which fits a rolling deploy where a couple of flaky hosts out of fifty should not halt the rollout. These two knobs govern blast radius on the Larkspur web tier — all-or-nothing for tightly coupled changes, percentage-tolerant for independent ones.

Common Mistakes

Leaving a shell task with no changed_when, so it reports changed every run, fires a notify handler every run, and makes the playbook never idempotent — exactly the trap the no-state-file spine warns about.
Using ignore_errors: true to paper over a task that fails for a real reason — the failure is hidden, the next task runs against a broken precondition, and the actual error surfaces three tasks later in a confusing place.
Writing failed_when and changed_when against a result you forgot to register:, so the expression reads an undefined variable and the task errors on the override itself.
Setting any_errors_fatal on a rolling deploy where you actually want max_fail_percentage — one transient host failure aborts the whole rollout instead of tolerating a small fraction.
Testing only the exit code on a tool that exits 0 while logging ERROR to stderr — the task reports success, the real failure goes unnoticed, and the broken state ships.

Best Practices

Add changed_when and a register to every command / shell task, deriving change from its output so reruns report ok and handlers only fire on real change.
Prefer failed_when: false over ignore_errors when a non-zero exit is genuinely acceptable, reserving ignore_errors for "let it fail but keep going and show me."
Use failed_when to catch tools that exit 0 on error, testing stderr or stdout for the real failure signal.
Choose any_errors_fatal for all-or-nothing coordinated changes and max_fail_percentage for rolling deploys where losing a small share of hosts is survivable.
Prefer a purpose-built idempotent module over command / shell wherever one exists, so you never need to hand-write the change and failure logic at all.

Comparable tools Puppet exec's onlyif / unless / returns shape exec idempotency similarly Chef not_if / only_if guards and custom guard blocks Shell set -e plus exit-code checks is the unstructured version

Knowledge Check

Why do command and shell need changed_when while idempotent modules like apt do not?

With no state file, an idempotent module checks reality and reports truthfully, but a raw command is a black box that can't tell whether anything changed
command and shell run inside a restricted sandbox that deliberately blocks them from reading the change-detection facts that every other module relies on internally
Idempotent modules write to a hidden per-host state file under the remote temp directory that command and shell are deliberately not permitted to read or touch
It is purely cosmetic — both modules report change identically on every single run, and only the wording printed on the status line differs at all

What does failed_when catch that a plain exit code cannot?

A tool that exits 0 but logs an error — you test stderr or stdout for the real failure signal
A host that becomes unreachable mid-task when the underlying SSH connection abruptly drops out
A YAML syntax error in the playbook source itself, caught at parse time long before the task ever runs
A handler that was notified earlier in the play but never actually ran before the end

How does ignore_errors: true differ from failed_when: false?

ignore_errors still records the failure and shows it red but keeps going; failed_when: false redefines the task as never-failed, so it shows green
They are identical aliases for the exact same behavior, differing only in spelling and producing the same status and recap output on every run
failed_when: false halts the play the moment a task errors, while ignore_errors quietly continues running the remaining tasks on that host
ignore_errors only works on a task nested inside a block, while failed_when can be applied to any task anywhere in the play

When should you choose max_fail_percentage over any_errors_fatal?

On a rolling deploy where losing a small share of independent hosts is survivable and should not halt the whole rollout
On a tightly coordinated cluster-wide change where any single host failing should stop the entire run for everyone immediately
Whenever you want a task to report change only on genuine differences detected in its output across runs
Only when the play happens to have exactly one host in its scope and no room for partial failure

You got correct