Chapter 11: Configuration, Plugins & Performance
Topic 68

Debugging

WorkflowDebugging

When a play does the wrong thing or fails on host four of six, Ansible gives you a ladder of visibility from a one-line summary down to the exact SSH command it ran. Turning up -v levels, reading the SSH invocation at -vvvv, stepping through tasks one at a time, dropping into the task debugger on failure, and printing values with the debug module are the tools that turn "it's broken" into a located cause.

The skill is reaching for the right rung instead of staring at the default output. Most logic bugs fall to -v and a debug of a registered result; most connection and escalation problems need the SSH command at -vvvv; and a long play debugs faster when you jump straight to the failing task instead of re-running everything before it. The sections below walk the ladder rung by rung.

Verbosity — -v to -vvvv

Each v adds a layer of detail. -v shows task results, -vv adds the task input, -vvv adds connection detail, and -vvvv adds connection-plugin debugging — including the literal ssh command Ansible ran. Reading that SSH line at -vvvv is how you confirm the user, port, host, and ControlPersist socket Ansible actually used, which is ground truth when the inventory says one thing and the connection behaves like another. More vs is more signal, but also more noise, so you climb only as high as the problem needs.

The debug Module

ansible.builtin.debug prints a variable with var: result or a free-form message at run time, and it is the most-used debugging tool in Ansible. Pair it with register to dump exactly what a task returned — the full result structure, the changed flag, the stdout — so you stop guessing at a variable's value or at the inputs to a failing when condition. When a conditional skips a task you expected to run, a debug of the variable behind it usually shows the assumption that was wrong.

The Task Debugger

Setting debugger: on_failed on a task — or exporting ANSIBLE_ENABLE_TASK_DEBUGGER=1 for the whole run — drops you into an interactive prompt the moment a task fails. There you can print variables with p task_vars, inspect task.args, even reassign a value and retry the task in place. That turns a failure into something you debug live, instead of editing the playbook and re-running the entire play to test one guess. It is opt-in, so a failure aborts normally unless you have enabled it.

A task wired for live inspection on failure
- name: Render the nginx vhost from a template
  ansible.builtin.template:
    src: vhost.conf.j2
    dest: /etc/nginx/sites-available/larkspur.conf
  register: vhost
  debugger: on_failed     # on failure, drop into the interactive task debugger

# at the (debug) prompt: p task.args, p vhost, then `redo` to retry the task

register: vhost captures the result so you can print it, and debugger: on_failed opens the prompt the instant the template task fails. From there p task.args shows what the task was actually called with and redo retries it after you fix a variable — the failure is inspected where it happened, not reconstructed after an abort.

Controlling Execution — --step and --start-at-task

--step prompts before every task so you walk a play interactively, confirming or skipping each one, which is useful when you want to watch a risky play unfold task by task. --start-at-task "name" jumps straight to a named task so you don't re-run the thirty tasks before the one that's failing. Both shorten the debug loop on a long Larkspur deploy: instead of running the whole play to reach the broken step, you start at it or step into it directly.

Dry Listing — --list-tasks and --list-hosts

--list-tasks prints the tasks a play would run, with their tags, and --list-hosts prints the hosts a pattern resolves to — both without executing anything. They answer "what is this run going to do, and to whom" before you let it touch prod. Running --list-hosts against a pattern catches the case where a typo in a host pattern would have hit the wrong tier, and --list-tasks confirms a tag filter selects exactly the tasks you meant, all before a single change lands.

Reading the SSH Command at -vvvv

At -vvvv Ansible prints the full ssh command line per task, and that line is the ground truth for any connection or escalation problem. It exposes whether multiplexing is active (the ControlPath socket), which key and user it used, the port, and whether pipelining is in effect. When a connection or escalation problem doesn't match what the inventory says — the wrong user, a missing jump host, a ControlPersist that isn't engaging — the printed SSH command shows what Ansible actually did, settling the question the inventory only describes.

Common Mistakes
  • Jumping straight to -vvvv for every problem and drowning in connection noise, when -v plus a debug of the registered result would have located a logic bug in seconds.
  • Editing the playbook and re-running the entire 40-task play to test a one-task fix instead of --start-at-task to jump to it or the task debugger to retry it in place.
  • Guessing at a variable's value behind a failing when instead of debug: var=the_var, then chasing the wrong cause because the variable wasn't what you assumed.
  • Running a risky play against prod without --list-hosts/--list-tasks first, so the blast radius and task set are assumptions rather than confirmed facts.
  • Forgetting the task debugger is opt-in and never enabling debugger: on_failed or ANSIBLE_ENABLE_TASK_DEBUGGER, so a failure that could be inspected live just aborts the run.
Best Practices
  • Reach for -v and a debug of the registered result first, and escalate to -vvvv only when the problem is connection, auth, or escalation — then read the printed SSH command for ground truth.
  • Use --start-at-task and --step to shorten the loop on long plays rather than re-running everything to reach one failing task.
  • Enable the task debugger with debugger: on_failed (or ANSIBLE_ENABLE_TASK_DEBUGGER=1) on plays you're actively debugging, so a failure drops you into an inspectable prompt instead of aborting.
  • Confirm scope with --list-tasks and --list-hosts before running anything against prod, so what-and-to-whom is verified, not assumed.
  • Read the -vvvv SSH line when a connection behaves unlike the inventory describes — the printed command is what Ansible actually ran, settling user, port, key, and multiplexing.
Comparable tools Terraform TF_LOG=DEBUG and terraform plan parallel -vvvv and --list-tasks SaltStack -l debug is the verbosity analog Python pdb the closest non-Ansible equivalent of the task debugger

Knowledge Check

What does -vvvv expose that lower verbosity levels do not?

  • Connection-plugin debugging including the literal ssh command Ansible ran, which shows the user, port, key, and ControlPersist socket actually used
  • A full before-and-after diff of the contents of every file changed on the managed node over the course of the run
  • A complete line-by-line execution trace of the Python module as it actually runs on the target host, with every individual statement logged out in order
  • The full resolved ansible.cfg with every default value printed, so you can audit the active configuration

A task fails and you want to inspect and fix it without re-running the whole play. What lets you do that?

  • The task debugger via debugger: on_failed — it drops into an interactive prompt where you can print variables, reassign them, and retry the task in place
  • -vvvv, which automatically pauses execution on the first failed task and lets you edit variables
  • --list-tasks, which re-runs only the single failed task in isolation while skipping all the others
  • The debug module, which halts the play right at the point of failure and then opens a full interactive shell session on the target host so you can poke around

A 40-task play fails on task 35. What is the purpose of --start-at-task versus --step?

  • --start-at-task jumps straight to a named task so you skip the 34 before it; --step prompts before every task so you walk the play interactively
  • Both flags re-run the entire 40-task play from the top, but --start-at-task does it twice in a row for confirmation
  • --start-at-task runs the tasks across hosts in parallel, while --step runs them serially one host at a time
  • --step jumps straight to a named task and --start-at-task prompts for confirmation before each one in turn — the two behaviors are exactly reversed here

Why run --list-hosts and --list-tasks before a play against prod?

  • They print which hosts a pattern resolves to and which tasks would run, without executing anything, so blast radius and task set are confirmed rather than assumed
  • They run the play in full check mode directly against prod and report back in detail every single change that it would otherwise have made to each managed host in turn
  • They are mandatory safety flags that Ansible refuses to run any production play without first being given
  • They pre-cache the resolved host list and task list so that the real production run starts up noticeably faster

You got correct