Chapter 13: Extending Ansible & Automation at Scale
Topic 77

Execution Environments

ScalePackaging

An execution environment is a container image that bundles ansible-core, the collections your playbooks need, and the Python dependencies those collections require, so every run uses an identical, versioned toolchain instead of whatever happens to be on a given laptop or CI runner. You build one with ansible-builder from a small definition file, push it to a registry, and run playbooks inside it.

From definition to a reproducible toolchain
execution-environment.yml
ansible-builder
EE image
reproducible run anywhere

This is what finally kills "works on my machine" for Ansible, and it is what every job on Ansible Automation Platform actually executes in. Once the toolchain is a tagged image, the question "which collection version produced this result?" has an exact answer — the image tag — rather than a shrug. For the Larkspur fleet, the deploy toolchain becomes larkspur-ee:2026.04, and that string is the whole runtime contract.

The Problem EEs Solve

A playbook depends on more than its YAML. It needs specific collection versions, Python libraries — boto3, the kubernetes client, a cloud SDK — and a particular ansible-core. When those differ between an engineer's laptop, the CI runner, and the controller, the same playbook behaves differently: a module that works locally fails in CI because the runner has an older collection, or a filter exists on one machine and not another. The execution environment freezes all of it into one image, so there is exactly one toolchain and one behavior.

What's Inside

An EE layers a base image (Red Hat ships supported ones), ansible-core, a pinned set of collections, the Python packages those collections declare, and any system packages a play needs — openssh-clients, a database client, a compiler. The result is a self-contained runtime you can run anywhere a container runs. Nothing about the play reaches outside the image for a dependency, which is exactly why the same image produces the same result on a laptop, in a pipeline, and on the controller.

ansible-builder

ansible-builder is the tool that turns a definition into a buildable container context and then the image. The definition file, execution-environment.yml, names the base image and points at a requirements.yml for collections, a requirements.txt for Python packages, and an optional list of system packages. Crucially, it reads each collection's own declared dependencies, so you do not hand-resolve which Python libraries amazon.aws or kubernetes.core need — the builder pulls them in from the collection metadata.

execution-environment.yml — the definition ansible-builder turns into larkspur-ee
version: 3

images:
  base_image:
    name: registry.redhat.io/ansible-automation-platform-25/ee-minimal-rhel9:latest

dependencies:
  ansible_core:
    package_pip: ansible-core==2.16.*   # pin the engine
  galaxy: requirements.yml             # collections, each version-pinned
  python: requirements.txt            # boto3, kubernetes, ...
  system: bindep.txt                  # openssh-clients, gcc, ...

You build and tag in one command — ansible-builder build -t larkspur-ee:2026.04 — and push the result to your registry. From that point the tag is the toolchain: anyone who pulls larkspur-ee:2026.04 runs the exact ansible-core, collections, and Python you pinned in this file.

Versioning and Distribution

An EE is a tagged image in a registry like any other container, so larkspur-ee:2026.04 pins the exact toolchain a deploy ran with, and rolling back is pulling the previous tag. Adopting a new collection version is a deliberate act: you bump the pin in the definition, rebuild, and tag a new image — a reviewed change with a changelog, not drift that crept in because someone ran ansible-galaxy install on the controller one afternoon. The image is a release artifact, and it earns the same discipline as one.

Why AAP Is Built on Them

Ansible Automation Platform runs every job inside an EE by default. That is how it guarantees a job template behaves identically whether it runs today or in six months — the toolchain is frozen in the image the template is bound to, not drawn from whatever the controller happens to have installed. It is also how different teams run on different curated images on the same controller: one team's job templates use a hardened minimal EE, another's a larger image with extra SDKs, and neither can disturb the other. The EE is not a convenience layer on top of AAP; it is the thing AAP runs.

Common Mistakes
  • Running playbooks from each engineer's local ansible install and treating green-on-my-laptop as green everywhere — collection and Python-version skew mean the EE-run result is the only one that counts.
  • Building an EE with unpinned collection and Python versions, so two rebuilds a month apart produce different toolchains and quietly reintroduce the drift the EE was meant to remove.
  • Forgetting a collection's Python dependency — a play uses amazon.aws but the image lacks boto3 — so the module fails at runtime inside the EE with an import error nobody saw coming.
  • Baking secrets or credentials into the EE image, where they then sit in every registry layer for anyone who can pull it — credentials are injected at run time, never built in.
  • Treating the EE as optional once you adopt AAP — AAP runs in EEs, so an out-of-date or wrong image is a production-behavior problem, not a missed nicety.
Best Practices
  • Pin ansible-core, every collection, and every Python dependency in the EE definition, so a given image tag is a reproducible toolchain you can roll back to by name.
  • Build EEs with ansible-builder from a checked-in definition and treat the image as a release artifact — tagged, in a registry, with a changelog of exactly which versions moved.
  • Run CI and local development against the same EE image you deploy with, so the laptop, the pipeline, and the controller share one runtime and one behavior.
  • Keep credentials out of the image entirely and inject them at run time through AAP credentials or a vault, since image layers are not a secret store.
  • Curate a small number of EE images deliberately rather than one sprawling image with every SDK, so each team runs the minimal toolchain its plays actually need.
Comparable tools A Docker image pinning a toolchain the general analog pip / virtualenv freeze the Python layer only, not collections or system packages A pinned Terraform CLI container the provisioning-side equivalent Nix the maximalist version of the same reproducibility goal

Knowledge Check

What does an execution environment bundle, and what problem does that solve?

  • ansible-core, pinned collections, and their Python and system dependencies in one image, so every run uses an identical toolchain
  • The inventory and the vault passwords baked right in, so a run carries its own targets and secrets without anything supplied at launch
  • A full snapshot of every managed host's filesystem, so the controller can diff live state against the image
  • The cached playbook output of the last run, so later reruns can simply skip any unchanged tasks

When resolving dependencies, what does ansible-builder read so you don't hand-resolve Python libraries?

  • Each collection's own declared dependencies, pulling in the Python packages it needs from the collection metadata
  • The running controller's currently installed package list, copying whatever Python libraries happen to already be there into the image
  • The managed nodes directly over SSH, querying each one to match its installed Python versions
  • The base image's published changelog, deriving the Python requirements from its prior releases

Why must an EE image be pinned and versioned rather than rebuilt from "latest" each time?

  • An unpinned rebuild produces a different toolchain over time, reintroducing the exact drift the EE was meant to remove
  • Container registries will simply reject and refuse to store any image at all that lacks an explicit, non-latest version tag
  • Pinning the version is the only safe way to bake the run credentials into the image at build time
  • ansible-navigator outright refuses to run any image whose bundled collections are not pinned

Why does Ansible Automation Platform run every job inside an execution environment?

  • So a job template behaves identically over time and different teams can run different curated images on the same controller
  • Because running inside a container is the only way the controller can open an SSH connection to its managed hosts at all
  • To store the full job audit log inside the EE image itself, right alongside the bundled toolchain
  • Because running in an EE entirely replaces the need for stored credentials and RBAC on the controller

You got correct