Chapter 12: Testing & Quality
Topic 73

CI/CD for Ansible

CI/CDPipeline

Tests that only run on a laptop protect nobody, so the linters, Molecule, and a check-mode run move into the pipeline and become the gate: no merge to the larkspur-io/infra repo without a green run. The pipeline runs yamllint and ansible-lint for the cheap layer, molecule test across a matrix of target OSes for the medium layer, and ansible-playbook --check against a staging-shaped inventory for the integration layer.

All of it runs inside the same execution environment image the controller uses, so CI runs the exact toolchain prod runs — same ansible-core, same collection versions, same system dependencies. That single discipline is what makes CI-green mean something: a play that passes in the pipeline passes on the controller, because nothing about the toolchain differs between them.

The Pipeline Stages

A GitHub Actions or GitLab CI workflow runs the stages in order: install pinned ansible-core, ansible-lint, and molecule; lint; molecule test; then ansible-playbook --check --diff against staging. Ordering cheap-to-expensive is deliberate — a lint failure stops the job in seconds instead of after a multi-minute Molecule run, so the fastest feedback is the feedback you get most often.

The Larkspur pipeline — cheap to expensive, gating the merge
push
yamllint + ansible-lint
molecule
--check
merge gate
.github/workflows/ci.yml — stages ordered cheap to expensive
jobs:
  lint:
    runs-on: ubuntu-latest
    container: ghcr.io/larkspur-io/ee:1.7.0   # same EE image the controller runs
    steps:
      - uses: actions/checkout@v4
      - run: yamllint .
      - run: ansible-lint

  molecule:
    needs: lint                    # don't burn Molecule minutes if lint is red
    runs-on: ubuntu-latest
    strategy:
      matrix:
        image: [ubuntu2204, rhel9]   # one cell per supported OS
    steps:
      - uses: actions/checkout@v4
      - run: molecule test
        env:
          MOLECULE_IMAGE: ${{ matrix.image }}

The needs: lint dependency makes the ordering enforceable rather than merely conventional: the molecule job will not start until lint is green, so a missing FQCN never burns the Molecule minutes that a multi-OS matrix run costs.

Matrix Across Target OSes

A CI matrix runs molecule test once per supported platform — Ubuntu 22.04 now, RHEL if the fleet adds it — so a role that is green on Ubuntu but broken on RHEL fails a specific matrix cell instead of shipping. The package name differs, the service unit differs, the default paths differ; a role that has never run on an OS has never been shown to support it.

The matrix is the only honest way to claim multi-OS support. A README that says "works on RHEL" without a green RHEL cell is a wish, not a fact, and the first RHEL host is where the wish meets the package manager. One cell per claimed OS turns the claim into something CI actually verified.

--check as the Integration Layer

ansible-playbook --check --diff against a staging inventory previews changes against a real, drifted host without applying them — the closest Ansible gets to a pre-merge "what would this do." It sits above Molecule in the pyramid, because Molecule tests against a pristine container while staging is a real machine with real drift, and the two catch different failures.

But check mode is best-effort and lies for any command or shell task, which cannot know what they would have changed without running. So --check is a signal, not a guarantee, and it supplements Molecule rather than replacing it. Treating a clean check as proof and merging on it is the mistake; a clean check raises confidence, it does not certify the apply.

Gating Merges on Green

A branch-protection rule makes the workflow a required status check, so an un-green pipeline physically cannot merge. This is the line that turns tests from advisory into enforced — the difference between "we have tests" and "you cannot break main." Without the required-check rule, the pipeline runs, goes red, and a merge proceeds anyway; the tests become documentation nobody is bound by.

The rule costs one setting and changes the meaning of the whole pipeline. Once the green check is required, the linters, the Molecule matrix, and the staging check stop being suggestions and become the gate every change has to clear, which is the entire reason for moving them off the laptop in the first place.

The Larkspur Pipeline End to End

The larkspur-io/infra workflow runs lint, then molecule test across the OS matrix, then --check against staging — all inside the execution environment image, on every pull request. Secrets, like the Vault password and any cloud credentials a cloud-driver Molecule run needs, come from the CI secret store, never the repo, so credentials never land in git history. The green check is required before merge.

End to end, the pipeline is the testing pyramid made enforceable: cheap static checks first, a real throwaway target in the matrix middle, a drifted staging host at the top, and a branch-protection rule binding the whole thing to the merge button. Run it in the same EE the controller uses and CI-green and prod behavior cannot quietly drift apart.

Common Mistakes
  • Running Molecule with no OS matrix and claiming the role supports RHEL, then having it break on the first RHEL host because the package name or service unit differs and CI never tested it.
  • Treating --check mode as a guarantee and merging on a clean check, when a command or shell task that is a no-op in check mode does real damage on a live apply — check mode is a signal above Molecule, not a replacement for it.
  • Not making the workflow a required status check, so the tests run but a merge can ignore a red pipeline and break main anyway — unenforced tests are documentation.
  • Installing un-pinned tool versions in CI, so ansible-lint or ansible-core silently upgrades between runs and a pipeline that was green yesterday fails today on behavior no one changed.
  • Building the CI image differently from the controller's execution environment, so a play that is green in CI fails on the controller because a collection version or system dependency differs between the two.
Best Practices
  • Make the lint, Molecule, and check workflow a required branch-protection check, so a red pipeline physically blocks the merge rather than merely warning.
  • Run molecule test in a CI matrix across every OS the role claims to support, since an untested platform is an unsupported platform no matter what the README says.
  • Run CI inside the same execution environment image the controller uses, with pinned ansible-core and collection versions, so CI-green and prod behavior cannot drift apart.
  • Order stages cheap-to-expensive — lint, then Molecule, then --check — and pull every secret from the CI secret store, never the repo, so failures surface fast and credentials never land in git.
Comparable tools GitHub Actions · GitLab CI the runners that host the pipeline Terratest-in-CI the provisioning-side analog Test Kitchen drives the same lint-converge-verify loop for Chef in CI

Knowledge Check

Why is an OS matrix required to honestly claim a role supports RHEL as well as Ubuntu?

  • Package names, service units, and paths differ by OS, so green on Ubuntu says nothing about RHEL — the matrix runs molecule test per platform
  • RHEL hosts refuse at runtime to execute any role that was not originally built and tested inside a multi-OS CI matrix first
  • The matrix parallelizes the converge across runners so the role runs measurably faster on every supported target OS at once
  • A matrix is only ever needed when packaging a published collection for Galaxy or a private hub, never for a plain role shared by copy-paste between teams

What does ansible-playbook --check prove, and where does it lie?

  • It previews changes against a real drifted host but is best-effort — any command or shell task can't know what it would change, so it's a signal above Molecule, not a guarantee
  • It guarantees the exact set of changes a real apply will make, so a clean check means a safe merge
  • It fully replaces Molecule at the top of the pyramid, since it runs against a real drifted staging host instead of a throwaway clean container, which proves considerably less about the apply
  • It proves idempotency entirely on its own by running the whole play twice in check mode and diffing the two preview passes against each other

Why must the workflow be a required status check to actually matter?

  • Without branch protection the pipeline can go red and a merge proceeds anyway, so the tests become advisory documentation rather than a gate
  • Marking the check required makes the whole pipeline run roughly twice as fast on every pull request
  • A required status check is the only supported way to get ansible-lint installed inside a CI job
  • It is not actually required at all, because a red pipeline already blocks the merge button by default on every repository everywhere without any extra branch-protection setting

Why must CI run the same execution environment image the controller uses?

  • So a different collection version or system dependency can't make a play green in CI but broken on the controller — same toolchain means CI-green equals prod behavior
  • Because GitHub Actions hosted runners offer no other supported mechanism for installing ansible-core onto a runner at all, so the controller's image is the only path that puts the binary on the PATH
  • To grant the CI job the write access and SSH credentials it needs to connect to and apply changes directly against the live production inventory during the run
  • Only so the pipeline's log output renders in the web console with the exact same font, color scheme, and timestamp format the controller uses

You got correct