Testing Collections

A role shared by copy-paste can be loose; a collection published to a Galaxy or private Automation Hub and pulled by teams you will never meet cannot. When larkspur_web is packaged as the larkspur.web collection, the testing bar rises to match the blast radius — and the tool changes too. ansible-test, built into ansible-core, runs the sanity, unit, and integration suites the wider Ansible ecosystem uses to vet every published collection.

A consumer cannot read your role's internals before depending on it. They run ansible-galaxy collection install larkspur.web, pin a version, and trust it — so the collection's own CI is the only thing standing between a bad release and everyone who installed it. That is why a published collection carries a heavier test suite than the local role it grew from, and why the gate sits on the release rather than on a single second-run check.

Why a Collection Needs More Rigor

A local role breaks one team — a team that can read the source, see what went wrong, and fix it in place. A published larkspur.web breaks every downstream consumer at once, none of whom wrote it, all of whom pinned a version and trusted it. The asymmetry in blast radius is the whole argument: the consequence of a bad release is multiplied by every install, so the collection carries its own full test suite and gates every release on it.

This is more than a second-run idempotency check. The local role's Molecule scenario still matters, but a collection adds a packaging contract — a manifest, a build, a namespace, doc blocks — and each of those is a surface that can break a consumer independently of whether the role itself converges. The rigor scales with who is downstream.

ansible-test Sanity

ansible-test sanity runs the ecosystem's static gauntlet: import checks, PEP 8 on any plugin code, documentation-block validation, and compatibility checks. The documentation check is strict — every module's DOCUMENTATION, EXAMPLES, and RETURN block must be present and well-formed — because those blocks are what ansible-doc renders for a consumer who cannot read your source. This is the exact bar Galaxy and certified collections are held to.

# static gauntlet: imports, PEP 8, DOCUMENTATION/EXAMPLES/RETURN, compatibility
ansible-test sanity --docker

# pytest unit tests for the collection's Python — modules, filter/lookup plugins
ansible-test units --docker

# target-based tests exercising modules and roles against real/containerized hosts
ansible-test integration --docker

ansible-test units

ansible-test units runs pytest-based unit tests for the collection's Python — custom modules, filter plugins, lookup plugins — in isolation, fast, with no target host. This is where a custom filter plugin shipped in larkspur.web gets tested as code before it is ever loaded into a play, the way you would test any Python function: feed it inputs, assert on outputs, cover the edge cases.

The payoff is where the failure surfaces. Without a unit test, a Python edge case in a filter plugin shows up as a cryptic Jinja error in a downstream user's play, far from the bug and impossible for them to debug. With one, the same edge case fails as a clean assertion in your CI, before the release is ever cut.

ansible-test integration

ansible-test integration runs target-based tests that exercise the collection's modules and roles against real or containerized hosts — the heavyweight layer that proves the packaged content works end to end. It complements the role's own Molecule scenarios rather than replacing them: Molecule proves the role converges a clean box, while collection integration tests prove the packaged module behaves correctly when invoked through the collection's namespace.

The distinction matters because packaging can break what the bare role did not. A module that works when called by short name in a local role can fail when resolved through the larkspur.web namespace, and only a test that runs the packaged collection catches it. Integration is the layer that exercises the collection as a consumer actually loads it.

Collection CI and the Release Gate

The collection repo runs ansible-test sanity, units, and integration on every pull request — often across an ansible-core version matrix — and a release is not cut unless all three are green. Versioning lives in galaxy.yml, and a changelog plus a real version bump are what let consumers pin and upgrade deliberately. That contract only works if each tagged version was actually tested, which is why the gate sits on the release tag.

namespace: larkspur
name: web
version: 2.4.0            # bump on every release; the changelog says what changed
readme: README.md
dependencies:
  ansible.posix: ">=1.5.0"

Testing the collection against only one ansible-core version is the quiet trap: a behavior differs between versions, and the consumer running the version you never tested breaks on a release that passed your CI. The matrix across the ansible-core versions you claim to support is what turns "supported" from a README line into something each tagged release actually verified.