Why Test and the Testing Pyramid

A role that configures production is code, and code is wrong until proven otherwise. The larkspur_web role decides what nginx, gunicorn, and the app config look like on every prod host; a bad when condition or a non-idempotent command ships a broken config to all six prod web hosts at once, with no plan to review before it lands. There is no separate artifact that says "this is what will happen" — the run itself is the change.

Ansible has no state file to diff against, so the question "did this run actually converge the box?" has only one honest answer: run it and check, then run it again and check that nothing changed. The testing pyramid orders the checks by cost — cheap static analysis at the bottom, a real throwaway target in the middle, full integration against something prod-shaped at the top — so the feedback you get most often is the feedback that is nearly free.

Automation Is Code

The larkspur_web role is not a script someone runs once and forgets. It is the definition of what every prod web host looks like — which packages are installed, what the rendered nginx vhost contains, which gunicorn socket is listening, what defaults the app config carries. A typo in a template or a wrong default in defaults/main.yml is a production change the moment the role runs, identical in consequence to a bug shipped in the application itself.

So the role earns the same testing rigor as the app it deploys, not a "it ran clean on my laptop" wave-through. The blast radius is the whole web tier at once, which is precisely the situation where you want a clean target, a verifier, and a gate standing between the change and production — the same machinery you would never ship application code without.

The Pyramid, Bottom to Top

The pyramid has three tiers, ordered by what they cost to run. At the bottom: syntax check and lint — milliseconds, no target needed, just static analysis of the files. In the middle: the idempotency check and a full Molecule run — seconds to minutes, against one throwaway container the test creates and destroys. At the top: full integration against a prod-shaped host — slow and expensive, because it stands up something close to the real thing.

The ordering is a budget. Run the cheap layers on every commit, because they cost nothing and catch the most common mistakes first; reserve the expensive layers for pre-merge, where a slow honest test is worth the wait. A lint failure that stops the job in milliseconds is strictly better than the same bug surfacing minutes into a Molecule run, so you put the fast checks where they fire most often.

The Second-Run Rule as the Cheapest Real Test

The Ansible-native correctness check is brutally simple: apply the role, apply it again, and the second run must report zero changed. A non-zero second run means a task is not checking state before acting and is not truly idempotent — it did work on the second pass that it should have recognized as already done. You catch that for the cost of one extra play, which is why it sits at the bottom of the pyramid as the cheapest test that measures something real.

This rule measures convergence, not completion. A play can finish green while a task quietly re-does its work every time, and only a second run exposes it. The spurious changed matters because it can notify a handler — an nginx reload, a service restart — on a run that should have been a silent no-op, so the defect is not cosmetic even when the end state looks correct.

No State to Diff, So You Re-Converge

Terraform proves correctness by running plan against a stored state file: it diffs desired config against the recorded reality and shows you what would change. Ansible has neither the plan nor the state — there is nothing to diff against, because the managed node itself is the only record of what is true. Convergence is therefore proven empirically: you re-run the role against the live machine and read the result.

That is why testing is not optional polish here — it is the only mechanism that tells you the role is right. With a provisioning tool you can read a plan before applying; with Ansible the equivalent reassurance comes from a clean converge on a fresh target followed by a zero-changed second run. The re-convergence is the test, and skipping it leaves you trusting a green run that proves only that the playbook finished.

What Each Layer Catches

Each tier fails for a different reason, which is why skipping one leaves a whole class of bug uncaught. Lint catches anti-patterns and deprecations before a target is even touched — a deprecated module call, a bare command where a real module exists, a missing FQCN. Molecule catches "the role does not actually produce the end state" by applying it to a clean box and inspecting the result. Integration catches "it works on a clean container but not on a real, drifted host," where a package already half-installed or a service in an odd state breaks an assumption the container never had.

Because the failure modes do not overlap, the layers are additive rather than redundant. A clean lint says nothing about whether the role converges a box; a green Molecule run says nothing about a drifted production host; and neither catches a deprecation that lint would have flagged in milliseconds. Run all three and each closes a gap the others leave open.