Topic 55

Secrets Beyond Vault

ComparisonSecurity

Vault files in git are the right default, but they have limits. The encrypted secret is a static blob that lives in the repo, rotation means a commit, and there is no audit trail of who read what. For a small fleet with a handful of static secrets, none of that matters. Once it does, you reach for something else.

An external secret manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager — moves the secret out of the repo entirely, and Ansible fetches it at run time with a lookup. You trade the simplicity of a checked-in file for rotation, audit logging, and dynamic credentials. This topic draws the line: what those managers give you, how Ansible reaches them, and when the trade is worth its operational weight.

Where the trade-off lands

Vault files

Simple, in-repo, needs only ansible-core. Rotation is a commit, reads leave no trace — right for a small fleet of static secrets.

external manager

HashiCorp Vault or a cloud manager — secrets out of the repo, with rotation, per-read audit, and dynamic short-lived credentials.

Lookups Fetch at Run Time

A lookup plugin reads a value from somewhere external when the play runs. The HashiCorp, AWS, and GCP managers each have one — community.hashi_vault.hashi_vault, amazon.aws.aws_secret, google.cloud.gcp_secret_manager — and they all work the same way: the playbook holds a path to the secret, not the secret itself. Nothing secret lives in the repo at all, because the value is fetched fresh from the manager every time the lookup is evaluated.

This is a sharper separation than a vault file gives you. With Vault, the ciphertext is in the repo and the key is elsewhere; with an external manager, even the ciphertext is gone — the repo holds only the address of the secret, and the value never enters version control in any form.

HashiCorp Vault

lookup('community.hashi_vault.hashi_vault', 'secret/data/larkspur:db_password') authenticates to a running Vault server and pulls the current value of db_password. The control node holds a Vault token, not the database password, and the password can rotate behind the lookup without a single change to the repo. The playbook keeps asking for secret/data/larkspur:db_password; what comes back can change underneath it.

A hashi_vault lookup — the repo holds the path, never the value

- name: Configure the Larkspur app with the current DB password
  ansible.builtin.template:
    src: app.env.j2
    dest: /etc/larkspur/app.env
  vars:
    db_password: "{{ lookup('community.hashi_vault.hashi_vault',
                  'secret/data/larkspur:db_password') }}"
  no_log: true           # the value never reaches the log — see topic 56

The token on the control node is governed by Vault policy, so "who can read db_password" is a Vault question, not a "who has the repo" question. That separation is most of why teams move here: access becomes a policy you can change centrally rather than a fact baked into who cloned the repository.

Cloud Secret Managers

amazon.aws.aws_secret and google.cloud.gcp_secret_manager read from AWS Secrets Manager and GCP Secret Manager using the control node's existing cloud credentials. No separate token to manage, no Vault server to run — the same IAM that governs the rest of your cloud governs who can read the secret. This is the natural fit when Larkspur already runs on that cloud and IAM already answers "who is allowed to do what."

The appeal is that you add nothing operationally. The secret manager is a managed service you are likely already paying for, the credential is the cloud credential the control node already has, and the access control is the IAM you already maintain. For a team on one cloud, this is often the lowest-friction step beyond a vault file.

Dynamic Secrets

HashiCorp Vault can do something no static file can: mint a short-lived PostgreSQL credential per run. The db role asks Vault for a database credential, Vault creates one that expires in an hour, and the role connects with a username and password that did not exist before the run and are useless after it. There is no long-lived password to leak, because the credential's whole life is the length of the run plus a short grace window.

This is the capability gap that justifies the operational weight. A static Ansible Vault file cannot do this at all — its secret is a fixed value that is exactly as compromised after a leak as a plaintext password would be. A dynamic secret limits the damage of a leak to the minutes before it expires, which for high-value credentials like a production database is worth a great deal.

Audit and Rotation

An external manager logs every read — who fetched the secret, when, and which version — and rotates a value in one place that every consumer picks up on the next run. A checked-in vault file has neither. Rotating it is a commit, and reads are invisible: there is no record that a particular engineer or pipeline ever decrypted the file, only that the file exists. When a compliance reviewer asks "who read the database password last quarter," the vault file has no answer and the manager has a log.

Rotation is the other half. With a manager, you change the secret once at the source and every consumer fetches the new value on its next lookup — no commit, no coordinated update across repos. With a vault file, rotation is a commit to every repo that holds the secret, and a stale copy lingers anywhere that has not pulled. The manager turns rotation from a distributed change into a single central one.

Vault Files vs an External Secret Manager

Ansible Vault files — encrypt secrets into git, need only ansible-core, and are the right call for a small fleet with a handful of static secrets and no audit requirement. The secret travels with the repo as ciphertext; rotation is a commit and reads leave no trace.

An external manager — HashiCorp Vault, AWS or GCP Secret Manager — keeps secrets out of the repo with per-read audit, centralized rotation, and dynamic short-lived credentials, and is worth its operational weight once you need rotation, an access log, or credentials that expire. Stay on vault files until rotation or audit is a real requirement, then move.

Common Mistakes

Reaching for HashiCorp Vault on day one for two static passwords, taking on a Vault server to operate when a vault.yml would have done the whole job.
Calling a hashi_vault lookup and assuming the value is cached — lookups run every time they are referenced, so a secret read in a loop hits the manager once per iteration unless you fetch it once into a variable.
Forgetting the control node still needs a credential to reach the manager — a Vault token, an AWS role — and storing that in plaintext; you have moved the secret, not removed the bootstrap-credential problem.
Using a static lookup against a manager that issues dynamic secrets and then wondering why the credential stopped working an hour later — match the lookup to whether the secret is static or short-lived.
Logging a fetched secret through a debug or a missing no_log, so moving the value out of the repo buys nothing because the run prints it anyway.

Best Practices

Default to Ansible Vault files and move to an external manager when rotation, an audit log, or dynamic credentials become a real requirement, not before.
Fetch a secret once into a variable rather than calling the lookup repeatedly, so a loop does not hammer the manager and inflate its audit log.
Govern reads with the manager's own access control — Vault policies, IAM on Secret Manager — so "who can read vault_db_password" is enforced centrally, not by who has the repo.
Use dynamic, short-lived credentials from HashiCorp Vault for the database where the manager supports it, so a leaked credential expires on its own instead of needing a rotation commit.
Protect the bootstrap credential the control node uses to reach the manager with the same care as any secret, since it unlocks all the others.

Comparable tools HashiCorp Vault via community.hashi_vault, tokens, policies, dynamic secrets AWS Secrets Manager · GCP Secret Manager via amazon.aws.aws_secret and google.cloud.gcp_secret_manager Azure Key Vault the third cloud-native manager SOPS-with-KMS the file-plus-cloud-key middle ground · CyberArk · 1Password Connect same external-manager role

Knowledge Check

What does an external secret manager give you that a checked-in vault file cannot?

Per-read audit logging, centralized rotation, and dynamic short-lived credentials
Stronger encryption than AES-256 on the very same blob still checked into the repo
The ability to run Ansible with no credential of any kind on the control node
Automatic no_log applied for you on every task that touches the fetched secret

A lookup is referenced inside a loop that runs fifty times. What happens, and how do you avoid it?

The lookup runs every iteration, hitting the manager fifty times; fetch the secret once into a variable and reference that
Ansible caches the first lookup result in the play's fact namespace automatically, so the manager is hit exactly once on the first iteration regardless of how high the loop count climbs
The loop fails right after the first iteration, because a lookup can only be evaluated once per play
The manager rate-limits Ansible to one read per second, slowing the loop but not duplicating reads

An external manager keeps secrets out of the repo. What problem does it leave behind?

The control node still needs a bootstrap credential — a Vault token or cloud role — to reach the manager, and that credential must itself be protected
The encrypted vault file must still be committed beside the lookup as a mandatory offline fallback that Ansible reads from automatically whenever the manager is down or unreachable during a run
Lookups cannot run unattended in CI, so a human has to approve every single secret fetch by hand
The manager re-introduces a local state file on the control node that records every secret it reads

When does a dynamic short-lived secret beat a static one?

For a high-value credential like a production database, where a per-run credential that expires in an hour limits the damage of a leak to minutes
Always — a static secret is simply never the right choice once an external manager is available
Only for read-only secrets that grant nothing but SELECT-style access, since the manager cannot mint a dynamic credential carrying write, admin, or schema-altering privileges on the target
When the secret must be committed to git anyway, because dynamic secrets compress better in the blob

You got correct