Topic 19

Secrets Management

Secrets

Applications need secrets — database passwords, API keys, signing keys, TLS certificates — and where those secrets live decides how bad a code leak or a stolen laptop turns out to be. A secrets manager stores them centrally, hands them out on authenticated request, rotates them, and audits access, replacing the anti-pattern of secrets in code, config files, and environment variables.

This topic builds Meridian's secret-handling and connects it to mutual TLS between its services — the endgame where app and api authenticate each other cryptographically instead of trusting the network.

Why Not Env Vars and Config Files

Secrets in a .env file, baked into an image, or committed to git are the single most common credential leak. They get copied, printed to logs, captured in crash dumps, and pushed to repositories — and they do not rotate. Committing a secret once and deleting it later does not help: it is in the git history forever, so the only fix is to rotate it. A real secret store removes secrets from all those leaky places.

What a Secrets Manager Does

A secrets manager provides central encrypted storage, authenticated and authorized access per service, automatic rotation, and an audit log of who read what and when. The application fetches a secret at runtime instead of carrying it, so the secret never lives in the code or the image — and access can be scoped so one service's compromise does not expose every other service's secrets.

Dynamic, Short-Lived Secrets

The strongest pattern is short-lived, on-demand credentials: a database password minted for this service for the next hour, then automatically expired. A secret that is valid for an hour is nearly worthless the moment after it leaks, which collapses the window an attacker has to use a stolen credential. Dynamic secrets turn credential theft from a lasting problem into a fleeting one.

Envelope Encryption and mTLS Identity

A key-management service holds master keys that never leave it and wraps the data keys applications use — envelope encryption, the model behind cloud encryption and Chapter 2's key-handling advice. The same idea gives services cryptographic identity: with mutual TLS, app and api each present a short-lived certificate and authenticate each other, so a service is trusted for who it cryptographically is, not for where on the network it sits.

Common Mistakes
  • Storing secrets in environment variables, config files, or the image — they leak through logs, crash dumps, docker history, and git, and they never rotate.
  • Committing a secret to git and thinking a later deletion fixed it — it is in history and must be rotated, not just removed.
  • Never rotating long-lived credentials, so a leak from years ago still works.
  • Granting every service access to every secret instead of scoping access per service, turning one compromise into all secrets.
  • Letting master keys leave the KMS, so the thing protecting every data key becomes stealable.
Best Practices
  • Keep secrets in a dedicated manager (Vault, cloud secret managers) and fetch them at runtime with authenticated, per-service access.
  • Prefer short-lived, dynamic secrets and automatic rotation so leaks expire quickly.
  • Use a KMS with envelope encryption for data keys, and never let master keys leave the KMS.
  • Give services cryptographic identities (mTLS with short-lived certs) instead of trusting network position, and audit every secret access.
  • Run a secret scanner in CI and pre-commit so a secret never reaches the repository (Chapter 12).
Comparable toolsManagers HashiCorp Vault · AWS/GCP/Azure secret managersKeys KMS / HSM · envelope encryptionService identity SPIFFE/SPIRE · service mesh mTLSEncrypted config SOPS · git-crypt

Knowledge Check

You accidentally committed an API key to git and deleted it in the next commit. What must you do?

  • Rotate the key, because it remains in the repository's git history
  • Nothing — the later deletion removed it completely
  • Rename the key variable so it no longer matches automated secret scanners
  • Make the repository private and consider it resolved

What makes dynamic, short-lived secrets stronger than long-lived ones?

  • A leaked credential expires quickly
  • They are encrypted while long-lived secrets are not
  • They never need to be stored in a secrets manager
  • They remove the need to authenticate the requesting service

In envelope encryption, why does the master key never leave the KMS?

  • The KMS wraps data keys internally, so the master key that protects everything is never exposed to be stolen
  • The master key is too large to transmit over the network
  • Applications encrypt faster when they hold the master key directly
  • The master key changes on every request, so it cannot be sent

You got correct