Secrets

The Payrail key was in settings.py, which was in git, which was on every developer's laptop, in every clone the CI runner had ever made, and in the CI log of every failed build, because the failure handler printed the environment to help with debugging. Nobody had done anything unusual. A secret is a value whose exposure is an incident, and it needs a different path through the system from ordinary configuration: it is never in the repository or the image, it reaches the process at startup from a store built for holding it, it never appears in a log line, a trace attribute or a stack trace, and it can be rotated without a deploy of new code. Stagedoor has seven such values, and the first version handled all seven the same way it handled the pool size.

Chapter 4 typed the config object and gave the key a SecretStr field, which was the first step. This topic is the rest of the path: what qualifies, where each of the seven lives, how it gets from the store into the process, the four places it leaks from and how each is closed, and the rotation procedure that has to exist before the day it is needed at speed.

What Counts

The test is one question: would this value in a public repository be an incident? The database host is not a secret; knowing pg-primary exists gets an attacker nothing without the password. The pool size, the hold duration, the Payrail URL: configuration, visible in any deployment record, harmless. The password for stagedoor_app is a secret, because with it the attacker has every order and every email address. By that test Stagedoor has exactly seven, and the table names each one, what it protects, and which process holds it.

Seven rows, and the columns on the right are the last section's subject. The api holds all seven because it is the process that authenticates, charges, verifies webhooks and signs tokens. The worker holds three: it reads and writes the database, it consumes the stream, and it issues refunds through Payrail, which is why it has the key and not the webhook secret. The migrator holds one, the database password, because a migration needs the database and nothing else. Nothing else in the config qualifies. Four of the seven are carried by the config object's fields, the two database URLs sharing one password, and the three secrets that live outside the config object, the pepper, the signing key and the cookie-signing key, are loaded by the same mechanism into the components that use them.

Never in the Repository or the Image

A secret committed to git is compromised from that commit onwards, and removing it in the next commit changes nothing, because the history keeps both. Every clone has it. The CI runner's cache has it. A fork made last year has it. Rewriting history with a force-push removes it from one remote and from none of the clones, and hosted platforms keep the reachable objects of a rewritten branch for a while besides. The only fix for a committed secret is rotation: treat it as leaked, issue a new one, revoke the old. Stagedoor rotated the Payrail key the afternoon Marek found settings.py in the history, and the rewrite of the history was done afterwards, for tidiness, not as the fix.

The prevention is two mechanisms. The developer's .env is in .gitignore from the first commit, and it holds sandbox values that work nowhere else. And a secret scanner runs in CI and as a pre-commit hook, matching on the shapes secrets have: Payrail's keys start pr_live_, Stagedoor's own API keys start sd_live_ as Topic 28 of Chapter 5 arranged precisely so that a scanner can find them, and a private key begins with a line that says so. The hosted platform's push protection does the same on the server side and refuses the push. Neither catches a secret that has no recognizable shape, which is one reason the pepper and the cookie-signing key are generated as long random strings with a prefix of their own. The image is the second repository: a COPY .env line or a RUN with a build argument leaves the value in a layer, docker history prints it, and every registry that mirrors the image holds it. The image contains code and dependencies and no values at all.

Delivered at Start

The platform holds the secrets in a store built for it and injects them into the process at start, either as environment variables or as files in a mounted directory. On Kubernetes that is a Secret object, backed by the cloud's secret manager through a sync operator or a CSI driver so that the value of record lives in the manager and the cluster holds a copy; on Cloud Run it is the manager mounted directly; on a VM under systemd it is a credentials directory the service manager populates. The alternative is for build_service to call the manager's API itself at startup, authenticated by the workload's own identity rather than by yet another key, and that is the right shape when the platform offers no injection. Either way the process reads each value exactly once, at startup, into the config object, and a missing one stops the process before the port is bound, as Topic 23 already arranged for every required field.

class Config(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",                 # the developer's shell only; gitignored
        secrets_dir="/run/secrets",      # one file per field, mounted by the platform
    )
    database_url: PostgresDsn            # the password is inside the URL
    redis_url: RedisDsn
    payrail_key: SecretStr               # /run/secrets/payrail_key, or PAYRAIL_KEY
    payrail_webhook_secret: SecretStr
    ...

cfg = Config()                           # a missing secret fails here, before the port is bound
payrail = PayrailClient(cfg.payrail_url, cfg.payrail_key.get_secret_value())
log.info("config loaded", payrail_key=cfg.payrail_key)   # logs '**********'

The settings model reads a field from three places in order: an environment variable, the developer's .env file, or a file named after the field in the secrets directory. Production mounts the directory; the laptop uses the file; the code is the same. Each secret is typed as SecretStr, whose string form is ten asterisks, so the startup line that logs the loaded config prints ten asterisks for it, and the one place the real value is needed, the Payrail client's constructor, asks for it by name with a method call that reads as a deliberate act in a code review. A process started without the webhook secret stops on the first line of build_service with a message naming the field, which is the same failure mode as any other missing config and the right one: the previous version keeps serving, and nobody discovers the missing secret from a webhook that failed verification an hour later.

Never in a Log

The logger of Chapter 13 has a redaction processor that runs on every line before it is written. It strips the Authorization and Set-Cookie headers wherever a request or response is logged, any field whose name contains key, secret, token or password, any string value matching a known prefix such as pr_live_ or sd_live_, and every field whose type is SecretStr, which handles itself. The demonstrated line is what a Payrail charge looks like in the log: the order, the amount, the status, the duration and Payrail's reference, and no key anywhere, because the key was never passed to the logger in the first place and would have been masked if it had been.

{"ts": "2026-09-18T19:41:07.412Z", "level": "info", "event": "payrail.charge",
 "request_id": "req_01J8...", "order": "3f9e...-...-a1c2", "amount_cents": 9000,
 "status": 201, "duration_ms": 412, "payrail_ref": "ch_7Kd...", "key_id": "pr_live_...c4e1"}

The line records which key was used by its prefix and last four characters, which is what the dashboard needs when two keys are valid during a rotation, and never the key itself. The leak this section exists for is not the deliberate log line; it is the stack trace. Python's plain traceback prints no local variables, but the pretty tracebacks that developers switch on for readability print every local in every frame, and the error tracker's SDK captures locals by default, so the frame that was about to call Payrail sends the key to a third-party dashboard on the first unhandled exception. Stagedoor's exception handler of Topic 15 prints the type and the frames and no locals, the tracker's local-variable capture is switched off, and its scrubber runs the same field-name list as the logger. A crash dump that prints the environment, which some process supervisors do on a fatal signal, is the same leak from below, and it is one of the two reasons the comparison box below prefers mounted files.

Rotation

Every secret has an owner and a written rotation procedure, and the procedure has been run at least once when nothing was wrong, because a secret that has never been rotated cannot be rotated in an emergency either. Marek learned this on the afternoon of the settings.py discovery: rotating the Payrail key took four hours, most of it finding out which processes held it and whether the worker's refund path would break, and the same rotation takes eight minutes now that it is a runbook that has been rehearsed. Each of the seven has a different shape, because the overlap between old and new is arranged differently for each.

The database password is checked by Postgres only when a connection is opened, so the pool's 20 open connections keep working after ALTER ROLE stagedoor_app PASSWORD and only a new connection needs the new value. The procedure is: write the new password to the store, change the role, and restart the api and the worker in a rolling fashion within minutes, so that each process reconnects with the new value, with the migrator picking up the store's value on its next run; the exposure is a pool that has to reopen a broken connection during those minutes, which the runbook accepts and the alert watches for. The stricter shape, two application roles with identical grants alternating between rotations, is what the cloud secret managers implement for managed databases, and Stagedoor will adopt it when the minute-long window matters. Payrail's API key rotates as Topic 28's API keys do: a new key issued, both valid, the store updated, the processes restarted, the old key revoked once the dashboard shows no calls on it. The webhook secret is the one Payrail signs with, and Payrail sends both signatures during its overlap, so the verifier of Topic 54 accepts either of two secrets for the window. The Ed25519 key rotates every 90 days under its kid, and the previous public key stays in the verifier until every 15-minute token it signed has expired, as Chapter 5 arranged. The pepper is versioned in the hash string and rotated by rehashing at login. The cookie-signing key has a ten-minute overlap in which both keys verify and only the new one signs.

Least Exposure

The table's right-hand columns are a policy: each process holds only the secrets it uses, and a process that holds a secret it does not use is a process whose compromise is larger than it needed to be. The worker issues refunds and therefore holds the Payrail key, which is a genuine need and is written down as one; it never verifies a webhook, never signs a token and never hashes a password, so it holds none of those four. The migrator is the sharpest case: it holds the database password and nothing that could charge a card or forge a login. A compromised migrator job can damage the database and nothing else.

The policy is enforced where the secrets are attached, not in the code. The worker's deployment mounts three files into its secrets directory and the api's mounts seven, and a worker that tried to read the webhook secret would fail at startup on a missing field, which is the correct failure. The same rule reaches the humans: the store's access list names the two people who can read the production Payrail key, and the CI runner is not one of them, because CI builds the image and the image contains no values. Every access to a production secret is a log line in the store, and the reconciliation of who read what is a monthly review that takes ten minutes and has found one stale grant so far.