Chapter Eleven · The Service in Production
The Service in Production
Stagedoor's first production deploy dropped 340 checkouts, shipped the Payrail key in a committed file, told the balancer it was healthy while the database was unreachable, and let one PDF render take the host down. Five topics make the process a good citizen of whatever runs it: the four twelve-factor rules that still bite, the seven secrets and the four places they leak from, two probes and a 30-second drain, the four resource limits and where their numbers come from, and the checklist the container expects before Docker Deep Dive and Kubernetes Deep Dive take over.
Chapters 1 to 10 built a service that is correct under load. This chapter is the evening it first met a platform, and the platform found four things the code had never been asked about. The deploy script killed the old process the instant the new one was listening, and 340 requests that were between accept and response were discarded with their sockets, some of them after Payrail had charged. The Payrail key sat in settings.py, in git, on every laptop and in the CI log of every failed build. The health check returned 200 while the database was unreachable, so the balancer kept sending traffic to two instances that turned every request into a 500. And the worker's container had no memory limit, so a 3 GB render took the host down with the two api processes on it. None of the three spring wounds reopened; these four were new, and every one of them was a question the platform asks that the service had no answer to.
The answers are short and they are all the process's to give. Nothing in memory or on disk between requests that another instance would need, config from the environment, logs to stdout, a fast start and a clean stop: Topic 58. Secrets on a separate path from configuration, delivered at start, redacted everywhere, rotated by a runbook that has been rehearsed: Topic 59. /healthz that checks the process and /readyz that checks its dependencies, and on SIGTERM a six-step drain that finishes inside the 30 seconds before SIGKILL: Topic 60. A memory limit that kills, a CPU limit that throttles and a descriptor limit that errors on an unrelated line, each set from measured numbers: Topic 61. And the seven-item contract with a tick per process, a twelve-line image, and migrate as a job that gates the rollout: Topic 62.
The chapter stops where the container begins. It says what the process must provide and declare, and hands the image build, the registry, the pod, the rollout and the scheduler to the two courses that own them by name. A service that has met the checklist runs unchanged on Kubernetes, on Cloud Run, under systemd on a VM and under Docker on Marek's laptop, and Stagedoor runs on all four.