Chapter 12: The Ecosystem
Topic 77

The Production Container Workflow

CapstonePipeline

This is the capstone: every chapter assembled into one narrative for Driftwood, from a Dockerfile to the line where Kubernetes takes over. Nothing here is new — it is the whole book seen as a single pipeline, the production path stated end to end so the pieces stop being separate lessons and become one workflow.

Follow driftwood/web from source to a hardened running container, and you have used everything this course taught. Build a good image, build it with BuildKit and then scan and sign it, tag and push it multi-arch to the private registry, and run it hardened on the host — then hand the exact artifact to the orchestrator when one host is no longer enough.

The whole book as one pipeline
multi-stage build
scan
sign
tag (semver + digest)
push multi-arch
run hardened

Build a Good Image — Dockerfile First

The artifact is small, hardened, and reproducible before it is ever pushed. A multi-stage build keeps the toolchain out of the final image: a builder stage compiles wheels, and the final stage is a slim or distroless base running as a non-root app user, with a HEALTHCHECK and a tight .dockerignore so the build context never drags in secrets or junk.

The Driftwood Dockerfile — multi-stage, non-root, healthchecked
# --- builder stage: the toolchain lives and dies here ---
FROM python:3.12-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip wheel --wheel-dir /wheels -r requirements.txt

# --- final stage: slim, non-root, no build tools ---
FROM python:3.12-slim
RUN useradd --system --uid 10001 app
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-index --find-links=/wheels /wheels/* && rm -rf /wheels
COPY --chown=app:app . .
USER app
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s CMD ["python", "healthcheck.py"]
ENTRYPOINT ["python", "-m", "driftwood.web"]

Every line here is something an earlier chapter argued for: the builder/final split keeps wheels and compilers out of what ships, the useradd plus USER app means the process never runs as root, and the exec-form ENTRYPOINT makes the app PID 1 so it receives SIGTERM cleanly on stop.

Build with BuildKit, Then Scan and Sign

BuildKit drives the build with cache mounts for fast dependency installs and build secrets via --secret so nothing sensitive ever lands in a layer — never ARG, which leaks into image history. CI then scans the image for CVEs and signs it, so the artifact is verified before it is allowed near distribution.

Build, scan, sign — the CI gates before anything is pushed
# build with BuildKit, passing a secret that never enters a layer
$ DOCKER_BUILDKIT=1 docker build \
    --secret id=pip_token,src=./pip_token.txt \
    -t driftwood/web:1.4.0 .

# scan for known CVEs; fail the pipeline on high/critical
$ trivy image --severity HIGH,CRITICAL --exit-code 1 driftwood/web:1.4.0

# sign the image so its provenance is provable
$ cosign sign --key cosign.key driftwood/web:1.4.0

The scan and the signature are gates, not formalities: a failing CVE scan stops the pipeline, and an unsigned image is rejected downstream. The one time a base-image CVE or a tampered layer would have shipped, these are the steps that catch it.

Tag, Multi-Arch, and Push

Tag by semantic versiondriftwood/web:1.4.0, never just :latest — and pin deployments by digest so what ran is always provable. docker buildx builds multi-arch for linux/amd64 and linux/arm64 as one manifest list and pushes it to the private registry registry.driftwood.example over the OCI distribution-spec API from topic 72.

One multi-arch manifest, pushed to the private registry
$ docker buildx build \
    --platform linux/amd64,linux/arm64 \
    -t registry.driftwood.example/driftwood/web:1.4.0 \
    --push .

# resolve the tag to a digest and deploy by that digest
$ docker buildx imagetools inspect \
    registry.driftwood.example/driftwood/web:1.4.0

A single tag now runs on both amd64 and arm64 nodes with no per-arch tags, and the digest the inspect command prints is what you pin in the run command — so a redeploy can never silently pull different code than the rest of the fleet.

Run Hardened on the Host

The run command is half the hardening — a non-root Dockerfile undone by a wide-open docker run is no protection at all. Drop all capabilities and add back only what is needed, mount the root filesystem read-only, deliver runtime secrets as files, set resource limits on memory and CPU, let the HEALTHCHECK drive the restart policy, and ship logs through a logging driver.

Running the hardened image with full single-host controls
$ docker run -d \
    --name driftwood-web \
    --read-only \
    --cap-drop ALL \
    --cap-add NET_BIND_SERVICE \
    --memory 512m --cpus 1.0 \
    --restart unless-stopped \
    --log-driver json-file --log-opt max-size=10m \
    -v /run/secrets/db_password:/run/secrets/db_password:ro \
    -p 8080:8080 \
    registry.driftwood.example/driftwood/web@sha256:<digest>

Each flag traces to a chapter: --cap-drop ALL and the single added capability are the least-privilege rule, --read-only closes off filesystem tampering, --memory and --cpus are the cgroup limits from Chapter 1, and pinning by @sha256: digest is the provability rule. Build-time and run-time hardening match — that is the whole discipline in one command.

Where Kubernetes Takes Over

That exact signed, multi-arch driftwood/web image, pulled from registry.driftwood.example, is the unchanged input the Kubernetes Deep Dive consumes when one host is no longer enough. The build-and-run job ends here; the orchestrate-across-many job begins there. Docker builds and runs containers on one host; Kubernetes orchestrates them across many — and it runs this same artifact, unchanged.

That is the end of the road this book covers. You can build a hardened, slim, signed image and run it under full single-host controls — and when Driftwood outgrows one machine, you hand that artifact across the boundary without rebuilding a thing. The sibling course picks it up from there.

Common Mistakes
  • Skipping the scan-and-sign steps under deadline pressure and pushing an unverified image — the one time a base-image CVE or a tampered layer ships, the missing gate is what you wish you had kept.
  • Tagging only :latest for a production push — the fleet pulls a moving pointer and a redeploy can silently run different code than the rest; tag by version and pin by digest.
  • Building single-arch and deploying to mixed amd64/arm64 nodes — containers fail to start on the unmatched architecture; one buildx multi-arch manifest avoids it.
  • Hardening the image but running it wide open — a non-root Dockerfile undone by docker run without --cap-drop, without a read-only rootfs, and with secrets passed as ENV; the run-time flags are half the hardening.
Best Practices
  • Build the image once through the full multi-stage, slim, non-root, healthchecked path and promote that exact artifact unchanged through scan, sign, push, and run.
  • Gate every push on a scan and a signature in CI so an unverified image cannot reach the registry, and pin deployed images by digest so what ran is always provable.
  • Publish a single multi-arch manifest with buildx so the same tag runs on amd64 and arm64 nodes without per-arch tags.
  • Pair the hardened Dockerfile with hardened run flags — --cap-drop ALL, read-only rootfs, file-based runtime secrets, resource limits, healthcheck-driven restart — so build-time and run-time hardening match.
Comparable tools Buildah · Podman · skopeo substitute for Docker build/run/push on the same OCI image trivy · grype · cosign the scan-and-sign gates the pipeline runs in CI containerd · Kubernetes the runtime and orchestrator the pipeline feeds across the boundary

Knowledge Check

What is the end-to-end order of the production workflow for driftwood/web?

  • Multi-stage build → BuildKit build → scan → sign → tag/digest → multi-arch push → hardened run → Kubernetes hand-off
  • Push the freshly built image straight to the private registry first, then scan it for CVEs and sign it only afterward
  • Build, push, and deploy the image straight away, then circle back to sign and harden it later if there is time
  • Hand off to Kubernetes first, and only then build, scan, and sign the image it will run

Which hardening lives in the Dockerfile versus the docker run command?

  • Non-root user, slim base, and HEALTHCHECK in the Dockerfile; cap-drop, read-only rootfs, limits, and file secrets at run time
  • All hardening, including cap-drop and the read-only rootfs, is baked into the Dockerfile itself, so the run command needs no extra flags
  • All hardening, even the non-root USER, is applied at run time through flags, so the Dockerfile stays minimal and bare
  • Resource limits go in the Dockerfile and the non-root user is set at run time with a flag instead

Why are scan, sign, and digest-pinning treated as non-negotiable gates?

  • The scan catches CVEs, the signature proves provenance, and the digest proves exactly what ran
  • They make the build run measurably faster by caching the already-verified image layers across runs
  • They let a single-arch image run unchanged on both amd64 and arm64 nodes without a rebuild
  • They shrink the final image so it pushes to and pulls from the private registry faster

Where does the single-host job end and orchestration begin in this pipeline?

  • At the hardened run — the same signed, multi-arch image is then handed unchanged to Kubernetes across many nodes
  • At a dedicated rebuild step where the image is recompiled into a cluster-native format before Kubernetes accepts it
  • At the CVE scan gate, after which Kubernetes takes over and drives the rest of the build and sign steps
  • At the registry push, which is the very last single-host step before the cluster picks the image up to run

You got correct