BuildKit — The Modern Builder

BuildKit is the engine that turns a Dockerfile into an image, and on current Docker it is the default — the old line-by-line builder is gone unless you deliberately ask for it back. Instead of marching through the file top to bottom, it parses the whole Dockerfile into a dependency graph, runs independent steps at the same time, and caches each step by the content of its inputs rather than by its position in the file.

Everything the rest of this chapter relies on exists because BuildKit is doing the work: --mount=type=secret, --mount=type=cache, and parallel multi-stage builds are all features the legacy builder never had. If you have written a Dockerfile in the last few years, BuildKit almost certainly built it, whether you invoked docker build or docker buildx build.

The Legacy Builder vs BuildKit

The old builder executed instructions strictly top to bottom, one throwaway container per step, with a linear cache keyed on instruction order. Change line 4 and every line below it rebuilt, regardless of whether line 9 actually depended on line 4. BuildKit instead builds a graph of the file, works out which steps depend on which, and runs unrelated branches concurrently — which is precisely why a multi-stage build with two independent stages builds both at the same time instead of one after the other.

The cache changes with it. BuildKit keys each step on the actual content of its inputs, so reordering an instruction that nothing below depends on no longer invalidates the rest of the file. The full cache mechanics land in topic 30 with the cache mount; the point here is that "instruction order is everything" was true of the old builder and is only partly true of BuildKit.

How It's Enabled

On current Docker Desktop and Docker Engine, BuildKit is the default builder — you get it from a plain docker build . with nothing extra to set. On an older daemon you opt in by exporting DOCKER_BUILDKIT=1 in front of the build, or by switching to docker buildx build, which is BuildKit's full-featured front door and the command the production chapters assume. There is no separate install on a recent Docker; the engine ships with BuildKit built in.

The # syntax Frontend Line

A Dockerfile whose first line is # syntax=docker/dockerfile:1 tells BuildKit to fetch that Dockerfile frontend at build time, so the build gets a current instruction set — including the --mount family — regardless of how old the daemon underneath is. This one comment is the prerequisite for cache mounts and build secrets later in the chapter; without it, an older daemon parses RUN --mount=type=secret as a syntax error and the build dies before it starts.

# syntax=docker/dockerfile:1
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "driftwood.wsgi"]

The directive pins the frontend — the parser that understands the Dockerfile — to the dockerfile:1 channel, which stays current with stable instruction additions. It does not pin the base image; that is FROM's job, and digest-pinning the base is topic 32's subject.

Parallelism and Smarter Caching

Because BuildKit knows the dependency graph, it parallelizes work that has no ordering relationship and caches by input content. Split genuinely independent setup into separate stages and BuildKit builds them concurrently rather than serializing a long RUN chain. The content cache also means an unrelated edit no longer cascades: touch a comment near the top and the steps below it stay cached as long as their own inputs are unchanged.

buildx as the Interface

docker buildx is the CLI plugin that exposes BuildKit's full surface: named builders, cache export and import, build secrets, and multi-platform output via --platform (previewed in topic 32, full treatment in Chapter 9). Plain docker build quietly uses BuildKit too, but buildx is where the advanced flags live, which is why the production chapters reach for docker buildx build as the default invocation.

For Driftwood, buildx is the command that carries the database password in as a build secret and keeps pip's wheel cache warm across builds. Both of those depend on the # syntax line and on BuildKit being the active builder — which, on a current install, it already is.