Debugging Containers

A container that won't stay up gives you more to work with than it looks. The exit code says how it died, docker logs says what it said on the way out, and docker inspect .State says whether the kernel killed it or it crashed on its own. Three artifacts, three different questions, and together they classify almost any failure before you touch the application code.

For a container that is running you exec a shell in and look around. For a distroless or scratch driftwood/web with no shell to exec (Chapter 2, topic 12), docker debug or an ephemeral debug container attaches the tools the image deliberately left out — so you keep the production image lean and still get a way in when it misbehaves.

Why It Exited — Start With the Exit Code

docker ps -a and docker inspect give the exit code, and the code narrows the cause before you read a single log line. The common ones each tell a story:

docker inspect --format '{{.State.ExitCode}} oom={{.State.OOMKilled}}' web

0 is a clean exit; a non-zero in the app's own range is the application's failure code; 137 is SIGKILL — usually the OOM killer or a docker stop that timed out, so cross-check .State.OOMKilled; 143 is SIGTERM (a clean stop signal); 139 is a segfault. The number alone often tells you whether to read the logs or the memory limit.

docker logs and .State

docker logs <container> replays the captured STDOUT and STDERR — driver permitting (topic 66) — and that output usually carries the stack trace or the fatal message. docker inspect .State adds the context the logs lack: Status, Error, OOMKilled, StartedAt and FinishedAt, and RestartCount. Read together, they distinguish "the app threw an exception" from "the kernel killed it" from "it never started at all" — three failures that can look identical from the outside.

Crash Loops

A container with restart: always or unless-stopped that keeps dying shows a climbing RestartCount and a Restarting status. The trouble is that the logs scroll past fast and the container is never sitting still when you look. The way to read a loop that won't hold still is to grab the previous run's docker logs plus docker events (topic 68) capturing the repeating die exit code — the failure reason survives each restart that way instead of vanishing.

exec Into a Running Container

docker exec -it <container> sh (or bash) opens a shell in a running container to inspect its filesystem, environment, and live processes:

docker exec -it web sh

It does nothing for a container that already exited — there is no process to attach to — and it depends on the image actually shipping a shell, which a slim or distroless image may not. When exec errors with "executable file not found," the image has no sh, and you need the next approach.

Distroless and No Shell — docker debug and Ephemeral Debug Containers

When driftwood/web is distroless or built FROM scratch (Chapter 2, topic 12), there is no sh to exec into — that is the point of those base images. docker debug <container> attaches a temporary toolbox image into the running container's namespaces, giving you a shell and utilities without rebuilding or bloating the production image. The older, runtime-agnostic pattern does the same by joining the target's namespaces directly:

docker run -it --rm \
  --pid=container:web \
  --network=container:web \
  nicolaka/netshoot sh

The debug container brings its own shell and tools but shares the target's process and network view, so you inspect the shell-less container's processes and connections from inside a container that does have a shell.

Failed Builds

A build that fails leaves the last successful layer cached, which is the fastest way back in. docker build shows which instruction failed and its output; BuildKit's --progress=plain un-collapses the log so you see the full output of the failing step rather than the folded summary. From there, running an interactive container from the last good intermediate image lets you reproduce the failing RUN step by hand — faster and more precise than editing the Dockerfile and rebuilding from scratch each time (ties to Chapters 4 and 5).