Pipes and Redirection

Every process starts life with three open file descriptors: standard input (fd 0), standard output (fd 1), and standard error (fd 2). A program reads from fd 0, writes its results to fd 1, and writes diagnostics to fd 2. The kernel attaches those descriptors to whatever the shell points them at — a terminal, a file, or another process — and the program neither knows nor cares which.

Redirection and pipes are how the shell rewires those three descriptors before the command ever runs. That wiring is the whole foundation of the Unix tool model: each command does one thing, reads stdin, writes stdout, and you compose them. Because stdout and stderr are separate descriptors, you can capture a command's results in a file while still seeing its errors on screen — but only if you get the ordering right, which is where most redirection bugs live.

The Three Standard Streams

A file descriptor is a small non-negative integer that indexes the kernel's per-process table of open files. When a shell launches a command it guarantees descriptors 0, 1, and 2 are open. On an interactive terminal all three point at the same device — the controlling TTY — which is why typed input, normal output, and error messages all share one window.

The streams are also reachable by path through /dev/stdin, /dev/stdout, and /dev/stderr, which the kernel maps back to descriptors 0, 1, and 2. Higher descriptors exist too: a shell or script can open fd 3 and above for extra channels, which is how a script keeps a dedicated log file open alongside its normal output.

# inspect the descriptors the shell handed this process
ls -l /proc/self/fd
# 0 -> /dev/pts/0   (stdin, the terminal)
# 1 -> /dev/pts/0   (stdout, the terminal)
# 2 -> /dev/pts/0   (stderr, the terminal)

Redirection and Truncation versus Append

Redirection points a descriptor at a file instead of the terminal. > sends stdout to a file, truncating it to zero length first; >> appends instead of truncating; < feeds a file into stdin. A descriptor number sits immediately before the operator with no space, so 1> is the same as > and 0< is the same as <.

The truncate-versus-append distinction is the one to internalize. echo line > log replaces the entire file on every run; echo line >> log adds a line and keeps the rest. A common data-loss event is running cmd > file inside a loop expecting the output to accumulate, then finding only the last iteration. On Debian and RHEL alike, bash also offers noclobber (set -C), which makes a bare > refuse to overwrite an existing file; >| overrides it when you do mean to clobber.

sort access.log > sorted.log      # truncate, then write stdout
date >> sorted.log            # append, keep existing contents
wc -l < sorted.log            # feed the file in as stdin

Redirecting Standard Error

A bare > moves only stdout; stderr still reaches the terminal. To redirect errors, name descriptor 2 explicitly — 2> truncates, 2>> appends. To send both streams to one place, duplicate one descriptor onto the other with 2>&1, which means "make fd 2 point wherever fd 1 currently points".

Order is the trap, because redirections apply left to right and 2>&1 copies the current target of fd 1 at the moment it is read. In cmd >file 2>&1 stdout is sent to the file first, then stderr is pointed at the same place — both land in the file. In cmd 2>&1 >file stderr is pointed at the terminal, where fd 1 still points, and only afterward is stdout moved to the file — so errors stay on screen. The bash shorthand &>file redirects both unambiguously, but it is a bashism — not POSIX — so a #!/bin/sh script under dash needs the explicit >file 2>&1 form.

# both stdout and stderr into build.log
make >build.log 2>&1
# WRONG: errors still hit the terminal; only stdout is captured
make 2>&1 >build.log
# bash shorthand: both streams to the file
make &>build.log
# discard stderr, keep stdout on screen
make 2>/dev/null

Pipes, Pipelines, and SIGPIPE

A pipe, written |, connects the stdout of one command to the stdin of the next through an in-kernel buffer (64 KB by default on Linux). Each stage of a pipeline runs as a separate process and they run concurrently — the reader consumes bytes as the writer produces them, so a pipeline streams rather than staging the whole output in memory or on disk. A pipe carries only stdout; each stage's stderr still goes to the terminal unless you redirect it.

When the reader exits early, the kernel sends SIGPIPE to the writer on its next write to the closed pipe. This is why producer | head -n 5 stops the producer shortly after the fifth line: head closes the pipe, the producer's next write triggers SIGPIPE, and the default action terminates it. Because each stage is its own process, a pipeline runs in subshells — variables set inside cmd | while read are lost when the pipeline ends, since the loop body never ran in the current shell.

# three concurrent processes; data streams through the buffer
journalctl -u nginx | grep ' 500 ' | wc -l
# head closes the pipe early -> producer gets SIGPIPE and stops
seq 1 100000000 | head -n 5
# tee: write to a file AND pass the stream on
dmesg | tee /tmp/boot.log | grep -i error

Pipeline Exit Status and pipefail

By default a pipeline's exit status is the status of the last command only. grep pattern bigfile | head reports success whenever head succeeds, even if grep matched nothing or bigfile never existed. Scripts that test $? after a pipeline silently miss failures in every stage but the last.

Bash offers two fixes. set -o pipefail makes the pipeline return the status of the rightmost command that exited non-zero, so an upstream failure propagates. The PIPESTATUS array holds each stage's individual exit status, letting a script pinpoint which stage failed. PIPESTATUS is a bashism dash does not implement, and the dash shipped in Debian 12 and Ubuntu 24.04 has no pipefail either — its set builtin rejects the option. Dash gained pipefail only in a later revision (Debian 13 "trixie"), so a #!/bin/sh script that needs either still wants a #!/bin/bash shebang.

# without pipefail: status is head's (0) even though grep failed
set -o pipefail
grep missing bigfile | head || echo "a stage failed"
# per-stage status: (1 0) means grep failed, head succeeded
grep missing bigfile | head
echo "${PIPESTATUS[@]}"

Here-docs, Here-strings, and Process Substitution

A here-document feeds a literal block into stdin. <<EOF reads lines until one containing exactly the delimiter; <<-EOF additionally strips leading tab characters so the block can be indented. Quoting the delimiter (<<'EOF') disables variable and command expansion inside the body — essential when the block contains literal $ signs. A here-string, <<<, feeds a single string into stdin without a temporary file.

Process substitution is a bashism that turns a command's output into a filename. <(cmd) exposes cmd's stdout as a path — a /dev/fd entry — that another command can open for reading; >(cmd) does the reverse for writing. It beats a temporary file for tools that demand filename arguments rather than reading stdin, such as diff comparing the output of two commands: no temp file to create, name, secure, or clean up. It also escapes the subshell trap, so while read x; do ...; done < <(cmd) keeps the variables the loop sets.

# here-doc: literal block to a config file, no expansion
cat <<'EOF' > /etc/motd
Maintenance window: Sunday 02:00 UTC
EOF
# here-string: one line straight into stdin
grep root <<< "$(getent passwd)"
# process substitution: diff two commands, no temp files
diff <(ssh web01 dpkg -l) <(ssh web02 dpkg -l)