awk

awk is a small programming language built for column- and record-oriented text. It reads input one record at a time — by default one line — splits each record into fields on whitespace, and runs your program against every record in turn. The whole language is a list of pattern { action } rules: for each record, awk tests the pattern, and where it matches, runs the action. That model is what makes a one-liner like awk '$5 > 100 { print $3 }' — "print the third column of every line whose fifth column exceeds 100" — a single expression instead of a script.

Every Debian and Ubuntu install ships an awk at /usr/bin/awk; on those systems it is usually mawk, a fast, lean implementation, while the fuller-featured GNU version is gawk (installed with apt install gawk, and the default on RHEL/Fedora). For the field-splitting, arithmetic, and counting that this page covers, mawk and gawk behave identically — the divergence only shows up in gawk extensions like gensub(), true multidimensional arrays, and FIELDWIDTHS. The practical consequence: reach for awk the moment a problem is about columns of variable-width data, where cut breaks and a shell loop would be ten times slower.

The Pattern–Action Model

An awk program is a sequence of rules, each of the form pattern { action }. Both halves are optional. A rule with only a pattern and no action uses the default action, which is print $0 — print the whole record. A rule with only an action and no pattern runs that action on every record. So awk '/error/' prints lines containing "error" (pattern only, default print), and awk '{ print $1 }' prints the first field of every line (action only, no pattern). This is why awk so often replaces grep | cut chains: the pattern filters and the action projects, in one pass.

Patterns are not limited to regular expressions. A pattern can be any expression that evaluates to true or false — $3 == "DOWN", NF > 5, NR % 2 == 0 — or a range written as /start/,/stop/ that switches on at the first match and off at the second. The action is a block of statements in a C-like syntax: assignments, if/else, for, while, print, and printf. Awk runs the entire program once per record, top to bottom, so multiple rules compose naturally against the same input stream.

Fields, Records, and Separators

Awk splits each record into fields and numbers them from $1; $0 is the entire record, $NF is the last field, and $(NF-1) is the second-to-last. Two built-in variables track position: NR is the running record number across all input, and NF is the field count of the current record. The default field separator is not a single space — it is "runs of whitespace, with leading and trailing whitespace ignored", which is exactly why awk handles columns aligned with multiple spaces or tabs that cut -d' ' mangles.

Four variables control splitting and output. FS (or the -F flag) sets the input field separator; RS sets the record separator; OFS sets the separator awk inserts between fields when you print them; and ORS sets what it appends after each record. For comma-delimited input you set -F,; to read paragraph-mode blocks you set RS="". The catch with OFS trips up nearly everyone: assigning it changes the output separator, but awk only rebuilds $0 with that separator if you assign to a field first.

# Count active SSH sessions per source IP from `who`-style output
who | awk '{ print $5 }' | sort | uniq -c

# Print user (col 1) and RSS in MB (col 3, in KB) where RSS > 100 MB
ps -eo user,pid,rss,comm | awk 'NR>1 && $3 > 102400 { printf "%-12s %6.1f MB  %s\n", $1, $3/1024, $4 }'

# Force OFS to take effect by touching a field ($1=$1)
awk -F: 'BEGIN{OFS="\t"} {$1=$1; print $1, $3}' /etc/passwd

BEGIN and END Blocks

Two special patterns bracket the main loop. A BEGIN { } block runs once before any input is read; an END { } block runs once after the last record. BEGIN is where you set FS, OFS, and counters, or print a header row. END is where you emit totals, averages, or the contents of arrays you accumulated. A program can have any number of each, plus the ordinary per-record rules in between, and awk runs them in the right order automatically.

This three-phase shape — initialize, process each record, summarize — is the structural reason awk replaces a whole class of shell scripts. Summing a column is the canonical example: awk '{ s += $1 } END { print s }' needs no array, no temporary file, and no second pass. The accumulator lives across records because awk variables persist for the life of the program, not just one record.

# Sum and average a numeric column, with a header and a footer
awk 'BEGIN { print "bytes" }
     { total += $1; n++ }
     END   { printf "sum=%d avg=%.1f over %d rows\n", total, total/n, n }' sizes.txt

Variables, Arithmetic, and Formatting

Awk variables are untyped: a value is a string, a number, or both, and awk decides which interpretation to use from context. In a numeric context a string like "100" becomes the number 100, and a string with no leading digits becomes 0. That dynamic typing is convenient until it surprises you in comparisons. $1 > $2 compares numerically if both fields look like numbers, but lexically — string order, where "10" sorts before "9" — if either side is non-numeric. Forcing a numeric comparison is a matter of adding zero ($1+0 > $2+0); forcing a string comparison is concatenating an empty string ($1"" == $2"").

For output, print is the blunt instrument and printf is the precise one. print joins its arguments with OFS and appends ORS; printf takes a C-style format string and emits exactly what you specify — field widths, decimal places, zero-padding, no automatic newline. When a report needs columns to line up or numbers to show one decimal place, use printf "%-20s %8.2f\n" and control every character, rather than fighting print and OFS.

Associative Arrays

The feature that elevates awk from a field-printer to a real data tool is the associative array: a hash map whose keys are arbitrary strings. You never declare or size one — referencing count[$1]++ creates the entry on first touch and increments it. This single idiom does grouping and counting in one pass over the data, with no prior sort, which is precisely the work that otherwise needs sort | uniq -c plus a second sort.

In an END block you iterate the array with for (key in arr) to print every group and its tally. The pattern generalizes far past counting: sum bytes per IP, track the max latency per endpoint, collect the set of users seen per host. The one gotcha is iteration order — for (key in arr) visits keys in an unspecified order in mawk and standard gawk, so pipe the output through sort when you need it ordered, or use gawk's PROCINFO["sorted_in"].

# Top talkers: count requests per client IP in an access log, show the top 10
awk '{ hits[$1]++ } END { for (ip in hits) print hits[ip], ip }' access.log \
  | sort -rn | head -10

# Total response bytes per HTTP status code (status in $9, bytes in $10)
awk '{ bytes[$9] += $10 } END { for (s in bytes) printf "%s\t%d\n", s, bytes[s] }' access.log