Topic 62

Packet Capture — tcpdump and Wireshark

Capture

When metrics say "it's slow" but not why, you capture packets. tcpdump grabs frames off the wire and filters them with BPF expressions before they ever hit disk; Wireshark takes that capture and dissects every byte into human-readable protocol detail. Packet capture is the ground truth of networking — it is the one tool that ends the argument about whether the problem is the network, the client, or the server, because it shows you exactly what crossed the wire and exactly what did not.

The discipline is knowing what to capture, where to capture it, and how to read the result without drowning. A capture on a busy 10 Gbps link with no filter will overrun the buffer and drop the very packets you needed; a capture at the wrong point will show clean traffic on one side of a loss you cannot see. Done well, a five-second trace settles a dispute that a week of log-staring could not. The three skills below are the whole game.

From wire to diagnosis — the packet-capture pipeline

capture
tcpdump

→

filter
BPF

→

dissect
Wireshark

→

read
the story

tcpdump and BPF Filters

tcpdump captures at the kernel level and applies a BPF filter — Berkeley Packet Filter — in the kernel before copying anything to userspace, so a tight filter is the difference between grabbing the 50 packets you care about and choking on a million you don't. The filter language is small and composable: host, net, port, tcp/udp, and the boolean operators and, or, not. The expression tcp port 443 and host 10.0.0.5 captures only that host's HTTPS traffic and lets everything else pass untouched.

Write the capture to a file with -w rather than printing it; printing decodes every packet inline and is slow enough to itself cause drops on a fast link. Capture to a .pcap file, confirm tcpdump reports 0 packets dropped by kernel when you stop it, and open the file in Wireshark for the heavy analysis. tcpdump is the scalpel for grabbing the right slice; it is not where you want to do the dissection.

# capture only this host's HTTPS, write to file, don't resolve names
tcpdump -i eth0 -n -w cap.pcap 'tcp port 443 and host 10.0.0.5'
# later, a quick headless read of the handshake + a reset
tcpdump -n -r cap.pcap
# 10.0.0.5.51234 > 93.184.216.34.443: Flags [S]      <- SYN
# 93.184.216.34.443 > 10.0.0.5.51234: Flags [S.]     <- SYN-ACK
# 10.0.0.5.51234 > 93.184.216.34.443: Flags [.]      <- ACK (handshake done)
# 93.184.216.34.443 > 10.0.0.5.51234: Flags [R]      <- RST (server slammed it)

Capture Points Matter

A capture only shows you what arrived at the point you captured. If packets are lost between client and server, a capture on the client shows them sent and a capture on the server shows them missing — neither side alone reveals the loss, but the two together pin it to the path in between. This is why "capture on both ends" is the rule for any drop you cannot localize: the asymmetry between the two traces is the diagnosis.

Where you tap also changes what you can see. Capturing on the host's own NIC is easy but shows traffic after the host's stack has already touched it; capturing on a mirrored switch port (a SPAN) shows the wire as the network sees it but can itself drop frames if the mirror is oversubscribed. A capture taken before a NAT shows original addresses; the same flow after the NAT shows rewritten ones. Decide what you need to prove, then choose the tap that can prove it.

Wireshark Analysis

Wireshark is where a raw .pcap becomes a story. Follow TCP Stream reassembles a scattered conversation into the bytes each side actually sent, in order. The Expert Information panel flags retransmissions, duplicate ACKs, zero-window events, and resets automatically, so you do not have to eyeball thousands of packets to find the three that matter. Display filters like tcp.analysis.retransmission or tcp.flags.reset==1 narrow a huge capture to just the symptom.

The real power is in the protocol dissectors. Wireshark knows the shape of TLS, HTTP, DNS, and hundreds of other protocols, so it can tell you that a handshake stalled after ClientHello, or that a DNS query went out and no response came back, without you decoding hex by hand. tcpdump captures fast and headless; Wireshark is where the time-consuming, click-heavy understanding happens — capture with one, analyze with the other.

Reading the Story in a Trace

A trace tells its story through a handful of recognizable patterns. A clean start is the three-way handshake — SYN, SYN-ACK, ACK — and its absence localizes the problem fast: a SYN with no SYN-ACK means the request never reached a listening server (firewall drop, wrong port, dead service), while a SYN immediately answered by a RST means the host is up but nothing is listening on that port.

Mid-stream, the tells are just as legible. A burst of retransmissions and duplicate ACKs means packets are being lost and TCP is resending — a network or congestion problem, not an application one. A zero window means the receiver's buffer is full and it has told the sender to stop, so the bottleneck is a slow consumer, not the link. A flood of RSTs means connections are being torn down abruptly, often by a firewall, a load balancer killing idle sessions, or an application crashing. Learn these four shapes and most captures read themselves.

tcpdump vs Wireshark

tcpdump is the fast, headless capture-and-filter tool. It runs over SSH on a box with no GUI, applies a BPF filter in the kernel so it can keep up on busy links, and writes a .pcap for later. Reach for it to grab exactly the right packets at the source with minimal overhead.

Wireshark is the deep, GUI dissection tool. It reassembles streams, flags retransmits and resets with Expert Info, and decodes hundreds of protocols field by field. Reach for it to understand a capture — the analysis you cannot do at a glance — typically on a file tcpdump already wrote.

Common Mistakes

Capturing without a filter on a busy link. The buffer overruns, tcpdump reports packets "dropped by kernel," and the frames it discarded are often the exact ones you were trying to catch — a capture full of holes is worse than none.
Capturing at only one point for a loss problem. The client trace shows packets sent and the server trace shows them missing, but neither alone localizes the drop; without both ends you cannot tell whether the path, the client, or the server lost them.
Expecting to read TLS payload from a capture. Past the handshake the bytes are encrypted, so the trace shows only the handshake and ciphertext; without the session keys you see that data flowed, never what it was.
Tapping on the wrong side of a NAT or proxy and chasing rewritten addresses. A capture after a NAT shows translated IPs and ports, so you correlate against the wrong host until you move the tap to where the addresses match what you expect.
Trusting a SPAN/mirror port blindly when the mirror is oversubscribed. The switch silently drops mirrored frames it cannot keep up with, and you diagnose "loss" that exists only in the capture, not on the real path.

Best Practices

Filter in the kernel with a tight BPF expression (host, port, tcp) so tcpdump keeps up on fast links, and confirm "0 packets dropped by kernel" before you trust the trace.
Write captures to a .pcap file with -w and analyze in Wireshark, rather than decoding inline with tcpdump — printing every packet is slow enough to cause its own drops.
Capture on both ends whenever you are chasing loss, and compare the two traces; the asymmetry between what was sent and what arrived is what localizes the drop to the path.
Use Wireshark's Expert Info and display filters like tcp.analysis.retransmission to jump straight to retransmits, zero-windows, and resets instead of scrolling thousands of packets.
Choose the tap to match what you must prove — before a NAT for original addresses, on the host NIC for stack behavior, on a SPAN for the wire view — and verify the mirror is not oversubscribed.

Comparable conceptseBPF (in-kernel observation)Flow logs (the sampled summary)

Knowledge Check

A client capture shows packets being sent, but the server capture never shows them arriving. What have you learned?

The loss is in the path between the ends, since one tap shows them sent and the other missing
The client's own TCP stack is buggy and never actually transmitted the packets onto the wire at all
The client application simply failed to generate the request in the first place before sending
The server received them but encryption hid them from its capture

A trace shows a SYN immediately answered by a RST from the server. What does that indicate?

The host is up but nothing is listening on that port, so the connection is refused
The destination host is unreachable, which is the reason no SYN-ACK ever comes back to the client
The three-way handshake completed normally and the session is now fully established between them
The server is rate-limiting the client and will accept a retry shortly

In a capture, the receiver advertises a TCP zero window and the sender stops. Where is the bottleneck?

The receiving application is consuming data too slowly, filling its buffer and pausing the sender
The network path between them is dropping packets, which is the reason the window collapsed to zero
The sender has simply run out of application data to transmit and so is sitting idle by its own choice
A firewall in the middle is resetting the connection mid-stream

You got correct