Routing

The routing table is the kernel's decision for every outbound packet: given a destination address, which interface does it leave from, and which next hop does it go to? The kernel compares the destination against every route it holds and picks the one with the longest matching prefix — the most specific route wins. If nothing else matches, the packet falls to the default route, 0.0.0.0/0, and is handed to the default gateway. Routing is per-packet and stateless; the kernel makes the same lookup for the first packet of a connection and the millionth.

This is where most connectivity faults live once you have ruled out DNS. A host that can reach its own subnet but not the internet has a missing or wrong default gateway. A host that reaches the internet but not one specific internal network is missing a static route. Both look identical from ping — "some things work, some don't" — and both are read directly from ip route in seconds. The skill is reading the table, not memorizing it.

The Routing Table

On the modern iproute2 toolset you read the table with ip route (the old route -n from net-tools still works but shows a less complete picture and is deprecated). Each line is one route: a destination prefix, optionally a gateway (via), the outgoing interface (dev), the source address the kernel will stamp on packets (src), and a metric that breaks ties when two routes have the same prefix length. Lower metric wins.

# Read the main routing table
ip route
# Typical output on a server with one NIC
default via 10.0.0.1 dev eth0 proto dhcp metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.42 metric 100

The second line is a directly connected route: the kernel added it automatically when the address was configured (proto kernel scope link), and it means "anything in 10.0.0.0/24 is on this wire, send it directly, no gateway." The first line is the default. Longest-prefix match decides between them: a packet for 10.0.0.7 matches the /24 (24 bits) and the default /0 (0 bits), and the /24 wins because it is more specific. A packet for 8.8.8.8 matches only the default and goes to the gateway.

The Default Gateway

The default gateway is the route of last resort. 0.0.0.0/0 matches every IPv4 address with a prefix length of zero, so it is always the weakest possible match and only ever chosen when nothing more specific applies. A server with no default route can still talk to every host on its own subnets — the directly connected routes handle those — but every packet aimed off-subnet has nowhere to go and the kernel returns Network is unreachable immediately, before a single packet leaves the box.

That failure mode is the signature of a missing or wrong gateway, and it is worth recognizing on sight: local hosts respond, the internet does not, and the error comes back instantly rather than after a timeout. On Debian/Ubuntu the gateway is normally set by DHCP or declared in Netplan (routes: with to: default, or the older gateway4:); on Red Hat it comes from NetworkManager or the legacy GATEWAY= in an interface file. A runtime fix with ip route add default via 10.0.0.1 works immediately but vanishes on reboot — it must also go into the persistent config.

Static Routes

A static route is any route you add by hand to reach a network that the default gateway does not serve — a second internal subnet behind a different router, a VPN range, a management network. You add it at runtime with ip route add, naming the destination prefix and the gateway that knows how to reach it. The change is live the instant the command returns, which is exactly why it is dangerous: it disappears on the next reboot, and a route that works in testing but was never persisted is a classic 3 a.m. surprise.

# Reach 192.168.50.0/24 via a router on the local wire
ip route add 192.168.50.0/24 via 10.0.0.254
# Confirm it landed
ip route get 192.168.50.10

To persist it on Debian/Ubuntu, add the route to the interface's Netplan YAML under routes: and run netplan apply, or let systemd-networkd own it via a [Route] section in a .network file. On Red Hat the equivalent is an nmcli connection modify ... +ipv4.routes entry or a legacy route-eth0 file. The rule is the same regardless of distribution: a static route is not finished when the packet flows, it is finished when it survives a reboot.

Policy Routing and Multiple Tables

By default the kernel makes its decision from a single table (main), choosing purely on destination. Policy routing lets you keep several tables and pick between them on more than the destination — source address, incoming interface, or a firewall mark — using ip rule. The common need is a multi-homed host that must reply out the same interface a request arrived on: you put each interface's default route in its own table (named in /etc/iproute2/rt_tables) and add ip rule entries that select the table by source address. It is the right tool for dual-uplink and source-based routing, but it adds real complexity, so reach for it only when one table genuinely cannot express the requirement.

Diagnosing Paths

Two questions sit behind every routing fault, and they need different tools. The first is "which route will this kernel pick for this destination?" — answered locally and definitively by ip route get <ip>, which runs the exact lookup the kernel would and prints the chosen route, gateway, interface, and source address without sending a packet. The second is "where does the packet actually die on the way there?" — answered by traceroute or, better, mtr, which probe the live hop-by-hop path and show which router stops responding.

A box meant to forward packets between networks — a router, a NAT gateway, a VPN endpoint — needs one more thing the routing table alone will not give it: forwarding has to be switched on. Linux drops packets not addressed to itself unless net.ipv4.ip_forward is set to 1. Set it at runtime with sysctl -w net.ipv4.ip_forward=1 and persist it in /etc/sysctl.d/. A forwarding host with correct routes but ip_forward=0 silently discards transit traffic, and nothing in ip route hints at why.