SSH

SSH is the encrypted channel every server administrator lives inside. One protocol carries an interactive shell, file transfer (scp, sftp, and rsync tunnelled over it), arbitrary TCP port forwarding, and the authentication that decides who gets in. On Debian and Ubuntu the daemon ships in the openssh-server package, runs as the ssh systemd unit, listens on TCP port 22, and reads its configuration from /etc/ssh/sshd_config. The client is openssh-client, configured per-user in ~/.ssh/config.

Almost everything that goes wrong with SSH is a key, a permission, or a configuration problem — not the network. Get the keypair, the file modes, and the agent right and remote access is silent and instant; get them wrong and you are typing passwords into every host, copying private keys onto bastions, or locked out of a box you just hardened. The protocol itself is solid; the operational surface around it is where the time goes.

The Protocol and Client/Server

Two programs make up the system. sshd is the daemon that listens for connections; ssh is the client that opens them. They negotiate a session key, then run everything — keystrokes, file bytes, forwarded ports — inside that encrypted tunnel. The same connection multiplexes all of it, which is why one SSH session can carry your shell and a forwarded database port at once.

On the very first connection to a host, the client has no way to know it is talking to the right machine. The server presents its host key — a public key that identifies the box itself, distinct from any user key — and the client shows you its fingerprint and asks whether to trust it. Say yes and the key is pinned in ~/.ssh/known_hosts; from then on the client verifies it silently and screams if it ever changes. That first prompt is the one moment the protocol cannot protect you, so an attacker who can intercept the initial connection can impersonate the host if you accept blindly.

# connect; first time, you are asked to trust the host key
ssh admin@10.0.0.12
# check the daemon and its listening socket on the server
systemctl status ssh
ss -tlnp 'sport = :22'

On Red Hat and its derivatives the unit is named sshd rather than ssh, and the package is openssh-server there too. The config file path and syntax are identical, so the only thing that changes across distributions is the systemctl unit name you type.

Key-based Authentication

A keypair is two halves of one secret: a private key that never leaves your machine and a public key you copy to every server. The server stores allowed public keys, one per line, in ~/.ssh/authorized_keys under the target account. At login the client proves it holds the matching private key without ever transmitting it, so there is no shared secret crossing the wire and nothing for the server to leak. Generate Ed25519 keys — they are shorter and faster than RSA, and 256-bit Ed25519 is the current default for ssh-keygen.

Keys beat passwords on every axis that matters. A password is phishable, reused across sites, and brute-forceable against an exposed port 22 at thousands of attempts per minute; a 256-bit key is none of those things. Add a passphrase to the private key and a stolen laptop still does not hand over your servers, because the file on disk is encrypted at rest. Once key auth works, you disable password auth entirely on the server and the bots hammering port 22 simply have nothing to guess.

# generate an Ed25519 keypair with a passphrase
ssh-keygen -t ed25519 -C "admin@laptop"
# copy the public half to the server's authorized_keys
ssh-copy-id admin@10.0.0.12

sshd is deliberately paranoid about file permissions. If ~/.ssh is group- or world-writable, or authorized_keys is readable by anyone but the owner, the daemon silently refuses the key and falls back to asking for a password — with no error to the client and a terse line in the server's journal. The directory must be 700 and the file 600, owned by the user, or key auth will not work no matter how correct the key itself is.

The SSH Agent and Config

Typing the key passphrase on every connection defeats the convenience that made you set up keys. The ssh-agent holds your decrypted private key in memory for the session: you unlock it once with ssh-add, and every subsequent ssh, scp, and git push uses it without prompting. The agent never reveals the key — clients ask it to sign challenges, and it signs them in process.

# start an agent and load your key once per session
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
# list keys currently held by the agent
ssh-add -l

The real productivity lever is ~/.ssh/config. Instead of remembering IP addresses, ports, and usernames, you define a Host alias once and connect by name. The most valuable directive is ProxyJump: it tunnels the connection to an internal host through a bastion in a single hop, so your private key authenticates end-to-end and never lands on the jump box. Copying private keys onto a shared bastion — the alternative people reach for — turns one compromised bastion into a compromise of every server those keys reach.

# ~/.ssh/config
Host bastion
    HostName 203.0.113.10
    User admin

Host db1
    HostName 10.0.5.20
    User admin
    ProxyJump bastion

With that in place, ssh db1 reaches an internal host that has no public address at all, routing through the bastion automatically. Agent forwarding (ForwardAgent yes or ssh -A) is the lazier alternative, but it exposes your agent socket to whoever has root on the intermediate host, so prefer ProxyJump for jumps and reserve forwarding for hosts you fully trust.

File Transfer and Tunnels

Because SSH already authenticates and encrypts, file transfer rides the same channel for free. scp copies single files, sftp gives an interactive file session, and rsync over SSH is the one to reach for on anything non-trivial: it transfers only the changed blocks, resumes interrupted runs, and preserves permissions and timestamps. A 50 GB directory that mostly already exists on the far side syncs in seconds with rsync where scp would recopy all 50 GB.

# single file with scp
scp report.csv db1:/tmp/
# incremental, resumable directory sync over ssh
rsync -avz --progress ./build/ db1:/srv/app/

Port forwarding turns the SSH tunnel into a way to reach services that are not exposed publicly. Local forwarding (-L) brings a remote service to a port on your machine — point a local database client at localhost:5432 and the bytes pop out on the server. Remote forwarding (-R) does the reverse, exposing a local service to the server side. Dynamic forwarding (-D) opens a local SOCKS proxy that routes arbitrary connections through the remote host.

# reach the server's Postgres on your own localhost:5432
ssh -L 5432:localhost:5432 db1
# a SOCKS proxy through the bastion for ad-hoc access
ssh -D 1080 bastion

Host Key Verification

The known_hosts file is the client's memory of which key each host presented. On every reconnect the client compares the presented host key against the pinned one; a match is silent, a mismatch is a loud, blocking warning that refuses to connect. That warning has exactly two causes: the server was legitimately rebuilt or reinstalled and got a new host key, or someone is sitting between you and the server presenting their own key — a machine-in-the-middle attack. The protocol cannot tell which, so it stops and makes you decide.

# WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
# someone could be eavesdropping on you right now ...
# after a *verified* rebuild, remove the stale entry:
ssh-keygen -R 10.0.0.12

The dangerous habit is treating that warning as noise and clearing the entry reflexively. Before you run ssh-keygen -R, confirm out of band that the host really was rebuilt — check the change ticket, the console, the new fingerprint against one recorded at provision time. For a fleet, distribute host keys ahead of time (or use certificate-based host keys signed by a trusted CA) so clients trust new machines without a first-connection prompt at all, and a genuine mismatch is unambiguous.