Topic 05

The Terminal, Shells, and Sessions

FoundationsConcept

Three things people lump together as "the command line" are separate components, and on a server the difference is operational. The terminal is a character device that moves bytes between you and a program. The shell (bash on Debian/Ubuntu for interactive logins, dash as /bin/sh) is an ordinary program that reads those bytes, parses commands, and runs them. The session is the kernel's bookkeeping that ties a login, its terminal, and the processes it spawned into one unit that can be signalled and torn down together.

Getting this wrong is how a long-running job dies the instant your SSH connection drops, or how a script that worked in your interactive shell behaves differently under cron. The terminal decides who receives SIGHUP when a connection closes; the shell type decides which startup files run and therefore what PATH and environment a process sees. Both are invisible until they bite.

Terminals, TTYs, and PTYs

A terminal is a kernel device, not an application. The historical hardware was a teletype, which is why the device class is still called a TTY. A physical Linux console — the screens you reach with Ctrl+Alt+F1 through F6 — is a real TTY backed by the kernel's virtual-console driver, exposed as /dev/tty1, /dev/tty2, and so on. These exist before any graphical session and are where you land when the network or the display manager is broken.

Everything else is a pseudo-terminal, a PTY: a kernel-emulated terminal with two ends. A terminal emulator or sshd holds the master side; your shell runs on the slave side at /dev/pts/0, /dev/pts/1, and so on. When you type in an SSH session, sshd writes your keystrokes to the master, the kernel's line discipline delivers them to the shell on the slave, and the shell's output flows back the same way. Run tty to see which device you are on; a job with no controlling terminal — most daemons, anything under cron — reports not a tty.

# Where am I, and what is on the other terminals?
tty                 # e.g. /dev/pts/2 over SSH, /dev/tty1 on the console
who                 # every logged-in session and its terminal
ps -o pid,tty,cmd -t pts/2   # processes attached to one PTY

The Shell Is Just a Program

The shell reads command lines, expands globs and variables, sets up pipes and redirections, forks the requested programs, and reports their exit status. It is not part of the kernel and not part of the terminal — swap it and nothing else changes. The login shell for an account is the last field in /etc/passwd, edited with chsh -s /usr/bin/zsh and listed in /etc/shells. On Debian and Ubuntu the interactive default is bash, but /bin/sh is a symlink to dash, a smaller, faster, strictly POSIX shell with none of bash's arrays, [[ ]], or process substitution.

That split is a recurring server footgun. A script with #!/bin/sh runs under dash, so bashisms fail; the same file with #!/bin/bash runs under bash and works. Red Hat and Fedora point /bin/sh at bash in POSIX mode instead, which is why a script that passes on RHEL can break on Ubuntu without a single line changing. Pin the interpreter you actually tested against in the shebang rather than trusting the platform default.

Login, Interactive, and Non-Interactive Shells

A shell is one of three kinds, and the kind decides which startup files run. A login shell is what you get at console sign-in or the start of an SSH session; bash reads /etc/profile and then the first it finds of ~/.bash_profile, ~/.bash_login, or ~/.profile. An interactive non-login shell — a new terminal tab, or bash typed at a prompt — reads /etc/bash.bashrc and ~/.bashrc instead, and never touches the profile files. A non-interactive shell, the one that runs a script or a cron job, reads neither; it only honours the file named in $BASH_ENV, which is usually empty.

This is why exported variables and PATH edits in ~/.bashrc silently vanish under cron, and why putting environment setup only in ~/.bash_profile leaves your second terminal tab without it. The convention that fixes both: keep environment and PATH in the profile, keep aliases and prompt tweaks in ~/.bashrc, and have the profile source ~/.bashrc so a login shell gets everything. Treat cron as having no environment at all and set PATH explicitly in the job.

# Classify the current shell
shopt -q login_shell && echo login || echo non-login
case $- in *i*) echo interactive;; *) echo non-interactive;; esac

Sessions, Process Groups, and Job Control

When the shell starts it becomes a session leader and the session's controlling terminal is bound to it. Every pipeline you launch becomes a process group; one group at a time is the terminal's foreground group and receives your keystrokes and signals, while the rest run in the background. Ctrl+C sends SIGINT to the whole foreground group, Ctrl+Z sends SIGTSTP to suspend it, and the shell's job-control built-ins — jobs, fg, bg, kill %1 — move groups between those states. This is why Ctrl+C kills an entire pipeline, not just the last command.

The dangerous edge is the controlling terminal going away. When an SSH connection drops or you close the terminal, the kernel sends SIGHUP to the session leader, which forwards it to its jobs, and anything that does not catch it dies. A backgrounded job is not safe — it is still in the session. nohup cmd & detaches one command from the hangup and redirects its output to nohup.out; disown -h %1 removes an already-running job from the shell's hangup list. Neither gives you the terminal back, which is what tmux and screen are for.

Persistent and Detached Sessions

tmux and screen solve the hangup problem properly by inserting their own PTY between you and the shell. The server process holds the real session and keeps your shells alive; your client merely attaches to and detaches from it. Detach with Ctrl+b d, drop the SSH connection, reconnect tomorrow, run tmux attach, and the long migration or compile is exactly where you left it. For any multi-hour command over SSH, starting it inside tmux is the difference between a flaky network being an annoyance and being a lost job.

For services, do not lean on tmux at all — that is what the init system is for. A process started in a detached session still dies if the machine reboots and has no log integration, no restart policy, and no resource limits. Run daemons under systemd as units; systemd also tracks interactive logins as sessions through logind, which is why loginctl lists them and why KillUserProcesses can reap stray background jobs when a user logs out. Use tmux for your own interactive work, systemd for anything that must survive a reboot.

# A long job that survives a dropped SSH connection
tmux new -s migrate     # named session; Ctrl+b d to detach
tmux attach -t migrate  # reconnect later from anywhere

# Inspect systemd-tracked login sessions
loginctl list-sessions
loginctl session-status 3

Terminal vs Shell vs TTY/PTY

Terminal emulator — the application that draws characters and owns the master end of a PTY (GNOME Terminal, the SSH client, the kernel's virtual console). It transports bytes; it does not interpret commands. Reach for this layer when the problem is keys, colours, resize, or line wrapping.

Shell — the program reading from the terminal that parses and runs commands (bash, dash, zsh). Reach for this layer when the problem is PATH, startup files, quoting, globbing, or a script behaving differently than at the prompt.

TTY / PTY — the kernel device in between, with its line discipline handling echo, line editing, and the Ctrl+C/Ctrl+Z signal mappings. Reach for this layer when a program complains it is not a tty, when piped output buffers differently, or when stty settings are wrong.

Common Mistakes

Backgrounding a long job with & and assuming it is safe — it is still in the session, so the SIGHUP on disconnect kills it. Use tmux, or nohup/disown at minimum.
Putting PATH and exports only in ~/.bashrc, then finding them missing under cron and in non-interactive SSH commands, which never read it.
Writing a script with bashisms ([[ ]], arrays, process substitution) under a #!/bin/sh shebang — on Debian/Ubuntu that is dash, which rejects them, even though the same file runs on RHEL where /bin/sh is bash.
Running production daemons inside a tmux window — they vanish on reboot and have no logging, restart policy, or resource limits. That is a systemd unit's job.
Editing environment in ~/.bash_profile only, then wondering why a new terminal tab (an interactive non-login shell) does not have it because that file is never read there.
Expecting Ctrl+C to stop only the last command in a pipeline — it signals the whole foreground process group, so the entire pipeline dies.
Hard-coding /dev/pts/0 or a specific TTY in scripts — pseudo-terminal numbers are assigned per connection and change every login.

Best Practices

Start every multi-hour command over SSH inside tmux new -s name so a dropped connection is an annoyance, not a lost job.
Pin the interpreter you tested against in the shebang — write #!/bin/bash when you use bash features, never rely on what /bin/sh happens to point at.
Keep environment and PATH in ~/.profile, keep aliases and prompt tweaks in ~/.bashrc, and source ~/.bashrc from the profile so login shells get both.
Set PATH explicitly inside every cron job and systemd unit; treat their shells as having no inherited environment.
Run anything that must survive a reboot as a systemd unit with Restart= and journald logging — not a detached terminal session.
Use tty and case $- in *i*) guards in shell config so interactive-only settings never run in scripts and break non-interactive use.
Reach the physical console (Ctrl+Alt+F2, /dev/tty2) to recover a box whose network or display manager is down — it exists independently of any PTY.

Comparable toolsWindows — Windows Terminal hosting PowerShell or cmd.exe; ConPTY is the pseudo-console layer, the rough analogue of a PTYmacOS — Terminal.app / iTerm2 over the same BSD-derived PTY model, with zsh as the default login shell since CatalinaBSD — the original job-control and TTY design Linux inherited; tcsh and sh defaults, same tmux/screen tools

Knowledge Check

You launch a 6-hour migration with ./migrate.sh & over SSH and the connection drops an hour in. What most likely happens?

The job receives SIGHUP and dies, because backgrounding does not remove it from the session whose controlling terminal just went away
It keeps running safely to completion — a backgrounded job is fully detached from the terminal and therefore immune to the disconnect
It pauses on the drop and then resumes automatically from where it stopped the moment you reconnect over SSH
It survives the disconnect only if you remembered to press Ctrl+Z to suspend it first

A script with #!/bin/sh uses [[ $x == y* ]]. It passes on RHEL but fails on Ubuntu. Why?

On Debian/Ubuntu /bin/sh is dash, which lacks [[ ]], whereas on RHEL /bin/sh is bash in POSIX mode and accepts it
Ubuntu ships an older build of bash that dropped support for the [[ ]] test operator entirely
The kernel parses the #! shebang line differently between the two distributions, so it picks a different interpreter
RHEL silently rewrites every [[ ]] into [ ] at runtime before executing the test

PATH additions in your ~/.bashrc work at the prompt but are missing in a cron job. What explains it?

cron runs a non-interactive shell, which reads neither ~/.bashrc nor the profile files, so its environment is nearly empty
cron reads your ~/.bashrc like any shell but then strips out the PATH entries it considers unsafe before it runs the scheduled job
The additions in ~/.bashrc only apply to the user's very first interactive login of the day
cron reads ~/.bash_profile instead of ~/.bashrc, and that is where the edit was never made

Why is a tmux session the wrong place to run a production database daemon, even though it survives SSH disconnects?

It dies on reboot and has no restart policy, log integration, or resource limits — those are what a systemd unit provides
tmux caps the CPU available to each pane, so the database process would be silently throttled under load
tmux cannot keep a process alive any longer than the original login session that first started the server
A daemon running inside a tmux pane loses its controlling PTY on disconnect and therefore can no longer open or hold listening sockets

You got correct