Job Control and History

Job control is the set of shell features that lets one interactive session run, suspend, resume, and switch between multiple processes. When you press Ctrl-Z, start a command with a trailing &, or type fg, you are operating on jobs: groups of processes that the shell tracks and that the kernel organizes into process groups attached to your terminal. Shell history is the parallel feature for commands themselves — every line you submit is recorded so it can be recalled, edited, and re-run without retyping.

For server work the operational consequence is concrete. Job control turns a single SSH session into a workspace where a long rsync runs in the background while you edit a config in the foreground, and history turns yesterday's exact tar invocation into one Ctrl-R away. Both features have failure modes that bite at scale: a backgrounded job dies when the session closes unless you account for the hangup signal, and a sloppy history setup either loses commands across concurrent sessions or silently records secrets you pasted on the command line.

Jobs, Process Groups, and the Controlling Terminal

A job is the shell's name for a pipeline you launched from the prompt. The kernel implements it as a process group — every process in the pipeline shares a process group ID, and the terminal has exactly one of those groups marked as the foreground group at any moment. Only the foreground group receives keyboard-generated signals and may read from the terminal; background groups that try to read are stopped with SIGTTIN. This is why a background job waiting on input appears frozen rather than failing.

List your jobs with jobs. The bracketed number is the job ID, distinct from the PID, and the + and - marks indicate the current and previous jobs that fg and bg default to. Reference a job by %n for job number, %+ for current, or %string to match by command prefix.

# start a long copy, then suspend it with Ctrl-Z
rsync -a /srv/data/ backup:/srv/data/
[1]+  Stopped    rsync -a /srv/data/ backup:/srv/data/

jobs -l           # -l also prints the PID
[1]+ 48213 Stopped  rsync -a /srv/data/ backup:/srv/data/

bg %1             # resume it in the background
fg %1             # pull it back to the foreground

Suspending, Backgrounding, and Surviving Hangup

Ctrl-Z sends SIGTSTP, which stops the foreground job and returns the prompt; bg then continues it with SIGCONT while detaching it from the keyboard. Starting a command with a trailing & skips the suspend step and launches it in the background directly. None of this, by itself, makes a job survive the terminal closing.

When the controlling terminal goes away — you close the SSH window or the link drops — the kernel sends SIGHUP to the controlling process, which is the shell itself, and bash forwards it to its jobs. To keep a job running, remove it from the shell's hangup list with disown -h %1, or launch it under nohup so the signal is ignored from the start. The modern alternative is to run inside a tmux or screen session, where the work lives in a server process that outlives any client.

# three ways to outlive the session
nohup ./long-import.sh > import.log 2>&1 &   # ignores SIGHUP, redirects output
./long-import.sh & disown -h %1               # detach an already-running job
tmux new -s import                          # run inside a persistent multiplexer

# systemd-run gives a transient unit with logging and cleanup
systemd-run --user --unit=import ./long-import.sh

On Debian and Ubuntu, systemd-run is often the better choice for anything that must truly persist: it places the work in its own cgroup, captures output in the journal, and is not tied to your login session at all. Red Hat systems ship the same systemd-run, so this approach is portable across distributions where nohup still leaves you guessing whether the process really detached.

Reading and Recalling Shell History

Bash keeps an in-memory history list during the session and persists it to the file named by $HISTFILE, which defaults to ~/.bash_history. The number of lines kept in memory is $HISTSIZE; the cap on the file is $HISTFILESIZE. By default bash writes history only on exit, which is why two terminals open at once tend to clobber each other's records — the last shell to close wins.

Recall is faster than retyping. Ctrl-R starts an incremental reverse search; !! repeats the previous command and !$ expands to its last argument; !cmd reruns the most recent line starting with cmd. Event designators are expanded before execution, so sudo !! reruns the prior command with elevated privileges — convenient, and a foot-gun if the prior command was not what you remember.

Tool	What it does	Common use
Ctrl-R	Incremental reverse search of history	Find a half-remembered command
!!	Expand to the previous command	sudo !! after a permission error
!$	Expand to the last argument	mkdir d && cd !$
history -d N	Delete entry number N	Remove a pasted secret
history -a	Append new lines to $HISTFILE	Sync across live sessions

History Across Concurrent Sessions and Secrets

The default write-on-exit behavior loses commands whenever sessions overlap. The standard fix is to append after every command and re-read the file, so each prompt sees what the others wrote. Set this in ~/.bashrc with PROMPT_COMMAND and the histappend option, and the history file becomes an ordered, shared log instead of a battleground.

# ~/.bashrc — durable, deduplicated, timestamped history
shopt -s histappend
HISTSIZE=100000
HISTFILESIZE=200000
HISTTIMEFORMAT='%F %T  '          # show date/time with `history`
HISTCONTROL=ignoreboth                  # skip dupes and lines that start with a space
PROMPT_COMMAND='history -a; history -n' # append, then read others' new lines

Secrets are the sharp edge. Anything typed on a command line lands in history and is also visible in /proc/PID/cmdline to other users while the process runs. With HISTCONTROL=ignorespace (included in ignoreboth), prefixing a command with a single space keeps it out of history entirely. To scrub a line already written, use history -d on its number followed by history -w, and prefer reading secrets from a file or environment variable over passing them as arguments.