Processes, jobs and signals

Why this matters

A program that is running is a process. A web server, a text editor, a build tool and the terminal you type in are all processes. Each one has a number, its process ID, usually shortened to PID.

Most of the time a process ends by itself. Sometimes it does not. A script loops forever, a server holds a port after you stopped using it, or a build keeps using all the memory. Then you need to see what is running and stop it. Stopping a process the wrong way can lose unsaved work or leave files half written, so the difference between the ways of stopping matters.

This lesson covers the terminal side of processes: looking at them, pausing and stopping them, and sending them signals. Reading and writing files, and the pipes that join programs together, are covered in the neighbouring lessons on working with files and on flags and pipes.

The shell starts processes

When you type a command and press Enter, the shell (the program that reads your typing, for example bash or zsh) starts a new process for it. The shell waits until that process ends, then shows you the prompt again. While it waits, the command is in the foreground. The shell can also start a process and return to the prompt straight away. Then the process runs in the background.

Every process has a parent, the process that started it. The shell is the parent of the commands you type. Each process also has an exit status, a number it returns when it ends. By convention 0 means it succeeded and any other number means it failed. You can read the last one with echo $?.

A process that stops because of a signal has an exit status above 128. The number is 128 plus the signal number. You will see this in the examples below: 130 is Ctrl+C, 143 is a normal stop request and 137 is a forced stop.

Looking at processes with ps

ps prints a list of processes. With no options it shows only the processes attached to your terminal, which is usually the shell and nothing else. Use -e to show every process on the machine, and -o to choose the columns.

Here is the first example. The program plain.py is a small Python program that starts, prints its PID and waits for a signal. We will use it several times in this lesson. Create it in a scratch folder:

import os, signal
print("ready", os.getpid(), flush=True)
while True:
    signal.pause()

signal.pause() waits until a signal arrives. It uses almost no CPU while it waits. The flush=True makes sure the line is printed at once, instead of sitting in a buffer.

To see its state, ask ps for the PID, the state column and the command:

$ ps -o pid,stat,cmd -p 1259215
    PID STAT CMD
1259215 S    python3 -I plain.py

The -p option picks one PID. The PIDs in this lesson's outputs are examples, so yours will differ. The column STAT is the state. Here it is S, which means sleeping: the process is waiting for something. The most common states are:

If you ask for a PID that does not exist, ps prints the header and nothing else, and exits with status 1. That is a quick way to check whether a process is still alive.

To find the heaviest processes, sort by CPU use:

$ ps -eo pid,user,stat,pcpu,cmd --sort=-pcpu | head -n 5

The column pcpu is the share of processor time, and --sort=-pcpu sorts from highest to lowest. The output depends on the machine, so your list will differ.

top for a live view

top shows the same information as a screen that updates every few seconds. The header gives the load on the machine, the number of tasks and their states, and the memory use. Below the header, a table lists the busiest processes. Press q to quit. In top, the key k asks for a PID and a signal to send, which is a menu version of the kill command used below.

For a single snapshot that you can save or pipe into another command, use batch mode:

$ top -b -n 1 | head -n 12

-b means batch mode, which prints plain text and does not wait for keys. -n 1 means one update. Without these two options top would take over the screen.

Stopping a program with the keyboard

In a terminal, the foreground program receives keyboard combinations as signals. A signal is a short message the operating system delivers to a process. It tells the process that something happened, for example that the user asked it to stop. The process can react to the signal, ignore it, or let the default action happen.

Three keys matter most:

Ctrl+C asks the program to stop. The program can catch the signal and clean up, for example by saving a file, before it exits. Ctrl+Z pauses the program and gives you the prompt back. The paused program keeps its memory and its open files, and it does nothing until you resume it. fg brings the most recent paused or background job back to the foreground. bg lets a paused job keep running in the background.

These keys only work in an interactive terminal with job control, the feature that tracks the processes of each command line. A script that runs in the background cannot receive them. To show what the keys do, this lesson sends the same signals with kill, which can target any PID.

Here is a short script that runs plain.py in the background, pauses it, resumes it and then sends an interrupt. It turns on job control with set -m so that the process behaves as it would at a real prompt. Save it as keys.sh in your scratch folder:

#!/bin/bash
set -m
cd ~/scratch
mkfifo ready
python3 -I plain.py > ready 2> plain.err &
pid=$!
read -r line < ready
kill -TSTP $pid
ps -o pid,stat,cmd -p $pid
kill -CONT $pid
ps -o pid,stat,cmd -p $pid
kill -INT $pid
wait $pid
echo "exit status after SIGINT: $?"
cat plain.err

Each line does one thing:

• mkfifo ready makes a named pipe, a file that two programs can use to talk. The line read -r line < ready waits until plain.py writes its "ready" line into it. This way the script never sends a signal before the program is waiting for one. It does not use sleep to guess.
• $! holds the PID of the last background process. The script saves it in pid.
• kill -TSTP sends the stop signal that Ctrl+Z sends. kill -CONT resumes a stopped process. kill -INT sends the signal that Ctrl+C sends.
• wait $pid waits for the process to end and sets $? to its exit status.

Running it gives this output:

$ bash keys.sh
    PID STAT CMD
1264535 T    python3 -I plain.py
    PID STAT CMD
1264535 S    python3 -I plain.py
exit status after SIGINT: 130
Traceback (most recent call last):
  File "plain.py", line 4, in <module>
    signal.pause()
    ~~~~~~~~~~~~^^
KeyboardInterrupt

The state changed from T (stopped by Ctrl+Z) to S (sleeping again after kill -CONT). Then the interrupt ended the program. Python turns SIGINT into an exception called KeyboardInterrupt, which is why the traceback is printed. The path in the traceback is shortened here. The exit status 130 is 128 plus 2, the number of SIGINT.

Your output will show the full path to plain.py and a different PID. The states and the exit status will match.

Why background jobs can ignore Ctrl+C

The script above turns on job control. Without it, a background process behaves differently. The shell ignores SIGINT for background commands, so a kill -INT does nothing to them. The test below starts plain.py in the background without set -m and reads the signal mask from /proc, a folder where Linux keeps a file for each process. The line SigIgn lists the signals the process ignores.

#!/bin/bash
mkfifo r5
python3 -I plain.py > r5 2> /dev/null &
pid=$!
read -r line < r5
echo "without job control, SigIgn of child: $(grep SigIgn /proc/$pid/status)"
kill -KILL $pid
wait $pid

$ bash no-control.sh
without job control, SigIgn of child: SigIgn:	0000000001001006

The bit for signal 2 (SIGINT) is set in the mask. A background process started this way will not stop on kill -INT. To stop such a job, use SIGTERM or SIGKILL, described next.

Running a program in the background

Put & after a command to start it in the background. The shell returns the prompt at once, and the program keeps running. The output still goes to the terminal unless you redirect it. A background job that writes to the terminal while you type can make the screen hard to read, so send its output to a file:

$ python3 -I plain.py > run.log 2>&1 &

The > run.log part sends normal output to a file. The 2>&1 part sends error output to the same file. Read the redirections from left to right: > run.log first points standard output at the file, then 2>&1 points errors at the same place standard output now points.

The shell keeps a list of its jobs. Show it with jobs. In a script that started the job with > r8 2> /dev/null &, jobs -l printed this line:

$ jobs -l
[1]+ 1271574 Running                    python3 -I plain.py > r8 2> /dev/null &

The number in square brackets is the job number, and the next number is the PID. The job number is only meaningful in that shell.

Each background job's PID is also stored in the special variable $!, which is what the scripts in this lesson use. To stop a job, you need its PID or its job number.

Keeping a job after the terminal closes

When a terminal window closes, the shell sends the hang-up signal, SIGHUP, to its jobs. A background job that ignores this signal keeps running. The command nohup runs another command with SIGHUP ignored. Here is a check of that, using the SigIgn line again:

$ nohup python3 -I plain.py > r9 2> nh.log &
$ grep SigIgn /proc/$!/status
SigIgn:	0000000001001007

Run the grep after the job has started, because the mask is set when nohup starts the program. The last bit of the mask, 1, stands for SIGHUP. The nohup command also writes a short note to its error file when it is started from a terminal. Here the error file was empty, because the test ran without a terminal.

Use nohup for a long job that must survive logging out, such as a copy of a large folder over a slow connection. For anything you need to watch or control again, a terminal multiplexer such as tmux is a better tool. It is outside this lesson.

Ending a process politely with SIGTERM

kill sends a signal to the PID you name. Despite the name, it does not always kill. The default signal is SIGTERM (terminate), which means "please stop". The program can catch it, run its cleanup code and exit. A program that does not catch it stops at once.

$ kill <PID>

The <PID> above is a placeholder: replace it with the number you got from ps or from $!.

Here is a program that catches SIGTERM. Save it as polite.py:

import os
import signal
import sys
 
 
def on_term(signum, frame):
    print("got SIGTERM, cleaning up", file=sys.stderr, flush=True)
    sys.exit(0)
 
 
signal.signal(signal.SIGTERM, on_term)
print("ready", os.getpid(), flush=True)
while True:
    signal.pause()

The line signal.signal(...) tells Python to call on_term when SIGTERM arrives. The function prints a message, and sys.exit(0) ends the program with status 0. In a real program the cleanup would close files and save state.

Here is the script that starts it, stops it and checks the result. It waits for the "ready" line through a named pipe, the same way as keys.sh:

#!/bin/bash
mkfifo ready1
python3 -I polite.py > ready1 2> polite.err &
pid=$!
read -r line < ready1
echo "child printed: $line"
echo "shell saw pid $pid"
ps -o pid,stat,cmd -p $pid
kill -TERM $pid
wait $pid
echo "exit status: $?"
echo "stderr: $(cat polite.err)"

$ bash polite-test.sh
child printed: ready 1259208
shell saw pid 1259208
    PID STAT CMD
1259208 S    python3 -I polite.py
exit status: 0
stderr: got SIGTERM, cleaning up

The program printed its message, cleaned up and exited with 0. That is the outcome you want when you ask a program to stop.

Ending a process forcefully with SIGKILL

The signal SIGKILL (signal number 9) cannot be caught, ignored or handled by the program. The operating system removes the process at once. The program gets no chance to clean up, so files it was writing may be left half written, and temporary data it was holding is lost.

Use SIGKILL only after SIGTERM did not work. Here is a program that ignores SIGTERM on purpose. Save it as stubborn.py:

import os
import signal
 
signal.signal(signal.SIGTERM, signal.SIG_IGN)
print("ready", os.getpid(), flush=True)
while True:
    signal.pause()

Here is the test: send SIGTERM, check the state, then send SIGKILL.

#!/bin/bash
mkfifo ready2
python3 -I stubborn.py > ready2 2> /dev/null &
pid=$!
read -r line < ready2
kill -TERM $pid
ps -o pid,stat,cmd -p $pid
kill -KILL $pid
wait $pid
echo "exit status: $?"

$ bash stubborn-test.sh
    PID STAT CMD
1259212 S    python3 -I stubborn.py
exit status: 137

The ps line shows the process still sleeping after the SIGTERM, so it ignored the request. The SIGKILL ended it. Bash also prints a line ending in Killed that names the PID, which is left out here. The exit status 137 is 128 plus 9, the number of SIGKILL.

Here is the same kind of test with the default SIGTERM on the plain program, which has no handler:

#!/bin/bash
mkfifo r7
python3 -I plain.py > r7 2>/dev/null &
pid=$!
read -r line < r7
kill -TERM $pid
wait $pid
echo "plain.py SIGTERM exit: $?"

$ bash term-test.sh
plain.py SIGTERM exit: 143

143 is 128 plus 15, the number of SIGTERM. A program with no handler stops at once, but it gets no chance to clean up. Compare that with polite.py, which exits with 0 because it chose to.

The signal list

kill -l lists the signal names and numbers that this system knows. Some names you will use most often:

Signal numbers can differ on other systems, so use the names in scripts and the numbers only when you have checked them. kill -l 9 prints KILL and kill -l 15 prints TERM, which is a quick way to translate a number.

Common mistakes

Mistake 1: stopping a PID that is gone. A process can end before you send it a signal. Then kill reports that the PID does not exist:

$ kill 999999
bash: kill: (999999) - No such process
$ echo $?
1

The exit status 1 means the signal was not sent. In a script, check the status and report the problem. Do not assume the process is still there.

Mistake 2: killing the wrong PID. PIDs are reused. A number you wrote down an hour ago may now belong to a different program. Before you send a signal, check what the PID is:

$ ps -o pid,stat,cmd -p 1259212
    PID STAT CMD
1259212 S    python3 -I stubborn.py

Read the CMD column. If it shows a program you did not expect, do not send the signal.

Mistake 3: reaching for SIGKILL first. A habit of kill -9 skips the cleanup code of every program. For a database, an editor or a program that writes files, this can cause the damage you were trying to avoid. Send SIGTERM first, wait for the process to end, and use SIGKILL only if it does not.

Mistake 4: expecting Ctrl+C to work on a background job. A background job started without job control ignores SIGINT, so kill -INT does nothing. The process kept running and the script waited forever. If a test hangs like this, stop it with the PID you saved, using SIGKILL:

$ kill -KILL 1264535

Use a timeout when you run a script that might hang, for example timeout 30 bash keys.sh. The timeout command stops the script after 30 seconds.

Mistake 5: a background job still writing to the terminal. A job started with & and no redirection prints into the middle of your typing. Send its output to a file, as shown in the section on background jobs.

Practice

Use the files from this lesson in your scratch folder. Each exercise has an answer under it.

Exercise 1. Start plain.py in the background with its output sent to run.log. Print its PID, check its state with ps, and stop it with the polite signal. What exit status does wait report?

Show the answer

$ python3 -I plain.py > run.log 2>&1 &
$ pid=$!
$ ps -o pid,stat,cmd -p $pid
$ kill -TERM $pid
$ wait $pid
$ echo $?
143

The default action for SIGTERM stops plain.py with status 143, because it has no handler. The wait command works here because the process is a child of this shell.

Exercise 2. How can you tell from the output of ps that a process is paused by Ctrl+Z and not running?

Show the answer

Look at the STAT column. A paused process shows T. The keys in this lesson sent SIGTSTP, and the state changed from S to T. After kill -CONT it went back to S.

Exercise 3. A program ignores SIGTERM. Which signal will still stop it, and what exit status does the shell report?

Show the answer

SIGKILL, the signal with number 9. The stubborn.py test ended with Killed and exit status 137, which is 128 plus 9. The program had no chance to run any cleanup code.

Exercise 4. What do these exit statuses mean: 130, 137 and 143?

Show the answer

Each one is 128 plus a signal number. 130 is 128 plus 2, so SIGINT (Ctrl+C). 137 is 128 plus 9, so SIGKILL. 143 is 128 plus 15, so SIGTERM.

Exercise 5. Write a short shell command that prints a message and a non-zero status when a PID is no longer running. Use the PID in the variable pid.

Show the answer

if ! kill -0 "$pid" 2> /dev/null; then
    echo "process $pid is not running"
fi

kill -0 sends no signal. It only checks whether the PID exists and that you may signal it. The ! reverses the test, so the message prints when the check fails. The 2> /dev/null hides the error text that kill would print.

Summary

• A process is a running program with a PID. ps -o pid,stat,cmd -p <PID> shows its state, and top shows the busiest processes.
• Ctrl+C sends SIGINT and stops the program. Ctrl+Z sends SIGTSTP and pauses it, and fg or kill -CONT resumes it.
• & runs a command in the background, and $! holds its PID. Send its output to a file so it does not write over your typing.
• kill <PID> sends SIGTERM, a request to stop that the program can handle. Use kill -KILL <PID> only when SIGTERM has no effect, because SIGKILL cannot be caught.
• An exit status of 128 plus a signal number means the process was stopped by that signal.