Process: Explanation & Insights

A running instance of a program, with its own process ID, memory, and resources, managed by the operating system kernel.

What It Is

A process is a program that is running. That's the whole of it, and it's worth sitting with for a second, because the distinction it draws is one of the deepest in all of computing: a program is a file sitting still on disk — a recipe, a list of instructions, dead text — and a process is what exists once the machine actually starts doing what the file says. The same program can be running as ten separate processes at once (ten people each running bash are ten processes from one file), the way one recipe can have ten cooks each making the dish in their own kitchen, each at a different step, each with their own half-chopped onions.

Concretely, a process on Linux is the kernel's unit of a thing that is running. Each one gets three things the moment it's born: a unique number called a PID (process ID) so the system can name it; its own private region of memory called an address space, walled off so no other process can reach in and scribble on it; and a set of open resources — files, network connections, a current directory, an owning user — that it carries around with it. The kernel keeps a small record for every live process (its slot in the process table), hands each one slices of CPU time in turn, and tracks whether it's running, sleeping, or finished. Everything you'll ever do on a server — every command you type, every web server, every database, the login shell itself — is a process, or a whole family of them.

And family is the right word, because no process on a Linux system appears from nothing. Every single one is created by another process, which makes the whole running system one enormous tree — every branch traceable back to a single root. That root, PID 1, is started by the kernel at boot; you can read about who it is and what it does on the init system page. Everything else descends from it.

How One Process Becomes Two

Here is the part that surprises everyone the first time, and it's the mechanism the rest of the system is built on. There is no Unix call that means "start this program." That sentence — so obvious you'd assume it must be the first thing anyone built — does not exist. Instead, starting a program is two moves, and you do them in order.

The first is fork() — a system call, a request a program makes to the kernel. fork() takes the process that calls it and makes a near-perfect copy: same code, same open files, same memory contents, same everything but the PID. For one instant there are two identical processes where there was one, a parent and a child, both sitting on the exact same line of code, about to walk forward into different lives. The only way each tells itself apart is the number fork() hands back: the child gets a 0, the parent gets the child's PID. One call, two return values, two futures.

The second move is exec(). The fresh child, now running, calls exec() and says: replace everything I am with this other program. The old code is thrown out, the memory wiped and reloaded from a new executable file, and the process — same PID, same slot in the table — wakes up as something entirely different. The shell does exactly this every time you run a command: it forks a copy of itself, and the copy immediately exec()s into ls or grep or whatever you typed.

Why two steps and not one?

Because the gap between the fork() and the exec() is where all the useful work happens. In that sliver of a moment the child is still a copy of the parent but isn't yet the new program — and that's exactly when the shell rewires things: redirect a file into the child's input, point its output at a pipe, drop its privileges. Splitting "make a new process" from "run a new program" turned out to be one of those decisions that looks like an accident and works like a master plan.

It looks like an accident partly because it was a happy one. On the PDP-7 the first Unix ran on in 1969, there was only enough memory for one process at a time; everything else lived on disk. So fork() was implemented by writing the running program out to the disk swap area and letting the copy still sitting in memory carry on as the child — in twenty-seven lines of assembly. Process creation began, quite literally, as the act of writing yourself down, walking away, and letting the version you left behind keep going. The machines got memory management, copy-on-write, and decades of cleverness since, but the shape of that 1969 shortcut is still exactly how every program on your server gets born.

The States a Process Lives In

A process is almost never running. That sounds wrong — surely a running program is running? — but on a server with hundreds of processes and a handful of CPUs, at any given instant nearly all of them are asleep, waiting for something: a key to be pressed, a network packet to arrive, a disk to answer. The kernel tracks which of a few states each one is in, and ps and top show them to you as single letters. Learning to read those letters is most of what process-watching is.

R — running or runnable. Either using a CPU right now, or sitting in the run queue ready to the instant a CPU frees up. On a healthy box only a few processes are R at a time; if the count climbs and stays there, that's the run queue backing up, and it shows up as high load average.
S — interruptible sleep. The normal resting state, and where most processes spend most of their lives. The process asked for something — a network reply, the next line of input — and is parked, costing nothing, until it arrives. "Interruptible" means a signal can still wake it. A web server idling at 3 a.m. is a wall of S.
D — uninterruptible sleep. This one's worth dwelling on. The process is waiting on the kernel for something it cannot be interrupted out of — almost always disk or network I/O deep in a system call. It won't answer a signal. It won't answer kill -9, the sledgehammer that ends almost anything. A D process is sulking in the corner with its arms folded, waiting on a disk that may never call back — and there is genuinely nothing you can do but wait or reboot. A pile of stuck D processes is the classic fingerprint of failing hardware or a dead NFS mount: the disk stopped answering, and everyone who asked it a question is frozen mid-sentence.
T — stopped. Frozen by a job-control signal — this is what happens when you press Ctrl-Z in a shell, or send SIGSTOP. Not dead, just paused, and fg or a SIGCONT brings it back exactly where it left off.
Z — zombie. The strangest of all, and the one everyone misunderstands. A zombie is a process that has already finished — its code has run, its memory is freed, it's done. What's left is a single line in the kernel's ledger holding one fact: how it died, its exit status. It stays there because that exit status belongs to the parent, and the kernel refuses to throw it away until the parent comes to collect it. You cannot kill a zombie; there is nothing left to kill. It's already dead — that's the whole point.

Note

The states overlap with extra flags ps tacks on: < for high priority, N for low (a "nice" process that yields to others), s for a session leader, l for multithreaded, + for a foreground job. So Ss is a sleeping session leader, R+ a running foreground process. The capital letter is the state; the lowercase trailers are footnotes.

On Zombies and the Parent Who Must Come

The zombie deserves its own moment, because the mechanism behind it is quietly lovely and it explains a whole class of server problems.

When a child process exits, it can't simply vanish. Its parent started it for a reason and is entitled to know how it ended — did it succeed, did it crash, what was its exit code? So the kernel keeps that one fact alive after everything else about the process is gone. The parent collects it by calling wait(), and the moment it does, the kernel erases the last trace and the zombie is reaped — a real Unix term, and exactly the right one. Until then, the dead process lingers as a Z.

Most of the time this happens in microseconds and you never see it. The trouble starts when a parent is written badly — it spawns children and never calls wait(), never reads their exit statuses. Each finished child then lies there unreaped. One zombie costs almost nothing; it holds no memory, burns no CPU, it's just a line in a table. But the table has a fixed number of lines, and a buggy parent forking forever can fill it until the system can't start a single new process. The fix is never to kill the zombie — you can't, it's already dead — but to fix or restart the parent, so it finally collects its children. Kill the parent and the zombies are re-parented, collected, and gone in an instant.

The Process Tree

Because every process is forked by another, the running system is a single tree with no loose branches. ps axf draws it for you, with indentation showing who launched whom (this is a trimmed slice of a real server — a live one has far more):

    PID    PPID STAT COMMAND
      1       0 Ss   systemd
    689       1 Ss   cron
   1031       1 Ss   sshd
3141250    1031 Ss    \_ sshd
3141343 3141250 Ss        \_ sshd
3141365 3141343 Ss            \_ bash
3141367 3141365 R                 \_ ps
   2655       1 Ssl  mysqld
  11571       1 Ss   haproxy
3312708   11571 Sl    \_ haproxy

Read it from the left edge inward. The root is PID 1, the first process the kernel starts at boot. Indented under it are the daemons — the long-lived background services like sshd and cron — and your login session. Notice the chain under sshd: it spawned another sshd for the logged-in session, which started a bash, under which sits the very ps command that drew this tree — a command catching itself in the act, because the shell had just forked and exec()'d it a heartbeat earlier. A service like haproxy shows the other classic shape: a master process with its worker beneath it — the master owns the configuration and the privileged port, the worker does the actual serving, and if the worker dies the master starts a fresh one. The whole shape of who-can-restart-whom is right there in the indentation.

When a parent dies before its children, those children become orphans — but they're never left truly parentless. They're immediately adopted, and the rules of that adoption belong to PID 1; the init system page tells that story. For our purposes here it's enough to know the tree never breaks: cut a branch and the orphaned twigs are grafted straight back to the trunk.

Where a Process Lives: /proc

Here's a thing that delighted me when it finally clicked. On Linux, every running process has a directory — a real, browsable folder — that the kernel conjures into existence the moment the process is born and deletes the instant it dies. It lives under /proc, named for the process's PID. The process running as PID 1234 has a folder at /proc/1234. None of it is on a disk anywhere; it's the kernel answering questions about its own internals, dressed up as files so you can read them with the same tools you read everything else. That's the Unix idea in its purest form — everything is a file, even a thing as abstract as "a program that is currently running."

Look inside one and the whole process is laid bare. The most useful file is status, a plain-text summary — here are the headline lines from a real database daemon:

Name:     mysqld
State:    S (sleeping)
Pid:      2655
PPid:     1
Uid:      108     108     108     108
Gid:      113     113     113     113
VmRSS:    58091960 kB
Threads:  130

Name is the program; State the letter we just met. Pid and PPid are the process and its parent — PPid 1 tells you this daemon was started straight by the init system at boot. Uid/Gid are the user and group it runs as: note this database isn't running as root (0) but as its own unprivileged user (108), which is exactly how a careful service limits the damage a break-in could do. Threads is how many threads of execution live inside this one process — 130 of them here, all sharing its single address space. And VmRSS is the actual physical memory it's holding in RAM right now — about 55 GB on this box. Other entries in the same folder are just as live: cwd is a link to its current directory, exe links to the very executable file it was loaded from, fd/ is a directory of every file and socket it has open, cmdline holds the exact command that launched it, and environ its environment variables. Every tool that "looks at processes" — ps, top, htop — is, underneath, just reading these files and tidying them into columns. There is no secret oracle; there's a folder, and you can read it yourself.

What Dies and What Lingers

When a process ends, most of it disappears cleanly and instantly. Its address space — its private memory — is handed back to the kernel; this is why a memory leak in a program is bounded by the program's lifetime, and why "have you tried restarting it" so often works. Its open files are closed, its network connections torn down, its CPU time forgotten. The walls came down, the room is cleared.

Three things, though, can outlive the process that made them, and knowing which is which saves you hours of confusion:

Children keep running. Killing a parent does not kill its children — they carry on, newly orphaned and re-adopted, as we saw. If you want a whole tree gone, you target the process group or the children explicitly; kill on the parent alone often leaves a surprising amount still alive.
Files on disk persist. Anything the process wrote — a log, a database row, a PID file — is on disk and stays there. The process was a verb; its files are the nouns it left behind.
The exit status lingers — as a zombie — until reaped. Which closes the loop back to where we started: a dead process's last word waits, patiently, for a parent who may or may not come to read it.

Pro Tip

When something "won't die," look at its state first with ps -o pid,stat,comm. A Z can't be killed because it's already dead — fix the parent. A D can't be killed because it's stuck in the kernel waiting on hardware — find the sick disk or hung mount, because no signal will touch it. Reaching for a bigger and bigger kill signal on either of these is effort spent on the wrong problem entirely.