CPU: Explanation & Insights

One number says the CPU is busy; underneath, that's eight different stories.

What It Is

The CPU — Central Processing Unit — is the part of the machine that actually does things. Everything else stores, moves, or feeds it: the disk remembers, the RAM holds the working set, the network ferries bytes in and out. The CPU is the only component that computes. It fetches an instruction, decodes it, executes it, writes the result, and reaches for the next — billions of times a second, faithfully, mindlessly, forever. A server is a pile of memory and storage wrapped around this one tireless engine, and almost everything you'll tune or worry about comes down to keeping that engine fed and not jammed.

It helps to be precise about a word that gets thrown around loosely. People call the CPU the "brain" of the computer, which is a charming image and almost entirely wrong — a brain learns, associates, and forgets; the CPU does none of that. It is closer to an impossibly fast clerk who can only follow instructions, one at a time, with no judgement and no memory of what it did a microsecond ago. The intelligence lives in the program; the CPU just executes it, very, very quickly. The whole art of running a server is arranging for that clerk to spend its time on your work instead of waiting around.

This page is the orientation map for the compute side of a Linux box. It ties together the ideas that branch off it: what a core really is versus a hardware thread, how the kernel's scheduler creates the illusion that hundreds of processes run at once on a handful of cores, what the single number "CPU usage" actually hides (it's at least eight different things rolled into one percentage), how clock speed and turbo and thermal throttling make "how fast is it" a moving target, and the difference between CPU usage and load average that trips up nearly everyone the first time. By the end, the numbers in top should read like a sentence instead of a wall of figures.

Why It Matters

The CPU is the loudest component you own. A disk that's quietly failing says nothing until it's too late; a saturated CPU screams in every tool on the box the moment it happens. That makes it the first thing people look at — and the most misread. "The CPU is at 100%, we need a bigger server" is the single most common reflex a new admin reaches for, and it's wrong about as often as it's right. Sometimes the cores are genuinely pinned doing real work. Sometimes they're "busy" doing nothing of the kind — blocked waiting on a slow disk, or quietly handed to someone else's virtual machine on the same physical host. The number lies by omission, and learning to take it apart is most of what this page is for.

It matters in the other direction too. The CPU is the resource that, unlike the disk, you genuinely can get back. A pinned core clears the moment the work finishes; a stuck process can be killed; a runaway can be reniced or capped. There's no permanent loss the way a dead drive loses data — which is exactly why CPU problems are worth understanding rather than fearing. Read the right numbers and the fix is usually cheap. Read the wrong number and you'll spend money on a bigger machine that solves nothing, because the bottleneck was never the cores at all. The two problem pages that hang off this one — high load and overheating — are both, at heart, stories about misreading what the CPU is doing.

Cores, Threads, and the SMT Sleight of Hand

Start with the physical reality and build up, because the marketing has muddied three different things into one.

A core is a complete execution engine — fetch, decode, arithmetic units, the lot. It can run one stream of instructions at a time. A chip with four cores can genuinely do four things at the same instant; that's real, physical parallelism. When you hear "quad-core," that's four clerks at four desks.

A hardware thread is subtler, and it's where Intel's Hyper-Threading (the generic name is SMT, Simultaneous Multithreading) earns its reputation for confusing people. A modern core has more internal machinery than a single instruction stream can usually keep busy — while one stream stalls waiting for data from memory, whole arithmetic units sit idle. SMT lets one physical core pretend to be two by interleaving a second instruction stream into those idle gaps. The operating system sees two "CPUs"; the silicon has one core doing double duty whenever it can steal a moment. It is not two cores — under a workload that genuinely saturates the core, the second thread buys you maybe 15–30%, not 100%. It's a clerk handling a second customer's paperwork in the seconds the first customer spends thinking.

This is exactly the gap between two numbers everyone confuses. On the box this page was written on, here's the truth straight from lscpu:

lscpu

Model name:            Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz
CPU(s):                8
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1

Read it bottom-up: one socket (one physical chip), four cores per socket, two threads per core — so 4 × 2 = 8, and the kernel reports CPU(s): 8. That 8 is what nproc prints and what every scheduler and thread pool will believe:

nproc

But there are only four real cores. If you size a thread pool for eight and your work is compute-bound, you've over-committed by a factor of two and the threads will fight over four engines. The fix isn't a rule of thumb; it's knowing the difference. Thread(s) per core: 2 is the line that tells you SMT is on and your "8 CPUs" are really 4 cores counted twice.

Note

A socket is a physical chip slot on the motherboard; a core is one execution engine inside it; a thread (in the SMT sense) is a logical CPU the kernel schedules onto. Big servers have multiple sockets, which adds a wrinkle called NUMA — memory attached to one socket is slower to reach from the other — but on the single-socket machines most people run, socket count is just 1 and you can forget it.

How the Scheduler Runs Hundreds of Things on Eight

Here's the question that should be nagging you: a busy server runs hundreds of processes, yet it has eight logical CPUs at most. How do they all run at once? They don't. They take turns, fast enough that you can't see the seam.

The kernel component that arranges this is the scheduler (on Linux, modern kernels use one called EEVDF, the successor to the long-serving CFS). Its job is to slice time. Each runnable process gets the CPU for a brief slice — often a few milliseconds — then the scheduler saves its state (registers, program counter), restores the next process's state, and lets that one run. This swap is a context switch, and it happens thousands of times a second per core. Stack enough of these slices end to end and a hundred processes each get a sip of CPU so frequently that, to a human and to the programs themselves, they all appear to run continuously and simultaneously. It's the same trick film uses: still frames flicked past fast enough to read as motion.

A process only competes for a slice when it actually wants the CPU. Most processes, most of the time, are asleep — blocked waiting for a network packet, a disk read, a timer, a keystroke. A web server idling at 3 a.m. has its worker processes parked in the kernel, costing nothing, until a request wakes one. The set of processes that do want CPU right now is the run queue, and its length over time is precisely what load average measures — which is why load and usage are different animals, a distinction we'll sharpen shortly.

The scheduler also decides who gets more. That's where nice values come in: a process's niceness (−20 to +19) biases how large a share of CPU it's granted when there's contention. A "nice" process (high number) yields to others; a low number is greedy. You nudge a running one with renice — renice 5 1234 makes PID 1234 more polite, handing more of the contested CPU to everything else. Niceness only bites when cores are scarce; on an idle box everyone runs flat out regardless.

What "CPU Usage" Actually Is

Now the heart of the page. "CPU usage: 100%" sounds like one fact. It's a summary of at least eight distinct states, and the kernel tracks every one of them separately. The raw source of truth is a file — because on Linux almost everything is — and you can read it with no tools at all:

cat /proc/stat

cpu  3468258 275873 943939 71221938 53546 0 74863 0 0 0
cpu0 403071 31242 171828 8901911 6602 0 6942 0 0 0
cpu1 390905 29417 112494 8946955 6967 0 9462 0 0 0

The first line aggregates all cores; the cpuN lines break it out per logical CPU. Each number is a running total of jiffies — clock ticks (typically 1/100th of a second) spent in that state since boot. The columns, in order, are the entire vocabulary of "busy":

Column	Name	What the CPU was doing
1	user	Running normal process code in user space — your application doing its actual work.
2	nice	Running user code that was reniced to a low priority. Same as user, just deprioritised — counted apart so you can see it.
3	system	Running kernel code on a process's behalf — system calls, handling I/O, managing memory.
4	idle	Doing nothing at all. A core with nowhere to be.
5	iowait	Idle, but only because every runnable thing is blocked waiting on the disk. See below — this is the trap.
6	irq	Servicing hardware interrupts — a device raising its hand to say "I need attention now."
7	softirq	Servicing deferred interrupt work, the kernel's "I'll finish that in a moment" queue (network packet processing lives here).
8	steal	Time the hypervisor ran someone else's VM while your vCPU wanted to run. The one nobody knows about.

The last two columns (guest, guest_nice) only matter if your machine is itself hosting VMs. Every monitoring tool you'll ever use — top, mpstat, the dashboard you're reading this to understand — does the same thing: read these counters, wait a second, read them again, and divide the difference into percentages. The numbers are cumulative; the rate of change is your usage.

The cleanest way to see all eight as live percentages, per core, is mpstat (from the sysstat package):

mpstat -P ALL 1 1

21:51:24   CPU   %usr  %nice  %sys %iowait  %irq  %soft  %steal  %guest  %gnice  %idle
21:51:25   all  15.66   0.00  4.11    0.13  0.00   0.51    0.00    0.00    0.00  79.59
21:51:25     0  18.56   0.00  1.03    1.03  0.00   0.00    0.00    0.00    0.00  79.38
21:51:25     1  15.00   0.00 10.00    0.00  0.00   2.00    0.00    0.00    0.00  73.00

That all row is the headline; the numbered rows are the same eight states broken out per logical CPU, which is how you catch the classic single-threaded-program-pinning-one-core pattern that the averaged number politely hides.

Steal and Iowait: Why "Busy" Is a Lie

Two of those columns dismantle the whole "100% usage = problem" reflex, and they're the magic of this page. Both describe a CPU that is not computing and yet shows up, to a careless glance, as a server in trouble.

Start with iowait (%iowait). A core in iowait is idle — it has no instructions to run — but the only reason it's idle is that every process that wants to run is blocked waiting for the disk to answer. The CPU isn't the bottleneck; it's sitting on its hands because the storage can't feed it fast enough. High iowait with low user/system time is the unmistakable signature of a storage-bound box, and the fix is a faster disk or less I/O — throwing more CPU at it does nothing, because the CPU was never the problem. This is the counterintuitive truth worth pinning down once and for all: a core can be "busy" in the sense of "not available for other work" while doing precisely zero computation. Busy waiting is still waiting. (See iowait for the full treatment.)

Then there's steal (%steal), the one that genuinely surprises people the first time they see it. If your server is a virtual machine — and in the cloud, it almost certainly is — your "CPU" is a virtual CPU that the hypervisor schedules onto a real physical core, sharing that core among several tenants. When your vCPU is ready to run but the hypervisor is busy running someone else's VM on the physical core you'd have used, that lost time is counted as steal. Non-zero %steal means, in plain words: you are being quietly metered, and right now a noisy neighbour is getting the silicon you're paying for. Your application feels sluggish, your own usage looks low, and nothing inside your VM explains it — because the cause is outside your VM entirely. A few percent of steal is normal background noise on shared tiers; sustained double-digit steal is your provider telling you (without telling you) that the box is oversold. The fix isn't in your code or your config; it's a dedicated-CPU instance, or a different host. On the machine this page was measured on, %steal is a clean 0.00 — it's bare metal, not a guest — but on a cheap cloud VM under load it's often the first number that explains an otherwise inexplicable slowdown.

So "busy" is at least three different stories wearing one number: busy computing (high user/system — real work, maybe genuinely needs more cores), busy waiting on disk (high iowait — needs faster storage, not more CPU), and robbed by the hypervisor (high steal — needs a better neighbour). Reading the breakdown instead of the summary is the whole skill.

Clock Speed, Turbo, and Throttling

"How fast is the CPU?" feels like it should have one answer — a number on the box, 1.30GHz on the chip above. It doesn't. Modern CPUs change their clock speed constantly, many times a second, and the sticker number is closer to a baseline than a promise.

The chip above is labelled 1.30GHz, yet lscpu reports a max of 3900.0000 MHz and a min of 400.0000 MHz. That's nearly a tenfold range. At idle the cores drop to a few hundred megahertz to save power and heat; under a single-threaded burst they leap to 3.9 GHz via turbo boost (Intel) or boost clock (AMD) — running far above the rated speed for as long as thermal and power headroom allows. Read the live frequency straight from /proc/cpuinfo and you'll catch them mid-flight, every core at a different speed:

grep MHz /proc/cpuinfo

cpu MHz : 1897.644
cpu MHz : 2140.473

What decides the speed at any instant is the frequency governor — a kernel policy. The common ones are powersave (favour low frequency and battery/heat, ramping up only under demand) and performance (hold high frequencies, latency over efficiency). The box here is on powersave:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

powersave

Counterintuitively, powersave is the sane default even on many servers — modern hardware ramps to full speed under load in microseconds, so the savings are nearly free. You only force performance when you're chasing the last slivers of latency.

And there's a hard ceiling above all of this: thermal throttling. A CPU that gets too hot will deliberately slow itself down to avoid cooking — it would rather be slow than dead. So a server with a clogged heatsink or a failed fan can show "low CPU usage" and still be inexplicably sluggish, because the cores are being held at a crawl by the thermal limiter. That's the link to overheating: a temperature problem masquerades perfectly as a performance problem, and the only way to tell them apart is to watch the temperature alongside the clock speed.

Warning

Don't benchmark on a laptop or a thermally constrained box and extrapolate. The first ten seconds run at turbo speed; then the chip heats up, throttles, and settles into a much lower sustained clock. A 30-second benchmark and a 5-minute one can disagree by 40% on the same hardware — purely because the later run is throttled. Measure the steady state, not the opening sprint.

CPU Usage Is Not Load Average

This is the confusion that costs the most hours, so let's settle it. They measure two different things and they answer to two different fixes.

CPU usage is what fraction of the time the cores spent doing something — the percentages from /proc/stat above. It's a utilisation figure, capped at 100% per core, and it tells you how hard the engines worked over the last interval.

Load average is how many processes wanted a core — the length of the run queue, averaged over 1, 5, and 15 minutes. It's a count, not a percentage, and it is not capped. On an 8-CPU box, a load of 8.0 means "on average, eight things wanted to run and there were exactly eight CPUs to run them" — full but not overcommitted. A load of 16 means twice as many things wanted CPU as there were CPUs to give, so half of them were always waiting in line.

The reason you need both: usage tells you the cores are full; load tells you how deep the queue behind them is. And here's the kicker that ties this page back to the iowait section — on Linux, processes blocked on disk I/O count toward load average even though they're using zero CPU. So you can see a load of 40 on a machine whose CPUs are 95% idle, because all 40 processes are stuck waiting on a dead disk mount, not crunching numbers. Load high, usage low, iowait high: that's a storage emergency dressed as a CPU one. (The full diagnosis lives in high load.)

The rule of thumb: a healthy box has a load average roughly at or below its core count. Above that, work is queuing. But always check why it's queuing — for CPU, or for something the CPU is merely waiting on.

How I Read It

Over the shoulder, the order of glances when a server feels slow:

top first, always, because it's everywhere and it's live. Glance at the %Cpu(s) summary line — us (user), sy (system), id (idle), wa (iowait), st (steal) are the abbreviations of the /proc/stat columns. If wa is high, I'm done thinking about CPU and I go look at the disk. If st is non-zero on a cloud box, I note that the host is overcommitted before I blame anything of mine. See top.
Press 1 in top to expand the per-core view. One core pinned at 100% while seven idle is a single-threaded program, and "buy a bigger server" won't help — it'll just have more idle cores. That one keystroke has saved more wasted hardware budgets than any tool I know.
mpstat -P ALL 1 when I want the same per-core breakdown as a clean, scriptable stream instead of an interactive screen. It's the right tool when I'm capturing a few seconds of behaviour to read carefully. See mpstat.
Compare usage against load average (the three numbers in top's header, or cat /proc/loadavg). Usage high and load high near core count: genuinely CPU-bound, scaling might help. Usage low but load high: something's blocked — almost always disk, sometimes a stuck mount. Check wa.
lscpu and nproc to know what I'm even working with — how many real cores hide behind the logical count, whether SMT is on, what the turbo ceiling is. You can't judge "8 CPUs at 100%" until you know it's really 4 cores.

The whole sequence is about refusing to trust the one big number until I've seen the breakdown behind it.

/proc/cpuinfo, Field by Field

/proc/cpuinfo is the chip's full ID card, one block per logical CPU. The fields worth knowing:

processor — the logical CPU number (0 through 7 here). One block each, so the file is long on a big box.
model name — the marketing name, e.g. Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz. The @ 1.30GHz is the base clock, not what it's running at now.
cpu MHz — the current live frequency of that logical CPU, the same turbo-and-governor dance from above.
cpu cores vs siblings — cpu cores: 4, siblings: 8. Siblings being double the cores is SMT confirmed once more: 8 logical, 4 physical.
cache size — the L2/L3 cache the core can see (8192 KB here). Cache is the small, blisteringly fast memory that keeps the core from waiting on main RAM — and cache misses are a huge slice of "why is this slow."
flags — the capability list: avx512f, aes, vmx, sse4_2… each a hardware feature. aes means hardware-accelerated encryption; vmx means it can host VMs; the avx family is wide vector math. This is how software at boot decides which fast path it's allowed to take.
bogomips — and here's a small joy.

Backstory

bogomips stands for bogus MIPS, and it is exactly as tongue-in-cheek as it sounds. Early in Linux's life the kernel needed to calibrate a tiny busy-loop for short delays, so at boot it spins a do-nothing loop and counts how many iterations fit in a known span of time. Linus Torvalds named the resulting figure BogoMIPS — "the number of million times per second a processor can do absolutely nothing" — and printed it at boot with a warning not to take it seriously as a benchmark. Decades later it's still there, in /proc/cpuinfo and the boot log, reading 2995.20 on this very machine. A joke that shipped, survived every rewrite, and outlived most of the hardware it ever measured. It tells you nothing useful about performance and everything lovely about how Unix actually got built — by people who labelled their hacks honestly and then never threw them away.

Cheat Sheet

The moves worth keeping within reach:

# --- What hardware am I on? ---
lscpu                              # cores, threads, sockets, turbo ceiling, flags — one screen
nproc                              # how many logical CPUs the scheduler sees
grep -c ^processor /proc/cpuinfo   # same count, straight from the file
cat /proc/cpuinfo                  # the full per-CPU ID card

# --- What is it doing right now? ---
top                                # live, interactive; press 1 for per-core, P to sort by CPU
htop                              # the friendlier interactive version (per-core bars)
mpstat -P ALL 1                    # per-core %usr/%sys/%iowait/%steal as a clean stream
cat /proc/stat                     # the raw counters every tool reads
cat /proc/loadavg                  # the three load numbers, no frills

# --- How fast is it clocking? ---
grep MHz /proc/cpuinfo                                   # current live frequency per CPU
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # powersave / performance

# --- Bias the scheduler ---
nice -n 10 some-heavy-job          # start a job at low priority
renice 5 1234                      # make running PID 1234 more polite