IOWait: Explanation & Insights

The CPU's most misunderstood number: idle, yet somehow accused of being the problem.

What It Is

IOWait — written %wa in top, wa in vmstat, %iowait in mpstat and iostat, and tucked away as the fifth column of /proc/stat — is the percentage of time a CPU core sat idle while at least one disk I/O request was still in flight. Read that sentence twice, because the whole point of this page lives in it. IOWait is not the CPU working. It is the CPU not working, with an asterisk: it had nothing runnable to do, and the only reason it had nothing to do was that something was stuck waiting on storage.

That distinction is the single thing every other explanation leaves muddy, so we'll nail it here, once and for all. When you see a CPU's time split into buckets — user, system, idle, iowait, and a few more — it's tempting to read the list top to bottom as "amounts of work." But iowait isn't work. It is a flavour of idle. The kernel could have lumped it in with plain idle time and nobody would have been technically wrong. It chose to count it separately for one reason: to give you a hint. The hint is "this core would have liked to keep going, but the disk wasn't ready." Everything useful about iowait, and everything dangerous about it, follows from understanding that it is idle wearing a label, not busy.

This page teaches that label from zero: where it shows up, what it actually counts, why it lies in both directions, and — the part that matters when a server is slow at midnight — how to stop trusting it and go read the disks directly. By the end, a high iowait number won't make you nervous and a low one won't reassure you. You'll treat both as breadcrumbs, not verdicts.

Why It Matters

IOWait matters because it's the most common false confession in Linux performance. A server feels slow, an admin opens top, sees %wa sitting at 40, and concludes the CPU is overloaded — so they reach for a bigger instance. They get the bill, they get the new box, and the slowness is exactly the same, because the CPU was never the bottleneck. It was politely standing around waiting for a disk that couldn't keep up. High iowait points at storage, not silicon. Throwing cores at it is like hiring a second chef when the problem is that the pantry is three rooms away.

The flip side is worse, because it's quieter. IOWait can read low — even zero — on a box that is being absolutely murdered by slow I/O. We'll get to exactly why below, but the headline is this: iowait only counts idle time, and if any other process on the machine can use a core while one process waits on the disk, that time gets counted as user or system, not iowait. So a busy multi-core server with one I/O-starved job will happily show a healthy-looking %wa while that one job crawls. The number that's supposed to warn you about disk trouble is the first to go silent on a busy machine. That's the trap, and it's why this page exists.

Here's the opinion this page will defend the whole way down: treat iowait as an invitation to go look at the disks, never as a verdict on them. It tells you where to point your attention, not how bad things are. The real story — which device is slow, by how much, and whether it's failing — is in per-device latency, and you read that with iostat -x, not by squinting at one summary percentage. IOWait is the doorbell. The disks are who's at the door.

What It Really Measures

Let's build the definition up properly, because the precise mechanics are what make the gotchas obvious instead of surprising.

A core, at any given microsecond, is in exactly one state. The kernel tallies those microseconds into buckets and publishes the running totals in /proc/stat, in units called jiffies (clock ticks, usually 1/100th of a second). You can read the raw counters with no tools at all:

cat /proc/stat

cpu  3468258 275873 943939 71221938 53546 0 74863 0 0 0

The columns, in order, are user nice system idle iowait irq softirq steal guest guest_nice. The fifth — 53546 above — is iowait. Every monitoring tool you'll ever touch reads these numbers, waits a second, reads them again, and turns the difference into the percentages you see on screen. The cumulative count itself is meaningless; the rate of change is your reading.

Now, the exact rule the kernel follows when it decides which bucket a tick belongs to. When the scheduler finds a core has nothing runnable — every process is asleep or blocked — it has to charge that idle tick somewhere. It asks one question: is there at least one process on this core that is blocked specifically on disk I/O? If yes, the tick goes to iowait. If no (everything's asleep for other reasons — waiting on a timer, a network packet, a keystroke), the tick goes to plain idle. That's the entire definition. IOWait is "idle, and the reason for the idleness is pending disk I/O." It is a subset of idle, carved out and given its own name so you can spot storage stalls without instrumenting anything.

Two consequences fall straight out of that rule, and they're the two things everyone gets wrong.

First: high iowait does not mean the CPU is the bottleneck. It means the opposite. The CPU is idle. It has spare capacity it can't use because the storage is too slow to feed it. High %wa is the kernel telling you "I could be computing, but I'm waiting on the platters." The fix is faster storage or less I/O — chasing a failing disk, moving from a SATA HDD to an NVMe SSD, adding RAM so the page cache absorbs more reads, or simply doing fewer, larger I/Os. More cores will not help, because the cores were never the constraint.

Second: iowait can read low even when I/O is killing you. This is the dangerous one. The iowait rule only fires on an idle tick. If, while one process is blocked on the disk, any other process can run on that core, the core isn't idle — it's running user or system code — so that tick is counted as user/system, never iowait. On a single-core box with one busy process, a disk stall shows up cleanly as high iowait. On an eight-core box running dozens of processes, the same storage emergency can hide almost entirely: there's nearly always something runnable to keep the cores occupied, so the idle ticks (and therefore the iowait ticks) never accumulate. The busier and more parallel your machine, the better it hides its own I/O stalls behind a respectable %wa. Counterintuitive, but it follows directly from "iowait is a flavour of idle" — no idle, no iowait, no matter how badly the disk is hurting.

Note

IOWait is also not divided cleanly per core in a way you can reason about. The kernel attributes a blocked-on-I/O process's wait to whichever core last ran it, and that process may migrate. So per-core %iowait in mpstat can look lumpy and arbitrary — one core showing 30, the next showing 0 — even though the underlying disk pressure is shared. Read the aggregate, not the per-core scatter, and don't try to map iowait to a specific core the way you might map a pinned compute thread.

How It Relates to Load and the Run Queue

If you've read about load average, a bell should be ringing. Load counts the processes that want a core right now — the run queue — averaged over 1, 5, and 15 minutes. And here's the connection that ties iowait to load and trips up nearly everyone: on Linux, processes blocked on disk I/O count toward load average even though they're using zero CPU.

The mechanism is a process state called uninterruptible sleep, shown as D in top and ps. When a process issues a disk read and the kernel parks it until the data arrives, it goes into D state — and you famously can't even kill a D-state process, not even with -9, until its I/O returns or errors out. Linux deliberately counts D-state processes as part of the run queue. So a pile of processes all stuck waiting on one slow or dead disk drives the load average to the moon while the CPUs sit idle. The classic, unforgettable shape of a storage emergency is: load high, CPU usage low, iowait high, a stack of D-state processes. Three numbers that look contradictory until you know that two of them — load and iowait — are both, in their own way, counting waiting rather than working.

This is why iowait and load are best read together. Load high and CPU usage high near the core count means genuinely CPU-bound work, where more cores might help. Load high but CPU usage low with iowait climbing means the queue is deep because everyone's stuck at the disk, where more cores help nothing. Same load number, opposite diagnosis — and iowait is the tiebreaker.

Where You See It

The same underlying counter, surfaced by four different tools, each suited to a different moment.

top — the live glance. The %Cpu(s) summary line carries wa as one of its eight fields:

%Cpu(s):  4.2 us,  1.1 sy,  0.0 ni, 94.5 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0 st

0.1 wa here is a calm box. This is where you first notice iowait — it's everywhere, it's live, and it's the natural starting point when a server feels off.

vmstat — the trend over seconds. Run vmstat 1 and watch the wa column tick by once a second alongside the b column (processes in uninterruptible sleep — your D-state count):

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  3      0 412600  20480 763400    0    0  8192     4 1203 2401  5  2 18 75  0

wa at 75 with b at 3 is the textbook I/O stall: three processes wedged in D state, the CPU three-quarters idle-but-waiting, and the bi (blocks in) column showing the disk straining to deliver. vmstat is my go-to for "is this momentary or sustained?" — a few seconds of it tells you whether you caught a blip or a problem.

mpstat — per-core, scriptable. mpstat -P ALL 1 breaks %iowait out per logical CPU as a clean stream, the right tool when you want to capture a few seconds of behaviour to read carefully rather than watch interactively. (Mind the per-core caveat in the Note above — read the all row.)

iostat — where you actually solve it. This is the one that matters most, and the rest of this page leans on it. Plain iostat shows the same CPU %iowait, but iostat -x adds the per-device columns that turn the doorbell into an answer. More on that next.

How I Inspect It

The order of glances when a server's slow and I suspect storage — and the discipline is never to stop at iowait.

top, check wa. If it's high, I have a hypothesis (storage), not a conclusion. I also glance at the process list for a clump of D-state entries — that confirms uninterruptible sleep, which confirms it's blocking I/O and not something else.
Cross-check against load average. Load high, CPU usage low, wa high: storage emergency dressed as a CPU one. If load's high but wa's near zero, I'm wrong about storage and I go look at compute instead.
iostat -x 1 — the verdict comes from here, not from iowait. This is the move that took me years to trust over the summary percentage:

iostat -x 1

Device   r/s   w/s   rkB/s   wkB/s  r_await  w_await  aqu-sz  %util
sda    312.0   8.0  9984.0   128.0   84.30    12.10   26.40  99.20
nvme0n1  2.0   1.0    64.0    16.0    0.21     0.18    0.01   1.10

The two columns I live by: await (average milliseconds a request waits, queue time plus service time) and %util (the fraction of time the device had at least one request outstanding). On sda above, r_await of 84.30 ms and %util of 99.20 is a disk pinned to the wall and answering reads glacially — that's your bottleneck, named, with a number. The healthy NVMe beside it answers in a fifth of a millisecond at 1% utilisation. IOWait pointed me at "the disks"; iostat -x told me which disk, how slow, and how saturated. That's the difference between a hint and a diagnosis.

If await is high and climbing for no good reason, suspect the hardware. A disk whose latency is creeping up, with read errors in the logs, may be on its way out — see disk failing. Sometimes high iowait isn't load at all; it's a drive dying, or a flaky SATA cable retrying every read.

Pro Tip

When you see high iowait, resist the reflex to read it as a magnitude ("40% means it's 40% broken"). It isn't a severity score — it's a pointer. The very next command should be iostat -x 1, and the very next columns your eyes hit should be await and %util. The percentage told you which room to walk into; the per-device latency tells you which thing in the room is on fire.

Reading It by Example

Build the instinct with a few readouts and their verdicts. Assume a 4-core box unless noted.

%wa at 0–2, steady — nothing to see. Storage is keeping up with whatever the box asks of it. This is the normal resting state, even on busy servers, because the page cache absorbs most reads in RAM and writes get batched.

%wa at 40, load average at 8, CPU mostly idle, three D-state processes. The signature storage stall. The cores are starving; something is hammering a disk that can't keep up, or a disk that's failing. Go straight to iostat -x and find the device with high await and high %util.

%wa near 0, but one specific job is crawling and the box has 16 busy cores. The hidden case — the one this page keeps warning about. Don't be reassured by the low %wa. There's always something runnable on a busy box, so I/O stalls never show as idle. Run iostat -x anyway; if one device shows high await, your low iowait was lying by omission.

%wa spikes to 90 for two seconds every minute, then drops. A periodic batch — a cron job, a backup, a log rotation, a database checkpoint flushing dirty pages. Often benign, but worth identifying so you know it's expected and not a creeping problem.

%wa at 95 with await in the hundreds of milliseconds and read errors in dmesg. This isn't a load problem; it's a hardware problem. A drive is failing, or a cable is throwing errors and forcing retries. Stop tuning queries and start replacing hardware — see disk failing.

Gotchas

The traps, roughly in order of how often they bite:

IOWait is idle, not busy. The whole page in one line. A core in iowait is doing nothing — it's just nothing-with-a-reason. Never add %wa to %us and %sy and call it "CPU load."
Low iowait doesn't mean healthy I/O. On a busy multi-core box, disk stalls hide inside user/system time because the cores never go idle. Confirm with iostat -x, never trust a low %wa to clear the disk of suspicion.
High iowait doesn't mean the CPU is the problem. It means the opposite — the CPU is waiting. Buying more cores is the classic wrong move; it just adds idle capacity.
IOWait is fuzzy under SMT and migration. Per-core %iowait can look erratic because the kernel attributes a blocked process's wait to whichever core last ran it. Read the aggregate.
It's disk I/O, not all I/O. Despite the name "Input/Output Wait," the Linux iowait counter is specifically about block-device (disk) waits. Time blocked on the network or on a pipe is plain idle/sleep, not iowait — a point the old textbooks blur. If a server's wedged on a dead network mount, you'll see it in load and D-state, not necessarily in %wa.
D-state processes won't die. A process stuck in uninterruptible sleep on a hung disk ignores even kill -9 until its I/O returns or the kernel gives up. A wall of them is what a serious iowait problem looks like in the process list.

History and Philosophy

IOWait is an old idea wearing a Unix coat. The accounting trick — charging idle time to "waiting on I/O" so an operator could tell a storage-bound machine from a genuinely idle one — goes back to the timesharing systems of the 1970s, when a single machine served a roomful of terminals and the operator's whole job was answering "why is it slow this time?" in one glance. Splitting idle into plain-idle and waiting-on-the-disk was a cheap, brilliant way to make the answer visible without any extra instrumentation. The counter was practically free: the scheduler already knew the core was idle and already knew whether a process was blocked on a block device. It just had to write the tick into a different column.

The honest twist is that on modern multi-core machines, iowait is a weaker signal than it was on those single-CPU mainframes — and for a reason that's almost poetic. The metric was designed for a world with one core, where "idle because of the disk" was unambiguous: if the one core was waiting on storage, the whole machine was. Give that machine sixteen cores and dozens of processes, and the very thing that makes it powerful — there's always something else to run — is what erodes the signal, because there's rarely a fully idle moment for iowait to capture. The number didn't get less truthful; the world got more parallel, and a metric built for a simpler machine now describes a corner of a busier one. It's a reminder worth keeping: a measurement is only as meaningful as the assumptions baked into it, and "idle" stopped being a simple word the moment we had more than one core to be idle on.

Which is the whole argument for not stopping at iowait. The kernel kept the column for backward-compatible courtesy and as a useful hint — but the ground truth migrated to the per-device counters that iostat -x reads straight from the block layer. IOWait is the historical signpost; await and %util are the map. Read the signpost, then trust the map.