IRQ: Explanation & Insights

How a device taps the CPU on the shoulder instead of making it sit and watch.

What It Is

IRQ stands for Interrupt Request, and the whole idea fits in one sentence: it's the mechanism a piece of hardware uses to say "I need attention now" without the CPU having to keep checking. A network card receives a packet, a disk finishes a read, a key is pressed, a timer expires — each of these raises an interrupt, a tiny electrical signal that yanks the CPU away from whatever it was doing so the kernel can deal with the event right then. The processor drops its current work mid-stride, saves enough state to come back to it, runs a short piece of code called the interrupt handler, and then picks up exactly where it left off. All of this happens in microseconds, millions of times a second, completely invisible to the program that was running.

To feel why this matters, you have to picture the alternative. Imagine the CPU had no way to be interrupted. The only way it could ever learn that a packet had arrived on the NIC would be to ask — to stop its real work, read the card's status register, see "nothing yet," go back to work, then ask again a moment later, forever. That's polling, and it's exactly as wasteful as it sounds: the processor burns cycles asking "anything? anything? anything?" instead of doing useful work, and a device that needs servicing right now still waits until the next time the CPU happens to look. Interrupts flip the relationship around. The CPU stops asking and gets on with its work; the device promises to raise its hand the instant it has something. The clerk doesn't stand by the mailbox refreshing it — the mail carrier rings the bell. That single inversion is one of the foundational tricks that lets a server stay fast, and it's the thread that runs through this whole page and its sibling, DMA.

This page teaches interrupts from zero: what hardware versus software interrupts are, what actually happens in the handler (and why the kernel splits the work into a fast half and a deferred half), how to read /proc/interrupts, what the hi and si numbers in top are really telling you, and the one pattern worth burning into memory — a single core pegged in software-interrupt time while the rest sit idle, almost always a busy NIC dumping all its work on one CPU, and what you do about it.

Why It Matters

Most days you never think about interrupts, and that's the system working as designed. They're plumbing — and like good plumbing, you only notice them when something backs up. But when you do notice them, it's usually one specific, confusing symptom: a server that feels slow or drops network traffic, where top shows the overall CPU barely warm, plenty of idle headroom on most cores — and yet one single core is pinned, not in your code, not in the kernel doing your work, but in si: software-interrupt time. Every other tool says the box has spare capacity. The box is nonetheless choking. If you don't know what si means, you can stare at that screen for an hour.

Here's the direction, stated plainly, because this page has an opinion: interrupts are how a server delegates instead of babysitting, and 99% of the time you leave them completely alone. They self-tune, the kernel and irqbalance spread them around, and hand-tuning interrupt affinity is an advanced move you'll rarely need. But the one time it pays off is exactly the pattern above — one core drowning in si while its siblings idle. That's a device firehosing all its interrupts at a single CPU, and now you know it has a name (interrupt affinity), a cause (every interrupt from one device landing on one core), and a fix (spreading them out, via IRQ affinity or RPS). Reading the breakdown instead of the headline number is, once again, the whole skill — the same lesson the CPU page hammers, applied to a different bucket.

Hardware Interrupts vs Software Interrupts

The word "interrupt" covers two related things, and keeping them straight saves a lot of confusion.

A hardware interrupt is the real, physical thing: a signal asserted by a device — a NIC, an NVMe drive, a SATA controller, the system timer, the keyboard — that travels to the CPU and forces it to stop and respond. This is the hi (hardware interrupt) bucket you'll see in top. On modern systems the routing is handled by a chip called the APIC (Advanced Programmable Interrupt Controller), the descendant of the old PC's PIC, which decides which core each device's interrupt is delivered to. Hardware interrupts are asynchronous — they can arrive at literally any instant, between any two instructions of whatever was running.

A software interrupt is the kernel's own mechanism for deferred work, and on Linux the everyday form is the softirq, which shows up as si in top. This is where the confusion usually starts, so let's resolve it now rather than later: a software interrupt isn't hardware raising its hand. It's the kernel saying "I'll finish that in a moment." When a hardware interrupt fires, the handler does the absolute minimum and then schedules the heavy lifting to run slightly later as a softirq — and that split is important enough to get its own section.

(There's also the historical sense of "software interrupt" — the int 0x80 instruction that old Linux used to enter the kernel for a system call. Modern CPUs use a dedicated syscall instruction instead, and system-call time lands in the sy bucket, not si. When a server admin says "softirq" today, they mean the deferred-work kind, and so do we for the rest of this page.)

Top Half and Bottom Half: The Handler, Split in Two

This is the idea that makes interrupts click, and most tutorials leave it muddy. When a device raises a hardware interrupt, the CPU is forced to drop everything — including, often, with other interrupts disabled. So the worst possible thing the handler could do is spend a long time in there. While it runs, that core can't do anything else, and other devices' interrupts may be stalled behind it. An interrupt handler must be fast.

But the work a device needs done often isn't fast. A packet arriving on a NIC needs to be pulled off the card, walked up the TCP stack, matched to a socket, and handed to an application — that's real processing. You can't do all of that with interrupts disabled without strangling the machine.

Linux's answer is to split every interrupt handler into two halves:

The top half (the hardware-interrupt handler proper) runs immediately, with interrupts disabled, and does only the urgent minimum: acknowledge the device so it stops yelling, grab the data into a buffer, and schedule the rest for later. Then it returns, fast. This is the hi time.
The bottom half (on Linux, the softirq) runs a moment later, with interrupts re-enabled, and does the heavy lifting — walking the packet up the stack, waking the waiting process. This is the si time.

It's the emergency-room model: triage at the door (top half — fast, just enough to stabilize and log), real treatment in the ward (bottom half — takes as long as it takes, but the door stays open for the next patient). The genius is that the door is only ever blocked for a heartbeat.

When the softirq load gets heavy — a NIC under a real traffic flood — the kernel doesn't try to handle it all inline. It hands the overflow to a dedicated kernel process you can actually see in top: ksoftirqd, one per core (ksoftirqd/0, ksoftirqd/1, …). So if you ever see ksoftirqd/3 eating CPU near the top of top, that's not a mystery process — that's core 3's softirq backlog made visible, almost always network packet processing that can't keep up. It's one of the few times the invisible plumbing surfaces with a name you can point at.

Reading /proc/interrupts

The kernel publishes the entire interrupt picture as a file, the same way it publishes everything else — read it with no special tools at all:

cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3
  0:         18          0          0          0   IO-APIC    2-edge      timer
  9:          0          0          0          0   IO-APIC    9-fasteoi   acpi
 16:      40213          0          0          0   IO-APIC   16-fasteoi   ehci_hcd
120:    1843922          0          0          0   PCI-MSI   524288-edge  eth0-rx-0
121:          0    1755012          0          0   PCI-MSI   524289-edge  eth0-rx-1
122:          0          0          0          7   PCI-MSI   327680-edge  nvme0q0
123:    2950481          0          0          0   PCI-MSI   327681-edge  nvme0q1
 NMI:         3          2          2          2   Non-maskable interrupts
 LOC:    9928104    9923551    9901223    9890876   Local timer interrupts

Read it left to right. The first column is the IRQ number (or an abbreviation for the special ones at the bottom). Then one count per core — a running total of how many times that interrupt has fired on each CPU since boot. Then the controller (IO-APIC, PCI-MSI), the trigger type, and finally the device name. The counts are cumulative, so the way you actually use this file is to read it twice a second apart and watch which numbers move — that tells you which device is busy right now, and which core is paying for it.

Two things jump out of the example above, and both are the point of the whole page:

The line eth0-rx-0 (a NIC receive queue) has 1.8 million interrupts, and every single one landed on CPU0. CPU1, 2, 3 are all zero for that line. That is the firehose-on-one-core pattern, sitting right there in the file. If the box also has a busy eth0-rx-1 pinned to CPU1, the card is spreading load across two queues (good); if everything piles onto CPU0, that's where your si time is coming from, and that's the thing IRQ affinity fixes.
nvme0q1 (an NVMe drive's queue) has 2.9 million interrupts on CPU0 too. Storage and network competing for the same core is a classic accidental bottleneck — and now you can see it instead of guessing.

The LOC line at the bottom is the local timer — the heartbeat that drives the scheduler's time-slicing, ticking on every core. The NMI line is non-maskable interrupts, the ones the CPU can't ignore even with interrupts disabled (used for hardware-fault watchdogs). Both are normal; you read past them to the device lines that matter.

IRQ Affinity: Pinning Interrupts to Cores

Every interrupt has an affinity — the set of cores it's allowed to be delivered on. By default the kernel, helped by a daemon called irqbalance, spreads them around so no single core carries everything. You can read and (carefully) change a given IRQ's affinity through /proc:

# Which cores is IRQ 120 (eth0-rx-0) allowed to run on? (bitmask)
cat /proc/irq/120/smp_affinity

# Pin it to CPU2 only (bit 2 set = the value 4)
echo 4 > /proc/irq/120/smp_affinity

The value is a CPU bitmask: 1 is CPU0, 2 is CPU1, 4 is CPU2, 8 is CPU3, and f (binary 1111) is "any of the first four." There's a friendlier smp_affinity_list next to it that takes plain core numbers (0-3, 2) if you'd rather not do binary in your head at 2 a.m.

The reason this exists is the pattern this page keeps circling back to. A high-traffic NIC on a single receive queue throws all its interrupts at one core. That core saturates in si while the others idle, and your network throughput hits a ceiling that has nothing to do with the link speed or the CPU's total capacity — one engine is overloaded while three sleep. The structural fixes, strongest first:

Multiple hardware queues (RSS). Modern NICs split incoming traffic across several receive queues by hashing each flow, and each queue gets its own IRQ that can land on its own core (the eth0-rx-0, eth0-rx-1… lines above). This is the real fix and it's mostly automatic on decent hardware.
RPS (Receive Packet Steering). A software version for cards that lack enough hardware queues — the kernel fans the softirq processing out across cores itself. Configured per-queue under /sys/class/net/eth0/queues/.
Manual IRQ affinity. Pinning specific IRQs to specific cores by hand, as above. The blunt instrument, and the one to reach for last.

Pro Tip

Before you touch any of this, check whether irqbalance is running (systemctl status irqbalance). On almost every general-purpose server it's already doing a fine job, and manually pinning IRQs while it's active just means it'll move them back. Hand-tuning affinity is for the specific case where you've seen one core pegged in si and confirmed the rest are idle — not a routine optimization to apply on a healthy box.

hi and si in top and mpstat

You'll meet interrupts most often not in /proc/interrupts but in the CPU summary line of top, where two of the eight time buckets belong to them:

%Cpu(s):  4.2 us,  1.1 sy,  0.0 ni, 88.5 id,  0.1 wa,  0.3 hi,  5.8 si,  0.0 st

hi — time spent in hardware interrupt handlers (the top halves). Normally a rounding error; you'll usually see 0.0.
si — time spent in software interrupts (softirqs, the bottom halves). Also usually tiny — but this is the number that climbs under heavy network load.

The trap is that the headline %Cpu(s) line averages across all cores, which hides the exact problem this page is about. A box where one of four cores is drowning at 23% si and the other three are at 0.0 averages out to a calm-looking 5.8 si — nothing alarming in the summary. You have to look per-core to see the disaster. In top, press 1 to split the line into one row per core. Better, reach for mpstat, which gives you the per-core breakdown as a clean stream:

mpstat -P ALL 1

21:51:25   CPU   %usr  %sys %iowait  %irq  %soft  %idle
21:51:25   all  4.20  1.10    0.10  0.30   5.80  88.50
21:51:25     0  3.00  1.00    0.00  1.20  23.10  71.70
21:51:25     1  4.50  1.20    0.20  0.00   0.10  94.00
21:51:25     2  4.90  1.10    0.10  0.00   0.20  93.70
21:51:25     3  4.40  1.10    0.10  0.00   0.00  94.40

There it is, undeniable: CPU0 at 23.10 %soft while the other three loaf along near zero. That's a single device — almost always a busy NIC — dumping all its softirq work on one core. The total looked fine. The per-core view tells the truth. This is the reading that justifies reaching for IRQ affinity or RPS; without it, you'd be tuning blind. (%irq is hi; %soft is si — mpstat just spells them out.)

MSI and MSI-X

A quick note on the modern delivery mechanism, because you'll see it in /proc/interrupts and it's worth a sentence. The old PC design had a small fixed number of physical interrupt lines (the famous "IRQ 0 through 15"), and when devices outnumbered lines they had to share a line — the source of the legacy "IRQ conflict" headaches. MSI (Message Signalled Interrupts) and its successor MSI-X did away with the wires entirely: instead of asserting a physical line, a PCI device signals an interrupt by writing a small message to a special memory address. That's why the controller column in the example above reads PCI-MSI rather than IO-APIC. The practical upshot for a server admin: vastly more interrupt "lines" available, no more sharing-induced conflicts, and crucially, MSI-X lets a single device have many independent interrupts — which is exactly what makes a multi-queue NIC (one IRQ per receive queue, each steerable to its own core) possible in the first place. The conflict-resolution era is mostly history; MSI-X is why.

The Interrupt Storm

The classic interrupt failure mode, and the reason "interrupts can hurt you" isn't just theory. An interrupt storm is when a device fires interrupts far faster than the CPU can possibly service them — sometimes because the hardware is faulty, sometimes because a driver is buggy and never properly acknowledges the interrupt (so the device keeps re-raising it), sometimes just because a NIC is genuinely receiving packets faster than one core can process them. The symptom is a core buried in hi or si time, doing almost nothing useful, because every time it tries to get back to real work, another interrupt drags it away. In the pathological case the machine can become unresponsive while looking, by total CPU usage, only moderately busy.

This is precisely why high-performance network drivers don't run on pure interrupts. Linux's NAPI mechanism is the elegant fix: under light load the NIC uses interrupts (low latency, no wasted cycles when idle); but the moment traffic gets heavy, the driver disables the interrupt and switches to polling the card in a tight softirq loop for a while — because when packets are arriving constantly, asking "any more?" and always getting "yes" is far cheaper than taking an interrupt for every single one. It's the best of both worlds, and notice the lovely symmetry: the same polling we dismissed at the top of this page as wasteful becomes the smart choice once the device is busy enough that the answer is always yes. Interrupts win when events are rare; polling wins when they're constant. NAPI just picks the right one for the moment.

Cheat Sheet

The moves worth keeping within reach:

# --- See the interrupt picture ---
cat /proc/interrupts                 # every IRQ, per-core counts, device names
watch -n1 'cat /proc/interrupts'     # watch which counts move = which device is busy now

# --- Spot the one-core-pegged-in-softirq pattern ---
top                                  # then press 1 to split %Cpu by core; watch the si column
mpstat -P ALL 1                      # per-core %irq (hi) and %soft (si) as a clean stream

# --- Read / set affinity (advanced — only when one core is provably pegged) ---
cat /proc/irq/120/smp_affinity_list  # which cores IRQ 120 may run on (friendly form)
echo 2-3 > /proc/irq/120/smp_affinity_list   # restrict it to CPU2 and CPU3

# --- The balancer that usually does this for you ---
systemctl status irqbalance          # is something already spreading IRQs around?