cgroup: Explanation & Insights

The kernel's accountant and bouncer — it meters what a group of processes may use, and cuts them off.

What It Is

A cgroup — short for control group — is a Linux kernel feature that takes a set of processes and puts a meter and a leash on them as a unit. It does three jobs at once: it limits what that group can use (no more than 2 GB of RAM, no more than half a CPU), it accounts for what the group actually uses (so you can read off exactly how much memory or CPU time these processes have consumed between them), and it isolates them, so one group's appetite can't starve another's. Limit, account, isolate — that's the whole feature in three verbs.

The mental shift that makes cgroups click is this: a normal Linux box treats every process as a free-roaming citizen competing for the same shared pool of CPU, memory, and disk. A runaway process under that model can eat the whole machine, because nothing told it where its allowance ends. cgroups draw boxes around groups of processes and write rules on each box: this group gets this much, and not a byte more. The kernel enforces those rules on every scheduling decision and every memory allocation, billions of times a second, without the processes inside even knowing they're being watched.

If you've never deliberately created a cgroup, here's the part that surprises everyone: you're already using them, heavily, right now. Every service on a modern Linux box runs inside its own cgroup, placed there by systemd the moment it started. Every Docker container is a cgroup wearing a costume. The feature isn't some exotic tool you reach for on special occasions — it's the quiet machinery underneath how Linux runs everything. This page teaches it from zero: the controllers that do the metering, the tree they live in, why there are confusingly two versions of the whole thing, and the one fact that explains more 3 a.m. pages than any other — why your container died with the host showing plenty of free RAM.

Why It Matters

Without cgroups, "this server is running ten things" means ten programs in one big free-for-all. With cgroups, it means ten programs each in a labelled box with a budget, and that changes everything about how you reason about a box.

The first reason it matters is blast radius. On an uncontrolled machine, one memory leak in one minor service can swell until the whole box is out of memory, and then the kernel starts killing things — sometimes the wrong things, because the global out-of-memory killer doesn't care which process was the actual offender. With a memory cgroup limit set on the leaky service, that same leak hits a wall: the group fills its allowance, the kernel kills something inside that group, and the other nine services never notice. The leak becomes a contained incident instead of a full outage. That containment is the entire reason cgroups exist, and it's why the opinionated advice below is "put limits on anything that can leak."

The second reason is accounting. cgroups are how monitoring tools answer "what is this service actually costing me?" — not the whole box, but this one group of processes. systemd reads cgroup counters to show you per-service CPU and memory in systemctl status. Container platforms read them to bill you. And the third reason is the big one: cgroups plus namespaces are how containers are built at all — there is no Docker, no Kubernetes, no pod without this feature underneath. Understand cgroups and a huge amount of "modern infrastructure" stops being magic and becomes plumbing you can see.

The Controllers

cgroups don't manage "resources" in the abstract. The feature is split into controllers (the kernel also calls them subsystems), each owning exactly one resource and knowing how to limit and account for that one thing. You attach the controllers you care about to a group; the rest stay out of the way. Four of them carry almost all the weight in practice.

`cpu` — Time on the Processor

The cpu controller decides how much processor time a group gets. It works two ways, and the difference matters enormously.

Shares (weight) are proportional: give group A a weight of 200 and group B a weight of 100, and when both are busy and the box is full, A gets twice the CPU time of B. But — and this is the kind part — if B is idle, A gets everything. Weights only kick in under contention. They never waste idle capacity.

Quota (cpu.max) is a hard cap, and it behaves nothing like weight. You tell it "this group may use 50 ms of CPU out of every 100 ms period," and the kernel enforces exactly that — even if the rest of the box is bored stiff. This is the setting that produces the single most misread symptom in all of cgroups, so it gets its own section below. For now, plant this flag: a CPU quota does not slow a process down. It freezes it.

`memory` — RAM and the Local OOM Killer

The memory controller caps how much RAM (and, separately, swap) a group may hold. Set memory.max to 2 GB and the processes in that group, all of them together, may never exceed 2 GB resident. This is the most consequential controller on the page, because of what happens when a group hits its ceiling — covered in full under "The Thing Everyone Gets Wrong" below. The headline: the kernel runs an out-of-memory kill scoped to that group, not to the whole machine.

`io` — Bandwidth to the Disk

The io controller throttles and weights block-device traffic — reads and writes to the disk. You can cap a group at, say, 50 MB/s of write throughput, or give it a proportional weight so a backup job can't drown your database's reads. It's the least-used of the four day to day (most leaks and runaways are CPU or memory, not I/O), but it's the answer when one noisy job is starving everyone else's disk access and your iowait is climbing.

`pids` — A Cap on Headcount

The pids controller does one small, lovely thing: it limits how many processes (and threads) a group may create. This is the fork-bomb seatbelt. A classic runaway process that spawns copies of itself without limit can exhaust the system's process table and wedge the entire box — the kind of thing that historically meant a hard reboot. With pids.max set, the offending group hits its ceiling, its fork() calls start failing, and the rest of the machine sails on. Small controller, outsized peace of mind.

There are more — cpuset (pin a group to specific cores), hugetlb, rdma — but cpu, memory, io, and pids are the four you'll meet on a real server.

The Hierarchy

cgroups are arranged as a tree, and the tree is the whole organizing idea. There's a single root at the top, and every group is a node hanging off some parent. A child can never exceed its parent's limits — if the parent cgroup is capped at 4 GB, the sum of all its children is capped at 4 GB too, no matter what generous numbers you write on the children. Limits flow downhill and tighten; they never loosen.

This nesting is exactly how systemd organizes a running system. At the top sit broad slices — system.slice for system services, user.slice for logged-in users — and each service hangs below its slice as its own group, with each service's processes inside that. So myapp.service lives at system.slice/myapp.service, and a memory limit you set on system.slice quietly caps every service beneath it at once. The tree turns "limit this one service" and "limit everything the system runs" into the same operation at different heights.

A process belongs to exactly one cgroup per hierarchy at any moment — it can't be half in one group and half in another. When systemd starts a service, it creates the group and drops the service's first process into it; every child that process forks inherits the same group automatically. That inheritance is why you don't have to label each worker thread by hand: put the parent in the box and the whole family tree comes with it.

cgroup v1 vs v2: Why v2 Won

Here is the muddiest bit of cgroup lore, and the one most tutorials wave at and move past. There are two versions of the cgroup interface, and a modern box runs v2.

cgroup v1 (the original, mid-2000s) made a defensible-sounding choice that turned out to be a trap: each controller got its own separate hierarchy. There was one tree for cpu, an entirely independent tree for memory, another for io, and so on. A process could sit in one position in the CPU tree and a completely different position in the memory tree. In theory, maximum flexibility. In practice, a coordination nightmare — the controllers couldn't easily reason about each other, because they didn't agree on what "the same group" even meant. The most painful symptom: the memory controller and the I/O controller couldn't cooperate on writeback (flushing dirty pages to disk), because a process's "memory group" and "I/O group" might be different boxes entirely. Tools had to juggle a dozen mount points and pray they stayed consistent.

cgroup v2 (merged 2016, the default on every current distro) threw that out and made one rule: a single unified hierarchy. One tree. A process sits in exactly one place in it, and all controllers apply at that one position. Now the memory and I/O controllers share a coherent view of which group a process is in, so they can finally cooperate — unified writeback accounting works, pressure information (PSI) becomes meaningful, and the whole thing is far easier to reason about. The trade-off cgroup v2 imposes is the "no internal processes" rule: a group either has child groups or it has processes directly, not both — which sounds annoying until you realize it's exactly what makes the accounting unambiguous.

That's why v2 won: not more features, but coherence. One tree where everyone agrees on the boxes beats a dozen trees where nobody does. systemd, Docker, and Kubernetes have all moved to v2; you'll only meet v1 on legacy boxes that haven't been rebooted into the unified mode.

Note

You can tell which version a box uses by what's mounted at /sys/fs/cgroup. On v2 it's a single unified tree with files like cgroup.controllers at the root. On v1 you'll see a directory per controller — cpu/, memory/, blkio/ — each a separate mount. A box can even run "hybrid" mode with both, which is exactly as fun to debug as it sounds.

Where It Lives: The `/sys/fs/cgroup` Tree

cgroups follow the deepest Unix instinct of all: they're exposed as files. The entire hierarchy is a virtual filesystem mounted at /sys/fs/cgroup, and you read and write the limits by reading and writing files in it. No special tool required — cat, echo, and a directory tree are the cgroup API.

Walk it on a v2 box and the structure tells the story:

ls /sys/fs/cgroup

cgroup.controllers   cgroup.procs   cpu.stat   io.stat
memory.stat          system.slice/  user.slice/  init.scope/

Those *.slice directories are systemd's top-level groups. Descend into a service and you find its controls and its counters side by side:

cat /sys/fs/cgroup/system.slice/myapp.service/memory.current   # bytes in use right now
cat /sys/fs/cgroup/system.slice/myapp.service/memory.max       # the ceiling (or "max" = unlimited)
cat /sys/fs/cgroup/system.slice/myapp.service/cpu.stat         # CPU time used, and throttling stats
cat /sys/fs/cgroup/system.slice/myapp.service/cgroup.procs     # the PIDs living in this group

Read cgroup.procs and you get the exact list of process IDs in that box. Read memory.current and you see live usage; memory.max shows the limit. This is the same trick top plays with /proc: the kernel publishes the live state as plain files, and every fancy tool on top — systemctl, docker stats, your monitoring agent — is just reading these files a few times a second. Once you've cat'd a memory.current yourself, the container dashboards stop being magic.

Pro Tip

The fastest read on a service's resource health is systemctl status myapp.service — it surfaces the cgroup's live Memory: and CPU: accounting right under the status line, no /sys spelunking required. If you want the tree at a glance, systemd-cgtop is top for cgroups: every slice and service sorted by CPU or memory, refreshed live.

How systemd Uses Cgroups

This connection deserves its own beat, because it's where most readers actually touch cgroups without realizing it. systemd is the init system — process 1, the thing that starts and supervises every service on a modern Linux box. And the way it keeps track of a service is by its cgroup.

When systemd starts myapp.service, it creates a cgroup, launches the service's main process inside it, and from then on, everything that process forks stays in that group — workers, helper scripts, the lot. This solves an old, ugly problem: a daemon that double-forks to background itself used to be able to "escape" its parent, leaving init unable to tell what belonged to the service. With cgroup tracking, escape is impossible. The group is the service's identity. When you run systemctl stop myapp.service, systemd kills every process in the group — no orphans, no stragglers, no kill games.

It also means setting a resource limit on a service is a one-liner. Drop this in the unit file:

[Service]
MemoryMax=2G
CPUQuota=50%
TasksMax=512

MemoryMax writes the group's memory.max, CPUQuota writes cpu.max, TasksMax writes pids.max. No cgcreate, no manual /sys editing — systemd manages the cgroup for you, and systemctl daemon-reload plus a restart applies it. This is, by a wide margin, the right way to use cgroups on a normal server: you almost never create them by hand; you describe limits in unit files and let systemd do the bookkeeping.

The Thing Everyone Gets Wrong

Two cgroup behaviours fool people constantly. Both are worth burning into memory, because misreading either one sends you debugging the wrong thing for an hour.

The Memory cgroup Is Who Actually Kills Your Container

Here's the scenario that pages people at 3 a.m.: a container dies. The orchestrator reports OOMKilled and exit code 137. You SSH to the host, run free or top, and stare in confusion — the host has gigabytes of free RAM. How can a process get out-of-memory-killed on a machine that isn't out of memory?

Because the kill didn't come from the global out-of-memory killer. It came from the memory cgroup. When a group's usage hits its memory.max, the kernel doesn't look at the whole machine — it runs an OOM kill scoped to that group, choosing a victim from the processes inside that box and only that box. The host's free RAM is irrelevant; the group's ceiling is what was breached. The container hit the limit you (or the orchestrator's default) set on it, and the kernel enforced it precisely as instructed.

This is the whole reason a container can OOM on a host with RAM to spare, and it's why exit code 137 (128 + 9, where 9 is SIGKILL) is so common in container land. It's not a bug and not a host problem. It's the memory cgroup doing exactly its job. The fix is almost always one of two things: the container's limit is too low for what it legitimately needs (raise memory.max), or the app has a memory leak and is climbing toward any ceiling you give it (fix the app). The cgroup limit didn't cause the problem — it revealed it, cleanly, instead of letting the leak take a neighbour down with it.

Warning

When you see OOMKilled / exit 137, check the cgroup's memory usage against its limit (memory.max and the memory.events file's oom_kill counter), not the host's. The host having free RAM doesn't clear the container — it confirms the limit was the binding constraint. Looking only at host-wide free is the single most common wrong turn here.

CPU Limits Are Throttling, Not Slowing

The second trap is the CPU quota. People imagine a cpu.max of "50%" means their process runs at half speed — a smooth, gentle slowdown. It does not. It means the process runs full tilt until it has used its slice of the period, then gets frozen for the rest of it.

Picture a 100 ms period with a 50 ms quota. The process runs flat out for 50 ms, then the kernel stops it dead for the remaining 50 ms — not running, not slowed, just suspended on the bench until the next period begins. Then it sprints again, then it's benched again. The averaged-out CPU usage looks like a calm 50%, but the lived experience inside the process is stop-start-stop-start, dozens of times a second.

This matters because of what it does to latency. A request that lands at the start of a run window flies through. A request that lands during a freeze window waits — sometimes tens of milliseconds, doing nothing, for no reason the application can see. The symptom is latency spikes and tail-latency misery, while average CPU% looks perfectly healthy. Teams chase this for days: profiling the code, blaming the database, adding caches — when the real culprit is a CPU quota throttling the process into periodic naps. The tell is in the cgroup's own books: cpu.stat exposes nr_throttled and throttled_usec counters. If those are climbing, your "slow app" is a throttled app, and the fix is to raise (or remove) the quota, not to rewrite the code.

The lesson behind both traps is the same: understand the cgroup before you blame the application. A surprising share of "the app is broken" incidents are really "the box's policy is doing exactly what it was told." Read the limits first.

How To Do It Right

You now know what cgroups do; here's the opinionated guidance on using them, strongest first.

Set a memory limit on anything that can leak. This is the highest-leverage cgroup move there is. Any service with a history of creeping memory — an app server, a worker pool, anything you've watched climb in top — should have a MemoryMax in its unit file. The point isn't to be stingy; it's that one runaway should never be able to take the whole box down. A limit turns "the server fell over and took everything with it" into "one service got restarted and the rest kept serving." That trade is almost always worth it. Pair it with Restart=on-failure so the killed service comes back automatically.

Cap process count on anything that forks. A TasksMax (which writes pids.max) is cheap insurance against a runaway process or fork bomb exhausting the process table. There's rarely a reason a normal service needs thousands of processes; a sane TasksMax costs nothing and closes a nasty failure mode.

Be careful and deliberate with CPU quotas. Given the throttling trap above, prefer CPU weights (shares) over hard quotas when your goal is just "be a good neighbour under contention" — weights never waste idle capacity and never inflict latency spikes. Reach for CPUQuota only when you genuinely need a hard ceiling (billing boundaries, strict multi-tenancy), and when you do, watch cpu.stat's throttling counters so you catch the latency cost before your users do.

Let systemd own the cgroups. Don't hand-craft groups with cgcreate and raw /sys writes on a systemd box — they'll fight, and systemd may reset what you set. Express limits in unit files (systemctl edit myapp.service for a drop-in), reload, restart. The unit file is the source of truth; the cgroup is its shadow.

Containers Are Just cgroups + Namespaces

The payoff that makes all of this connect: a container is not a tiny virtual machine. There's no emulated hardware, no second kernel, no hypervisor. A container is an ordinary group of processes on your host, wrapped in two kernel features working together.

cgroups are half of it — the limit half. They cap how much CPU, memory, and I/O the container's processes may use, and account for what they consume. That's where the resource limits on a Docker container or a Kubernetes pod actually live: as memory.max, cpu.max, and pids.max on a cgroup the runtime created.

Namespaces are the other half — the isolation half. Where cgroups control how much, namespaces control what you can see. A PID namespace gives the container its own process numbering, so its main app sees itself as PID 1 and can't see the host's processes. A mount namespace gives it its own filesystem view; a network namespace its own interfaces and ports. The processes are right there on the host — you can spot them in top — but inside their namespaces they believe they're alone on the machine.

Put them together — cgroups for the budget, namespaces for the blinkers — and you have a container: a process that's metered like a VM and blinkered like a VM, but is really just a normal process the kernel is being strict with. Docker and Kubernetes are, at bottom, very sophisticated tools for setting up cgroups and namespaces and then running a process inside them. Which is why, when a pod dies with OOMKilled and the node has free RAM, you now know exactly what happened: the memory cgroup hit its memory.max and did its job. The whole tower of modern orchestration rests on the plain kernel feature this page is about.