/var: Explanation & Insights

On Unix and Linux systems, /var holds the machine's variable data — logs, caches, spooled mail and print jobs, databases, and other files that grow and change while the system runs.

What It Is

/var is where a Linux system keeps everything it writes down while it's awake. The programs themselves — the compiled binaries, the libraries, the fixed data they ship with — sit perfectly still in /usr, the same after a hundred reboots as before the first. /var is the other half of the machine: the drawer for everything that won't hold still. Logs pile up. Caches swell and drain. Mail waits in a queue, print jobs shuffle forward, databases grow a row at a time, and every service with something to remember between restarts tucks it somewhere under this one branch.

The name gives the whole game away, and it's short for exactly what you'd guess: variable. Not variable as in a name-for-a-value in a program — variable as in changeable, the plain opposite of fixed. The people who laid out the Unix filesystem sorted their directories by one blunt question — does this change while the machine is running, or doesn't it? — and everything that answered yes got swept in here. That is the whole organizing principle of /var. It isn't sorted by what the files are for: a web server's cache, a mail queue, and the package manager's database have nothing in common — nothing except that all three get written to while the machine is awake. That shared restlessness is what puts them under the same roof.

So /var is less a category than a habit. Everywhere else you can hold a file up and ask "what kind of thing is this?" — programs in /usr, configuration in /etc, devices in /dev. /var is the one top-level directory sorted by a verb instead of a noun. It holds the things that move.

Why It Matters

Two reasons, and the first is the one that'll introduce you: /var is usually where the disk fills up. Look at what lives here — logs that grow every second, caches that only ever fatten until someone clears them, databases that pile up, mail and job queues that back up the moment something downstream jams. Everything under /var is, by its very nature, the stuff that grows. The rest of the system holds a roughly fixed size; /var is the part with an appetite. So when a server runs out of disk at three in the morning, the culprit is under this one directory far more often than not, and the first move a seasoned admin makes is to point du at /var and see which child got greedy.

The second reason is quieter, older, and quietly elegant once it clicks. Split the changing data (/var) away from the fixed programs (/usr) and the fixed half can be nailed down — mounted read-only, so nothing can scribble on your installed software by accident; or, back when this rule was written, mounted off a CD-ROM, or served across the network from one machine to a whole room of them. Picture a computer lab in the nineties: thirty identical workstations, every one reading the same /usr off a single file server, not one of them able to write a byte to it — and each keeping its own private /var for the logs and state that are nobody's business but its own. That split is what made it work: one read-only copy of the programs for everybody, a separate writable scratchpad for each. /var exists so that /usr never has to change.

Note

If your system parks /var on its own partition — plenty of production servers do, precisely to fence in the growth — then filling /var won't drag the root filesystem down with it. That's the good news. The bad news is that a full /var still breaks nearly everything anyway, because services can't log, can't cache, and can't queue. A separate partition turns a fatal wound into a bad day. It doesn't stop the wound from hurting.

What Lives Here

Here's the top level of /var on a typical Debian system, with the sizes beside it — a real, healthy machine, nothing wrong with it:

$ ls -la /var/
drwxr-xr-x   backups
drwxr-xr-x   cache
drwxr-xr-x   lib
drwxr-xr-x   local
lrwxrwxrwx   lock -> /run/lock
drwxr-xr-x   log
drwxrwsr-x   mail
drwxr-xr-x   opt
lrwxrwxrwx   run -> /run
drwxr-xr-x   spool
drwxrwxrwt   tmp

$ du -sh /var/* | sort -rh
5.4G   /var/cache
4.0G   /var/lib
3.0G   /var/log
6.8M   /var/backups
52K    /var/tmp
52K    /var/spool
4.0K   /var/mail
0      /var/run
0      /var/lock

Notice how lopsided it is: three directories carry nearly all the weight and the rest is a rounding error. That's normal. Here's what each one is for.

Directory	What it holds
`/var/log`	The machine's diary — boot messages, kernel chatter, logins, and a log per service. The first place you look when something breaks.
`/var/cache`	Work a program did once and saved so it needn't do it twice — downloaded packages, rendered fonts, thumbnails. All of it regenerable.
`/var/lib`	Application state — the real, irreplaceable data: databases, the package manager's records, whatever a service must not forget.
`/var/spool`	Queues — work accepted but not yet done: outgoing mail, print jobs, cron tasks waiting for their moment.
`/var/mail`	Users' inboxes — the classic Unix mailbox, one file per person, where local mail comes to rest.
`/var/tmp`	Temporary files, like `/tmp` — with one crucial difference we'll get to.
`/var/backups`	On Debian, daily copies of a few critical system files (the account list, the package database) — a small safety net, not a real backup.
`/var/opt` and `/var/local`	Variable data for hand-installed software — `/var/opt` for packages under `/opt`, `/var/local` for what you built and dropped in `/usr/local`.
`/var/run`, `/var/lock`	Not really here anymore — signposts pointing elsewhere. More on that below.

Two lines in that listing hide a clue in their permission bits, if you know where to look. /var/mail reads drwxrwsr-x — that little s sitting where you'd expect an x is the setgid bit, which quietly hands every file created inside the directory's own group (mail) instead of the creator's, so all the mail programs can reach each other's dropped-off messages. And /var/tmp reads drwxrwxrwt — that trailing t is the sticky bit: anyone may write to the directory, but you can only delete files you own. It's the one sane way to let the whole system share a single scratch folder without users quietly clobbering each other. The permissions aren't decoration. They're the house rules of a shared room, written right into the listing.

The Difference That Bites: Cache Versus State

Look again at the two heavyweights sitting just behind the logs: /var/cache at 5.4 GB, /var/lib at 4.0 GB. From across the room they're twins — two big opaque folders stuffed with a program's private files. They could hardly be less alike, and mixing them up is how people lose data.

/var/cache is scratch paper. Every last thing in it was derived from something else, and every last thing in it can be built again from nothing. It's the package manager's hoard of downloaded .deb files, fonts baked into a fast binary form, thumbnails a file browser made so it wouldn't have to make them twice. Delete the lot and the machine just shrugs: the next time it wants a cached file that's gone, it does the work over and saves it again. Slower for a heartbeat, whole in every way that counts. When the disk is full and you need room this second, /var/cache is the safe place to reach.

/var/lib is the filing cabinet. It holds state — the data that was derived from nothing and can be rebuilt from nothing. Your database's actual tables are here. The record of every package on the system is here. Free up space by deleting /var/lib and you haven't cleared a cache, you've wiped the machine's memory. This is the one place under /var where a stray rm is a catastrophe and not just an inconvenience.

The whole difference folds into a single line: a cache is something you keep because rebuilding it is slow; state is something you keep because rebuilding it is impossible. They sit side by side under the same parent, look identical from ten feet away, and one is scratch paper while the other is everything the machine can't afford to forget. Learn which is which before you ever need the space — because the moment you do need it you'll be in a hurry, and a hurry is precisely when a hand reaches into the wrong drawer.

Danger

Never claw back disk space by deleting things under /var/lib. That's application state — databases, package records, service data — and it does not come back. Clear /var/cache instead, or reach for the proper tool (for the package manager's own hoard, that's its clean command), which knows what's safe to throw away.

The Two That Packed Up and Left

Two of the entries in that listing — /var/run and /var/lock — aren't directories at all anymore. Follow the arrows: run -> /run and lock -> /run/lock. They're symlinks now, pointing up and clean out of /var, to a directory called /run that hangs directly off the root.

It's worth the detour, because the why is a lovely little bootstrapping puzzle. Both hold runtime data — the ID numbers of running processes, lock files that flag "this resource is busy" — and that data has two habits. First, it's pointless to keep across a reboot: a process ID means nothing once the process (and the whole machine under it) is gone, so writing it to a real disk is effort spent on nothing. Second — and here's the sharp end — services want this scratch space very early in boot, sometimes earlier than /var itself is even mounted. If a program's first act is to drop its process-ID file into /var/run, and /var is a separate partition not yet attached, you've got a chicken and no coop to put it in.

The fix was to lift the data out of /var and into /run, a tmpfs — a filesystem that lives entirely in memory, never so much as touches the disk, wakes up empty at every boot, and can be stood up in the first breath of startup, before any real disk is mounted. It's fast, because it's RAM and not spinning platters; it's spotless on its own, because memory starts empty when you power on; and it's there when the earliest services come knocking. The old paths keep working thanks to those symlinks, so decades of programs that still expect /var/run never notice they've been quietly rerouted. The switch rolled across the Linux world around 2011, and it's the same flavour of housekeeping as the /bin symlinks: an old address kept alive as a forwarding pointer, long after the thing it named moved out.

/var/tmp: Temporary, but Patient

A Linux system has two temporary directories, and the whole difference between them fits in one rule: /tmp gets wiped, /var/tmp gets kept.

Both are scratch space any program can scribble in. But /tmp is cleared out on a schedule — on many systems at every reboot, and sometimes it's a memory-backed filesystem that can't outlive a reboot even if it wanted to. It's for the truly fleeting: the working file a program needs for the next half-minute. /var/tmp is for temporary things that have to survive a restart — a long job's half-finished files, something still wanted tomorrow but not forever. It lives through reboots; it just isn't promised to live through a cleanup. The name pins it exactly: it sits under /var, the changeable stuff, so it's temporary — but it's the patient kind, the drawer you can leave something in overnight. Put the half-minute file in /tmp and the two-day file in /var/tmp, and you'll never be caught out wondering which one vanished.

How I Inspect It

Nothing exotic required — the everyday tools cover all of it. The move you'll make again and again is the same one: find out where the space went.

# Which child of /var is eating the disk? Biggest first.
sudo du -sh /var/* | sort -rh | head

# Go one level deeper into the biggest offender
sudo du -sh /var/lib/* | sort -rh | head

# How full is the partition /var actually sits on?
df -h /var

# What's the split between cache (safe) and state (not)?
sudo du -sh /var/cache /var/lib

Fire that first command the instant a disk-full alarm goes off and it answers the only question that matters right then: what do I delete? Nine times in ten the sorted list points dead at a runaway log under /var/log or a swollen cache — and both of those you can clear without a second thought.

Pro Tip

df -h /var tells you how full the disk is; du -sh /var/* tells you why. Reach for df first — it's instant, because the filesystem already keeps its own free-space tally — and only then set the slower du crawling through the directory tree to find the culprit. Asking du first is like counting every item in the fridge to learn whether it's full, when the door would have told you.

Cheat Sheet

Command	What it does
`du -sh /var/* \\| sort -rh`	Find which part of `/var` is eating the disk.
`df -h /var`	How full is the partition holding `/var`?
`ls -la /var`	See the layout, the symlinks, and the permission bits.
`sudo du -sh /var/cache /var/lib`	Compare the safe-to-clear cache against the never-touch state.
`ls -la /var/spool`	Peek at the queues — stuck mail or print jobs live here.
`readlink /var/run`	Confirm it now points at `/run`.

History and Philosophy

For most of early Unix, /var simply wasn't there. The changing data was scattered through /usr in a jumble of directories whose names survive today as fossils: /usr/adm for administrative and accounting logs, /usr/spool for the mail and print queues, /usr/preserve for the half-written files an editor rescued when it crashed. The variable stuff was just mixed in among the programs, and on a single machine with a single disk, that was perfectly fine.

What broke it was the wish to share. By the early 1980s, sites were running rooms full of workstations and wanting each one to mount a single central /usr over the network instead of hauling its own copy — cheaper, and everybody gets the same software the same day. But you cannot share a /usr read-only if the machines need to write their logs and queues into it. The changing data had to come out. So in the great reorganization that shipped with 4.4BSD in the early 1990s, all the per-machine variable directories were gathered up out of /usr and poured into a single new mount point: /var. The Filesystem Hierarchy Standard later wrote the rule down for the whole of Linux, and stated the reason flat out — /var exists "in order to make it possible to mount /usr read-only."

And that's the part worth keeping. /var wasn't dreamed up because someone found variable data fascinating and decided it deserved a home. It was carved out as the negative space around a different goal: making /usr shareable. Someone wanted to lock the programs down and hand them out across a network, realized the one thing standing in the way was software's habit of scribbling as it runs, and swept all that scribbling into a single directory to get it out from underfoot. /var is the pile that everything-that-changes got swept into so that everything-that-doesn't could finally sit still. The whole directory is the answer to a question about a different directory.

Gotchas

/var filling up breaks far more than you'd guess. When services can't write logs, can't cache, and can't queue, they don't fail politely — they hang, they crash, they refuse to start, often with an error that never once mentions the disk. When several services misbehave at once for no reason you can see, check df -h /var before anything else.
/var/lib is not scratch space. It's the most dangerous directory under /var to "tidy up." Databases and package records live here; deleting them to reclaim space is how a full disk turns into a lost machine.
A deleted log may not hand the space back. If a process still holds the log file open, rm-ing it frees nothing until that process restarts — the space stays claimed by the open file. Truncate it instead (truncate -s 0), or restart the service.
Don't count on /var/run surviving a reboot. It's memory-backed now (by way of /run) and wakes up empty every boot by design. Any program assuming its files persist there across a restart is buggy — that's the entire point of the move.
/var/backups is not a backup plan. On Debian it's a handful of daily copies of a few critical files, sitting on the same disk as the originals. It'll save you from a fat-fingered edit to the account list; it will not save you from a dead drive. Real backups live on another machine.