/var/log: Explanation & Insights

On Linux, the directory where the system and its services write down what happened: boot messages, kernel output, login records, and per-application logs, kept together in one place.

What It Is

/var/log is the directory a Linux machine writes its own history into. When the kernel brings a disk online, when someone types the wrong password, when a web server hands back a page or refuses to start — the record of it lands here, in a file. Nobody sits and reads these files for fun. You open them at exactly one moment: when something has gone wrong and you need to know what the machine was doing in the seconds before it did.

It sits under /var, and the name is the whole story. var is short for variable — the part of the filesystem that is expected to grow and change while the system runs, as opposed to /usr, which holds the installed programs and is meant to sit still. Logs grow. Mail spools grow. Print queues, caches, and databases grow. All of it was gathered under one branch so that the static half of the system — the programs — could, in principle, be mounted read-only, or shared across machines, or burned onto a disk that never changes. /var is the drawer for everything the machine scribbles in the margins as it goes.

Inside, you'll find three kinds of thing. Plain-text logs you can read with your eyes — the system messages, the kernel ring buffer, the authentication record. Per-service subdirectories, one folder each for the busier programs — a mysql/, an nginx/, an apache2/. And a handful of files that look like logs, are named like logs, and will fill your terminal with beeping garbage the instant you try to read them the obvious way — because they aren't text at all.

Why It Matters

Every server you will ever run keeps a diary here, and /var/log is the first place you look when the patient is already sick. A service won't start, a login is being refused, the machine rebooted at three in the morning and nobody knows why — the answer is almost always sitting in a file under this directory, written plainly, waiting for someone to read it. Learning to find it fast is most of what "debugging a server" actually is.

There's a second reason, and it's the one that bites people who've never run a machine before. Logs grow without end, and a disk does not. A service caught in a loop can write the same error a thousand times a second; a rotation job can quietly fail; and /var/log swells until the partition it lives on is completely full — at which point the whole machine begins to fall over, because a Unix system that cannot write to disk cannot do much of anything. It is one of the most common real outages there is, and it has a particular cruelty to it: the disk fills with a program dutifully writing down, over and over, that something is wrong — and eventually one of the things going wrong is the disk filling up, which it also writes down.

Warning

A full disk caused by runaway logs will take services down long before you get a tidy "disk full" error. Watch free space on the partition holding /var/log — when it crosses into the high nineties, you are already in trouble.

What Lives Here

Run ls on a real machine and you'll see something like this — trimmed here to the entries worth knowing, and taken from an actual system:

-rw-r--r--  root  root   alternatives.log
-rw-r--r--  root  root   alternatives.log.1
-rw-r--r--  root  root   alternatives.log.2.gz
-rw-------  root  root   boot.log
-rw-rw----  root  utmp   btmp
-rw-r--r--  root  root   dpkg.log
drwxr-sr-x  root  systemd-journal  journal/
-rw-rw-r--  root  utmp   lastlog
drwxr-x---  root  adm    mysql/
-rw-rw-r--  root  utmp   wtmp
-rw-r--r--  root  root   Xorg.0.log

The plain-text logs are the ones you'll spend your life reading. Their names vary a little between families of Linux — Debian and its relatives use syslog and auth.log; the Red Hat side calls the same things messages and secure — but the idea is identical:

File	What it holds
`syslog` / `messages`	The general firehose — most services log here unless they have their own file.
`auth.log` / `secure`	Authentication: every login, every `sudo`, every failed password. The security diary.
`kern.log`	Messages from the kernel itself — hardware, drivers, the disk that's starting to fail.
`dmesg`	The kernel's boot-time ring buffer, dumped to a file. What the hardware said as the machine woke up.
`boot.log`	The output of services starting up during boot.
`dpkg.log`	On Debian systems, a record of every package installed or removed.

Notice the three alternatives.log files stacked together — alternatives.log, then alternatives.log.1, then alternatives.log.2.gz. That is not clutter. That is logrotate doing its job: aging out the log so it never grows without bound. More on that below — it's the single most important housekeeping mechanism in this directory.

The Files That Aren't Text

Three names in that listing sit under the group utmp and will punish you if you treat them like the others: wtmp, btmp, and lastlog. Point cat at one and your terminal fills with control characters and beeps — because these are not lines of prose. They are arrays of fixed-size C structures, packed end to end. A record in wtmp is a struct utmp: a slot for a username exactly 32 bytes wide, one for the terminal, one 256 bytes wide for the hostname a login came from, a timestamp, a type code. Same size, every record, no delimiters. It's a tiny binary database, and it was built that way on purpose — because you can jump straight to the hundredth record by multiplying, without reading the ninety-nine before it.

You read them with the tools written for the job:

last walks wtmp and prints the login history — who logged in, from where, and how long they stayed.
lastb walks btmp and prints the failed logins — the ones that got the password wrong. On an internet-facing box this fills up fast, and it's the first thing worth checking when you suspect someone's knocking.
lastlog records the single most recent login for every user account on the system, indexed by user ID, which is why it's read with its own command of the same name.

Text, Binary, and the Old Argument in the Corner

Here is a tension worth being honest about, because you will run straight into it. For decades, the deal was simple: logs were plain text. You could grep them, follow one live with tail, page through it with less, or just open the thing and read it. No special tool, no format, no ceremony — a log was a file full of lines, and every admin on Earth already knew how to read a file full of lines.

Then systemd arrived with journald, and moved the system's logging into a structured binary journal, kept under /var/log/journal. You read it with journalctl. The case for it is real and worth understanding: because each entry carries labelled fields instead of a flat line of text, you can ask precise questions — show me only this service, only this boot, only errors and worse — and get answers back quickly, the way you would from a database. It manages its own size, so it never needs the rotation dance. And it can be made tamper-evident, which a plain text file an intruder can simply edit cannot.

The case against is just as real, and it's mostly one sentence long: I used to be able to just open the file. A binary journal is useless without the tool to read it; you can't grep it, can't tail it onto a slow terminal over a bad connection, can't scroll it in the pager you've used for twenty years. Plain text goes anywhere and outlives everything. The two camps have been arguing about this for over a decade, and both are right about what they've each given up.

What almost nobody notices is that this argument was already settled inside this very directory — twice, in opposite directions, forty years ago. Login records went binary and stayed binary; that's wtmp, sitting right there, read by last. Everything else went text and stayed text; that's syslog, sitting right next to it. Structured-binary-you-query versus plain-text-you-read has been living in /var/log, side by side, in one folder, since long before journald reopened the case. The pitch that sounds so modern is one the login records made in the 1970s.

When It Fills the Disk

The mechanism that keeps /var/log from eating the machine is logrotate, and it's worth understanding because when it fails, you need to know what was supposed to be happening. It's a small program, run once a day by cron, that walks a set of rules in /etc/logrotate.conf and the drop-in files beside it. When a log gets too old or too big, logrotate rotates it: auth.log is renamed to auth.log.1, the previous .1 slides to .2, older ones are compressed to save room — auth.log.2.gz — and anything past the keep-limit is deleted for good. The service goes on writing to a fresh, empty auth.log, none the wiser. That whole staircase of numbered, gzipped files you saw in the listing is just history sliding gently toward the exit.

It fails in ordinary, unglamorous ways. A service holds the old file open and keeps writing to it after the rename, so the "rotated" copy quietly keeps growing and the new one stays empty. A single log balloons faster than a once-a-day job can trim it. A config typo means a directory is simply never rotated at all. And because logs are the one thing a stressed system produces more of, the failure tends to arrive exactly when everything else is already going wrong. When you get paged for a full disk, /var/log is the first place to point du — on a real machine I just checked, a perfectly healthy /var/log was holding 2.9 GB, and that's the quiet case.

How I Inspect It

The moves that cover most of what you'll ever do here:

# What's here, biggest first — where is the space going?
sudo du -sh /var/log/* | sort -rh | head

# Watch a log live as it happens (the classic)
sudo tail -f /var/log/syslog

# Find something specific across a log
sudo grep -i "out of memory" /var/log/kern.log

# The kernel's boot-time messages
dmesg | less

# Who logged in, and who tried and failed
last
sudo lastb

# The systemd journal: this boot only, errors and worse
journalctl -b -p err

Pro Tip

On a systemd machine, journalctl -b shows only the current boot and journalctl -b -1 the one before it — which is exactly what you want after an unexplained reboot. It's the fastest way to read the machine's last words before it went down.

Cheat Sheet

Command	What it does
`ls -lh /var/log`	See the log files and their sizes at a glance.
`sudo du -sh /var/log/* \\| sort -rh`	Find which log is eating the disk.
`sudo tail -f /var/log/syslog`	Follow the main log live.
`sudo grep -i error /var/log/syslog`	Search a text log for a term.
`dmesg`	Read the kernel's boot / hardware messages.
`last`	Login history (reads `wtmp`).
`sudo lastb`	Failed logins (reads `btmp`).
`journalctl -b -p err`	Journal: this boot, errors and up.
`journalctl -u nginx`	Journal: one service's messages.

History and Philosophy

The reason so many different programs write to /var/log in the same style is one man's mail server. In the early 1980s, Eric Allman was building sendmail at Berkeley — the program that carried a great deal of the early internet's email — and he needed a way for it to record what it was doing: what got delivered, what bounced, and why. So he wrote syslog, and it shipped with the 4.2 release of Berkeley Unix in 1983. It was written for one program's benefit. It just turned out that every daemon on a system has the same need — somewhere to jot down what happened — so one by one they all learned to speak syslog, and a logging format built for mail quietly became the way Unix logs.

For its first eighteen years or so it wasn't even a written standard, just a thing everyone copied from everyone else, with all the small incompatibilities you'd expect. Only in 2001 did anyone bother to write down what the de-facto behaviour actually was. That's a very Unix story: the convention that everything depends on, standardised long after everything already depended on it.

And the choice to put it all under /var traces to the same instinct that shaped the whole filesystem — sort things by how they behave, not by what they're for. The Filesystem Hierarchy Standard drew the line explicitly so that /usr, the installed software, could be mounted read-only and even shared between machines, while the parts that must change as the system runs — logs, spools, caches — were penned into /var where writing is expected. A read-only program directory and a writable diary, kept firmly apart. It's the plainest sort of good idea, and it's held up for decades.

Gotchas

cat wtmp is not how you read it. Those files are binary. Use last, lastb, and lastlog. If your terminal starts beeping and filling with garbage, run reset and step away from the binary log.
journalctl may show nothing after a reboot. By default many systems keep the journal only in memory (/run), not on disk, so it's wiped every boot. If /var/log/journal doesn't exist, the journal isn't persistent — create it and restart the service to keep history across reboots.
A "rotated" log that keeps growing means a service didn't reopen its file. After logrotate renames a log, the writing service must be told to reopen it; if it wasn't, it keeps writing to the renamed file by its inode, and your fresh log stays empty while the old one balloons. The copytruncate option and proper postrotate hooks exist for exactly this.
Reading logs needs root. Most of /var/log is readable only by root or the adm group — for good reason, since auth.log records security-sensitive events. Reach for sudo.
Deleting a full log with rm may not free the space. If a process still holds the file open, the disk space isn't returned until that process closes it or is restarted. Truncate it instead: sudo truncate -s 0 /var/log/huge.log.