Logging: Explanation & Insights

Every server keeps a diary it never stops writing, and the day it goes quiet is the day you wish you'd read it.

What It Is

Logging is the habit a server has of writing down what just happened, the instant it happens, so that later — when something has gone wrong and you weren't watching — there is a written account to read.

That's the whole idea, and it's worth saying plainly because it sounds too simple to need a name. A program does something noteworthy: it starts up, it refuses a login, it runs out of room, it crashes. At that moment it scribbles a line of text — a timestamp, who it was, and what happened — and appends it to a file. A log is just that growing pile of lines. Nothing more exotic than a diary the machine keeps about itself, one entry per event, forever.

Here is a real one, from an ordinary Linux server's authentication log:

Jun 18 04:11:52 web01 sshd[3801952]: Failed password for root from 203.0.113.7 port 47804 ssh2

Read it left to right and it tells you a small story: at 04:11:52, on the host web01, the SSH daemon (process 3801952) turned away someone trying to log in as root from a particular address. Nobody was awake to see it. The line was written anyway, and it'll still be there in the morning. That patience — write it down now, in case someone needs it later — is the entire discipline.

To feel why it matters, imagine a server with no logs at all. It boots, it serves, and when it falls over it simply… stops, telling you nothing about why. You'd be a detective at a scene with no witnesses, no fingerprints, no notes — just a body and a shrug. Every serious failure would be a fresh mystery with the evidence already erased. Logging exists so the evidence survives the event. The machine is the only witness that's awake at 4 a.m., and the log is its statement.

Why It Matters

Servers fail when you're not looking. That's not cynicism — it's arithmetic: you watch a screen for maybe a few hours a day, and the machine runs every second of every one of them. The interesting moment — the disk filling, the service crashing, the burst of failed logins — almost always lands in the hours you weren't there. Without a log, that moment is gone the instant it passes. With one, it waited for you, written down in full.

So nearly everything you'll ever do to fix a broken server starts the same way: you go and read what it wrote. "Why did the website go down at 3 a.m.?" is not answered by staring at the website now; it's answered by opening the log and reading the minutes leading up to 3 a.m. The log is the flight recorder — the black box you pull from the wreck to find out what the machine was thinking on the way down. A green admin learns the commands; a good one learns the reflex: something's wrong → read the logs first, guess second.

And logs do more than explain crashes. They're how you notice the slow things — a service quietly retrying all night, an attacker patiently working the front door, an error that's been happening for weeks under a load you only just got big enough to feel. The crash announces itself. The slow rot only shows up in the diary.

A Short History: From a Block of Wood to /var/log

Before logging had a standard, every program improvised. One wrote to its own file, another to a different one in a different format, a third printed to the screen and lost it the moment you looked away. To investigate a problem you had to know each program's private habits — where it stashed its notes, in what shape — and stitch the story together yourself across a dozen incompatible diaries. It worked, barely, the way a town works when everyone speaks a different language.

In the early 1980s at the University of California, Berkeley, a programmer named Eric Allman was building sendmail, the program that moved email around the early internet. Sendmail had a lot to say — every message handed off, every delivery that bounced — and Allman needed somewhere to say it. So he wrote a little companion system to collect those messages: syslog. The idea was almost embarrassingly good: instead of each program inventing its own logging, programs would hand their messages to one service whose only job was to receive log lines and decide where to file them. Write once, in one shape, to one place.

It was too useful to stay inside sendmail. Other programs started speaking syslog, then the kernel did, then everyone did, and a private convenience quietly became the way Unix logs. It ran as the de-facto standard for two decades before anyone bothered to write it down formally — as RFC 3164 in 2001 (so faithful to its origins it's literally titled "BSD syslog"), then modernised as RFC 5424 in 2009. That's the usual order of things in Unix, by the way: the thing gets built and used and loved for twenty years, and then someone writes the specification describing what everybody already does.

Where the word "log" actually comes from

Now indulge a small detour, because the name on all of this turns out to be wonderful, and once you know it you can't un-know it.

Centuries before computers, a sailing ship had a genuinely hard problem: how fast are we going? There's no roadside to watch slide past, just open water. The solution, in use by the 1600s, was beautifully crude. You took an actual chunk of tree — a log of wood — weighted one edge so it floated upright and bit into the water, and you threw it off the back of the ship on a long rope. The log stayed put in the water while the ship sailed away from it, paying the rope out. The rope had knots tied along it at even spaces; a sailor let it run through his hands and counted how many knots slipped past while a small sandglass emptied. More knots in the same time meant more speed — and that is, to this day, why a ship's speed is measured in knots.

Then he wrote the number down. In a book. Kept by the helm, filled with these readings entry after entry, day after day — the ship's running record of speed, heading, weather, events. They called it, naturally, the log-book: the book of what the log told them. By the 1840s sailors had clipped it to just the log, and the word had quietly slipped its mooring — it no longer meant the wood, it meant the record. Any record. The diary of a journey.

That's the word sitting in /var/log on every Linux server alive. When your machine writes a line about a failed login, it is keeping a log-book in the exact sense a ship's mate did three hundred years ago — the running record of a voyage, written as it happens, so someone can read later what the journey was like. The server log is the direct descendant of a block of wood on a rope. The technology changed completely; the idea — write down where you've been, as you go — never had to.

How the Pieces Fit: Facilities, Severities, and /dev/log

Allman's design had two small, sharp ideas in it, and they're still exactly how Linux logging works today. When a program emits a log message, it tags it with two labels.

The first is the facility — who's talking, roughly by category. It's a fixed, old list: kern (the kernel itself), mail, cron (scheduled jobs), daemon (general background services), auth and authpriv (logins and security — authpriv for the sensitive ones), plus a handful of historical relics like lpr (the line printer) and uucp (a way computers copied files over phone lines, long dead). And — the practical ones — local0 through local7, eight deliberately blank slots left for you to assign to your own applications. The facility lets the logging system sort the mail: kernel chatter to one file, mail-server chatter to another.

The second label is the severity — how bad is it — and this is the part worth memorising, because it's a real, fixed scale of eight levels, numbered 0 to 7, most urgent first:

#	Name	Meaning
0	`emerg`	System is unusable — the lights are out
1	`alert`	Act immediately — something critical is failing
2	`crit`	Critical condition — a real failure
3	`err`	An error occurred
4	`warning`	Something's off, but it's still running
5	`notice`	Normal, but worth noting
6	`info`	Routine "here's what I'm doing" chatter
7	`debug`	Firehose — every tiny step, for developers

The scale runs from "the building is on fire" (emerg) down to "I just took a breath" (debug), and the numbers go the opposite way to what you'd guess — 0 is the worst, 7 the most trivial. The point of the scale is the filter: you tell the system "keep everything at warning or worse and throw away the rest," and the routine chatter evaporates while the problems stay. Crank a misbehaving service up to debug and it tells you everything, line after line, until you find the bug — then turn it back down before the firehose drowns your disk.

So how does a message physically get from a program to the log? Through a door in the filesystem called /dev/log. It looks like a file but it isn't one — it's a socket, a tiny mouth that the logging service holds open and listens at. A program with something to say opens /dev/log, posts its line (facility, severity, timestamp, message) through the slot, and goes back to work. It never has to know where the log ends up, or in what format, or whether the log file is being rotated out from under it that very second. It drops the message through the slot and trusts the postal service on the other side to deliver it. That decoupling — the writer doesn't manage the log, it just posts — is the quiet genius of the whole design, and it's why a hundred unrelated programs can all log sanely to the same place without ever colliding.

Where Logs Live: The journald and /var/log Split

For most of Unix history, the answer to "where are the logs?" was one word: /var/log. A directory full of plain text files you can open with any tool you own, one or a few per subsystem:

/var/log/syslog        # the general catch-all (Debian/Ubuntu)
/var/log/messages      # the same idea (RHEL/Fedora family)
/var/log/auth.log      # logins, sudo, SSH — every door event
/var/log/kern.log      # the kernel's own voice
/var/log/nginx/        # many services keep their own subdirectory

The receiver writing those files is a syslog daemon — today almost always rsyslog (configured in /etc/rsyslog.conf) or syslog-ng. They are the modern descendants of Allman's original, still doing the original job: listen at /dev/log, sort by facility and severity, write text to /var/log. The great virtue of this world is that a log is just a text file, and forty years of Unix tools — grep, less, tail, awk — already know how to read it. tail -f /var/log/syslog and you're watching the diary fill in real time.

Then, around 2010, systemd arrived to manage services on most Linux distributions, and it brought its own logger: systemd-journald. The journal made a controversial choice — it stores logs not as plain text but as a structured binary file (under /var/log/journal/), indexed so you can query it fast. You read it not with cat but with its own tool, journalctl:

journalctl -u nginx --since "1 hour ago"   # one service, recent
journalctl -p err -b                        # errors only, this boot
journalctl -f                               # follow live, like tail -f

Because every line is captured with its metadata — which service, which severity, exact microsecond, the process ID — you can slice the entire system's history by any of them instantly. Ask for "only errors, only from nginx, only since the last reboot" and it comes back in a single command. The cost is that you've traded a plain text file any tool can read for a binary blob only journalctl can. That trade is the heart of the long, slightly grumpy debate between the two camps.

On most servers today you have both, side by side, and that confuses everyone at first. journald collects everything firsthand — kernel, boot, every service's output — into its binary journal, and then politely forwards a copy to rsyslog, which writes the familiar text files in /var/log. So the same login failure can appear both in journalctl and in /var/log/auth.log. They're not fighting; one is the indexed master record, the other is the old-fashioned text copy kept for the tools and habits that expect it.

Note

By default, journald often keeps its logs only in memory (under /run), which means they vanish on reboot. If you want the journal to survive a restart — and after a crash you very much do — make sure /var/log/journal/ exists. With it, journald persists to disk; without it, your most important evidence evaporates exactly when you reboot to recover.

Log Rotation: Why Logs Don't Eat the Disk

A diary that's never closed has an obvious problem: it grows forever. A busy server can write gigabytes of logs a day, and a log file left to its own devices will calmly expand until it swallows the entire disk — at which point the server, now unable to write anything, falls over. A surprising number of "the disk is full" emergencies are really "nobody was rotating the logs."

The fix is rotation, handled by a small tool called logrotate that runs quietly each day. Rotation is just good filing: when today's log gets big enough (or old enough), close it, rename it out of the way, and start a fresh empty one. You end up with a little stack of history:

syslog          # today, being written right now
syslog.1        # yesterday
syslog.2.gz     # older still — and compressed to save room
syslog.3.gz
...
syslog.7.gz     # the oldest kept; tomorrow it's deleted

Each night the stack shuffles down one, the newest yesterday gets compressed, and the oldest falls off the end and is gone. You keep a rolling window of recent history — long enough to investigate last week's incident, short enough that the logs never devour the disk. journald does the same thing internally on its own, by size limits, without needing logrotate at all.

Pro Tip

Rotation has one classic foot-gun. If you're watching a live log with tail -f /var/log/syslog and rotation happens overnight, your tail keeps faithfully watching yesterday's file — now renamed syslog.1 — and shows you nothing new, because the action moved to a fresh syslog it never reopened. Use tail -F (capital F): it notices the file was rotated and follows the name, not the old file. One letter, and you stop wondering why your live log went silent at midnight.

The Discipline of Good Logging

Logging is half plumbing, half judgement, and the judgement is the part nobody teaches. A few habits separate a log you'll thank yourself for from one you'll curse:

Log the why, not just the what. Connection failed tells you nothing at 4 a.m. Connection to database db01:5432 failed: timeout after 30s hands you the answer. Write the line you'd want to find, not the line that was easiest to emit.
Pick the right severity, honestly. If everything is an err, nothing is — you've trained yourself to ignore your own alarms. Reserve the high levels for things that genuinely need a human; let the routine be info.
Never, ever log secrets. Passwords, tokens, full credit-card numbers — a log is a file that gets copied, shipped to other servers, and read by people who were never meant to see the live data. A secret in a log is a secret in a hundred places you forgot about. (This is exactly why the authpriv facility exists — to keep the sensitive auth details out of the world-readable logs.)
Logs are write-once truth — guard them. The first thing a competent intruder does after breaking in is edit the logs to erase the break-in. That's why serious shops ship logs off the machine to a separate box the moment they're written: a log an attacker can rewrite is a log you can't trust. The witness is only useful if nobody can get to it.

Get these right and the log stops being a wall of noise you skim in a panic and becomes what it was always meant to be: a calm, honest account of everything that happened while you were asleep.