systemd: Tutorial & Best Practices

The first program your kernel starts and the parent of everything else — your whole server, as one graph.

What It Is

systemd is the thing that turns a freshly-booted Linux kernel into a running server. The kernel boots, mounts your root disk, and then does exactly one more thing: it launches a single program and steps back. On almost every Linux box you'll ever touch — Debian, Ubuntu, Fedora, Arch, RHEL — that program is systemd. It starts your database, your web server, your SSH daemon; it restarts them when they crash; it brings the network up in the right order; it captures every log line; it schedules your nightly jobs. When people say "the server is up," what they almost always mean is systemd brought everything up.

Here's the orientation that makes the whole thing click, and it's worth saying on line one: systemd is PID 1. The very first process, the one the kernel hand-starts, the ancestor of every other process on the machine. Every database connection, every shell, every stray python script traces its family tree back to this one program. You don't normally talk to it directly — you talk to it through a remote control called systemctl, and through a log reader called journalctl. This page is about the world behind those two commands: what systemd actually is, the one mental model that makes it stop feeling like sorcery, and how the people who've run servers for years actually use it. By the end, "the service won't start" will be a five-second investigation, not a panic.

One bit of vocabulary up front, because the whole rest of the page leans on it. systemd doesn't think in "programs" or "scripts." It thinks in units — and a unit is just a thing systemd manages. A service is a unit. A mounted disk is a unit. A scheduled timer is a unit. A network socket is a unit. Even the abstract goal "the system is fully booted" is a unit (a target). Once you see that everything is a unit, and that units declare dependencies on each other, your server stops being a pile of scripts and becomes what it really is: a graph.

Your First Look

Let's just look at one. The single command you'll type ten thousand times in your career is systemctl status, pointed at a service. Here's the real output for SSH on this machine:

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-05-18 09:14:26 CEST; 2 weeks 2 days ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 838136 (sshd)
      Tasks: 1 (limit: 37810)
     Memory: 160K (peak: 7.8M, swap: 1.1M, swap peak: 1.1M)
        CPU: 98ms
     CGroup: /system.slice/ssh.service
             └─838136 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

That little dot in the top-left is doing real work: it's green when the service is happily running, red when it has failed, white when it's stopped. Read this block top to bottom and it tells you the whole story of one service:

Loaded — systemd has read the unit file (from /usr/lib/systemd/system/ssh.service), and the service is enabled (it'll start on boot — more on that distinction soon).
Active: active (running) — it's up, and has been for 2 weeks 2 days. That uptime is the service's, not the machine's; a service that restarted ten minutes ago on a box that's been up for a year will say so right here, which is often your first clue that something crashed.
Main PID — the actual process ID, 838136, the same number top and ps would show you. systemd owns this process.
Tasks / Memory / CPU — live resource accounting for this service and everything it spawned. systemd knows that sshd is using exactly 160K of RAM right now, with a peak of 7.8M, because of a trick we'll get to that is genuinely the best part of the whole system.
CGroup — the most important line on the page, and the one nobody explains. Hold that thought.

That's a service that's perfectly fine. Now let's understand why it's fine — and what's really underneath.

How It Works: Units, Dependencies, and the One True Mental Model

Every other tutorial leaves one thing muddy, and it's the thing that matters most, so let's nail it once and for all: what actually happens when you boot, and how does systemd know what to start, in what order?

The answer is the graph. Picture every manageable thing on your server as a node:

.service units — long-running programs: your database, web server, sshd. The bread and butter.
.target units — named groupings, milestones in the boot. multi-user.target means "the system is up and ready for normal multi-user work, but no graphical desktop." graphical.target means "all of that, plus a login screen." These targets are the modern replacement for the old SysV "runlevels" — and they're strictly better, because a target is just a unit that depends on other units, so you can invent your own.
.socket units — a network port or pipe systemd opens on behalf of a service, so it can start the service lazily the moment the first connection arrives.
.timer units — "run this service on a schedule," the modern replacement for cron.
.mount / .automount units — filesystems. Yes, mounting a disk is a unit, derived automatically from your /etc/fstab.
.device units — hardware, synthesized from the kernel as devices appear.

Now the edges between the nodes. Each unit file can declare relationships, and only two of them really matter day to day:

Wants= / Requires= — what must also be running. "Bring this up, and bring that up too." Requires= is strict (if the dependency fails, I fail); Wants= is the gentle version (try to start it, but don't fail me if it doesn't).
After= / Before= — ordering. This is the one that trips everyone up, so etch it in: After= and Before= control only the order; they do not pull anything in. Wants=/Requires= decide whether; After=/Before= decide when. A unit that says After=network.target is saying "if the network is being started, wait for it first" — not "start the network." You need both kinds of edge to express "start the database, and not until the disk is mounted."

So how does boot actually work? The kernel starts systemd as PID 1. systemd looks at its default target (graphical.target on this desktop; on a server it's almost always multi-user.target), which Wants= a pile of other units, which Want still others, all the way down. systemd computes the dependency graph, figures out everything that can run in parallel, and fires it all off at once — only serializing where an After= forces it to wait. That parallelism is most of why a modern Linux box boots in seconds, not minutes. You can watch the graph resolve:

$ systemctl get-default
graphical.target

$ systemctl list-dependencies ssh.service
ssh.service
● ├─-.mount
○ ├─sshd-keygen.service
● ├─system.slice
● └─sysinit.target
●   ├─apparmor.service
●   ├─nftables.service
●   ├─systemd-journald.service
...

That tree is the graph, rendered. ssh.service won't start until sysinit.target is reached, which won't be reached until apparmor, nftables, the journal, and a dozen others are up. systemd worked all of that out from the unit files. You never wrote the order; you only declared the relationships, and it derived the rest.

Note

The order systemd starts things in is not the order you'd read in any file. It's recomputed every boot from the dependency graph, which means adding one new service with the right After= line slots it into exactly the right place automatically — no editing a numbered init script, no guessing. That's the entire reason systemd replaced the old /etc/init.d scripts, where ordering was a fragile sequence of filenames like S20mysql, S30apache.

Going Deeper: The cgroup Trick Nobody Tells You About

Remember that CGroup line in the status output we said to hold onto? This is the magic, and once you see it you'll never trust the old way again.

Here's the problem it solves. A "service" is rarely one process. Your web server forks workers; your database spawns helper threads; a shell script ExecStart might launch three children and a watchdog. The old init systems had no reliable way to know which processes belonged to which service — they tracked a single PID in a file and hoped, and when a service crashed and orphaned its children, those children just… kept running, leaked, haunted the box. Stopping a service often left ghosts behind.

systemd kills that whole class of problem by leaning on a kernel feature called cgroups (control groups). A cgroup is a kernel-enforced box you can put processes in: every process spawned by a process inside the box is automatically also in the box, with no way to escape. systemd puts each service in its own cgroup. So when you start ssh.service, every process it ever spawns is tagged, by the kernel itself, as belonging to that service — and stopping the service means telling the kernel "kill everything in this box," which is atomic and complete. No orphans. No ghosts. Ever.

That's also where all those numbers came from. Because the kernel groups the processes, it can account for them as a group — so Memory: 160K, CPU: 98ms, Tasks: 1 aren't systemd estimating; they're the kernel's own per-cgroup ledger, read straight off. You can see the whole machine arranged this way:

$ systemd-cgls
CGroup /:
-.slice
├─user.slice
│ └─user-1000.slice
│   └─user@1000.service …
│     └─session.slice
│       └─pipewire.service
│         └─2577 /usr/bin/pipewire ...
└─system.slice
  ├─ssh.service
  │ └─838136 sshd: /usr/sbin/sshd -D [listener]
  └─mariadb.service
    └─... mysqld

Look at that tree. system.slice holds your daemons; user.slice holds everything you are running. Every process on the box lives in exactly one branch, and the branch tells you who's responsible for it. The first time you grasp that the entire process table is secretly organized into these accountable boxes — and that this is what lets systemd report exact per-service RAM and guarantee a clean stop — the server stops being a swamp of unrelated PIDs and becomes a tidy org chart. That's the deepest "whoa" on this page: systemctl is just a remote control, and the kernel's cgroup tree is the thing it's actually steering.

How I'm Using It

Here's the actual handful of commands I run, in roughly the order I reach for them. Master these eight and you can operate any Linux server alive.

Look before you leap — status. Always the first move. Green dot, recent restart, last ten log lines, all in one screen:

systemctl status mariadb.service

The four verbs. Start it now, stop it now, restart it (stop then start), or reload it (re-read config without dropping connections — sshd and nginx both support this, and it's gentler than a restart):

sudo systemctl start  mariadb
sudo systemctl stop   mariadb
sudo systemctl restart mariadb
sudo systemctl reload  ssh

(You can drop the .service suffix — systemd assumes .service when you don't say otherwise. systemctl restart ssh and systemctl restart ssh.service are the same command.)

The distinction that confuses everyone — start vs enable. This is the single most important thing on the page after the cgroup trick, so let me be blunt about it: start affects right now; enable affects the next boot. They are completely independent.

sudo systemctl enable  mariadb     # start automatically on every boot (does NOT start it now)
sudo systemctl disable mariadb     # don't start on boot (does NOT stop it now)
sudo systemctl enable --now mariadb   # the one you actually want: boot-persist AND start immediately

The number of times someone installs a service, runs start, sees it working, walks away happy — and then reboots a month later to find it gone, because they never enabled it… it's a rite of passage. Use enable --now and skip the heartbreak. To check, systemctl is-enabled mariadb answers enabled or disabled in one word.

See the whole board. What's running, and — sharper — what has failed:

systemctl list-units --type=service --state=running
systemctl --failed

systemctl --failed is the single best "is anything wrong?" command on a Linux box. On a healthy machine it's beautifully empty:

0 loaded units listed.

Any red line there is a service that tried and died, waiting for you to read its logs — which is the next command.

Read the logs — journalctl. systemd captures the stdout/stderr of every service into a structured binary log, the journal, and journalctl reads it. The two invocations I use constantly:

journalctl -u mariadb.service        # everything this one service has ever logged
journalctl -u mariadb.service -f     # follow it live, like tail -f

journalctl -u <service> is the answer to "why did it crash?" nine times out of ten. The logs are right there, scoped to exactly the service you asked about, no hunting through /var/log for the right file. Add -b for "this boot only," -p err for "errors and worse only," --since "1 hour ago" to bound the time. (BTW — the journal being a binary format was, and still is, the single most controversial decision in systemd's history. The upside is real: every line carries structured metadata — which service, which PID, which priority, which boot — so you can slice it like a database. The downside that purists never forgave: you can't grep the raw file, and a corrupted journal can eat its own tail. You'll meet people with strong feelings; now you know what the fight is about.)

How Services Are Built: The Unit File

Eventually you'll write or read one. A unit file is a small INI-style text file, and they're refreshingly readable. Here's the real, complete cron.service from this machine — a typical minimal service:

[Unit]
Description=Regular background program processing daemon
Documentation=man:cron(8)
After=remote-fs.target nss-user-lookup.target

[Service]
EnvironmentFile=-/etc/default/cron
ExecStart=/usr/sbin/cron -f $EXTRA_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

Three sections, and that's the whole grammar:

[Unit] — identity and relationships. Description= is the human label you see in status. Documentation= points at the man pages. After=/Before=/Wants=/Requires= are the graph edges from earlier. (The leading - in EnvironmentFile=-/etc/default/cron means "don't fail if this file is missing" — a tiny, common idiom.)
[Service] — how to run it. The fields you'll actually set:
- ExecStart= — the command to launch. The heart of the file.
- Type= — how systemd decides the service is "up." simple (the default): the process is the service, up the instant it's launched. forking: the old-school daemon that backgrounds itself, so systemd watches for the parent to exit. oneshot: runs once and exits (a setup task), and systemd considers it "done," not "running." notify: the program actively tells systemd "I'm ready" via a socket — the gold standard, because then After= ordering is genuinely accurate. dbus: ready when it claims a D-Bus name.
- Restart= — the auto-healing knob, and a huge reason to use systemd at all. on-failure restarts only on a crash or non-zero exit; always restarts no matter what; no (default) never does. Pair it with RestartSec=5 to avoid a tight crash-loop hammering the box.
- User= / Group= — drop privileges and run as an unprivileged account. Always do this for anything network-facing.
- ExecStartPre= / ExecStartPost= / ExecStop= / ExecReload= — hooks for "before start," "after start," "how to stop," "how to reload."
- KillMode= — control-group (default, kill the whole cgroup), process (just the main PID), mixed, or none.
- Environment= / EnvironmentFile= — env vars, inline or from a file.
[Install] — what enable does. WantedBy=multi-user.target means "when someone runs systemctl enable cron, hook me onto the multi-user target so I start at boot." This section is only consulted by enable/disable; it's the bridge between a unit file sitting on disk and it actually being part of the boot graph.

Pro Tip

Never edit the unit files under /usr/lib/systemd/system/ — a package update will overwrite your changes without warning. To tweak a service, run sudo systemctl edit <name>, which creates a small override file in /etc/systemd/system/<name>.service.d/override.conf that layers on top of the vendor file. You change only the one line you care about, and it survives upgrades. After any unit-file change, run sudo systemctl daemon-reload so systemd re-reads from disk — forgetting that is the classic "but I changed it and nothing happened" trap.

Timers: cron, Done Right

If you've ever written a crontab, systemd has a replacement that's strictly more powerful: the .timer unit. A timer is a unit whose only job is to start another unit (a service) on a schedule. They come in pairs — backup.timer triggers backup.service. Here are the real timers on this box:

$ systemctl list-timers
NEXT                         LEFT      UNIT                    ACTIVATES
Thu 2026-06-04 00:00:00 CEST 16min     dpkg-db-backup.timer    dpkg-db-backup.service
Thu 2026-06-04 00:13:48 CEST 29min     logrotate.timer         logrotate.service
Thu 2026-06-04 06:32:13 CEST 6h        apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2026-06-04 07:31:43 CEST 7h        anacron.timer           anacron.service

NEXT is when it fires next, LEFT is the countdown, ACTIVATES is the service it'll launch. Why bother over cron? Three real wins: a timer's job runs as a full service, so it's logged in the journal (cron's output famously vanishes into the void or an unread mail spool); it gets the same cgroup accounting and Restart= logic as any service; and the killer feature — Persistent=true, which means "if the machine was asleep or off when this should have fired, run it the moment we boot back up." cron just silently skips missed runs; that's the bug behind a thousand "why didn't my nightly backup run?" mysteries. (You'll still meet cron everywhere — it's simpler for a quick one-liner. But when missed runs matter, that's your cue to reach for a timer instead.)

When Things Break: Troubleshooting

A service won't come up. Here's the exact five-second loop, every time:

systemctl status <service> — read the dot and the last log lines it shows you. Often the error is right there.
journalctl -u <service> -n 50 --no-pager — the last 50 log lines, the full story status only teased. This is where the actual error message lives ("address already in use," "permission denied," "config syntax error on line 12").
systemctl --failed — confirm it's actually failed and not, say, oneshot-finished or never-started.
systemd-analyze verify <unit> — if you just wrote or edited the unit file, this validates the syntax before you waste time wondering why it won't load.

A few specific failure modes worth recognizing instantly:

Unit X.service not found — typo in the name, or you forgot sudo systemctl daemon-reload after creating the file.
status=203/EXEC in the journal — systemd couldn't execute your ExecStart line. Almost always a wrong path or a script that isn't chmod +x.
start request repeated too quickly — your service crashes instantly and Restart=always is dutifully restarting it in a tight loop, until systemd gives up and rate-limits it. Read the logs above this line for the real crash; the loop is a symptom, not the cause.
Service is active (exited), green but no process — that's a Type=oneshot unit that ran and finished cleanly. Not a bug; that's the design.

And the boot itself: systemd-analyze blame lists every unit by how long it took to start, slowest first — the first stop when a box boots slowly:

$ systemd-analyze blame
2min 15.927s fstrim.service
      6.282s fwupd.service
      5.555s NetworkManager-wait-online.service
       773ms mariadb.service

That NetworkManager-wait-online.service taking 5.5s is a near-universal boot-time offender, BTW — it's a unit that deliberately blocks until the network is fully up, and on a box that doesn't need to wait, it's pure dead time you can often disable.

Where Everything Lives

The file map, so it stops feeling like magic:

/usr/lib/systemd/system/ — unit files shipped by your distro and packages. Look, don't touch — upgrades overwrite these.
/etc/systemd/system/ — your local units and overrides. This wins over /usr/lib when both define the same unit, which is how you customize without losing changes. Your systemctl edit overrides land here.
/run/systemd/system/ — runtime-generated units, gone on reboot. systemd's generators write here (e.g. one .mount unit per line in /etc/fstab).
~/.config/systemd/user/ — user services. Yes — systemd runs a second copy of itself per logged-in user (systemctl --user), managing things like your audio daemon, all in user.slice. The same model, scoped to you.
/etc/systemd/system.conf and /etc/systemd/journald.conf — global tuning (default timeouts, journal size limits, whether the journal persists to disk or lives only in RAM).
/var/log/journal/ — where the journal stores its binary logs if persistent storage is enabled (otherwise it's volatile in /run and wiped each boot — a common surprise on minimal installs).

History & Philosophy

systemd arrived in 2010, written by Lennart Poettering and Kay Sievers at Red Hat, and it replaced the venerable SysV init — a system of numbered shell scripts in /etc/init.d that started services strictly one at a time, in filename order, with each script reinventing start/stop/PID-tracking by hand. It was simple, hackable, and slow (serial boots) and unreliable (every script tracked its processes differently, and orphans leaked everywhere). systemd's bet was: model dependencies declaratively, let the computer parallelize, and lean on the kernel's cgroups for bulletproof process tracking. It won — and by the late 2010s essentially every major distribution had adopted it.

It also remains the most ferociously argued-over piece of software in modern Linux, and it's worth knowing why, because the critique is sharp. The old Unix philosophy is "do one thing well, and connect small tools with pipes." systemd, by contrast, grew to manage services and logging and timers and device mounting and network config and DNS resolution and login sessions — a sprawling, integrated suite. To admirers, that integration is exactly the point: these things were always tangled, and pretending otherwise (a dozen incompatible little daemons) was the real mess. To critics, it's a monolith swallowing the base system, and a single PID-1 process with that much surface area is a scary place for a bug to live (and they've happened). There's no neutral ground; pick any Linux forum and you'll find the argument still warm. Knowing both sides means you understand the system and the people who run it.

The idea worth carrying away is the one we opened with: a server is a graph of units with dependencies, and systemd is the engine that resolves it. Once that lands, the daily commands aren't a list of incantations to memorize — they're obvious operations on a structure you can see. status reads a node. start/stop flips a node. enable wires a node into the boot graph. list-dependencies draws the edges. journalctl -u reads what a node has been saying. And systemctl itself is nothing more than a polite little messenger, carrying your wishes to PID 1 — the one process that has been running, quietly orchestrating everything, since the very first instant the kernel let go.