Backups: Tutorial & Best Practices

The copy that survives the disaster — what a real backup is, and the one simple setup that delivers it.

What It Is

Let's start with the uncomfortable truth that this whole page is built to fix: the backups you think you have probably aren't backups. A Linux server backup is one of those things everyone assumes is handled — by the hosting provider, by the RAID array, by the nightly snapshot someone set up two jobs ago — right up until the morning a disk dies or a DROP TABLE goes to the wrong database, and the assumption evaporates along with the data.

So before anything else, three things that masquerade as backups and are not:

RAID is not a backup. RAID keeps you running when a disk dies. It does nothing when you delete the wrong file — the deletion is faithfully mirrored to every disk in the array, instantly, perfectly. RAID is uptime insurance, not a backup, and confusing the two has cost more data than almost anything else. (We have a whole page on the day a RAID goes offline and what it does and doesn't save you from.)
A snapshot is not a backup. A ZFS or btrfs snapshot lives on the same pool as the data it froze. When the pool dies, the snapshot dies with it. When ransomware (or a careless root) reaches the box, it can delete the snapshots too. We'll come back to this — snapshots are wonderful, but not for the reason most people think.
Replication is not a backup. A streaming replica of your database, a mirrored object bucket, a synced folder — they all copy your data and your mistakes in real time. Corrupt a row, and the corruption arrives at the replica before you've finished pressing Enter.

What, then, is a backup? A separate, independent, restorable copy that survives the original — and its entire environment. Survives the disk failing. Survives the whole server burning down. Survives the datacenter flooding, the cloud account getting suspended, the rm -rf typed into the wrong terminal, the ransomware that encrypts every byte it can reach. If a single catastrophe can take out both the data and the copy, you don't have a backup. You have one copy in two places, which is a very different and much sadder thing.

Here's the promise, and I mean it: backups have a reputation for being a tedious, fiddly, never-quite-finished chore, and that reputation is mostly outdated. By the end of this page you'll have one setup — ten or fifteen lines of config — that genuinely protects a typical server, the kind running a website and a database on one box. We'll survey everything first so you can stop worrying about the ninety percent you don't need, then land hard on the part you do. Backup. Yes, seriously. Let's make it boring.

The One Principle: 3-2-1

Almost everything sensible anyone has ever said about backups collapses into one little number sequence, and if you remember nothing else from this page, remember this: 3-2-1.

3 copies of your data. The live one you're using, plus two backups. Why three and not two? Because backups fail silently — a corrupt archive, a drive that quietly went bad, a job that stopped running weeks ago. With only one backup, the day you reach for it is the day you discover it's no good. The third copy is your margin for the backup that lets you down.
2 different media or locations. Don't keep both copies on the same disk, the same array, or even the same machine. Two files in one place share one fate — a power surge, a filesystem bug, a thief. Spread them so that no single physical failure can claim more than one.
1 offsite. At least one copy lives somewhere else entirely — a different building, a different city, a different company. This is the rule that survives the fire, the flood, and the burglar who walks off with the whole rack. Your local backups protect you from the common disasters; the offsite copy protects you from the rare, total one.

That's the classic version, coined back when "offsite" meant a box of tapes driven to a bank vault. The modern world added two numbers, giving 3-2-1-1-0, and both additions are worth their place:

1 immutable or offline copy. One of your copies must be impossible to alter or delete even by you — or by whatever has compromised your server. Ransomware's whole business model is finding and encrypting your backups along with everything else; an immutable copy is the thing it can't touch. (We'll see exactly how to get this cheaply in a moment — it's the single most important knob on the whole page.)
0 errors — verified by restoring. A backup you have never restored from is a hypothesis, not a backup. The 0 means you have actually proven a restore works, end to end, with your own eyes. This is the rule everyone skips and the one that bites hardest, so it gets its own section near the end.

Everything else on this page — which tool, which storage, how often, how long to keep it — is just the practical machinery for satisfying these five numbers on a real server without it taking over your life. Keep 3-2-1-1-0 in your head as the spine. We're about to hang the meat on it.

The Techniques, Honestly

There's a sprawling world of backup tools and approaches, and most tutorials either drown you in all of them or pick one with no explanation. Let's do the honest tour instead — what each thing is for, and when it's overkill — so you can confidently ignore the parts that don't apply to you. (Spoiler: that's most of them.)

File-level backup tools

This is the category that backs up your actual files — /etc, /srv, your web root, your database dumps — and for a single server it's where you'll live. The field splits cleanly into "the old way" and "the modern way," and the modern way won decisively.

The old way is rsync, tar, or even dd, wired up to cron. It still works, and you'll see it in a thousand blog posts. But it has three quiet problems that the modern tools fixed: no deduplication (back up a 2 GB file that changed by one byte, and you store another 2 GB), no built-in encryption (your data sits in plain text wherever you copied it), and no integrity verification (nothing checks the copy is still readable until you desperately need it). A tar cron job is a backup the way a lean-to is a house — technically shelter, but you wouldn't want to weather a storm in it.

The modern tools — restic, BorgBackup, Kopia — solve all three at once. They split your data into chunks, store each unique chunk exactly once (so a year of daily backups of a slowly-changing server might cost barely more than one full copy), encrypt everything before it leaves the machine, and verify integrity on demand. They are, frankly, a different class of thing, and there's no good reason to hand-roll rsync snapshots in the year you're reading this.

Tool	Dedup	Encryption	Native cloud (object storage)	Notes
restic	Yes	Yes (always on)	Yes — B2, S3, etc. built in	Single Go binary, dead simple, our pick
BorgBackup	Yes (excellent)	Yes	No — needs an SSH target (or `rclone`)	Leaner RAM, superb dedup, slightly more setup
Kopia	Yes	Yes	Yes	Newer, has a GUI, strong but smaller community
rsnapshot	No (hardlinks)	No	No	`rsync` + hardlink rotation; the best of the old guard
rsync / tar / dd	No	No	No	The raw materials; fine for one-off copies, not a backup system

Two of these matter for us. restic and BorgBackup are both excellent, and the choice between them comes down to one practical question we'll settle in the verdict: where does the backup go? Hold that thought.

Filesystem snapshots (ZFS, btrfs, LVM)

Here's where the most expensive misconception in backups lives, so let's kill it cleanly: a snapshot is not a backup.

A snapshot — whether from ZFS, btrfs, or LVM — is a frozen, instant, point-in-time view of your filesystem. It's nearly free to make (it stores only what changes afterward) and it's genuinely magical: you can roll your whole system back to 3 a.m. before that bad deploy, in seconds. But it lives on the same disk or pool as the live data. The pool dies, the snapshot dies. root (or the ransomware wearing root's face) runs zfs destroy, and the snapshot is gone in the same breath as everything else. A snapshot shares the fate of the thing it's protecting — which is the exact opposite of what 3-2-1 demands.

So what are snapshots good for? Two real, valuable things:

A consistent image to back up from. This is the underrated one, and it ties directly into the database section coming up. A snapshot freezes the entire filesystem at one instant, so you can copy from the frozen view while the live system keeps writing. No "the file changed while I was reading it" races.
Cheap local time-travel. Instant rollback of a bad change, recovering a file you deleted an hour ago — without touching your offsite backup at all. Lovely for the small, common "oops."

A snapshot only becomes a backup the moment you send it off the box — zfs send to another machine, or backing up the frozen mount with restic. On its own pool, it's a convenience, not a safety net.

Note

If you run plain ext4 or XFS — as most servers do — do not convert your filesystem to ZFS or btrfs just to get backups. You don't need to. Reach for an LVM snapshot only in one specific situation: grabbing a consistent copy of a live database that's too big to dump. For everything else, a file-level tool reading your files directly is all you need. Filesystem choice and backup strategy are separate decisions; don't let one drag the other.

Push vs pull — the security crux

This sounds like a dry architectural detail. It is actually the difference between surviving a ransomware attack and not, so stay with me for sixty seconds.

Every backup either pushes or pulls:

	Who initiates	Who holds the credentials	The risk
Push	The server sends its data out	The server holds keys to the backup target	A compromised server can reach in and delete the backups too
Pull	The backup server reaches in and grabs the data	The backup server holds the keys; the server holds nothing	A compromised server cannot touch the backups — it doesn't even know where they are

Read that right column twice. The whole game of modern ransomware is: break into your server, then find and destroy your backups so you have no choice but to pay. If your server holds the credentials that can delete from the backup target — the normal setup for push backups — then whoever owns your server owns your backups. They encrypt your live data and wipe your safety net in one motion.

Pull flips this: the backups live on a machine your server has no power over. Even total compromise of the server leaves the backups untouched. It's the safer model, which is why serious shops run pull. The catch is you need a second machine to do the pulling, which is more than a one-server setup has lying around.

So for the typical single server, we push — but we push carefully. The trick is to push to a target that won't honor a delete: an append-only or object-locked destination where your server's credentials can write new backups but cannot delete or overwrite old ones. The ransomware can connect, can even upload garbage, but the existing backups are physically beyond its reach. This is the 1 immutable copy from 3-2-1-1-0, made real with one setting — and it's why the verdict below insists on a delete-disabled key. Don't skip it. It is the difference between an inconvenience and a catastrophe.

How many, where, and how long

Three practical questions, three clean answers.

How many — retention. You don't keep every backup forever; you thin them out on a schedule called GFS (Grandfather-Father-Son). The sensible default is 7 daily, 4 weekly, 12 monthly: every day for the last week, every week for the last month, every month for the last year. That's about 23 backups covering a full year — versus 365 if you kept every daily — and thanks to deduplication they cost a fraction of even that. You get a year of history for pocket change.

Why does that time-depth matter so much? Because the disasters you actually need protection from are often the slow ones. A database row quietly corrupts. A config change breaks something subtle that nobody notices for three weeks. By the time you spot it, a backup that only goes back two days has faithfully captured the broken state in every copy — the damage is already in all your backups. Time-depth is what lets you reach back to before the rot started. Shallow backups protect you from the dramatic disasters; deep ones protect you from the sneaky ones, and the sneaky ones are more common.

Where — offsite storage. The offsite copy wants to be "cheap, dumb, and far away." You don't need a clever server; you need bulk object storage. The standouts for a small operation:

Backblaze B2 — around $6/TB/month, S3-compatible, supports object lock (immutability). The easy, cheap default, and our lead recommendation.
Hetzner Storage Box — a flat-rate SSH/SFTP box, excellent value if you're in or near the EU, and a natural fit for BorgBackup.

Warning

Beware the big-cloud egress trap. Amazon S3's storage looks cheap, but you pay to pull data back out (egress) and sometimes per-request on top — so the day you actually need a full restore, the bill can be a nasty surprise on top of a bad day. Backblaze B2 and Hetzner are priced sanely for the case that matters: getting your data back. Always know your restore cost before the emergency, not during it.

Databases: The Trap That Eats Startups

Now the section that, if you run a database, matters more than everything above combined. This is where confident, technically-sound people lose their data — not through neglect, but through a backup that looked perfect and was quietly poison the whole time.

Picture the canonical setup, because it's probably yours: a web server and a database — MySQL/MariaDB or PostgreSQL — running on the same host. The classic startup box. You set up restic to back up /var, it dutifully copies the database's data directory along with everything else, the job turns green every night, and you sleep well.

Here's the trap. A database is constantly writing, and it spreads a single logical change across many files (and many pages within a file). When your backup tool walks that directory copying files one at a time, it photographs each file at a slightly different instant — this file at 03:00:00.1, that file at 03:00:00.4 — while the database keeps mutating all of them in between. The result is a copy where the pieces don't agree: torn pages (a single page half-old, half-new), half-applied transactions, indexes that point at rows that moved, foreign keys that reference rows the snapshot didn't catch. It's a photograph of a running engine with the parts in mid-stroke.

Danger

The cruelest part: a copied-live-files database backup usually restores cleanly and looks fine — the server starts, tables are there, casual queries work. Then days or weeks later it corrupts, crashes, or returns silently wrong answers, because the damage was baked in at backup time and only surfaces under the right query. This is the worst failure mode there is: a backup that lies to you, that passes a glance and fails in the disaster. Never back up a live database by copying its raw data files.

The vocabulary worth knowing: a copy is crash-consistent if it looks like the state right after a power cut — the database can usually recover from that, the way it recovers from a real crash, by replaying its log. It's application-consistent if it's a clean, transactionally-coherent point with no recovery needed. Naively copying live files often gives you neither — it gives you a smear across time, which no recovery can fix. The fix is to capture a real, consistent point. Two ways:

Dump it, don't copy it. A dump asks the database itself to export a consistent snapshot of its data as a file — and the database, which understands its own transactions, gets it right. For MySQL/MariaDB:

mysqldump --single-transaction --routines --triggers \
  -u backup mydb > /srv/dumps/mydb.sql

The --single-transaction flag is the hero: on InnoDB it opens one consistent-read transaction so the dump sees a single frozen instant without blocking writers — your site stays up while it runs. One honest caveat: this consistency is InnoDB-only. Any MyISAM tables can still tear, because MyISAM doesn't do transactions; the cure is simply to not use MyISAM (every modern MySQL/MariaDB defaults to InnoDB anyway). Full flag reference lives on the mysqldump page. For PostgreSQL, pg_dump gives you this consistency automatically — it runs inside a snapshot-isolated transaction by default, so a plain pg_dump mydb > mydb.sql is already a coherent point-in-time export with no special flag needed.

For small and medium databases — which is to say, most databases on most servers — dumping is simply the answer. It's a clean SQL file, it restores anywhere, and it's beautifully boring.

Or snapshot-then-copy. When a database is too large to dump in a reasonable window, freeze a consistent instant with an LVM, ZFS, or btrfs snapshot, then back up from the frozen mount while the live database keeps running. The snapshot gives you the single-instant consistency that copying live files lacked. (PostgreSQL and MySQL can both recover from a properly-snapshotted filesystem image as if from a crash — crash-consistent, which for them is enough.)

Wire it as a pre-backup hook. The point is automation: the dump must run automatically, right before the file backup, so the fresh dump is what gets backed up every night. You don't want to remember to do it by hand — you'll forget exactly once. borgmatic does this natively: declare postgresql_databases: or mysql_databases: and it dumps before every run. restic users add a pre-backup hook (a small script that runs the dump first), or use a wrapper like backrest or resticprofile that has hook support built in. Running a Docker stack? Dump from inside the database container (docker exec mydb pg_dump …), or stop the container briefly to release its locks before snapshotting its volume — either way, the rule doesn't change.

And the whole section in one sentence, worth taping to your monitor: never back up a live database's raw files — dump first.

What You Should Actually Do

Enough survey. Here's the verdict — the setup I'd put on a typical single server running a website and a database, no hedging. It's deliberately small, because that's the entire point: a real, robust, offsite, immutable, verified backup is about fifteen lines of config, not a weekend project.

The shape: restic → Backblaze B2 (or Hetzner Storage Box) → over a delete-disabled application key → daily via a systemd timer → keep 7 daily / 4 weekly / 12 monthly → prune weekly → restic check every run → with a database dump hook → and one restore drill a quarter. That's it.

Why restic over the equally-excellent Borg? One reason, and it's decisive for this reader: restic talks to cheap object storage natively. Point it straight at a B2 bucket and you're done. Borg needs an SSH target or an rclone bridge to reach object storage, which is more moving parts on a one-server setup. (Borg earns its honored runner-up spot when RAM is precious on a tiny VPS — its dedup is leaner — paired with a Hetzner Storage Box as the SSH target and borgmatic driving it. A completely respectable choice; just slightly more assembly.)

1. Install and initialize. On Debian/Ubuntu, apt install restic. Create the repository once, pointed at your B2 bucket:

export RESTIC_REPOSITORY="b2:my-backup-bucket:server01"
export RESTIC_PASSWORD="a-long-random-passphrase-you-store-safely"
export B2_ACCOUNT_ID="...delete-disabled-key-id..."
export B2_ACCOUNT_KEY="...delete-disabled-key..."
restic init

Tip

Create the B2 application key with delete and overwrite disabled (a writeFiles/listBuckets key without deleteFiles), and turn on Object Lock on the bucket. Now the credentials sitting on your server can add backups but can never remove them — so even if the box is fully owned by ransomware, your history is untouchable. This one setting is what turns "a copy offsite" into the immutable 1 of 3-2-1-1-0. It costs nothing and it's the most important line on this page.

2. The backup command. Dump the database first, then back up files and the dump:

mysqldump --single-transaction --routines --triggers -u backup mydb > /srv/dumps/mydb.sql
restic backup /etc /srv /home /var/www --tag daily
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12
restic check --read-data-subset=5%

That forget line is your GFS retention — restic works out which snapshots to keep. check --read-data-subset=5% re-downloads and re-hashes a rotating 5% of the actual data each run, so over a few weeks you've verified the whole repository is genuinely readable, not just present. (Pruning — the slow part — we do separately and less often; see below.)

3. A systemd timer to run it daily. Drop the four commands above into /usr/local/bin/backup.sh, then create a .service and a .timer:

# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=1h

[Install]
WantedBy=timers.target

Persistent=true means a backup missed because the box was off runs at next boot instead of being silently skipped — the single most important line in the timer. RandomizedDelaySec=1h jitters the start so you're not hammering B2 at exactly midnight with every other server on earth. Enable it with systemctl: systemctl enable --now backup.timer. (cron works too, but the timer's Persistent=true and clean status in systemctl list-timers make it the better tool here.)

4. Prune weekly, separately. Pruning is the expensive operation — it repacks the repository to actually reclaim space from forgotten snapshots — so you don't want it in the daily path slowing things down. Once a week is plenty:

restic prune

A second tiny timer (OnCalendar=weekly) running just this. Backup is cheap and frequent; prune is costly and rare — keeping them apart is what keeps the nightly job fast.

One honest wrinkle, and it's the only genuinely fiddly part of the whole setup: the delete-disabled key we just insisted on can't prune — pruning is deletion, and we deliberately took that power away from the server. That's not a mistake, it's the point; you just move the one deleting operation somewhere ransomware can't reach. Two clean ways, pick one:

Run prune from a trusted machine — your laptop, or a separate admin box — using a full-access key that never lives on the server. The server's nightly key stays delete-only; the rare, privileged prune happens from somewhere a compromised server can't touch. This is the purist's answer.
Use B2 Object Lock with a retention window (say 90 days) instead of a hard delete-disabled key. Now even a full key can't remove anything younger than the window — ransomware included — while your weekly prune still reclaims space from data older than it. One setting, no second machine, and it's the least fuss for a single box.

Either way the principle holds: the credentials doing your nightly backup must never be able to wipe your history. Don't let the convenience of self-pruning quietly hand that power back.

That's the whole system. Install, init with a delete-disabled key, dump, backup, forget, check, time it, prune weekly. If you'd rather not live in the CLI at all, backrest is a single Go binary that wraps restic in a clean web UI — same engine, same B2 repo, same delete-disabled key, with buttons and a schedule editor and dump hooks. No shame in it; it runs the exact setup above and shows you it's working.

The One Rule You Cannot Skip: Test Your Restore

Everything above is the easy ninety percent. Here's the asterisk on "backup is simple," and it's a real one: an untested backup is not a backup — it's a wish. This is the most-violated rule in all of systems administration, and the failure is always the same shape: the backups ran flawlessly for two years, and on the one day they were needed, the restore didn't work — wrong path, missing database grants, an exclude that quietly skipped the one directory that mattered. Nobody knew, because nobody had ever tried.

The fix is not complicated. It's just a thing you have to actually do, once, deliberately:

Restore files to a scratch directory. restic restore latest --target /tmp/restore-test and let it pull a recent snapshot down somewhere harmless.
Diff it against the source. diff -r /etc /tmp/restore-test/etc (or spot-check the directories that matter). You're confirming the bytes that came back are the bytes that went up.
Bring the database back from the dump, in a throwaway instance. Spin up a scratch database (a quick Docker container is perfect), load the dumped .sql into it, and run one real query — count a table you know the size of, fetch a recent row. Watching real data come back from cold storage is the moment a backup stops being faith and becomes fact.

Do this once a quarter — put it on the calendar like a dentist appointment. Fifteen minutes, four times a year, and you've satisfied the 0 of 3-2-1-1-0: you have proven, with your own eyes, that the disaster you're insured against is a disaster you'll actually walk away from. Simple to set up — yes. But you must prove it works once, or all the cleverness above is just hope with good production values.

Where Simple Honestly Stops

I promised you simple, and the one-server recipe above genuinely is. But I won't oversell it — there's a real boundary where this setup stops being enough, and knowing where it is keeps you honest. The recipe stops scaling when:

You're running a fleet, not a server. Five, fifty, five hundred hosts need central configuration and — crucially — monitoring that confirms every single host actually ran its backup last night. One server you can eyeball; a fleet hides the one box that's been silently failing for a month.
You have multi-TB of data over the internet. Backing it up is fine; restoring it is the problem. Pulling 8 TB back down a 1 Gbps link is most of a day — your recovery time (RTO) blows past anything acceptable, and you need a local copy or a seeded restore.
You have strict RPO/RTO targets. A daily backup means up to 24 hours of data loss if disaster strikes at 23:59. If losing a day is unacceptable, you need more frequent snapshots and database point-in-time recovery (continuous WAL/binlog archiving), which is a real step up in complexity.
Your database is clustered or huge. A 500 GB database won't mysqldump in a sane window. That's the territory of pgBackRest, Percona XtraBackup, or WAL archiving — physical, incremental, cluster-aware database backup tools that are their own discipline.
You answer to compliance. Enforced WORM retention, audit trails, legal hold, proof-of-immutability — these need infrastructure and process beyond a delete-disabled key and a good intention.

None of these make the simple recipe wrong; they're just the line where "one server, fifteen lines" graduates into "backup engineering." Cross any of them and it's time to bring in the heavier tools — eyes open, knowing exactly why you outgrew the simple thing.