The 3-2-1-1-0 backup rule, applied to a Linux server

3-2-1-1-0 is easy to say:

author
Zahara Schaefer
published
updated
read
8 min
tested
restic 0.14 or newer (0.16.1+ for RESTIC_REST_*), rest-server 0.13 or newer. Commands checked against restic 0.19.1 and rest-server 0.14.0; Ubuntu 22.04's apt restic 0.12.1 fails the --from-repo flags.
outline {
  1. What each number means here
  2. What you need
  3. 1. Install restic
  4. 2. Create the local repository
  5. 3. Take the first backup
  6. 4. Add the off-site repository in object storage
  7. 5. Add the append-only copy
  8. 6. Schedule it with systemd
  9. 7. Check the repositories
  10. 8. Restore something, on purpose
  11. Troubleshooting
  12. References
  • 3 copies of your data;
  • on 2 kinds of storage;
  • 1 off-site;
  • 1 offline or immutable;
  • 0 errors on the last restore test.

It's harder to actually set up. This guide does it for one Linux server with restic, a free, encrypted, deduplicating backup tool. If your servers are Proxmox guests, use the Proxmox backup checklist for the guests and this guide for anything outside them.

What each number means here

NumberWhat it means on this server
3 copiesThe live data, a local restic repository, and an off-site repository
2 kinds of storageA local disk or NAS, plus object storage
1 off-siteThe object storage repository, in another provider's building
1 offline or immutableAn append-only repository that the server can add to but can't delete from
0 errorsrestic check on a schedule, plus a restore you actually test

The point of the fourth copy: if someone gets root on the server, they also get every credential stored on it. An append-only target means those credentials can add snapshots but can't remove old ones.

What you need

  • A Debian or Ubuntu server, with the data you want to back up. You need restic 0.14.0 or newer (--from-repo). RESTIC_REST_USERNAME and RESTIC_REST_PASSWORD need 0.16.1 or newer.
  • A second disk or a NAS share mounted on the server, for example at /mnt/backup.
  • An S3-compatible object storage bucket, with an access key limited to that bucket.
  • A second small machine for the append-only rest-server. This can be a VM at another site, or a box on a separate network.
  • About an hour.

Install restic

Ubuntu 22.04's apt package is restic 0.12.1, which does not have --from-repo (that arrived in 0.14.0). Debian 12 ships 0.14.0, which still lacks RESTIC_REST_* (those arrived in 0.16.1). On those releases, skip apt and install the official binary. restic self-update only works on that official binary, not the distro package.

On a current Debian 13 host, apt is usually new enough:

bash
sudo apt update
sudo apt install restic
restic version

Check it worked: restic version prints 0.14.0 or newer. If you will use RESTIC_REST_USERNAME, it must be 0.16.1 or newer.

Create the local repository

Restic encrypts every repository with a password. Generate one, then store it in your password manager before you go any further:

bash
sudo install -d -m 700 /etc/restic
openssl rand -base64 32 | sudo tee /etc/restic/password >/dev/null
sudo chmod 600 /etc/restic/password

Create the repository:

bash
sudo restic -r /mnt/backup/restic --password-file /etc/restic/password init

Check it worked: the command prints created restic repository ... at /mnt/backup/restic.

Take the first backup

Pick what to include. For most servers that's /etc, /home, /srv, selected /var/lib state, and any database dumps. Back up a dump file rather than a running database's data directory. The include list below still has /var/lib, so the exclude list must drop live database directories and Docker volumes, or you back up inconsistent files and waste space:

bash
# Example for Postgres; writes a dump that restic then backs up
sudo sh -c 'umask 077; install -d -m 700 /var/backups; sudo -u postgres pg_dumpall > /var/backups/pg_dumpall.sql'

Write the include and exclude lists to files, so every run uses the same set:

bash
printf '%s\n' /etc /home /srv /var/lib /var/backups | sudo tee /etc/restic/include.txt
printf '%s\n' \
  '/var/lib/postgresql' \
  '/var/lib/mysql' \
  '/var/lib/docker/overlay2' \
  '/var/lib/docker/volumes' \
  '/var/lib/apt/lists' \
  '*.cache' | sudo tee /etc/restic/exclude.txt

Run the backup:

bash
sudo restic -r /mnt/backup/restic --password-file /etc/restic/password \
  backup --files-from /etc/restic/include.txt --exclude-file /etc/restic/exclude.txt \
  --tag nightly

Check it worked: restic snapshots lists one snapshot with the nightly tag.

Add the off-site repository in object storage

Restic speaks S3 directly. Put the credentials in an environment file that only root can read:

bash
sudo tee /etc/restic/offsite.env >/dev/null <<'ENV'
AWS_ACCESS_KEY_ID=<key id>
AWS_SECRET_ACCESS_KEY=<secret>
RESTIC_REPOSITORY=s3:https://<s3-endpoint>/<bucket>/restic
RESTIC_PASSWORD_FILE=/etc/restic/password
ENV
sudo chmod 600 /etc/restic/offsite.env

Initialise it with the same chunker parameters as the local repository, so restic copy can deduplicate between them:

bash
sudo bash -c 'set -a; . /etc/restic/offsite.env; set +a; \
  restic init --from-repo /mnt/backup/restic --from-password-file /etc/restic/password \
  --copy-chunker-params'

Then copy the snapshots across:

bash
sudo bash -c 'set -a; . /etc/restic/offsite.env; set +a; \
  restic copy --from-repo /mnt/backup/restic --from-password-file /etc/restic/password'

Copying from the local repository, rather than running a second full backup, means both copies hold the same snapshots.

Check it worked: running restic snapshots against the off-site repository lists the same snapshot IDs as the local one.

Add the append-only copy

rest-server serves restic repositories over HTTP. With --append-only, clients can add data but can't delete or overwrite it.

On the second machine:

bash
# Download the rest-server release binary for your platform from
# https://github.com/restic/rest-server/releases and put it in /usr/local/bin
sudo useradd --system --home /srv/restic restic
sudo install -d -o restic -g restic /srv/restic
sudo htpasswd -B -c /srv/restic/.htpasswd server01

A .lan name will not get a publicly trusted certificate. Create one (or use your internal CA) and make the key readable by the restic service user:

bash
sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 825 -nodes \
  -keyout /etc/ssl/rest.key -out /etc/ssl/rest.pem \
  -subj '/CN=backup2.example.lan' \
  -addext 'subjectAltName=DNS:backup2.example.lan'
sudo chown restic:restic /etc/ssl/rest.key /etc/ssl/rest.pem
sudo chmod 600 /etc/ssl/rest.key

Copy rest.pem to the machine you are backing up, as /etc/restic/rest-ca.pem (mode 600, root-only). If you already run an internal CA, put that CA file there instead.

Run rest-server with append-only and private repositories (each user can only see its own path):

bash
rest-server --path /srv/restic --append-only --private-repos \
  --listen :8000 --tls --tls-cert /etc/ssl/rest.pem --tls-key /etc/ssl/rest.key

Run it as a systemd service under the restic user. The project ships an example unit in its repository. htpasswd comes from the apache2-utils package.

On the server, initialise and copy, as before. RESTIC_REST_USERNAME and RESTIC_REST_PASSWORD need restic 0.16.1 or newer. On older versions, put the credentials in the URL instead. In the nightly script, load these from a root-only file the same way as offsite.env.

bash
export RESTIC_REPOSITORY='rest:https://backup2.example.lan:8000/server01'
export RESTIC_REST_USERNAME=server01
export RESTIC_REST_PASSWORD='<password>'
export RESTIC_PASSWORD_FILE=/etc/restic/password
export RESTIC_CACERT=/etc/restic/rest-ca.pem
restic init --from-repo /mnt/backup/restic --from-password-file /etc/restic/password --copy-chunker-params
restic copy --from-repo /mnt/backup/restic --from-password-file /etc/restic/password

Include RESTIC_CACERT=/etc/restic/rest-ca.pem in /etc/restic/append.env. Without it, restic init over HTTPS to a .lan host fails with an x509 error.

Pruning this repository has to happen on the rest-server machine itself, with direct file access, on a schedule you control there. That's the point: the server being backed up can't do it.

After the first copy, the append-only repository has one snapshot. restic forget --keep-last 1 keeps that snapshot, deletes nothing, and exits 0, so it is not a test yet. Take a second backup, copy it, then forget:

bash
sudo restic -r /mnt/backup/restic --password-file /etc/restic/password \
  backup --files-from /etc/restic/include.txt --exclude-file /etc/restic/exclude.txt --tag nightly
sudo bash -c 'set -a; . /etc/restic/append.env; set +a; \
  restic copy --from-repo /mnt/backup/restic --from-password-file /etc/restic/password'
sudo bash -c 'set -a; . /etc/restic/append.env; set +a; \
  restic forget --keep-last 1'

Check it worked: the second forget fails with 403 Forbidden (restic often exits 3). On a rest-server without --append-only, the same command succeeds.

Schedule it with systemd

Create a script that runs the backup, the two copies and the retention. It expects /etc/restic/append.env to hold the rest-server variables from step 5, with mode 600:

bash
sudo tee /usr/local/sbin/restic-nightly >/dev/null <<'SH'
#!/bin/sh
set -eu
umask 077
LOCAL="-r /mnt/backup/restic --password-file /etc/restic/password"
install -d -m 700 /var/backups
sudo -u postgres pg_dumpall > /var/backups/pg_dumpall.sql
restic $LOCAL backup --files-from /etc/restic/include.txt --exclude-file /etc/restic/exclude.txt --tag nightly
restic $LOCAL forget --tag nightly --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
( set -a; . /etc/restic/offsite.env; set +a
  restic copy --from-repo /mnt/backup/restic --from-password-file /etc/restic/password
  restic forget --tag nightly --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune )
( set -a; . /etc/restic/append.env; set +a
  restic copy --from-repo /mnt/backup/restic --from-password-file /etc/restic/password )
SH
sudo chmod 700 /usr/local/sbin/restic-nightly

The retention values are examples. Choose yours from how far back you'd ever need to restore.

Add a service and a timer:

ini/etc/systemd/system/restic-nightly.service
# /etc/systemd/system/restic-nightly.service
[Unit]
Description=Nightly restic backup
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/restic-nightly
ini/etc/systemd/system/restic-nightly.timer
# /etc/systemd/system/restic-nightly.timer
[Unit]
Description=Run restic nightly

[Timer]
OnCalendar=*-*-* 02:00
Persistent=true
RandomizedDelaySec=15m

[Install]
WantedBy=timers.target

OnFailure= needs a notify-failure@.service unit that sends you a message (mail, a chat webhook, anything a person reads). Write that one to match your setup, or drop the line and alert from your monitoring instead.

bash
sudo systemctl daemon-reload
sudo systemctl enable --now restic-nightly.timer

Check it worked: systemctl list-timers restic-nightly.timer shows the next run, and journalctl -u restic-nightly shows a clean run after the first night.

Check the repositories

restic check verifies the repository structure. --read-data-subset=5% picks a random 5%, so a weekly run is only likely to cover the whole repository over time. Use a deterministic n/t subset instead, so each ISO week of the year reads a different slice:

bash
sudo restic -r /mnt/backup/restic --password-file /etc/restic/password \
  check --read-data-subset="$(date +%-V)/53"

Run the same against the off-site repository on a separate weekly timer.

Check it worked: the output ends with no errors were found.

Restore something, on purpose

This is the zero. Restore to a scratch directory, never on top of live data:

bash
sudo restic -r /mnt/backup/restic --password-file /etc/restic/password \
  restore latest --target /tmp/restore-test --include /etc/nginx
diff -r /etc/nginx /tmp/restore-test/etc/nginx && echo "matches"

Once a quarter, restore the database dump into a scratch database and run a query you know the answer to. Once a year, restore from the off-site and append-only copies as well as the local one.

Write each test down: date, snapshot ID, what you restored, what you checked, and anything that went wrong.

Check it worked: the diff prints matches, and the test is in your log.

Troubleshooting

  • `Fatal: wrong password or no key found`: the password file doesn't match that repository. Each repository keeps its own key, even with --copy-chunker-params.
  • `repository is already locked`: a previous run crashed. Confirm nothing is running, then run restic unlock.
  • The off-site copy is much bigger than the local one: it was initialised without --copy-chunker-params, so deduplication between the two doesn't line up. Recreate it.
  • `forget` against rest-server fails: that's append-only doing its job. Prune on the rest-server host instead.

References