Proxmox backup checklist: jobs, PBS, off-site, restores

Plenty of Proxmox setups have backups. Fewer have backups that anyone has restored. This guide takes a Proxmox VE host from "there is a nightly job somewhere" to encrypted backups on a Proxmox Backup Server (PBS), a copy at a second site, and a restore test you've written down.

author
Zahara Schaefer
published
read
8 min
tested
Proxmox VE 9.x and Proxmox Backup Server 4.x (from the official docs; menu paths not re-checked on a live host)
outline {
  1. What you need
  2. 1. Write down what you're protecting
  3. 2. Create a datastore on the backup server
  4. 3. Add PBS to Proxmox VE, with encryption
  5. 4. Create the backup job
  6. 5. Schedule prune, garbage collection and verify on PBS
  7. 6. Set up failure notifications
  8. 7. Add the off-site copy
  9. 8. Back up the host configuration
  10. 9. Restore, and write it down
  11. Troubleshooting
  12. References

What you need

  • A Proxmox VE 9 host or cluster with at least one VM or container.
  • A separate machine for Proxmox Backup Server 4. A backup server that runs as a VM on the host it protects won't survive that host.
  • Optional: a second PBS at another location for the off-site copy.
  • Root shell access to both, and about an hour.

Write down what you're protecting

Before you touch a setting, list your guests and two numbers for each: how old a restore can be (your recovery point), and how long the service can be down (your recovery time). A table in your runbook is enough.

GuestIDTypeMax data lossMax downtime
mail relay (example)101VM1 day4 hours

Add the things that aren't guests: /etc/pve, /etc/network/interfaces, the PBS configuration, and the encryption key you'll create in step 3. Proxmox VE backup jobs back up guests, not the host.

Check it worked: every guest from qm list and pct list appears in your table.

Create a datastore on the backup server

Install PBS from the ISO on the backup machine and open https://<pbs-host>:8007.

  1. Go to Datastore > Add Datastore. Give it a name (for example main) and a path on dedicated disks.
  2. Under Configuration > Access Control, create a user for Proxmox VE to back up with, for example pve-backup@pbs.
  3. Give that user the DatastoreBackup role on /datastore/main (Datastore > main > Permissions). That role can create backups and read its own, without being able to change the datastore.
  4. Copy the fingerprint from the PBS dashboard (Show Fingerprint).

You can do the same from the PBS shell:

bash
proxmox-backup-manager datastore create main /mnt/datastore/main
proxmox-backup-manager user create pve-backup@pbs
proxmox-backup-manager acl update /datastore/main DatastoreBackup --auth-id pve-backup@pbs

Set the user's password in the web UI under Access Control > User Management > Change Password, so it never lands in your shell history.

Check it worked: proxmox-backup-manager datastore list shows the datastore, and the user appears under Access Control.

Add PBS to Proxmox VE, with encryption

On Proxmox VE, go to Datacenter > Storage > Add > Proxmox Backup Server and fill in the server, datastore, user and fingerprint. On the Encryption tab, choose Upload an existing client encryption key or Auto-generate a client encryption key. Proxmox VE then encrypts backups before they leave the host.

The CLI equivalent:

bash
pvesm add pbs pbs-main \
  --server pbs.example.lan \
  --datastore main \
  --username pve-backup@pbs \
  --fingerprint '<fingerprint>' \
  --password '<password>' \
  --encryption-key autogen

If you lose the key, the backups are unreadable. Download it when Proxmox VE offers it and put it in your password manager. Print a paper copy too: on a machine with proxmox-backup-client installed, proxmox-backup-client key paperkey <keyfile> --output-format text prints it in a form you can type back in.

Check it worked: pvesm status lists pbs-main as active, and you can find the key without logging in to the host.

Create the backup job

Go to Datacenter > Backup > Add:

  • Storage: pbs-main.
  • Schedule: start from your recovery-point column. A nightly job (for example 21:00) suits most guests. Give the ones that need more frequent backups their own job.
  • Selection mode: All with exclusions, or Pool based. Both pick up new guests without anyone remembering to add them. Hand-picked IDs drift.
  • Mode: Snapshot keeps guests running. For VMs, install the QEMU guest agent and enable it in the VM's Options, so Proxmox VE can freeze the file systems during the snapshot.
  • Retention: set it either on the job or on the datastore's prune job, not both. With PBS, I put it on the datastore (step 5).

Databases need one more thing. A crash-consistent disk snapshot isn't the same as a clean dump, so also schedule pg_dump or mysqldump inside database guests, writing to a path that's included in the backup.

Check it worked: click Run now, then open the task log. Each guest should end with INFO: Finished Backup of VM <id>, and the job should end with TASK OK.

Schedule prune, garbage collection and verify on PBS

PBS needs three kinds of jobs. In the PBS UI, under Datastore > main:

  1. Prune & GC > Add prune job. Set your retention, for example keep-daily 7, keep-weekly 4, keep-monthly 6. Those values are an example, not a recommendation. Take yours from your recovery-point table and how much disk you have.
  2. Prune & GC > Garbage Collection schedule. Pruning removes snapshots from the index, but disk space only comes back when garbage collection runs. Schedule it daily.
  3. Verify Jobs > Add. Verification reads the chunks back and checks them against their checksums. Schedule it weekly, and skip snapshots that verified recently.

Check it worked: after one cycle, the Summary tab shows a recent GC run, and the Content tab shows a verify state of OK on your snapshots.

Set up failure notifications

Proxmox VE 9 sends notifications through Datacenter > Notifications: you add a target (an SMTP server, Gotify, or a webhook) and a matcher that decides what goes where. PBS 4 has the same model under Configuration > Notifications.

Make sure failed backup and verify jobs go somewhere a person actually reads. Then test it: in the job settings, temporarily point a backup job at a storage that's offline, and run it.

Check it worked: the failure message arrives.

Add the off-site copy

The simplest off-site copy is a second PBS that pulls from the first:

  1. Install PBS at the second site and create a datastore (for example offsite).
  2. On the first PBS, create a read-only user for syncing, with the DatastoreReader role on /datastore/main.
  3. On the second PBS, go to Configuration > Remotes > Add and enter the first server, the sync user and the fingerprint.
  4. On the second PBS, go to Datastore > offsite > Sync Jobs > Add. Choose the remote, the source datastore and a schedule.
  5. Give offsite its own prune, GC and verify jobs.

Pulling means the main site never holds credentials that can delete the off-site copy. Because the backups are encrypted on the client, the second server never needs the key either.

PBS 4.0 also added S3-compatible object storage as a datastore backend (a technology preview in 4.0) and a run-on-mount flag that runs a sync job whenever a removable datastore is mounted. Both are useful for off-site and offline copies. Check the release notes for your exact version before you rely on them.

Check it worked: the sync job's task log ends in TASK OK, and the offsite content view lists the same backup groups as main.

Back up the host configuration

Guest backups don't include the host. Keep a copy of:

bash
tar czf /root/pve-config-$(hostname)-$(date +%F).tar.gz \
  /etc/pve /etc/network/interfaces /etc/hosts /etc/fstab /etc/apt/sources.list.d

Copy that archive somewhere that isn't the host. The PBS client can also back up host paths directly:

bash
export PBS_REPOSITORY='pve-backup@pbs@pbs.example.lan:main'
proxmox-backup-client backup pveconfig.pxar:/etc/pve

Also write down the install details: version (pveversion), disk layout, storage names and bridges.

Check it worked: open the archive (tar tzf) and confirm /etc/pve/qemu-server/ is in it.

Restore, and write it down

The test that counts is a restore. Restore to a new ID, so you never overwrite the running guest:

bash
# Find the archive name
pvesm list pbs-main --vmid 101

# Restore a VM to ID 9101 on local-lvm
qmrestore pbs-main:backup/vm/101/<timestamp> 9101 --storage local-lvm

# Restore a container to ID 9201
pct restore 9201 pbs-main:backup/ct/201/<timestamp> --storage local-lvm

Before you start the copy, detach its network card or move it to an isolated bridge. Two copies of the same server on one network cause their own outage.

For a single file, select the backup in the PBS storage view in Proxmox VE and click File Restore.

Record each test in the runbook:

DateGuestBackup usedRestored toWhat I checkedTime takenProblems

"What I checked" should be the service working. A VM that boots isn't enough. Log in, open the app, and read a record you know exists.

Check it worked: the restored service answers on the isolated network, and the row is in the runbook.

Troubleshooting

SymptomLikely causeFix
fingerprint ... not verified when adding storageWrong or missing fingerprintCopy it again from the PBS dashboard. It changes if the PBS certificate changes.
permission check failed during backupThe user lacks rights on the datastoreGive it DatastoreBackup on /datastore/<name>.
Datastore keeps filling after pruningGarbage collection isn't scheduled, or hasn't run yetSchedule GC. GC also keeps chunks that are recent enough to still be in use, so space comes back after a delay.
Snapshot backup of a VM is slow or inconsistentGuest agent missing or disabledInstall qemu-guest-agent in the guest and enable it in the VM options.
Restore fails with a decryption errorWrong key or no key on the hostAdd the original key to the storage configuration.

References