Running Proxmox in production: clusters, HA and upgrades
One Proxmox host is a server. Three hosts with shared storage, a quorum and an upgrade routine is a platform. This guide covers the jump: forming a cluster, keeping quorum, restarting guests automatically when a node fails, moving guests without downtime, making new VMs from a template, and upgrading without surprises.
It assumes you've done the single-host setup from Install Proxmox VE 9 on one server on every node.
Contents
- Plan the cluster network
- Create the cluster
- Keep quorum with two nodes
- Shared storage
- High availability
- Live migration
- Cloud-init templates
- API tokens for automation
- Upgrades
What you need
- Two or more Proxmox VE 9 nodes on the same version, each with a unique hostname and a static IP.
- No guests on the nodes that will join (a joining node can't have guests with IDs that clash).
- Ideally, a dedicated network link for cluster traffic.
- For HA: shared or replicated storage, and three votes (three nodes, or two nodes plus a QDevice).
Step 1: Plan the cluster network
Proxmox clusters use Corosync, which is sensitive to latency. The cluster manager docs recommend a dedicated, low-latency network for it, separate from storage and guest traffic, because a busy link can make nodes drop out of the cluster. Corosync supports several links, so give it a second one as a fallback.
Write down each node's name and its cluster-network address before you start. Every node must resolve every other node's hostname. Check /etc/hosts on each one.
Check it worked: each node can ping the others on the cluster network, by name.
Step 2: Create the cluster
On the first node:
pvecm create prod --link0 10.10.10.11On each other node, join through the first node's address and give the joining node's own cluster address:
pvecm add 10.10.10.11 --link0 10.10.10.12You'll be asked for the first node's root password and to accept its fingerprint. You can also do this in the UI under Datacenter > Cluster > Create Cluster, then Join Information and Join Cluster.
Check it worked:
pvecm status
pvecm nodespvecm status shows Quorate: Yes and the expected number of votes.
Step 3: Keep quorum with two nodes
A cluster needs a majority of votes to make changes. With two nodes, losing one leaves one vote out of two, which isn't a majority, so /etc/pve turns read-only and HA stops. The fix is an external vote, called a QDevice: a small Linux machine outside the cluster, such as a VM on other hardware.
On the external machine:
apt install corosync-qnetdOn every cluster node:
apt install corosync-qdevicepvecm qdevice setup needs root SSH from a cluster node to the qnetd host. Debian's default is PermitRootLogin prohibit-password, so set up key-based root access first (or temporarily allow password login, then turn it off):
ssh-copy-id root@10.10.10.50
ssh root@10.10.10.50 trueThen, from one node:
pvecm qdevice setup 10.10.10.50With three or more nodes, you don't need one (and with an even number of nodes, check the docs on whether to add one).
Check it worked: pvecm status lists the QDevice, and the expected vote count goes up by one.
Step 4: Shared storage
Live migration and HA need every node to reach a guest's disk. Your options:
- Ceph, built into Proxmox VE (Datacenter > Ceph). It needs at least three nodes and a fast network. Proxmox VE 9 offers the Ceph Squid and Tentacle releases.
- NFS or iSCSI from an external storage box:
pvesm add nfs .... Simple, but the storage box becomes the one thing that can't fail. - ZFS replication, which isn't shared storage, but copies guest disks to another node on a schedule (guest > Replication). HA can then restart a guest from the last copy, so you can lose the changes made since the last replication.
Choose based on what you can afford to lose. I'd rather have replication I understand than Ceph that nobody on the team can repair.
Check it worked: the storage appears on every node in pvesm status, or replication jobs show a recent successful sync.
Step 5: High availability
HA watches resources and restarts them on another node when their node fails. Add a guest as an HA resource:
ha-manager add vm:100 --state started --max_restart 1 --max_relocate 1
ha-manager statusIn the UI, this is Datacenter > HA > Add. To control where a guest may run, Proxmox VE 9 uses HA rules under Datacenter > HA > Rules. Node affinity rules pin resources to chosen nodes. Resource affinity rules keep resources together, or spread them across nodes, for example two copies of the same app. HA groups are deprecated, and existing groups are migrated to node affinity rules automatically once every node runs 9. Details are in the HA docs.
Two things to know before you rely on it:
- HA fences a node that loses quorum. The node's watchdog resets it, so the guest doesn't run in two places at once. That's by design.
- HA only helps if the guest's disk is reachable from the node it moves to (step 4).
Check it worked: in a maintenance window, shut down a node cleanly and watch ha-manager status. The guest should start on another node within a few minutes.
Step 6: Live migration
Move a running VM to another node with no downtime:
qm migrate 100 pve2 --onlineOn local storage, add --with-local-disks. This copies the disk over the network, so it takes longer. Containers can't be live-migrated. pct migrate 200 pve2 --restart stops the container, moves it, and starts it again.
For node maintenance, set the node to maintenance mode, so HA moves its guests off first:
ha-manager crm-command node-maintenance enable pve1
# do the work, reboot
ha-manager crm-command node-maintenance disable pve1Check it worked: the VM shows the new node in the tree, and an open SSH session to the VM survived the move.
Step 7: Cloud-init templates
Instead of installing every VM from an ISO, build one template from a cloud image and clone it:
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2
qm create 9000 --name debian13-tmpl --memory 2048 --cores 2 \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --agent enabled=1
qm set 9000 --scsi0 local-lvm:0,import-from=/root/debian-13-genericcloud-amd64.qcow2
qm set 9000 --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0
qm template 9000Then for each new VM:
qm clone 9000 120 --name app02 --full
qm set 120 --ciuser admin --sshkeys ~/.ssh/id_ed25519.pub \
--ipconfig0 ip=192.0.2.120/24,gw=192.0.2.1
qm resize 120 scsi0 +18G
qm start 120Check the image URL on cloud.debian.org before you use it. The genericcloud image doesn't include the guest agent, so install qemu-guest-agent with a cloud-init snippet or after the first boot.
Check it worked: you can SSH in as admin with your key a minute after qm start.
Step 8: API tokens for automation
For Terraform, Ansible or scripts, use an API token on a dedicated user, not root's password:
pveum user add automation@pve
pveum role add Provisioner --privs "VM.Allocate VM.Clone VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Cloudinit VM.Config.Options VM.PowerMgmt VM.Audit Datastore.AllocateSpace Datastore.Audit SDN.Use"
pveum acl modify / --users automation@pve --roles Provisioner
pveum user token add automation@pve tf --privsep 0The token secret is shown once. Put it straight into your secrets manager. Trim the privilege list to what your tooling actually needs; the list above is a starting point, and the exact names your provider expects are in its documentation. With --privsep 1, the token gets its own, narrower permissions instead of the user's.
Check it worked:
curl -sS --cacert /etc/pve/pve-root-ca.pem \
-H 'Authorization: PVEAPIToken=automation@pve!tf=<secret>' \
https://pve1.example.lan:8006/api2/json/versionThe PVE web certificate is self-signed. curl -s without --cacert exits 60 with no body, so you see neither JSON nor a 401. Copy pve-root-ca.pem off the node if you are calling from another machine. -k is fine for a quick local test; -sS still shows errors. The response is JSON with the version, not a 401.
Step 9: Upgrades
Minor upgrades (9.x to 9.y), one node at a time:
ha-manager crm-command node-maintenance enable pve1
apt update && apt full-upgrade
reboot
ha-manager crm-command node-maintenance disable pve1Wait until the node is back and pvecm status is quorate before moving to the next one.
Major upgrades (8 to 9) follow the official upgrade guide. In outline:
- Bring every node up to the latest 8.4 and take fresh backups.
- Run the checker and fix every warning:
pve8to9 --full. - Change the Debian and Proxmox repositories from
bookwormtotrixie. apt update && apt dist-upgrade, answer the configuration prompts, reboot.- Repeat node by node, and upgrade Ceph first if you use it, as the guide describes.
Proxmox VE 8 reaches end of life in August 2026, so any 8.x host still running is past it.
Check it worked: pveversion -v matches on every node, and pvecm status is quorate.
Troubleshooting
- `/etc/pve` is read-only: the node has lost quorum. Fix the network or bring nodes back. Only use
pvecm expected 1as a last resort on an isolated node, because it can cause split-brain. - Join fails with a certificate or hostname error: check
/etc/hostson both nodes, and that the joining node has no guests. - Live migration fails on CPU flags: use the same CPU model on all guests (step 8 of the single-host guide).
- HA guest stuck in `fence` or `error`: read
journalctl -u pve-ha-crm -u pve-ha-lrm. Runha-manager set vm:100 --state disabled, fix the cause, then set it back tostarted.
References
- Cluster manager (
pvecm, QDevice): https://pve.proxmox.com/wiki/Cluster_Manager - High availability (
ha-manager): https://pve.proxmox.com/wiki/High_Availability - Cloud-init support: https://pve.proxmox.com/wiki/Cloud-Init_Support
- Storage replication: https://pve.proxmox.com/wiki/Storage_Replication
- User management and API tokens: https://pve.proxmox.com/wiki/User_Management
- Upgrade from 8 to 9: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9
- Ceph on Proxmox VE: https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster