Proxmox Cluster Management: Build a 2-Node Cluster or 3-Node HA (2026)

By LK Wood IV · 2026-06-16 · ~18 min read · St. Louis County, MO

A single Proxmox node is fine until it isn’t. The drive fails. The NIC dies. You need to reboot for a kernel update and everything goes down.

A Proxmox cluster solves this. It is two or more nodes that share cluster state and can migrate VMs between them. It can also restart VMs automatically when a node fails.

This guide covers Proxmox cluster management from the first node onward. You get the correct 2-node architecture with a QDevice and the cleaner 3-node architecture that doesn’t need one. It then covers shared storage, live migration, HA groups, the fencing question you can’t skip, and the day-to-day work of adding and removing nodes.

What a Proxmox cluster gives you

Proxmox clustering has two distinct features that people sometimes conflate:

1. Cluster management (no shared storage required)

• Unified web UI: manage all nodes from one panel
• Centralized user and permission management
• Pool-based resource allocation across nodes
• Planned VM migration between nodes, with shared disks or disk transfer

2. High Availability (HA) - requires recoverable disks + fencing

• Automatic VM restart on surviving node when a node fails
• HA watchdog groups with priority ordering
• Fencing to prevent split-brain and data corruption

You can run a cluster for migration and unified management without HA. HA also needs reliable quorum, fencing, spare capacity and every required guest disk on an eligible recovery node. After a node fails, its guest restarts from disk on a surviving node once fencing permits recovery, so the guest’s RAM contents are lost and its applications may need to recover from the interrupted writes. Expect downtime during that restart.

Architecture: 2-node vs 3-node

3-node cluster (recommended)

Three nodes each have a vote in corosync. Quorum is 2 votes (majority). If any single node fails:

• 2 surviving nodes form quorum
• Cluster continues operating
• HA restarts VMs on surviving nodes

This is the clean architecture. No workarounds, no QDevice, straightforward HA.

2-node cluster with QDevice

Two nodes cannot hold quorum on their own. Each losing the other would leave one node alone, and one vote does not equal a majority of two.

The solution is a QDevice: a third machine that provides a tie-breaking vote. It runs corosync-qnetd as the arbiter, and each cluster node runs the corosync-qdevice client.

Node 1 (vote) + Node 2 (vote) + QDevice (vote) = 3 votes total
Quorum = 2 votes required

If Node 1 fails: Node 2 + QDevice = 2 votes → quorum → cluster continues. If the QDevice fails: Node 1 + Node 2 = 2 votes → quorum maintained (QDevice is just a tiebreaker). If Node 1 + QDevice both fail: Node 2 alone = 1 vote → no quorum → node fences itself.

The QDevice can be anything running Debian/Ubuntu with a network connection. A Raspberry Pi, an LXC on a third server and a VM on a NAS all work. It requires minimal resources: 256MB RAM, a few hundred MB of disk, and a stable network connection to both nodes.

Proxmox’s Cluster Manager documentation, read on 2026-10-10, adds details that matter for a 2-node cluster:

• QDevices are supported for even-numbered clusters and recommended for 2-node clusters that need higher availability.
• The QDevice talks to the nodes over TCP/IP and can sit outside the cluster LAN. Corosync’s low-latency requirement does not apply to it.
• In a tie between two equal halves that cannot see each other but can both see the QDevice, the QDevice votes for the partition with the lowest node ID.
• With only two plain votes and no QDevice, losing one node loses quorum. Proxmox says the cluster then switches to read-only mode.
• For odd-numbered clusters, Proxmox currently discourages a QDevice. If it fails, no other node may fail without losing quorum. The setup command’s own message says “Clusters with an odd node count are not officially supported!”

Prerequisites

For a 2-node + QDevice cluster:

• 2 Proxmox hosts (any hardware, same or different specs)
• 1 QDevice host (Raspberry Pi, mini PC, or any Debian/Ubuntu system)
• All three on the same network (or VLANs with routing between them)
• No existing Proxmox clusters on any node. Clusters can only be created, not merged
• UDP ports 5405-5412 open between the nodes for corosync, plus SSH on TCP port 22
• Synchronized date and time, and the same Proxmox VE version on every node
• Each node installed with its final hostname and IP address. Proxmox says neither can be changed after the cluster exists

For HA (optional, beyond basic clustering):

• Shared storage accessible from eligible recovery nodes, or local ZFS replicated to those nodes
• Watchdog device for fencing (the kernel softdog works everywhere; a hardware/IPMI watchdog where the board offers one)

Step 1: Create the cluster on node 1

Log into node 1’s Proxmox web UI or SSH in:

# Create the cluster (run on node 1 only)
pvecm create mycluster --link0 <node1-ip>

Verify the cluster is created:

pvecm status
# Should show: Quorum information, Node ID 1, Member count 1

Step 2: Join node 2 to the cluster

From node 2, join the existing cluster:

# Run on node 2
pvecm add <node1-ip> --link0 <node2-ip>

This prompts for node 1’s root password. After joining, verify from node 1:

pvecm nodes
# Should show both nodes listed

At this point, both nodes appear in the web UI under Datacenter. The cluster exists but cannot survive a node loss. With 2 nodes, both votes are needed for quorum, so losing either node means no quorum.

Two cautions from Proxmox’s documentation before you join a node:

• Joining overwrites everything in /etc/pve on the joining node. A node that already holds guests cannot join, and it inherits the cluster’s storage configuration.
• To put corosync on its own network, pass the link addresses: pvecm add <node1-ip> --link0 <node2-ip> (as above), and add --link1 for a redundant second link. Corosync supports up to 8 links.

Step 3: Set up the QDevice

On the QDevice machine (Debian/Ubuntu) install corosync-qnetd. This machine is the arbiter:

apt update && apt install -y corosync-qnetd

corosync-qdevice is the node-side client, and it goes on every cluster node. Proxmox’s documentation lists this as a manual step before the setup command:

# Run on node 1 and node 2
apt install -y corosync-qdevice

Make sure all nodes are online, then on node 1 configure the QDevice:

pvecm qdevice setup <qdevice-ip>

This copies the cluster SSH key to the QDevice and configures corosync on all nodes to include the QDevice vote. It needs root SSH access to the QDevice machine. Use key-based access, or allow root password login temporarily during setup. If setup reports “Host key verification failed”, run pvecm updatecerts and try again.

Verify:

pvecm status
# Quorate: Yes
# Total votes: 3 (node1 + node2 + qdevice)
# Expected votes: 3
# Quorum:        2   <- the threshold, printed separately

The Qdevice flag should appear in the pvecm status output. If the QDevice shows as Not Alive (NA), check that TCP port 5403 on the QDevice machine is reachable from the nodes.

Step 4: Choose storage for migration and HA

The VM migration reference allows transfer of local disks. The storage replication reference explicitly allows HA with replicated local ZFS. Choose a recovery design:

Local ZFS replication

Create a local ZFS storage with the same storage ID on each target and sufficient free space. In the guest’s Replication panel, add a job targeting the intended recovery node, set its schedule, and verify a successful initial sync with pvesr status and the job log. Every local disk needed to boot and run the guest must be covered; a disk excluded from replication will not appear on the recovery node.

Restrict HA placement to replica targets. Use a restricted group on PVE 8 or a strict node-affinity rule on PVE 9 to keep recovery, placement and migration within nodes that hold all the guest’s disks and dependencies. Check replicas and job errors before enabling HA.

The recovery point is the last successful sync. Failed jobs can leave it older than the configured interval. Replication and HA still need independent backups.

The shared-storage alternatives follow below. A single NAS can remain a failure point even when several Proxmox nodes use it.

NFS (easiest for homelab)

If you have a TrueNAS or any NFS server:

In Proxmox web UI: Datacenter → Storage → Add → NFS

After adding, both nodes should see the same NFS storage in their storage list.

Verify from both nodes:

# On both node1 and node2
ls /mnt/pve/nfs-shared/
# Both should show the same directory contents

iSCSI

For better performance than NFS, an iSCSI target (TrueNAS, OpenMediaVault, or a dedicated SAN):

# Discover iSCSI targets
iscsiadm -m discovery -t st -p <iscsi-server-ip>

# Log in to target
iscsiadm -m node -T <target-name> -p <iscsi-server-ip> --login

Add in Proxmox UI: Datacenter → Storage → Add → iSCSI, then put plain LVM on top. Do not use LVM-thin here: it is not cluster-aware and cannot be shared between nodes, so it cannot back HA or live migration. That is the whole point of this section.

Ceph (advanced, distributed storage)

Ceph runs on the Proxmox nodes themselves — each node contributes disks to a distributed object store. No separate NAS required.

Ceph requires:

• 3+ Proxmox nodes (Ceph quorum is formed by an odd number of monitors, not OSDs)
• A dedicated disk per node for Ceph OSD (separate from the OS disk)
• Low-latency network between nodes (10GbE strongly recommended)

Set up via Proxmox web UI: Datacenter → Ceph → Install → follow wizard. Ceph is powerful but adds complexity; for a 2-node homelab, NFS is simpler.

Step 5: Live migration

For an eligible VM, live migration can use shared disks or transfer local disks. In the web UI:

1. Select a VM
2. Right-click → Migrate
3. Target node: select the other node
4. Select online migration for a running VM; review target storage mappings if local disks need transfer
5. Click Migrate

From the CLI:

# Live migrate VM ID 100 to node2
qm migrate 100 node2 --online
# For a VM with local disks, permit their transfer; check target storage first
qm migrate 100 node2 --online --with-local-disks 1

Use qm help migrate for the options on your installed release. Check CPU compatibility, target disk space and matching guest networks. PCI/USB passthrough and other resources tied to one host can block live migration. A container migration is a separate workflow; these qm commands are for VMs.

Time depends on effective throughput, memory dirty rate and how much disk data must move. Test a noncritical representative guest on your installed PVE release, keep its application active during migration, and record task duration and any lost connections. Test a return migration too. No new cluster migration or failover test was performed for this documentation correction.

Two migration settings from Proxmox’s documentation are worth knowing:

• Secure vs insecure mode. Secure mode encrypts the migration channel. Insecure mode sends guest RAM unencrypted, which can expose secrets, so Proxmox recommends secure mode unless you control the network.
• Migration network. By default, migration traffic uses the cluster network. To use a dedicated network, pass it for one migration with qm migrate 100 node2 --online --migration_network 10.1.2.0/24, or set migration: secure,network=10.1.2.0/24 in /etc/pve/datacenter.cfg. Always set the migration type alongside the network there.

Proxmox recommends CPUs from the same vendor with similar capabilities for online migration. Its reference says cross-vendor migration can sometimes work but is not guaranteed; validate the guest CPU model before relying on it.

Step 6: Enable High Availability

Fencing is what makes HA safe. Without it, a node that becomes unreachable (but is not actually down) could keep running VMs while the surviving node restarts the same VMs, corrupting shared storage.

How Proxmox fencing works: watchdog self-fencing

Proxmox VE does not use external fence agents. There is no fencing dialog in the GUI, which trips up people arriving from Pacemaker/corosync setups.

Instead, every node running HA resources arms a hardware watchdog timer. If the node loses quorum, pve-ha-lrm stops petting the watchdog and the node hard-resets itself after about 60 seconds. Only after that window does the cluster recover the VMs elsewhere. (Proxmox HA manager documentation)

Out of the box Proxmox uses the Linux kernel softdog module, which loads automatically the first time a node runs an HA resource. On a homelab you typically configure nothing at all. It is software fencing rather than hardware fencing, but it is the supported default and it prevents split-brain.

Verify the watchdog device exists once an HA resource is active:

ls /dev/watchdog

Using a hardware watchdog (boards with IPMI/BMC)

Boards with a BMC usually expose a true hardware watchdog (ipmi_watchdog; many Intel boards expose iTCO_wdt). It keeps fencing working even if the kernel itself wedges, which softdog cannot guarantee. Select it in /etc/default/pve-ha-manager (not resources.cfg, which holds HA resource stanzas — writing this key there corrupts it), then reboot the node:

WATCHDOG_MODULE=ipmi_watchdog

Blacklist competing watchdog modules if the BMC and chipset both expose one. Exactly one driver should own /dev/watchdog.

Enable HA for a VM

Fencing is armed automatically with the watchdog, so this is the only step most homelabs perform:

1. Web UI → Select VM → More → Manage HA
2. Click “Add HA”
3. Set Group (optional: assign to a specific HA group)
4. Max Restart: 3 (restart attempts before abandoning)
5. Max Relocate: 3 (move attempts to another node)

Or CLI:

ha-manager add vm:100 --state started --max_restart 3 --max_relocate 3

Check HA status:

ha-manager status
# Shows: vm:100 in state started, node: node1

When node1 fails, a quorate cluster waits for fencing and restarts VM 100 on an eligible node with its disks available. With local ZFS replication, that is a replica target and recovery uses the last successful sync. Test the chosen failure/recovery path with a noncritical guest before placing a service under HA.

Step 7: HA groups and priorities

HA groups let you prefer specific VMs on specific nodes while allowing failover to others:

# Create a group that prefers node1 but allows node2
ha-manager groupadd ha-group1 --nodes node1:2,node2:1 --restricted 0
# node1:2 = priority 2 (preferred), node2:1 = priority 1 (failover)
# restricted 0 = allow running on any listed node

Assign VMs to the group:

ha-manager set vm:100 --group ha-group1

With this config, VM 100 prefers node1. After a failure, it can restart on node2 if its disks are available there. When node1 returns, the default failback policy moves it back. For replicated local storage, use a restricted group covering only valid replica targets; the non-restricted example above is for disks available on every eligible node.

On Proxmox VE 9 this becomes a node-affinity rule. Per the official HA manager reference, “HA Groups are deprecated and migrated to HA Node Affinity rules since Proxmox VE 9.0” (ha-manager docs, checked 2026-08-22), and the 8 to 9 upgrade converts existing groups for you. The same preference on 9.x is one node-affinity rule:

# PVE 9.x equivalent: prefer node1 (priority 2), allow node2 (priority 1), non-strict
ha-manager rules add node-affinity ha-rule1 --resources vm:100 --nodes node1:2,node2:1

--restricted 0 from the old group becomes the default non-strict rule; Datacenter → HA → Affinity Rules shows the same thing in the web UI.

Managing a Proxmox cluster: add, remove and edit

These tasks come from Proxmox’s Cluster Manager documentation, read on 2026-10-10.

Remove a node

Do this in order:

1. Move every guest off the node and back up any local data.
2. Remove replication jobs that involve the node. Otherwise, Proxmox warns, the job becomes irremovable.
3. If you use a QDevice, remove it first with pvecm qdevice remove.
4. If you run Ceph, destroy the node’s OSDs one at a time, wait for HEALTH_OK, and remove its monitor and manager first.
5. Power the node off and make sure it cannot come back on this network with its current configuration.
6. From a remaining node, run pvecm delnode <nodename>, then confirm with pvecm nodes.

A “could not kill node (error = cs_err_not_exist)” message can be ignored. Afterwards, remove /etc/pve/nodes/<nodename> once you have confirmed the name, remove the node from any HA rules, and remove its key from /etc/pve/priv/authorized_keys. To reuse the hardware, reinstall Proxmox VE and join it again.

Change the node count with a QDevice in place

Proxmox says to remove the QDevice first, add or delete the nodes, and set the QDevice up again once the node count is even.

Edit corosync.conf safely

Edit a copy and leave the live file untouched. Raise config_version in the copy:

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new
# edit corosync.conf.new and increase config_version
cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak
mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf
systemctl status corosync
journalctl -b -u corosync

If the change is not applied automatically, run systemctl restart corosync.

After a full power loss

A cluster is not quorate while every node is off. On startup, pve-guests waits for quorum and then starts guests flagged to start on boot, so guest startup is delayed until enough nodes are back. Proxmox recommends a UPS, especially with HA.

Monitoring cluster health

# Cluster-wide status
pvecm status

# Node status  
pvecm nodes

# HA manager status
ha-manager status

# Corosync ring status (network health between nodes)
corosync-cfgtool -s

# Check for quorum issues
corosync-quorumtool

In the web UI: Datacenter → Summary shows all nodes with CPU/RAM/disk graphs and cluster status.

Common issues

“Cluster not quorate” — Usually means the QDevice is unreachable or a node went offline without fencing. Check corosync-cfgtool -s for ring status. Confirm QDevice is running systemctl status corosync-qdevice.

Local-disk migration fails - Read the migration task log and check destination storage IDs, free space, disk mappings and local-resource restrictions. Local disks are supported; moving everything to shared storage is not the only remedy. If the target reports an existing guest or conflicting volume, inspect that conflict before retrying or deleting anything.

HA manager won’t start VMs — check ha-manager status. The usual causes are lost quorum (the surviving side can’t fence-and-recover until quorum returns) or a resource stuck in an error state after a failed recovery; clear that with ha-manager set vm:<id> --state started once quorum is back.

Corosync ring 0 errors — Network instability between nodes. Check physical connectivity, switch port speed (ensure both nodes are at full duplex), and that the corosync network isn’t saturated. Corosync is very sensitive to dropped packets.

Network recommendations

• Dedicate a network interface or VLAN for corosync (--link0 parameter)
• VM traffic on a separate interface or VLAN from cluster communication
• 10GbE between nodes for live migration (1GbE works but migration is slow)
• Storage traffic on a third interface if possible

The VLANs for the homelab guide covers network segmentation for cluster + storage + VM traffic with OPNsense and managed switches.

Cluster storage architecture connects to ZFS. The ZFS on Proxmox guide covers local storage pool creation; for cluster scenarios, set up local ZFS before joining nodes to the cluster.

Sources

• Proxmox VE Cluster Manager – official wiki for pvecm, corosync, quorum, and QDevice setup. The pve-docs chapter was read on 2026-10-10 for requirements, node removal, corosync editing, QDevice setup, and migration settings.
• Proxmox VE High Availability – official docs for ha-manager, groups, fencing, and the watchdog.
• Proxmox VE Storage – official reference for NFS, iSCSI, and other shared storage used by HA and live migration.
• Proxmox VE Storage Replication and VM migration – local ZFS replicas, their recovery-point limitation, and local-disk migration. Checked 2026-10-12 in the project’s replication source and VM source.
• Deploy Hyper-Converged Ceph Cluster – official wiki for the Ceph distributed storage option run on cluster nodes.