My 3-node Proxmox cluster, as it runs today

Live log · last update 2026-10-17

Three boxes, one cluster, no shared storage, no Ceph. Everything below is pasted from the running cluster. Copy what you need. When something changes I add a dated line to the log and update the output.

Log

2026-10-17 sevro moved to 10G. The old node-to-node DAC now goes to the switch, bonded active-backup with the 2.5G port. reaper and sevro updated to 9.2.21. skynet BIOS F41 broke GPU passthrough, fixed in the BIOS (that post).
2026-10-16 MS-A2 BIOS 1.03 skipped on both minis. It breaks PCI passthrough and the option that fixes it is locked [3]. Staying on 1.02.

The boxes

Node	Hardware	PVE	Pools
reaper	Minisforum MS-A2, Ryzen 9 9955HX, 32G, BIOS 1.02	9.2.21, kernel 7.0.14-24	rpool 472G, data-zfs 952G
sevro	Minisforum MS-A2, Ryzen 9 9955HX, 32G, BIOS 1.02	9.2.21, kernel 7.0.14-24	rpool 472G, data-zfs 952G, nas-pool 952G
skynet	Gigabyte B650 AORUS ELITE AX V2, Ryzen 9 7950X3D, 96G, RTX 5070 Ti	9.2.20, kernel 7.0.14-19	rpool 1.81T, tank 476G

skynet is the odd one. It's a desktop board with the GPU passed to my daily desktop VM, so it updates last.

Quorum

$ pvecm status
Cluster information
-------------------
Name:             howlers
Config Version:   9
Transport:        knet
Secure auth:      on

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.1.14
0x00000002          1 192.168.1.13 (local)
0x00000003          1 192.168.1.166

Three votes, need two. Any one box can go down and the other two keep going. With two nodes you need a QDevice; a third real node fixed that for me.

corosync.conf

$ cat /etc/pve/corosync.conf
nodelist {
  node {
    name: reaper
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.1.14
  }
  node {
    name: sevro
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.1.13
  }
  node {
    name: skynet
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.1.166
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: howlers
  config_version: 9
  interface {
    knet_link_priority: 10
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  token_coefficient: 125
  version: 2
}

One link on the house LAN. The docs want a second, separate corosync link [1]. I don't have one yet. That's the next thing.

Network: 10G with a 2.5G fallback

$ cat /etc/network/interfaces
auto bond0
iface bond0 inet manual
    bond-slaves nic2 nic0
    bond-mode active-backup
    bond-primary nic2
    bond-miimon 100

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.13/24
    gateway 192.168.1.1
    bridge-ports bond0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    bridge-stp off
    bridge-fd 0

$ grep -E "Mode|Active|Interface|Speed" /proc/net/bonding/bond0
Bonding Mode: fault-tolerance (active-backup)
Currently Active Slave: nic2
Slave Interface: nic2
Speed: 10000 Mbps
Slave Interface: nic0
Speed: 2500 Mbps

nic2 is the 10G DAC, nic0 the 2.5G copper. Pull the DAC and traffic moves to nic0. Active-backup needs nothing from the switch [2]. The bridge is VLAN-aware, so a guest just gets tag=101 on its NIC.

Apply it with a safety net. The rollback runs as a systemd timer, so it still fires if your ssh session dies:

cp /etc/network/interfaces /root/interfaces.bak
systemd-run --on-active=2min --unit=net-rollback sh -c 'cp /root/interfaces.bak /etc/network/interfaces && ifreload -a'
ifreload -a
# still in? cancel the rollback
systemctl stop net-rollback.timer

storage.cfg

$ cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content vztmpl,import,iso,backup

zfspool: local-zfs
    pool rpool/data
    content rootdir,images
    sparse 1

zfspool: data-zfs
    pool data-zfs
    content images,rootdir
    mountpoint /data-zfs
    nodes sevro,reaper
    sparse 1

cifs: nas-iso
    path /mnt/pve/nas-iso
    server 192.168.1.20
    share share
    content iso
    subdir /lab/images/iso
    username neal

Replication needs ZFS storage with the same storage ID on both nodes [4]. That's why both minis have data-zfs. ISOs live on the NAS so every node sees the same list.

Replication instead of shared storage

$ pvesr status
JobID      Enabled    Target         LastSync             NextSync             Duration  FailCount State
100-0      Yes        local/reaper   2026-10-17_14:50:00  2026-10-17_14:55:00  1.647333          0 OK
103-0      Yes        local/reaper   2026-10-17_14:50:02  2026-10-17_14:55:00  1.470997          0 OK
106-0      Yes        local/reaper   2026-10-17_14:50:03  2026-10-17_14:55:00  1.581741          0 OK
108-0      Yes        local/reaper   2026-10-17_14:50:05  2026-10-17_14:55:00  1.914492          0 OK
109-0      Yes        local/reaper   2026-10-17_14:50:07  2026-10-17_14:55:00  1.929232          0 OK

Every 5 minutes each guest snapshots and ships the difference to the other mini. Takes under 2 seconds. Make one:

pvesr create-local-job 109-0 reaper --schedule "*/5"

Moving a replicated guest is quick because most of the disk is already there:

qm migrate 105 reaper --online --with-local-disks
pct migrate 109 reaper --restart

HA is off. No fencing device, so after a node dies I start the replica on the other one by hand. Worst case I lose 5 minutes.

Updating a node

# empty the node first
qm migrate <id> <other-node> --online --with-local-disks
pct migrate <id> <other-node> --restart

apt update && apt full-upgrade -y
reboot

pveversion

One node at a time. Two stay up, quorum holds.

What I'd change

A second corosync link on its own network.
Every pool here is one disk. Two 12TB drives are on the way as a ZFS mirror.
reaper and skynet to 10G like sevro. The DACs are ordered.