Proxmox Cluster Quorum Lost: What It Means and How to Fix It

Proxmox cluster quorum loss is one of those events that looks catastrophic from the GUI and is often straightforward to fix. The cluster goes red, /etc/pve turns read-only, operations start failing – and the cause is usually not a Corosync bug or a corrupted filesystem. It is usually vote math (Corosync and fencing explained here). Either a node went down and the remaining nodes no longer hold a majority, or maintenance removed too many votes at once, or a two-node cluster was running without a third vote to begin with.

That said, the consequences vary significantly depending on whether HA is active. Understanding that distinction before you start recovery is more important than running the first fix you find.

Quorum Lost, Corosync Down, or pmxcfs Still Starting? Tell Them Apart First

A red GUI and a failing pvecm status do not always mean the same thing. Before choosing a fix, separate four distinct failure classes – the recovery path is different for each:

• Quorum lost – Corosync membership exists, but this partition does not hold enough votes to be authoritative.
• Corosync service failed – the local corosync process is stopped, crashed, or cannot parse its own configuration.
• pmxcfs waiting for Corosync/quorum – boot or service ordering is still waiting for cluster membership to form; this is not automatically filesystem corruption.
• Node communication degraded but still quorate – red X marks or stale status in the GUI while the cluster as a whole still holds a majority.

Run this set before picking a recovery command. If corosync.service itself is failed or inactive, the fix is a service-level restart and log review, not a vote-math conversation. If pve-cluster.service is still starting, give it a few seconds and re-check rather than assuming proxmox cluster quorum is broken.

What Quorum Actually Does

Proxmox is designed to prefer temporary unavailability over data corruption. When proxmox cluster quorum is lost, the cluster cannot prove which partition is authoritative. Rather than risk two nodes modifying the same VM configuration or storage simultaneously, Proxmox blocks writes until a majority is restored. That is why the GUI turns red and operations stop – not because something is broken, but because Proxmox is refusing to guess.

Proxmox clusters use a voting system built on Corosync to decide which partition of nodes is authoritative. Each node has one vote by default. A partition needs more than half the total configured votes – a strict majority – to remain quorate and allow writes to the cluster filesystem (pmxcfs).

pmxcfs replicates cluster configuration in real time across all nodes: VM configs, storage definitions, HA rules, VMID allocations. Only the partition with a strict majority of votes stays writable. The minority partition switches to read-only and blocks new cluster operations.

Split Brain: The Failure Quorum Exists to Prevent

Split brain is the scenario proxmox cluster quorum exists to prevent. Understanding it makes the rest of the article make sense.

A network partition divides the cluster into two groups that can no longer communicate. Each group still has nodes running. Each group may still have access to shared storage. Neither group knows whether the other side is alive or dead – and this is exactly the scenario quorum and fencing are built to contain, not a scenario where both sides are free to act.

Under normal proxmox cluster quorum protection, only the partition holding a majority of votes stays writable. The minority side’s pmxcfs goes read-only and cluster-managed operations there are blocked – it does not get to keep operating as if nothing happened. Fencing adds a second layer in HA clusters: a node that loses quorum removes itself from the equation via watchdog reboot rather than waiting to confirm whether the other side is alive.

Real split brain – two partitions independently believing they are authoritative and both writing to the same VM disk – shows up when these safeguards are bypassed or incorrectly overridden, for example:

• pvecm expected forced low on more than one side of a partition
• fencing disabled or interrupted mid-sequence
• an operator manually starting a guest outside supported cluster controls while the other partition is still active
• storage access and locking misconfigured so both sides can reach the same disk image

Storage corruption from simultaneous writes is not always immediately visible. The image looks intact. The VM boots. The corruption surfaces later – during a backup, a restore, or a filesystem check. By then the original cause is long gone and the damage is hard to trace. That is the reason overriding expected_votes without confirming the other side is offline is dangerous – it removes the exact protection that prevents this.

What Happens to Running VMs During Quorum Loss

This is the question most operators search first when proxmox cluster quorum drops and the cluster turns red.

Without HA:

Quorum loss without HA is primarily a management freeze. Existing guests normally continue if the underlying host, network, and storage remain healthy – most management actions that require a pmxcfs write or a distributed lock fail while the node is non-quorate. Do not infer guest health solely from the quorum state: if the same network partition or power event that caused quorum loss also affected storage, running guests can be impacted independently of the quorum status shown in the GUI. You can SSH into nodes, observe the system, and run diagnostics. What you typically cannot do is change anything cluster-managed until quorum is restored.

With HA:

With HA the stakes are higher, but the behavior is not automatic for every node and every quorum-loss event. On an armed HA cluster with a functioning watchdog, loss of the quorum/HA control path can stop the local resource manager (LRM) from resetting the watchdog, and the node may then self-fence – commonly around a minute, though exact timing depends on watchdog configuration. This is not a crash. It is Proxmox preventing the node from continuing to run HA-managed resources in a partition that may conflict with another active partition. If a surviving partition still holds quorum and has access to the required storage, HA can attempt to recover the affected resources there once the failed node is considered safely fenced – not before. Verify HA and watchdog evidence rather than assuming every reboot after quorum loss was fencing:

Also check the BMC reset reason where available – it confirms whether the reboot was a watchdog fence or something else entirely.

In HA clusters, quorum loss is not always “things slow down and I have time to investigate.” Depending on how the cluster splits, it can trigger fencing on more than one node.

Situations that look scary but usually are not:

• One node rebooted during maintenance → cluster GUI turns red → running VMs still healthy
• Power blip took one node offline → /etc/pve read-only → nothing actually corrupted
• Cluster shows red X marks → SSH still works on all nodes

In non-HA clusters, this is quorum protection working as designed. The cluster is not broken. It is waiting for a safe majority before allowing writes again.

Six Causes of Proxmox Cluster Quorum Loss

The order below reflects practical investigation priority – the sequence most likely to find the actual cause fastest. It is not a statistical frequency ranking.

1. Vote Math Problems

The cluster simply does not have enough votes online to form a majority. This is a recurring, high-yield first check for proxmox cluster quorum loss in homelab and SMB environments – rule it out before investigating anything else.

The most recurring pattern: a two-node cluster with no QDevice. One node goes down for any reason – reboot, shutdown, power loss – and the remaining node has one vote out of two. That is 50%, not a majority. Quorum is immediately lost.

Cold boot after power loss is a pattern that trips up homelab operators more than any other. The full sequence: power fails, all nodes shut down simultaneously, power returns, one node boots faster than the others. That single node has one vote out of the cluster’s expected total – not a majority. The cluster shows Quorate: No. onboot guests do not start. The operator assumes something is broken. Nothing is broken. The cluster is waiting for the rest of the nodes to come online and restore a majority. Using pvecm expected 1 here is unnecessary and adds split-brain risk if any other node is in an unknown state.

Symptoms: quorum dropped immediately after a deliberate shutdown, reboot, or power event; the cluster was healthy before; no network errors in the logs.

Recovery: restore a majority first. Bring the missing node back online, or restore QDevice reachability. Only after verifying the missing side is genuinely offline should you consider pvecm expected <N> as a temporary override. For a persistent two-node quorum problem, the right fix is architectural: add a third voter (QDevice or third node), not recurring use of expected_votes.

2. Corosync on a Shared or Congested Network

This is a frequent transport-level cause of proxmox cluster quorum loss on shared networks. Corosync traffic is competing with VM, storage, backup, or migration traffic on the same link. Corosync has tight latency and timing requirements. When the link is congested or jittery, Corosync misses heartbeat tokens and starts forming new membership configurations – which can cascade into quorum loss and, in HA clusters, fencing.

Proxmox explicitly recommends a dedicated physical NIC for cluster communication and warns against running storage or migration traffic on the Corosync path – see the Proxmox networking guide for how to lay out bridges and VLANs so Corosync gets its own path.

Symptoms: quorum flaps while SSH and ping look mostly fine; instability correlates with backup runs, VM migrations, or storage replication; multiple nodes may reboot in HA clusters after a period of network stress.

Log patterns:

Recovery: move Corosync onto a dedicated low-latency network. Stop routing bulk traffic across the Corosync path. Token and consensus timeout tuning (token_coefficient) can help in high-latency setups, but it is a compensating measure – the real fix is the network.

3. Bonding and Asymmetric Connectivity Mistakes

Operators add a Linux bond to the Corosync interface expecting redundancy, but Corosync can already use multiple independent networks and switch between them – a bond is not required for that redundancy. Load-balancing bond modes carry a documented risk of asymmetric connectivity, where some nodes see different subsets of peers at any given moment. Not every LACP deployment produces this, but Corosync may fail to form stable quorum in this state, or the cluster may fence itself after a link or switch event that looks harmless on the surface.

Proxmox explicitly advises against load-balancing bond modes for Corosync. If LACP must be used, Proxmox recommends bond-lacp-rate fast on both the node and the switch, and the design should be validated under real cable/switch failure, not assumed safe from the config alone.

Log patterns:

Recovery: prefer multiple Corosync links on independent physical paths over hiding all redundancy inside one bond. If a bond is still required, switch to active-backup mode rather than LACP or balance-rr, and validate failure detection and convergence under a real cable or switch failure before trusting it.

4. Address, Hostname, and Configuration Drift

Corosync configuration gets out of sync with actual network state. This happens after hostname changes, IP reassignments, moving Corosync to a new VLAN, adding or replacing nodes, or manual edits to corosync.conf that increment config_version incorrectly. Proxmox recommends using IP addresses rather than hostnames in cluster configuration, because hostname resolution can change over time.

Symptoms: all nodes appear individually reachable, but the cluster shows red X marks, ghost nodes, or inconsistent views; /etc/pve becomes read-only despite basic connectivity looking fine.

Log patterns:

Recovery: if the cluster is quorate, update /etc/pve/corosync.conf once and let pmxcfs distribute it to every node, then verify with systemctl status corosync after the update propagates. Do not manually edit a node-local Corosync configuration as a normal repair step – creating different corosync.conf variants on separate nodes is how config divergence, mismatched config_version, and a larger split-brain risk get introduced. If the cluster is not quorate or Corosync cannot start at all, use the documented Proxmox recovery procedure for that specific failure and preserve a copy of the original configuration before making any change.

5. QDevice Placement or Reachability Errors

QDevice is the standard fix for two-node clusters, but the fix only works if the QDevice is genuinely independent from the cluster nodes. The most common design mistake: placing qnetd as a VM inside the same Proxmox cluster or on one of the cluster hosts. When that host goes down, the QDevice goes with it – exactly when you need it most.

QDevice is primarily intended to improve quorum behavior in even-node clusters, especially two-node designs. For a normal three-node cluster, a QDevice is usually unnecessary – do not add one without understanding how the specific qdevice algorithm changes vote behavior and failure scenarios on an odd-node cluster.

Recovery: move qnetd to an external physical host, NAS, SBC, or non-cluster hypervisor. Re-run pvecm qdevice setup. Remember that Proxmox requires removing the QDevice before adding or deleting cluster nodes. Losing the QDevice does not always immediately make a healthy two-node cluster non-quorate while both real nodes stay connected – but it silently removes the tiebreaker that would have let one real node fail safely, so treat qnetd reachability as something worth monitoring, not just checking after an incident. A QDevice also only supplies vote arbitration – it does not replace redundant shared storage, independent switches, or compute capacity.

6. MTU and Edge Transport Cases

This is the least common cause and worth checking only after the five above are excluded – or when logs point directly at MTU discovery failures.

Corosync uses KNET’s PMTUD (Path MTU Discovery) to determine the usable packet size between nodes. An MTU mismatch across NICs, VLANs, or switches can cause repeated PMTUD aborts and unstable peer reachability, even when basic ping works. This surfaces most often after enabling jumbo frames on some but not all segments.

Log patterns:

Diagnosis: verify a consistent MTU end-to-end – NICs, VLANs, switches, any tunnels. Run corosync-cfgtool -n to see the live MTU Corosync is using. Recovery: normalize MTU across the full path and restart Corosync after the path is clean.

Log Patterns and First-Pass Triage

When proxmox cluster quorum breaks, start here:

pvecm status answers the vote math question in seconds. systemctl status corosync confirms whether the Corosync process is actually running and shows recent service errors at a glance – most operators run this immediately after pvecm status. corosync-cfgtool -n shows live KNET link state and the MTU Corosync is actually using – on newer Corosync releases corosync-cfgtool -s and corosync-cmapctl expose the same state differently, so check corosync-cfgtool --help on the installed version rather than assuming one flag works everywhere. pvesh get /cluster/status gives a structured JSON view of node states useful in scripts and automated health checks.

If the KNET/TOTEM symptoms above point at the network layer specifically, a few targeted checks go further than a generic connectivity test:

Use these as needed rather than running every command on every incident – the goal is confirming route symmetry, packet loss, and whether Corosync’s UDP traffic is actually leaving the expected interface, not building a giant indiscriminate command dump.

Healthy cluster output:

Broken cluster output:

One vote out of three expected. Two nodes are offline or unreachable from this node’s perspective. The first question is whether that is intentional.

Treat every message below as evidence that Corosync communication failed, not as the root cause itself – correlate it with kernel, switch, BMC, and workload timelines (CPU scheduling delay, host lockup, NIC/driver issues, congestion, MTU, a wrong ringX_addr, firewall, or packet loss can all produce the same symptom) before naming a fix.

Pattern recognition:

Expected votes > Total votes, Quorate: No – vote math problem. Count online nodes. Check QDevice state. Do not chase packet-level causes until vote math is ruled out.

[KNET] link: host X link Y is down or host X has no active links – Corosync path failure. Think NIC, switch, bond mode, VLAN, or wrong ringX_addr. The host itself may be fully up.

[TOTEM] Token has not been received, A processor failed, forming new configuration – congestion, latency, or jitter on the Corosync path. Check whether the link is shared with bulk traffic.

[QUORUM] This node is within the non-primary component and will NOT provide any services – this node is in the minority partition. Do not force writes here until you have confirmed the other side is offline.

pmxcfs ... cpg_send_message retried 100 times – cluster filesystem cannot get reliable group communication. /etc/pve behavior will degrade even if some node-to-node connectivity still exists.

Nodes rebooting roughly 60 seconds after quorum loss in an HA cluster are self-fencing. That is expected behavior, not a secondary failure.

Exact Proxmox Cluster Quorum Errors and Fast Checks

These are the exact strings operators paste into search when proxmox cluster quorum breaks. Each one maps to a specific first check.

“cluster not ready – no quorum”

The node cannot perform a cluster operation because its current partition lacks a majority of configured votes. Run pvecm status, restore missing nodes or QDevice connectivity, and do not lower expected votes until every missing partition is confirmed offline. If a specific guest also fails to start while this error is showing, the VM won’t start troubleshooting guide covers the no-quorum branch alongside other startup failure causes.

“waiting for quorum”

This is a symptom, not a diagnosis on its own. Work through the branches in order rather than reaching for expected 1 first: remaining nodes have not finished booting, the Corosync service failed, KNET addresses are unreachable, QDevice is unavailable while a real node is also down, the configuration no longer matches actual network addresses, or a firewall/MTU/link issue is blocking transport.

“waiting for pmxcfs mount to appear and get quorate”

This can mean a normal boot-time wait for cluster membership, a pve-cluster/pmxcfs startup problem, Corosync not running, no majority currently online, or an invalid cluster configuration. It is not automatically filesystem corruption.

“cluster consists of less than three quorum-providing nodes”

This is a topology warning, not necessarily an active failure. A two-node cluster can operate normally while both nodes are online. Without an external QDevice, losing either node removes quorum. A QDevice improves vote arbitration but does not add storage or compute redundancy – a third full node remains the stronger architecture for HA workloads.

Corosync “KNET link is down”

Check the configured ring/link address, NIC state, routing, VLAN, bond configuration, switch, firewall, MTU, and kernel driver logs – in roughly that order, cheapest checks first.

Common Proxmox Cluster Quorum Incidents and First Check

Typical patterns by environment:

Before You Run pvecm expected

Recovery: Safe Use of expected_votes

pvecm expected exists for legitimate emergencies. It is not a workaround for bad cluster design.

When it is appropriate:

• Repairing a broken corosync.conf when the cluster cannot otherwise reach quorum
• Running pvecm delnode after a permanently failed node has been powered off
• Getting critical guests back online after confirming the missing nodes are physically offline

The rule: only use pvecm expected to complete one narrowly defined recovery action after eliminating split-brain risk, and restore normal vote membership as soon as possible. The runtime override resets when nodes rejoin – it is a temporary state, not a persistent config change. Do not use it as a steady-state solution, and do not leave expected 1 as the final state of a two-node cluster.

Capture evidence before changing votes or configuration on any proxmox cluster quorum incident, so it can be reviewed later without guessing:

If a node is permanently lost (dead hardware, not coming back), the emergency override is a bridge, not the endpoint: confirm the node is powered off and cannot return, restore quorum safely if required, remove it with the documented pvecm delnode workflow, clean up residual configuration only per official guidance, then restore normal expected votes.

Planned Maintenance vs Unexpected Failure

These require different approaches.

Planned maintenance: on PVE 9.2 and later, Proxmox provides a cluster-wide HA Arm/Disarm workflow for HA-safe maintenance windows – confirm the exact commands and modes for the installed version with ha-manager help before relying on them. In freeze mode, current resource state remains as-is and HA pauses new reactions and requested state changes. In ignore mode, HA stops managing resource state changes so you can move them manually. While HA is disarmed, automatic fencing and failover are intentionally suspended – appropriate for Corosync or network work, and best kept as short as possible. This is not a universal requirement for every single-node reboot; check current state first with ha-manager status, and re-arm HA and verify all resources once maintenance is done. On PVE 8.x releases without this workflow, use the earlier per-resource maintenance-mode approach instead.

Safe node shutdown sequence for a 3-node cluster:

• Verify cluster is quorate before starting (pvecm status)
• Migrate HA guests off the target node if required
• Put the target node into maintenance mode
• Shut down one node at a time only
• Verify quorum is still intact after each shutdown
• Never remove majority votes simultaneously

Unexpected failure: assume split-brain risk until proven otherwise. Do not modify cluster state. Do not lower expected votes before verifying which side holds shared storage and which side is truly offline. In HA clusters, let fencing complete – interrupting it mid-sequence can leave guests in an undefined state.

The distinction matters because the tooling is the same but the risk model is completely different. During planned maintenance you know the full picture. During unexpected failure, you do not.

Preventing Your Next Proxmox Cluster Quorum Incident

These architecture decisions are what separate a cluster that never sees a proxmox cluster quorum incident from one that hits the same failure every few months.

Minimum safe cluster designs:

Beyond topology, five operational decisions eliminate most of the recurring incidents documented in Proxmox forum threads:

• Use one or more low-latency Corosync links whose bandwidth and availability are protected from storage, backup, and guest traffic – physical separation is the strongest design, and carefully controlled VLAN separation may be acceptable for smaller environments
• Use IP addresses instead of hostnames in corosync.conf – hostnames introduce a DNS dependency into cluster availability
• Test node failure before production use – shut down one node deliberately, verify quorum survives, bring it back
• Never shut down more nodes simultaneously than your cluster can afford to lose
• Run qnetd on a physically separate host – NAS, SBC, Raspberry Pi – not inside the cluster

None of these require significant extra hardware in most homelab setups.

FAQ

What does proxmox cluster quorum “Quorate: No” actually mean in practice?

The cluster does not have enough votes online to be authoritative. Most cluster operations that write to /etc/pve are blocked. Running VMs generally keep running. Starting new VMs, editing configs, running migrations, and most management operations will fail with cluster not ready - no quorum until quorum is restored.

Can I just use pvecm expected 1 to get back online?

You can, but you should not do it casually. If the other nodes are offline and confirmed not coming back on their own, it is a valid emergency tool. If there is any chance another partition is still active – especially one with access to the same shared storage – you risk split brain and VM image corruption. Verify the missing nodes are genuinely offline before using it.

Why does a two-node cluster lose quorum when one node reboots?

One vote out of two is 50%, not a majority. Proxmox’s quorum model requires more than half. A two-node cluster needs either a third node or a QDevice to provide the tiebreaker vote that keeps the surviving node quorate during any single failure.

What is a QDevice and where should I run it?

A QDevice is an external voting participant provided by corosync-qnetd, running on a host outside the cluster. It adds one vote to the cluster without adding a full Proxmox node. For a two-node cluster, it gives the surviving node a 2/3 majority on single failure. It should run on a physically separate host – NAS, SBC, VM on a different hypervisor – never as a VM inside the same cluster it is arbitrating.

If my HA nodes self-fence after quorum loss, is something broken?

No, provided HA was armed on that node. The LRM cannot reset the watchdog without quorum, so the node reboots to prevent HA-managed resources from being active in two places simultaneously. Confirm it with ha-manager status and the HA/watchdog logs rather than assuming – not every reboot after quorum loss is a fence event. If it was a fence, it means HA is working as designed. The question to investigate is why quorum was lost in the first place.

Does losing the QDevice immediately break a two-node cluster?

Not necessarily. Both real nodes can stay quorate together as long as they’re both online and can talk to each other. What losing the QDevice actually does is remove the tiebreaker silently – the next single node or link failure now takes the whole cluster non-quorate, with no external vote left to prevent it.

Does a QDevice make a two-node cluster fully HA?

No. A QDevice only supplies a tiebreaker vote for quorum arbitration. It does nothing for shared storage redundancy, independent switching, or spare compute capacity – those still have to be designed separately for a two-node cluster to actually tolerate a node failure in practice.

Final Thoughts

Proxmox cluster quorum loss is the cluster doing exactly what it was designed to do: refuse to guess which partition is authoritative. The GUI turns red because Proxmox is choosing temporary unavailability over the risk of data corruption. That is the correct trade-off.

The failure modes worth preventing are the ones that make quorum loss unavoidable by design – two-node clusters without a tiebreaker vote, Corosync sharing a link with backup or storage traffic, QDevice running inside the cluster it is supposed to arbitrate. Most of these are one-time architecture decisions that cost nothing to get right the first time and are painful to fix after an incident.

When quorum does drop unexpectedly, vote math and network are the right starting points – not config edits and not expected_votes overrides. Restore a majority first. Everything else follows from there.