High Availability (guests)
This page is about keeping your guests running when a Proxmox node fails. High availability for PegaProx itself (a group of PegaProx instances with a leader and standbys) is described under PegaProx High Availability.
PegaProx provides application-level HA monitoring on top of Proxmox's built-in HA:
- Node Failure Detection: Monitors node health and detects failures
- Auto-Restart: Automatically restarts VMs on a healthy node when a node goes down
- Configurable Timing: Set detection interval and restart delay per cluster
- Affinity-Aware: Respects affinity rules during failover placement
How PegaProx HA Differs from Proxmox HA
Application-Level Monitoring
Checks VM responsiveness and service health, not just node heartbeat. Detects application-level failures that cluster HA misses.
Cross-Cluster Awareness
Can coordinate failover across multiple clusters when site recovery is configured. Proxmox HA only operates within a single cluster.
External Observer
Monitors from outside cluster quorum. Can detect and react to split-brain scenarios where Proxmox HA may be deadlocked.
Configuration
| Parameter | Default | Description |
|---|---|---|
failure_threshold |
3 | Consecutive failed node checks before a node is considered down |
recovery_delay |
30s | Wait after confirming failure before recovery starts |
node_timeout |
60s | Node must be unreachable this long before it counts as dead |
ha_check_interval |
10s | Health-poll interval (internal, not user-tunable) |
Alert-only mode is controlled by the HA enable/disable toggle, not an auto_restart flag.
Failover Process
Detect Failure
Node unreachable or VMs stopped responding during health check.
Wait Threshold
Must persist for failure_threshold consecutive checks to rule out transient issues.
Select Target
Pick optimal node based on resources, affinity rules, and current load.
Restart VMs
VMs restarted in priority order. Higher-priority VMs (databases, DCs) come up first.
Which guests are recovered
Only guests that can start on another node are moved:
- Shared storage: every disk, the EFI and TPM state and any mounted ISO are on storage all nodes reach. These are recovered.
- No disks at all: recovered, they start anywhere.
- Local storage: a guest with any volume on the failed node's own storage (local disk, local ISO, a bind mount not marked
shared=1) is skipped and named in the log. Move its disks to shared storage to make it recoverable. - Unknown: when the storage of a guest cannot be determined, it is skipped to be safe rather than started without its disks.
The guest configs are read from the cluster's own configuration on the remaining nodes, because Proxmox would forward that read to the failed node, which does not answer.
Note: Maintenance mode is not treated as a failure. VMs are gracefully live-migrated, no failover alerts triggered. Integrates with rolling updates for zero-downtime patching.
Storage Heartbeat
Uses shared storage files to distinguish true node failure from network partition. Nodes write timestamps; if a node stops writing but storage is accessible, it's a real failure.
Advanced HA Configuration
| Parameter | Default | Description |
|---|---|---|
storage_heartbeat_enabled |
false |
Enable storage-based heartbeat for split-brain detection |
storage_heartbeat_path |
(none) | Path on shared storage for heartbeat files |
dual_network_mode |
false |
Use secondary NIC as additional heartbeat channel |
self_fence_installed |
false |
Node can self-fence on split-brain |
recovery_delay |
30 |
Seconds before initiating recovery after failure |
Safety rules for node recovery
These settings are in the cluster's HA settings.
Fencing per node
How PegaProx powers a failed node off before it starts the guests elsewhere. Only an IPMI fence with the address and password of the node's BMC can be read back: after the power off, the power state is read from the BMC, and only that counts as proof that the node is off. A fence that cannot be read back is shown as No proof.
Two-node clusters
A two-node cluster has to force quorum on the node that is left. By default PegaProx does that only after the failed node was powered off and read back as off. Without a node that has such a fence, a failed node is not recovered automatically, and the HA settings say so.
Unsafe two-node recovery switches that proof off: quorum is forced without it. In a network split both nodes can then run the same guests. Leave it off unless you understand and accept that risk; it only matters where the two-node cluster mode is enabled.
Self-fence agent per node
A small agent on each node stops the node's guests when the node has been out of contact for the fence delay, so a node that lost its network does not keep running guests that were started elsewhere. Check agents shows per node which agent version runs, whether it answers, the earliest time a recovery can start, and which PegaProx instances it reaches. An agent from an earlier PegaProx keeps running as it is until you install the self-fence agent again; the page offers the upgrade.
Cluster claim
Relevant when you run a PegaProx group. With the claim switched on, the leader writes a claim of its own into /etc/pve of the cluster, and node recovery runs only while the cluster carries the claim of this instance. If another PegaProx instance holds the claim under a higher epoch, this instance is the stale one and recovers nothing; under the same epoch, recovery is refused because two instances would act on the cluster. Take over the claim writes the claim of this instance over the existing one: do it only when the other instance no longer acts on this cluster. Only administrators switch the claim.
Interrupted node recoveries
If a former leader stopped while it recovered a failed node, nothing resumes that on its own. The HA settings list such runs: guests that were moved and not started (start them with Start the moved guests), guests held on purpose because the failed node was online when they were moved (check that node first, it may still run them), and steps that were begun for guests the cluster no longer lists. Dismiss removes a run from the list without changing anything on the cluster.
HA Endpoints
| Method | Endpoint | Description |
|---|---|---|
POST |
/api/clusters/{id}/ha/enable |
Enable HA |
POST |
/api/clusters/{id}/ha/disable |
Disable HA |
GET |
/api/clusters/{id}/ha/status |
HA status + heartbeat info |
Failover Scenario Example
Health Check Failure
Node fails to respond to periodic probe.
Confirmation
3 consecutive failures over ~90s (default 30s interval).
Node Offline
Status updated, new operations blocked.
Recovery Delay
30s default wait to prevent thrashing.
Target Selection
Best node by free resources + affinity rules.
VM Restart
VMs restarted in priority order on target.
Alert Sent
Email/webhook notification with failover details.
Note: Total time from failure to VM restart: ~2 minutes with defaults (90s detection + 30s delay).