Site Recovery & DR

PegaProx includes a full site recovery engine for defining recovery plans between clusters, assigning VMs with boot ordering, mapping network and storage resources, and executing planned, emergency, or test failovers-all from the web UI or API.

Recovery Plans

Define source → target cluster pairs with network and storage mappings. Add VMs with boot groups and delays to control startup order.

Failover Modes

Planned (graceful), emergency (source down), test (non-destructive clones), and failback (reverse direction).

Readiness Checks

Pre-validate cluster connectivity, replication status, target resources, and network mappings before failover.

Auto-Failover

Enable automatic failover with configurable timeout. Source unreachable triggers the recovery plan automatically.

Adding VMs to a Plan

Each VM has a boot group and boot delay. VMs in group 0 start first, then group 1, etc. The delay (seconds) controls wait time before proceeding. Critical for dependencies-e.g. database (group 0) before app servers (group 1).

Field	Type	Description
vmid	Integer	Proxmox VM ID on the source cluster
boot_group	Integer	Boot priority group (0 = first, default: 0)
boot_delay	Integer	Seconds to wait after starting this VM (default: 30)
replication_job_id	String	ID of cross-cluster replication job for RPO tracking

Failover Modes

Planned Failover

Graceful migration when source is reachable. VMs shut down cleanly, final replication sync runs, VMs start on target in boot group order.

Emergency Failover

Source cluster unreachable. VMs start on target using last replicated data. Some recent changes may be lost (RPO depends on replication schedule).

Test Failover

Non-destructive: creates cloned VMs on target using replicated data without affecting source. Use cleanup endpoint to remove test VMs.

Boot screenshots (from 1.3): after a test failover has started the clones, PegaProx takes one console screenshot of each started guest about a minute after boot and keeps it with the test event as evidence that the guest came up. Expand the test event in the plan's failover history to see them, with the time each took. At most 20 guests per test are captured, three at a time; a guest that cannot be captured is listed with the reason, and nothing about the screenshots changes the outcome of the test. The screenshots of the last five tests of a plan are kept. Viewing a screenshot needs vm.console on the protected guest and on its test clone.

Failback

Reverses direction-migrates VMs back to original source. Both clusters must be reachable.

Warning: Emergency failover may result in data loss. Configure frequent replication schedules for mission-critical VMs. Auto-failover timeout should be at least 120 seconds to avoid false triggers from transient network issues.

Re-Running After a Failed Plan v0.9.12

Planned and emergency failover are now re-runnable from a completed or failed plan state - fix the underlying issue and re-trigger from the UI or API without manual DB intervention. Prior to v0.9.12, the atomic state-transition's WHERE status='ready' guard rejected legitimate retries with a misleading "concurrent failover in progress" message. The guard is now ('ready', 'completed', 'failed'); concurrent-run protection is still enforced via the per-plan lock. Reported by @blackshocks (#413).

Pre/Post Webhooks

Each plan can define webhook URLs called before and after failover (HTTP POST with JSON payload). Integrate with alerting systems (PagerDuty, Slack), DNS updates, or external automation.

RBAC Permissions

Permission	Description	Required For
site_recovery.view	View recovery plans and events	List plans, get details, view events
site_recovery.manage	Create/edit/delete plans and VMs	CRUD operations on plans and VMs
site_recovery.failover	Execute failover operations	Readiness check, failover, failback, test, cancel

Step-by-Step Setup

Configure Replication

Set up cross-cluster replication jobs for each VM. Go to Replication → Cross-Cluster.

Create Recovery Plan

Site Recovery → New Plan. Choose source and target clusters, configure network and storage mappings.

Add Protected VMs

Assign VMs with boot groups. Group 0 for infrastructure, group 1 for databases, group 2 for apps. Link replication jobs.

Run Readiness Check

Validate connectivity, replication status, and target resources. Resolve any errors before proceeding.

Test Failover

Run a test first. Verify recovered VMs work, then cleanup. No production impact.

Enable Auto-Failover

Optionally enable with timeout (min 120s). Configure webhooks for notifications.

Test Failover NIC isolation & pre-flight

Disconnect NICs on Test Failover

A new per-plan option Disconnect NICs on Test Failover is available in the plan's Test settings. When enabled, test-failover clones are brought up with every NIC disconnected (link down) before they start, so a DR test cannot collide with production IPs on the live network.

Note

This option applies to QEMU guests only.

Planned / Failback pre-flight

When the target VMID is already present on the target cluster (pre-seeded by replication), PegaProx now fails fast during Planned and Failback pre-flight with clear guidance to use Emergency Failover - which starts the already-replicated target - instead of launching a cross-cluster live migration that can only abort.

Warning

There is no data loss: the pre-flight fails before touching the source.

API Endpoints

Method	Endpoint	Description
GET	/api/site-recovery/plans	List all recovery plans
POST	/api/site-recovery/plans	Create a new recovery plan
GET	/api/site-recovery/plans/{id}	Get plan details with VMs and replication status
PUT	/api/site-recovery/plans/{id}	Update plan settings
DELETE	/api/site-recovery/plans/{id}	Delete a recovery plan
GET	/api/site-recovery/plans/{id}/vms	List VMs in plan
POST	/api/site-recovery/plans/{id}/vms	Add VM to plan
PUT	/api/site-recovery/plans/{id}/vms/{vm_id}	Update VM settings
DELETE	/api/site-recovery/plans/{id}/vms/{vm_id}	Remove VM from plan
POST	/api/site-recovery/plans/{id}/readiness	Run readiness check
POST	/api/site-recovery/plans/{id}/failover	Execute planned failover
POST	/api/site-recovery/plans/{id}/emergency	Execute emergency failover
POST	/api/site-recovery/plans/{id}/test	Execute test failover
POST	/api/site-recovery/plans/{id}/test/cleanup	Clean up test failover VMs
POST	/api/site-recovery/plans/{id}/failback	Execute failback
POST	/api/site-recovery/plans/{id}/cancel	Cancel running action
GET	/api/site-recovery/plans/{id}/events	Get event history