Multi-Cluster Management
PegaProx's core capability is managing multiple Proxmox VE clusters from a single interface.
Unified Dashboard
- Real-time CPU, RAM, and storage metrics via Server-Sent Events (SSE)
- Cluster health overview with node status indicators
- Global VM/CT search across all clusters
- Aggregated resource utilization charts
All Clusters overview (from 1.3)
- Storage of every cluster in one list.
- Inventory export of all guests with vCPU, memory, disks, IPs, HA state, tags and pool.
- Guests without a backup: guests that no backup job covers are listed and can raise an alert. New guests get a grace period, and a
no-backuptag marks intentional gaps.
Node details (from 1.3)
- Temperatures are read from the kernel's hwmon interface, so lm-sensors is no longer needed on the nodes.
- ZFS pools of a node show their device tree with state and read, write and checksum error counts, the last scrub or resilver and any data errors. An alert can fire when a pool is not ONLINE or shows errors.
QDevice
New in 1.4.0
The QDevice view, its alert and its metric come with 1.4.0. They are not in 1.3.0.
A cluster with a QDevice (corosync-qdevice, the external vote for two-node and other even-sized clusters) shows it next to its nodes: in the Corporate tree right after the nodes, in Modern after the node cards, with a dot that is green when every node that answered is connected, yellow when some are not, red when none is, and grey when the last read failed (the details then show what was read before). Clicking it opens its details, read-only on every instance and on a standby as well:
- QNetd host, model, algorithm, tie-breaker, state, last poll and echo reply, as the QDevice daemon prints them (
corosync-qdevice-tool -sv). - Answered by: the node whose answer these fields are, the API host when it has a daemon.
- Per node: the state of the QDevice daemon on each node, "No QDevice daemon" where it is not running, and "Not asked" for a node PegaProx has no address of its own for.
Where it comes from: Proxmox VE answers GET /cluster/config/qdevice from the QDevice daemon of the node that serves the call; the call has no node parameter. PegaProx therefore asks every online node it can reach at an address of its own, the API host and the fallback hosts of the cluster, with the cluster's own credentials and TLS settings (no SSH). The fallback hosts are found on the first connection: every node that was online then. A node behind none of those addresses is listed but not asked; its address can be added to fallback_hosts through the cluster API. A cluster is read at most every 30 seconds, only while someone has it open (the selected cluster, the clusters open in the Corporate tree) or while a QDevice alert rule watches it, and never per guest. The account needs node.view; users limited to pools or single guests do not see it. The API route is GET /api/clusters/{cluster_id}/qdevice.
Alert: the event rule QDevice not connected (Automation > Alerts) raises one alert per node whose QDevice daemon is not connected to the QNetd host, or that runs no daemon while the cluster has a QDevice, and sends an all-clear once it is connected again. It goes out through e-mail, push and the webhook channels like every alert, can be acknowledged and muted per rule or per node, and is evaluated on the active instance only. It turns critical when none of two or more nodes asked is connected. When every node asked answers without a daemon, PegaProx takes the QDevice as removed and closes its open alerts quietly. Prometheus gets pegaprox_cluster_qdevice_connected{cluster_id,cluster,node} (1 or 0) from the last read; a scrape never reads it itself.
Limitation
The QNetd host is not a Proxmox node. PegaProx sees it only through the QDevice daemons of the cluster nodes: its connection state and answer times, not its own CPU, memory or updates. Monitor the QNetd host itself with your usual host monitoring.
Check connection (from 1.3)
Check connection on a cluster runs read-only probes (API and fallback hosts, TLS fingerprint, the token's privileges, versions, clock, quorum, SSH) and names a fix for each item that fails. It needs cluster.config.
Broadcast banner (from 1.3)
Administrators can show a banner to every user or to chosen tenants and roles, for example before planned maintenance. Users can close it; a changed text shows again.
API reference (from 1.3)
The HTTP API is described in OpenAPI, generated from the routes and naming the permission each route needs. Every signed-in user can browse it under API reference in the user menu, or download the openapi.json from there.
Cross-Cluster Operations
- Cross-Cluster Migration: Move VMs between different Proxmox clusters
- Cross-Cluster Load Balancing: Automatically distribute workloads across clusters based on resource utilization
- Cross-Cluster Replication: Replicate VMs across clusters for disaster recovery
From 1.3, removing the source guest after a cross-cluster migration needs the vm.delete permission on it (moving without removing needs vm.migrate). A user limited to single pools or guests cannot place a guest on a target cluster where they are limited the same way, and a tenant's VMID range applies to the target VMID.
Layout Modes
- Modern Layout: Card-based dashboard with cluster tiles
- Corporate Layout: Tree-based sidebar with compact tab strips and reduced whitespace for managing 50+ clusters efficiently. From 1.3 the sidebar has a Tools section (Topology, World Map, Automated Installations, Hypervisor Migration and more) below the clusters.
- Cloud Layout: an airy card grid; from 1.3 it can be picked at the first login and set as the default theme.
The layout is a per-user setting in Preferences. From 1.3 there is also a System theme that follows the operating system's light or dark mode.
Cluster Groups
Organize clusters into hierarchical groups for streamlined management. Each group can have its own load balancing, replication, and failover policies. Tenants can be assigned to specific groups for scoped access.
PVE Node Subscription Management v0.9.12
Datacenter → Subscriptions tab. Set / refresh / delete subscription keys per Proxmox node directly from PegaProx, plus a cluster-wide aggregator that lists every node's status, level, and expiry in one view.
Key handling: the aggregator masks the key (read via cluster.view); the per-node endpoint returns the raw key (read via node.view) and set/refresh/delete require admin.settings. Stored encrypted at rest in the SQLCipher DB.
| Method | Endpoint | Description |
|---|---|---|
GET |
/api/clusters/{id}/datacenter/subscriptions |
Aggregate view - all nodes, keys masked |
GET |
/api/clusters/{id}/nodes/{node}/subscription |
Per-node detail (key not redacted) |
PUT |
/api/clusters/{id}/nodes/{node}/subscription |
Set / update key |
POST |
/api/clusters/{id}/nodes/{node}/subscription |
Force online validation |
DELETE |
/api/clusters/{id}/nodes/{node}/subscription |
Remove key |
Contributed by @gyptazy / credativ GmbH (PR #508).
Cross-Cluster Load Balancing
Auto-Migration
Score-based algorithm using weighted CPU + RAM utilization. VMs auto-migrated when imbalance exceeds threshold. Exclusion rules for specific nodes, VMs, and pools.
Migration Tolerance
Deadband parameter (0–20) prevents VM ping-pong between nodes. 15-minute cooldown per VM after migration. Applies to both intra-cluster and storage balancing.
Pool Exclusion
Exclude entire Proxmox resource pools from auto-balancing. All VMs in excluded pools are skipped by the balancer.
Dry-Run Mode
Test the algorithm without migrating. Reports what it would do, allowing safe configuration validation.
Cross-Cluster Replication
Snapshot-based replication between clusters with cron scheduling (hourly, daily, weekly). After initial full sync, transfers are incremental (delta between snapshots) to minimize bandwidth.
Predictive Migration Analysis
Analyzes resource utilization trends and forecasts future imbalances. Suggests proactive migrations before bottlenecks occur - shifting from reactive to preventive cluster management.
Note: Predictive analysis requires at least 7 days of historical metrics data for reliable recommendations.
Cross-Cluster Replication
Snapshot-based replication copies VM data between clusters. After initial full sync, transfers are incremental (delta only).
Create Job
Replication → Cross-Cluster → Create Job. Select source VM, target cluster, target storage.
Set Schedule
Cron syntax: 0 */4 * * * (every 4h), 0 2 * * * (daily 2 AM).
Monitor RPO
Dashboard shows time since last successful sync per job.
Understanding RPO
| Schedule | Cron | RPO |
|---|---|---|
| Every hour | 0 * * * * |
1 hour |
| Every 4 hours | 0 */4 * * * |
4 hours |
| Daily | 0 2 * * * |
24 hours |
Link replication jobs to Site Recovery plans for automated failover. Readiness check verifies replication status before failover.
| Method | Endpoint | Description |
|---|---|---|
GET |
/api/cross-cluster-replications |
List replication jobs |
POST |
/api/cross-cluster-replications |
Create job |
POST |
/api/cross-cluster-replications/{id}/run |
Run the job now |
DELETE |
/api/cross-cluster-replications/{id} |
Delete job |