Skip to content

Setting up a PegaProx group

Available from PegaProx 1.3

High availability for PegaProx itself is part of PegaProx 1.3. Automatic failover and the witness are a beta of 1.4.0, see Automatic failover (beta).

Everything here is under Settings → High Availability, on each instance. Only administrators see and use it.

Before you start

  • The same PegaProx release on every instance. A member on another release can skip columns it does not know; the sync then lists them and asks you to run the same release everywhere.
  • An HTTPS address per instance that the other instances can reach, for example https://pegaprox-b.example.com:5000. It has to start with https://. This is the address of the normal PegaProx web interface, so no extra port is needed between the instances.
  • Network access from each instance to every cluster. A member that takes over, or serves users, connects to the same Proxmox, PBS and other endpoints as the leader, with the credentials it received from the leader.
  • Correct clocks. Run NTP (systemd-timesyncd or chrony) on every instance.
  • A fresh second instance is easiest. Pairing replaces the whole configuration of the instance that joins (see below).

Pair a second instance

  1. On the instance that stays in charge (it becomes the leader): open Settings → High Availability, enter its own address under Address of this instance, and use Create pairing code. Confirm with your password. The code is shown only once and is valid for 15 minutes; creating a new code replaces the old one.
  2. On the new instance: open Settings → High Availability, go to Pair this instance as a standby, paste the code, tick I understand that the configuration of this instance is replaced and confirm with your password.
  3. The new instance restarts as a standby. Everything configured on it before (users, clusters, settings) is replaced by the leader's configuration, and it connects to the leader's clusters with the credentials stored there.

To add more members, create another code on the leader under Add a standby. A group holds up to four instances, the leader included; when it is full, remove a member before adding another.

The Group list on the leader shows every member with its address, its last contact, how it was last seen (for example Standby), its key and whether it is confirmed (seen as a standby under the current epoch).

Live view and forwarding

Two switches on each member decide what users can do there. Both are on by default.

  • Connect to the clusters while following the leader (live view): the member shows live data from the clusters using the credentials synced from the leader. Without it a standby connects to no cluster and only shows its synced configuration. Changing this switch restarts the instance.
  • Carry out changes through the leader (forwarding): what is done on the member, such as starting a VM or changing a setting, is carried out by the leader under the same account and synced back right after. Without it the member only shows and changes nothing.

While the leader does not answer, a forwarding member is read only and says so in its banner; live data and consoles keep working.

Active members

On the leader, a standby can be made an active instance (Active instance: {name} in the member list). An active member serves users as the leader does: consoles, shells and SPICE open there, and the leader carries out every change. It needs both the live view and forwarding switched on, on that member, and takes effect at its next sync (shown as Active (pending) until then).

At most three instances are active, the leader included. Automation (HA monitoring, balancing, schedules, alerts) still runs only on the leader.

Sync interval

Interval in seconds sets how often the instance contacts the rest of its group: 5 to 3600 seconds, 30 by default. Each instance keeps its own value. Sync now pulls the leader's configuration at once.

The member card shows the last successful sync, the content of the last one (rows and tables), and the configuration version: a counter in the form epoch.count. A member is in step with its leader when both show the same version.

Time zone of the schedules

Scheduled actions, scheduled tasks, snapshot policies and scheduled updates fire by the wall time of one time zone on whichever member leads. Set it on the leader under Time zone of the schedules with an IANA name such as Europe/Vienna. The last-run times move to the new zone first, and the members take it with their next sync.

Until a zone is set, each instance evaluates the schedules in its own zone, and a failover to a member in another time zone shifts them. If a host has no time zone data for the chosen zone, the page says so; install tzdata there.

Automatic failover (beta)

Beta

Automatic failover is new in 1.4.0 and ships as a beta: the Automatic failover card and the witness card carry a Beta badge. Try it in your setup before you rely on it; you can switch back to manual failover at any time. Every group stays in manual mode until an administrator switches it on, also after an update.

In automatic mode the members elect the leader by majority: when the leader fails, another member takes over on its own. The leader holds a lease that a majority of the votes keeps renewing. A leader that cannot renew it stops acting, and a member takes over only after the old lease has run out.

Before you switch it on

  • At least three votes. Every data member votes unless you set its vote off in the member list; a group of two adds a witness. A member that joins a group which is automatic already has no vote until you give it one.
  • The same release on every member, and every member answering.
  • Clocks within 5 seconds of each other. Run NTP on every member and on the witness.
  • A time zone for the schedules of the group, see above.
  • A site for every member, set in the member list on the leader. The checklist warns about a site whose loss would stop the group.
  • Every cluster with node HA ready for a leader at any site.

Switch it on

  1. On the leader, open Settings → High Availability. The Automatic failover card lists what stands in the way under Before it can be switched on.
  2. Red items keep the switch off until they are fixed. Amber items are warnings: tick I understand at each one to go ahead.
  3. Turn on the Automatic failover switch and check the lease: 15 to 120 seconds, 20 by default. The card says how long a failover pauses changes and automation at that length, about 54-59 seconds at 20.
  4. Confirm with your password and use Switch on. The card shows Switching on until every member, the witness included, holds the new voter config, then Automatic with the leader, its lease and the majority. A member that does not take it within ten minutes takes the group back to manual mode.

To go back, turn the same switch off and confirm with your password. The group is in manual mode once a majority of the members holds the change; from then on a member is promoted by hand again.

The witness

A witness is a third vote that holds no data: a small process at a third site, with its own network path to every member. It never leads and never serves users. It wants a host of its own (never one that runs PegaProx), about 40 MB of memory, TCP port 5005 open for the members, and its clock within 5 seconds of theirs.

On the leader, Add a witness makes a one-time code, good for 15 minutes, and shows one command per way to install the witness (Linux, offline from the leader, Docker) with the code in it. Run the one you want on the witness host. The card then lists the witness with the last time it was heard and its clock; Remove witness takes it out again. The full guide, with the firewall and updates of the witness, is The PegaProx witness.