Failover and operation

Available from PegaProx 1.3

High availability for PegaProx itself is part of PegaProx 1.3. Automatic failover is a beta of 1.4.0, see Automatic failover.

When the leader is gone

While the leader does not answer, the other members say so in their banner: changes are paused, live data and consoles keep working. In manual mode, the default of every group, nothing else happens on its own: the group fails over by hand.

To make another member the leader:

On that member, open Settings → High Availability and use Promote to leader (on a plain standby the button reads Promote to active).
Type PROMOTE and confirm with your password.
If the old leader still answers, the member first fetches its latest configuration. If that fetch fails, the promotion stops and offers Promote anyway, without the latest configuration; changes from the last few minutes may then be missing.
The member becomes the leader: it acts on all clusters and runs schedules, alerts and every other background job. The other members follow it.

The old leader steps down to standby as soon as it sees the new one, also when it comes back hours later. Promote only when the current leader is really gone or you want to move the lead on purpose.

Automatic failover

Beta

Automatic failover is new in 1.4.0 and ships as a beta. Try it in your setup before you rely on it; you can switch back to manual failover at any time. How to switch it on: Automatic failover (beta).

In a group switched to automatic failover nobody promotes by hand; Promote is not offered there. When the leader stops renewing its lease, the members wait until the lease has run out, elect a new leader by majority, and the winner takes over. Meanwhile every signed-in user sees that the group has no leader or that a member is taking over, and for ten minutes afterwards that the leader changed. At the default lease of 20 seconds a failover pauses changes and automation for about a minute (54-59 seconds); live data and consoles keep working.

Make leader hands the lead to another member on purpose. Changes pause while that member catches up (up to 10 seconds), then the leader hands it the next term and restarts as a standby, and the member restarts as the leader.
Force leader, in the Danger zone of a member, is only for a group that lost its majority for good. The member becomes the manual active of the group without the members that do not answer; tick each of them only once it is powered off or destroyed. An alert goes out, the group is in manual mode afterwards, and automatic failover comes back only through the switch, with all its checks.

What you notice of the clocks and the lease

A member whose clock is more than 5 seconds off the majority of the members it measures waits one lease length longer before it asks for votes. A member with a right clock wins whenever one asks, and a group whose clocks all drifted apart still elects a leader, one lease later at most. The member keeps voting meanwhile. Fix its time sync (NTP).
Writes need 2 seconds of lease left. In automatic mode the leader takes a change, also one handed on from a member, only while at least 2 seconds of its lease remain. Otherwise the request is answered with 503 and the code HA_NO_LEASE; send it again. Scripts against the API should retry on it.
A leader whose clock steps by more than 120 seconds loses the lead. The other members refuse its calls from then on, its lease runs out and another member takes over. A step of more than 5 seconds on the acting instance raises the alert HA group: clock jump on the usual alert channels (critical beyond 120 seconds). It names the instance and the step, so you know why the lead moved: check the time sync of that host.

Removing a member

On the leader, Remove takes a member out of the group (type REMOVE and confirm with your password). If the member answers, it lets go of the group at once and stays passive. If it cannot be reached, it does not know yet that it was removed: unpair it there before you use it again.

A member that was not confirmed as a standby under the current epoch may still be running as an active instance somewhere. Removing it then needs your confirmation I confirm this instance is shut down for good, because a member removed without being told would keep acting on the clusters.

A removed member is refused by every other member. On the removed instance the page says so; unpair it to use it on its own, or unpair and pair it again with a new code to make it a standby again.

Unpairing

Unpair (type UNPAIR and confirm with your password):

on the leader dissolves the whole group. Every standby stops receiving changes and stays passive until it is unpaired or promoted there;
on a standby makes that instance standalone: it starts acting on its own (cluster actions, schedules, alerts). Do this only when the leader is gone for good, or two instances will act on the same clusters.

Changes that were not carried over

A sync replaces a member's configuration with the leader's. When that replaces rows the member held and the leader did not, the member keeps them as an encrypted copy instead of dropping them. A banner points to them: copies of changes that were not carried over - review them in Settings > High Availability.

Each copy shows when it was kept, why, which rows exist only on the member and which only in the synced configuration, and who made the changes. Download asks for your password again, since a copy holds rows of the instance (accounts included), like a configuration backup. Dismiss deletes a copy for good. Nothing else deletes copies, so review them and dismiss the ones you no longer need; the page warns when they take up a lot of space.

Troubleshooting

"The HA state file cannot be read": The instance stays passive until config/ha_state.json is fixed or restored; promoting and changing the interval stay locked. Restore the file and restart PegaProx.
"These columns from the active instance do not exist here and were skipped": The members run different releases. Update them to the same release.
"The active instance sends a snapshot format this version does not read": Same cause: update both instances to the same release.
A standby shows no clusters: Its live view is off, or it has not synced yet. Switch the live view on (the instance restarts) or use Sync now.
Changes are refused on a member: Forwarding is off there, or the leader does not answer. Make the change on the leader, or switch forwarding on.
Consoles do not open on a standby: Consoles and shells run on the leader and on active members only. Make the standby an active instance on the leader, or open the console there.
The pairing code is refused: It is valid for 15 minutes and only once, and a newer code replaces it. Create a new one on the leader.
No Automatic failover card, or "Automatic failover is not available in this release yet": The instance runs 1.3.0 or older, which has no automatic failover: its HA page shows no such card and no witness, and its API refuses them with that text (409 HA_AUTO_NOT_SHIPPED). Update every member to the same, newer release.
The automatic failover switch stays off: A red item in the list Before it can be switched on holds it. Fix that item, for example the time sync of a member whose clock is more than 5 seconds off, or add a witness to a group of two.
A change is refused with HA_NO_LEASE: The group is in automatic mode and the leader has less than 2 seconds of its lease left, or no leader holds a lease right now. Send the change again; during a failover, wait until the new leader has taken over.