High availability (warm standby)

Gardience can run as a pair of servers: a primary that does the work, and a warm standby that holds a recent copy of the primary's data and keys. If the primary is lost, you promote the standby. The console is back about a minute after you promote it, and agents move to it on their own with no re-approval.

Failover is a command you run, or, if you turn on automatic failover, a decision the standby makes on its own under strict rules (below).

How the pair works#

• Roles and generations. Each server has a role (primary or standby) and a generation, a number that goes up by one at every promotion. The newest generation is the real primary.
• The standby runs no Salt master. It keeps the pair's keys and publishes its status, but accepts no agents and its console is stopped. Restarting, upgrading or rebooting the primary therefore never moves the fleet to the standby.
• Agents know both servers. Installing with --standby-address sets agents up with both addresses in Salt's failover mode. They keep trying both until one answers, and never move back on their own.
• Sync. The primary sends the standby an encrypted backup every 15 minutes, and its Salt keys every minute.
• Fencing. Each server publishes its role and generation, signed with the Salt master key the pair shares. A primary that sees its partner is the primary of a newer generation stops its own Salt master and makes no changes. This is called being fenced.

Set up a pair#

1. Prepare both servers#

• Install Docker on both. Size both the same, as described in the sizing guide.
• Keep both clocks synced with NTP (for example chrony). A signed status whose time is more than 10 minutes away from the reader's clock is ignored, and then fencing cannot act. The console reports it when this happens.
• Create the same backup passphrase file at the same path on both servers, for example /root/.gardience-backup-pass. The standby uses it to check and decrypt what the primary sends.
• Let root on the primary reach the standby over SSH with a key, without a password prompt. The account used on the standby must be able to run the gardience command there.

2. Install both servers#

Use the same --name on both. Give both the standby's agent address, and give each the other's console URL:

# on the primary
sudo ./install/install.sh --name gardience-server \
  --standby-address standby.example.com \
  --peer-url https://standby.example.com \
  --witness-dir /mnt/gardience-witness \
  --backup-passphrase-file /root/.gardience-backup-pass

# on the standby
sudo ./install/install.sh --name gardience-server \
  --standby-address standby.example.com \
  --peer-url https://primary.example.com \
  --peer-agent-address primary.example.com \
  --witness-dir /mnt/gardience-witness \
  --backup-passphrase-file /root/.gardience-backup-pass

--peer-agent-address on the standby means agents enrolled there after a promotion still list both servers. --witness-dir (recommended) is the same shared folder mounted on both servers, for example an NFS export or an AWS EFS file system mounted at /mnt/gardience-witness; see "Network splits and the witness" below.

If Gardience is installed somewhere other than /opt/gardience on the standby, add STANDBY_INSTALL_DIR=/that/path to the primary's .env file.

3. Make the standby a standby#

On the standby:

/opt/gardience/gardience standby-mode

Its console stops, and its Salt master side holds keys but accepts no agents. It stays a standby after upgrades.

4. Keep the standby current#

On the primary, add two cron jobs as root:

# /etc/cron.d/gardience-standby
*/15 * * * * root /opt/gardience/gardience standby-sync admin@standby.example.com:/var/lib/gardience-standby --passphrase-file /root/.gardience-backup-pass
* * * * *    root /opt/gardience/gardience standby-sync admin@standby.example.com:/var/lib/gardience-standby --passphrase-file /root/.gardience-backup-pass --keys-only

The full sync:

1. Checks that this server is the live primary and is not fenced.
2. Asks the standby for its role over SSH, and refuses to send unless it is a standby of the same or an older generation. This stops older data ever being pushed over a newer primary.
3. Takes an encrypted backup, copies it with its checksum and MAC to the standby, and keeps the newest 5 there. The newest is also linked as latest.tar.gz.enc.
4. Loads the primary's Salt master and agent keys into the standby.
5. Records the result in config/standby-sync.json. gardience doctor on the primary fails if no full sync has succeeded in the last hour.

The --keys-only job sends only the Salt keys, which takes seconds. With it, agents approved since the last full sync do not need approving again after a failover. The standby keeps the newest 3 key files.

The destination directory defaults to /var/lib/gardience-standby if you leave out :DIR.

Automatic failover (optional)#

With a witness configured, the standby can take over by itself:

# on the primary
/opt/gardience/gardience autofailover on 5
# on the standby (as root: it installs a once-a-minute check)
/opt/gardience/gardience autofailover on 5 --passphrase-file /root/.gardience-backup-pass

5 is the number of minutes the primary must be silent before the standby takes over (3 or more).

How it decides:

• The primary writes a signed heartbeat to the witness every 30 seconds while it serves. Only a member of the pair can sign it, so nothing else can fake a live or a dead primary.
• The standby promotes itself only when all of these hold: the primary's heartbeat at the witness is older than the limit, the primary does not answer the standby either, and the standby can read the witness. A standby that cannot see the witness never takes over, and a primary that still answers is never replaced.
• A primary cut off from both the witness and its standby stops serving one minute before the limit. So during a network split the old primary has already stopped when the standby promotes: there are never two primaries. When it is back in touch with the witness it resumes on its own, unless the standby took over meanwhile, in which case it stays fenced as described below.
• Promotion is the same gardience promote an operator runs, from the newest backup the standby holds, and it is logged in ha/autofailover.log on the standby.

gardience autofailover status shows the setting and, on the standby, how old the primary's heartbeat is. Turn it off with gardience autofailover off on both servers. Review Held work and Approvals after an automatic failover, as after a manual one.

Fail over#

1. Make sure the primary is really down: powered off, or stopped with gardience stop on that server.

2. On the standby, run:

   Copy

   /opt/gardience/gardience promote --passphrase-file /root/.gardience-backup-pass

3. Review the Held work and Approvals pages on the new primary before releasing anything. They are as of the last sync.

promote takes the newest full backup at /var/lib/gardience-standby/latest.tar.gz.enc unless you name another. Before it changes anything, it:

• Refuses if this server has no --peer-url, because it then cannot check whether the old primary still runs.
• Asks the old primary for its status, and refuses if it still answers as a primary.
• Checks the backup's MAC, and refuses a backup older than the newest full backup this standby has received.
• Keeps the standby's own Salt keys if the every-minute key sync is newer than the backup.

It then restores the backup while still a standby, and only becomes the primary once the restore has worked. A failed promotion leaves a standby, never a half-started primary. The new generation is one more than the highest it knows of: its own, the backup's, and any it has seen from the old primary.

--force skips the checks for a peer URL, a still-running primary, and an older backup. Use it only when you know the old primary is cut off and will be powered off.

Agents set up with --standby-address reconnect to the new primary on their own. Agents without it need the DNS name they use pointed at the new primary.

After a failover#

On the old primary, remove the standby-sync cron jobs. If it comes back (after a reboot, for example), it reads the new primary's signed status, sees a newer generation, and fences itself: no Salt master, and its console makes no changes. Its sync jobs also refuse to run, and the new primary refuses its keys.

The fence stays until you act on that server. To make it the new standby:

/opt/gardience/gardience standby-mode

Then set up standby-sync on the new primary, pointing at it.

Move back later#

Do the same in the other direction: keep the old primary current as the standby, then promote it. Each promotion raises the generation, so the other server fences itself when it sees the newer one.

Fencing in detail#

• Each server publishes its status at /gardience/ha-status every 30 seconds, signed with the Salt master key. This works even while the console is stopped. Only a member of the pair holds that key, so a reply from anything else is ignored.
• A running primary checks its partner every 30 seconds. If the partner proves it is the primary of a newer generation (or of the same generation, promoted later), this server stops its Salt master and its console stops making changes.
• A partner that cannot be reached, has not received the pair's keys yet, or sends a status more than 10 minutes off the clock is never a reason to fence, and never delays a start.
• A primary that cannot verify its partner's status reports an ha-peer problem in the console, because fencing cannot work until it can.
• The console of a standby or a fenced server makes no changes: no schedules, no released held work, no commands.

Network splits and the witness. Without a witness, one case is not covered: if you promote the standby while the old primary is still running but cannot reach it, both accept agents until they can see each other again. Give the pair a witness to close it: a folder on shared storage (NFS, SMB, AWS EFS) mounted on both servers, set with --witness-dir PATH on both installs. Then:

• gardience promote records the new generation at the witness, signed with the pair's key, before anything changes, and refuses to promote if it cannot.
• A primary that cannot reach its partner reads the witness every 30 seconds. If it shows a newer promotion, the primary stops its Salt master and stays fenced, while running or after a restart.
• gardience doctor fails when the witness folder is not writable.

If the old primary loses both its partner and the witness, it keeps running (it has no evidence either way), so the first step of a failover is still to make sure the old primary is down. Tested by the HA drill: the primary is cut off from its standby, the standby is promoted, and the cut-off primary stops its master from the witness record.

How much data can be lost#

Commands used by the pair#

Related#

• Backup and restore
• Monitoring
• Installer options
• Disaster recovery runbook