Disaster recovery runbook

What a backup contains#

gardience backup writes one archive with:

• the Gardience data volume: fleet state, runs, runbooks, workflows, schedules, groups, the audit log, the vault
• the Salt PKI: the master key and every accepted node key, so restored agents reconnect without re-trusting
• config (users, roles, SSO, license, audit key), pillar, .env and compose.yml

Gardience pauses for a few seconds while the snapshot is taken so nothing is captured mid-write. The archive is owner-only (0600) and its SHA-256 is printed. With --passphrase-file it is encrypted (AES-256, PBKDF2, 200,000 iterations).

The archive holds the vault key and node keys. Store it encrypted and off the Gardience server.

Recovery objectives#

During an outage of the Gardience server the managed machines keep running normally, but nothing is managed: no patching, no drift repair, no runbooks. Agents reconnect on their own when the master returns. Held changes run then; a scheduled run whose time passed during the outage does not run late, so run it by hand if needed.

Warm standby#

For a recovery time of about a minute after you decide to fail over:

1. Keep both servers' clocks synced (NTP/chrony). Each server's signed status carries a time and is ignored when the clocks differ by more than 10 minutes, which would leave fencing unable to act; the console reports it.
2. Install Gardience on two servers with the same --name, both with --standby-address <standby's agent address>, and each with --peer-url https://<the other server>. Agents are set up with both addresses and Salt's failover mode: they keep trying both until one answers, and never fail back on their own.
3. On the standby: gardience standby-mode. Its console stops and its Salt master holds the primary's keys but accepts no agents, so restarting, upgrading or rebooting the primary never moves the fleet to the standby.
4. On the primary, every 15 minutes from cron: gardience standby-sync admin@standby:/var/lib/gardience-standby --passphrase-file /root/.gardience-backup-pass This ships an encrypted, checksummed backup over ssh, keeps the newest 5, and loads the primary's master and agent keys into the standby. It refuses to run on a server that is not the live primary, and refuses to send to a server that is not a standby of the same or an older generation. Add a second cron line every minute with --keys-only (same arguments): it ships only the Salt keys, in seconds, so machines approved since the last full sync do not need approving again after a failover. A failed sync shows as FAIL in gardience doctor.
5. If the primary is lost: make sure it is really down (powered off, or stopped with gardience stop). Then on the standby run gardience promote --passphrase-file /root/.gardience-backup-pass. It becomes the primary of a new generation and brings up its console with the latest data; agents reconnect to it on their own with no re-approval.
6. On the old primary, remove the standby-sync cron job. If it comes back (for example after a reboot), its Salt master reads the promoted server's signed status, sees a newer generation, and stays fenced: no Salt master, and its console makes no changes. The fence stays until you act on that server. Make it the new standby with gardience standby-mode, then set up standby-sync on the new primary.
7. To move back later, do the same in the other direction: keep the old primary current as the standby, then promote it. Each promotion raises the generation, so the other server fences itself when it sees the newer one.

How the pair decides: each server publishes its role and generation at /gardience/ha-status, signed with the Salt master key the pair shares, so a reply from anything else is ignored. A peer that cannot be reached never delays a primary's start. A network split (promoting the standby while the old primary is still running but cut off from it) is covered when the pair has a witness (--witness-dir, a folder on shared storage mounted on both): every promotion is recorded there first, and a primary that cannot reach its partner stops its Salt master within about 30 seconds of seeing a newer promotion at the witness. Without a witness, or if the old primary loses the witness too, both would accept agents until they can reach each other again. That is why step 4 still says to make sure the old primary is down first.

Tested every release by tools/tests/ha-drill.sh, with the agent settings the product hands out: standby holds keys with no master; a primary restart keeps the agent on the primary; primary stopped and standby promoted, agent reconnects with no re-approval; old primary restarted (fenced: no Salt master, console makes no changes, its sync job refused, and the promoted server refuses its keys); promoted console scans and manages the agent; then a full failback to the original server, after which the other server fences itself.

Data written on the primary after the last standby-sync (at most the sync interval) is not on the standby; with the every-minute --keys-only job, agent approvals are at most a minute behind. The audit trail is also on your SIEM. A promotion restores the queue of held changes and pending approvals as of that sync: review the Held work and Approvals pages on the new primary before releasing anything.

Schedule backups#

# /etc/cron.d/gardience-backup  (as root)
15 2 * * * root cd /opt/gardience && ./gardience backup /backups/gardience-$(date +\%a).tar.gz --passphrase-file /root/.gardience-backup-pass

This keeps 7 rolling daily copies. Copy /backups off the host (object storage, backup system, tape).

Restore onto a new server#

1. Provision a Linux host with Docker (see SIZING.md), with the same hostname or DNS name the agents use.
2. Install the same Gardience version: ./install/install.sh --dir /opt/gardience --hostname <same name> (offline bundle on air-gapped networks). The installer refuses to start on top of leftover data from an earlier install with the same name, so an old audit log or vault can never be mixed with new keys by accident.
3. Copy the backup over and restore: /opt/gardience/gardience restore /path/gardience-Mon.tar.gz.enc --passphrase-file /root/.gardience-backup-pass
4. Check: gardience status shows healthy; the console lists every node; Reports shows the audit chain intact.
5. Point DNS (or the IP) agents use at the new server. Agents reconnect within a few minutes with their existing keys.

Test it#

Every release runs this drill automatically (tools/tests/fresh-install.sh): install, attach agents, back up, delete the install folder and all its data, install fresh into an empty folder (new secrets, new configuration), restore, and confirm the same nodes answer with no agent re-trusted, the audit chain is intact with no entries lost, and the vault secrets are back. The drill reinstalls on the same machine, so it does not exercise moving DNS to a different host; do that once in your own environment. Customers should run their own restore drill at least once a year.

Upgrades and rollback#

gardience upgrade gardience-X.Y.Z.tar.gz refuses a bundle whose checksum or signature does not verify (or when the install has no trusted release key), takes a backup and stops if that fails, updates the install's configuration for the new release while keeping its secrets, users and certificate, and on a failed health check rolls back images, configuration and data on its own.