Backup and restore

gardience backup writes one archive that holds everything needed to bring a Gardience server back, including the keys that let agents reconnect without being approved again. gardience restore puts it back.

What a backup contains#

• The Gardience data volume: fleet state, runs, runbooks, workflows, schedules, groups, single sign-on settings, API keys, the vault and its key, and the whole audit log.
• The Salt PKI: the Salt master's key and every accepted agent key, so agents reconnect after a restore without being trusted again. Attach-mode installs have no Salt PKI of their own, so their backups do not include one.
• The install's config directory: users, roles, the license, the TLS certificate, the audit log key and checkpoint, and the generated Salt and web server settings.
• The pillar (srv/pillar): system locks and drift rules.
• .env and compose.yml.

The Salt job cache is not backed up.

A backup holds the vault key and every agent key, so it is always encrypted unless you ask otherwise:

• with --passphrase-file F when you give one (or GARDIENCE_BACKUP_PASS_FILE in .env);
• otherwise with this server's own backup key, config/backup.key, which the first backup creates. Copy that key into your password manager. A restore on the same server finds it by itself; a restore on a new server needs it (--passphrase-file), and without it the backup cannot be opened.
• --plaintext writes an unencrypted archive, only when you ask for it by name.

Store backups off the Gardience server.

Take a backup#

/opt/gardience/gardience backup /backups/gardience-$(date +%a).tar.gz --passphrase-file /root/.gardience-backup-pass

1. Gardience pauses the console for a few seconds while it takes a snapshot, so nothing is captured half-written. Sealed audit log segments, which never change again, are linked rather than copied, which keeps the pause short.
2. The console starts again and the archive is built from the snapshot.
3. The archive is encrypted (file name ending .enc, plus a .mac integrity file) and the plain archive is removed.
4. A checksum file is written next to it, and the command prints the file name and how long the console was paused.

Without a file name, the backup is called gardience-backup-YYYYMMDD-HHMMSS.tar.gz. Relative paths, for the backup and for the passphrase file, are taken from the install directory, not your current directory. Use full paths to avoid surprises.

Files a backup writes#

Keep these files together. Restore looks for them next to the archive.

Encryption#

The passphrase file holds the passphrase on its first line. Create one like this and keep a copy somewhere other than the backup's disk:

openssl rand -base64 32 > /root/.gardience-backup-pass
chmod 600 /root/.gardience-backup-pass

Encryption is AES-256 with a key derived from the passphrase by PBKDF2 with 200,000 iterations and a random salt. The passphrase is never put on a command line.

The MAC#

An encrypted backup also gets a .mac file: an HMAC-SHA256 over the encrypted file, keyed from the passphrase (PBKDF2, 200,000 iterations). It also covers the kind of backup and the time it was made. Restore checks it before decrypting anything, so a swapped, altered or back-dated archive is refused. A wrong passphrase fails the same check.

The checksum#

The .sha256 file is a plain SHA-256 of the archive. It catches damage in storage or transfer. Unlike the MAC it needs no passphrase, so it is checked for every backup, encrypted or not.

Schedule backups#

Set them up in the console under Administration > Backups (admins only):

1. Schedule: every day or every week, at a time in your time zone, and how many copies to keep (1 to 365).

2. Destination, one of:

   • Folder on this server, such as /mnt/backups. Mount a network share (NFS or SMB) or a USB disk there to keep copies off the server. System folders such as /etc, /usr and /var/lib/docker are refused, also when a link leads there.
   • S3-compatible storage: Amazon S3, Backblaze B2, Wasabi, Cloudflare R2, MinIO and others. Give the endpoint, region, bucket, an optional folder in the bucket, and an access key that can only write, list and delete in that bucket. The secret key is stored encrypted and never shown again. Each copy is uploaded as one file, up to 5 GB.
   • SFTP server: host, port, user and folder. Gardience makes its own key pair and shows the public key; add it to that user's authorized_keys on the backup server. Get host key shows the server's keys with their fingerprints: compare one with the backup server before you choose Use. Gardience then connects only to a server holding that key.
3. Save, then Test destination: Gardience writes, lists and deletes a small test file there.

Back up now makes one right away. The Status panel shows the last backup, the last good one, the last test and the next scheduled time. Every backup, test and settings change is in the audit log, and a failed backup sends an alert to your notification channels.

Scheduled backups are the same backups as gardience backup, always encrypted with this server's backup key (config/backup.key, or the passphrase file in GARDIENCE_BACKUP_PASS_FILE), so a copy that leaves the server cannot be read without that key. They are named gardience-scheduled-YYYYMMDD-HHMMSS.tar.gz.enc, with their .sha256 and .mac files. When old copies are removed, only files with that name pattern are touched.

How it runs#

The console never holds the Salt keys, so it cannot make a backup itself. It asks, and the server's once-a-minute check does the work: gardience backup-scheduled, run by a systemd timer (gardience-backups.timer) or, without systemd, by cron. The installer and upgrades turn it on.

sudo /opt/gardience/gardience backups-cron status   # on, and the last few results (backups.log)
sudo /opt/gardience/gardience backups-cron on       # turn the check on (off turns it off)

If the check is not running, the Backups page says so and an alert is sent. The check looks for a request file and stops; it does work only when a backup is due or someone chose Back up now or Test destination. It checks a folder destination again on its own side. Copies to S3 or SFTP are made by a short-lived Gardience container that sees only the finished, encrypted backup, so the credentials never leave Gardience.

With your own cron instead#

You can also schedule gardience backup yourself:

# /etc/cron.d/gardience-backup  (as root)
15 2 * * * root cd /opt/gardience && ./gardience backup /backups/gardience-$(date +\%a).tar.gz --passphrase-file /root/.gardience-backup-pass

This keeps seven rolling daily copies. Copy /backups off the host.

gardience doctor warns when it finds no backup less than 7 days old in the install directory. That warning is fine if you keep backups somewhere else.

Automatic backups before upgrades#

gardience upgrade always takes a backup first, named pre-upgrade-VERSION-DATE.tar.gz, and keeps the newest two. These are encrypted only when the install has a backup passphrase file, set with the installer's --backup-passphrase-file option (stored as GARDIENCE_BACKUP_PASS_FILE in .env). gardience doctor warns when it is not set.

Restore#

Restore onto an install of the same Gardience version. To restore onto a new server, first install Gardience there (see Installer options). The installer refuses to start on top of data left by an earlier install with the same name, so old and new keys can never be mixed by accident.

/opt/gardience/gardience restore /backups/gardience-Mon.tar.gz.enc --passphrase-file /root/.gardience-backup-pass

1. Gardience asks you to type restore to confirm. --yes skips the question.
2. It checks the archive against its .sha256 file. If they differ, nothing is changed.
3. For an encrypted backup, it checks the .mac, then decrypts.
4. It checks that the archive really holds Gardience data. If not, nothing is changed.
5. It stops Gardience and replaces the data volume and the Salt PKI. Each is copied to a staging folder first; the old contents are removed only once the copy is complete. If a copy fails, the current data is left in place and Gardience starts again.
6. It copies back config and the pillar, and the secrets from the backup's .env (see below).
7. It starts Gardience, shows its health, and asks every agent to refresh its pillar so group targets match right away (not in attach mode).

On its next start, the service adds a system.restored entry to the audit log. It records which backup was restored, by whom, and how many entries the log held before and in the backup.

Restore options#

What is restored and what is not#

Files in config or the pillar that exist on this server but not in the backup are left in place.

Audit log entries written after the backup was taken are not in the restored log. If you forward the audit log, they are in your SIEM. See Forward the audit log to a SIEM.

After restoring onto a new server, point the DNS name or address the agents use at it. Agents reconnect within a few minutes with their existing keys.

Test your backups#

Run a restore drill at least once a year: install on a spare server, restore a real backup, and check that gardience status is healthy, the console lists every node, and Reports shows the audit chain intact.

Related#

• High availability
• CLI reference
• Forward the audit log to a SIEM
• Disaster recovery runbook