Upgrade

gardience upgrade moves an install to a new release. It checks the release's signature, takes a backup, installs the new release, and rolls everything back on its own if the new version does not become healthy.

The easy way: the newest release#

sudo /opt/gardience/gardience upgrade --latest --check   # what would change, nothing is changed
sudo /opt/gardience/gardience upgrade --latest

This asks gardience.com for the newest release with the install's license, downloads it with its signature, and then upgrades exactly as below. A free license always gets new releases; a trial or paid license until 7 or 30 days after it ends. A lapsed license gets no new releases, and everything installed keeps working.

Before you start (a bundle you downloaded)#

• Download the new bundle with its .sha256 and .sha256.sig files, and put all three in the same folder. Do not rename them: the signed checksum names the bundle file, and the upgrade refuses a file with a different name.
• Use the full path to the bundle. The gardience command switches to the install folder before it runs, so a relative path is read from there.
• Make sure the install encrypts its pre-upgrade backup. See "Encrypt the pre-upgrade backup" below.

Check first (dry run)#

To check an upgrade without changing anything:

sudo /opt/gardience/gardience upgrade /root/gardience-X.Y.Z.tar.gz --check

The check verifies the signature, the checksum and that the new version is newer than the installed one. Then it reports:

It exits with code 1 if any check fails, and 0 otherwise.

Upgrade#

To upgrade:

sudo /opt/gardience/gardience upgrade /root/gardience-X.Y.Z.tar.gz

What happens, in order:

1. Verify. The install checks the bundle with the code it already runs: the signature must verify against the release key the install already trusts, the signed checksum must be for this exact file, and the new version must be newer than the installed one. If any check fails, it stops and changes nothing.
2. Unpack. The bundle is unpacked into releases/<date-time>/ in the install folder. From here on, the new release's own code runs the upgrade.
3. Back up. A full backup is written to the install folder as pre-upgrade-<old version>-<date-time>.tar.gz (with .enc added when it is encrypted). If the backup fails, the upgrade stops and nothing is changed.
4. Snapshot. The current images are tagged for rollback, and the settings (.env), compose.yml, config/, srv/, the gardience command and VERSION are copied aside.
5. Update the configuration for the new release. Secrets, users, the TLS certificate, the license, system locks, drift rules and any setting you added to .env are kept.
6. Build or load images. A normal bundle builds them. An offline bundle loads its prebuilt images after checking the CPU architecture.
7. Apply roles and permissions of the new release, then start the services.
8. Health check. It waits up to about 4 minutes for the console to report ready: Salt connected and no component reporting a problem.
9. Finish. It records the new version, prints upgraded to X.Y.Z, and keeps only the two newest pre-upgrade backups.

Options#

Release signatures#

Each release has a SHA-256 checksum file (.sha256) and a signature over that checksum (.sha256.sig). The install trusts the release public key at install/release-public-key.pem in the release folder it was installed or last upgraded from.

The upgrade refuses a bundle when:

• the .sha256 or .sha256.sig file is missing,
• the install has no trusted release key and you did not pass --trust-key,
• the signature does not verify,
• the signed checksum names a different file, or the file does not match the checksum,
• the version is not newer and you did not pass --allow-downgrade.

Before you pass a key with --trust-key, compare its SHA-256 fingerprint with the one published at gardience.com/security:

openssl pkey -pubin -in gardience/install/release-public-key.pem -outform DER | sha256sum

Rollback#

You do not run rollback yourself. If any step after the backup fails, or the new version does not report ready in time, the upgrade:

1. stops the services,
2. puts back the configuration, .env, compose.yml, Salt content, the gardience command and VERSION,
3. puts back the previous images,
4. restores the data from the pre-upgrade backup,
5. checks health again.

If the old version is not healthy, read the service log with sudo /opt/gardience/gardience logs gardience and run sudo /opt/gardience/gardience doctor. If the data restore itself fails, the message says so, and you can restore the pre-upgrade backup in the install folder with gardience restore.

Encrypt the pre-upgrade backup#

The pre-upgrade backup holds every key, including node keys and the vault key. It is encrypted when the install has a backup passphrase file:

• At install time, pass --backup-passphrase-file /root/.gardience-backup-pass.
• On an existing install, add the line GARDIENCE_BACKUP_PASS_FILE=/root/.gardience-backup-pass to .env in the install folder.

The file must be readable when the upgrade runs. gardience doctor shows a warning while pre-upgrade backups are not encrypted.

Upgrade from 1.1.x or earlier#

Installs from before 1.2 have an older upgrade command that does not update the configuration. Use the new bundle's command against your install instead:

cd /root
tar xzf gardience-1.2.X.tar.gz
openssl pkey -pubin -in /root/gardience/install/release-public-key.pem -outform DER | sha256sum   # compare with the published fingerprint
sudo GARDIENCE_HOME=/opt/gardience /root/gardience/install/gardience upgrade /root/gardience-1.2.X.tar.gz \
     --trust-key /root/gardience/install/release-public-key.pem

Use full paths, as shown: the command switches to the install folder named by GARDIENCE_HOME before it reads the bundle and the key.

After that, gardience upgrade works as described above.

Primary and standby pairs#

A warm standby is a second Gardience server that holds a recent copy of the primary's data and Salt keys, ready to take over with gardience promote. Run the same upgrade on both servers.

• The standby stays a standby. After a successful upgrade on the standby, the upgrade stops its console again and restarts its Salt master in standby mode, and prints this server is the standby: console stopped again.
• Keep the copies current. On the primary, gardience standby-sync copies an encrypted backup to the standby (run it from cron, for example every 15 minutes), and gardience standby-sync --keys-only copies just the Salt keys (run it every minute). Both refuse to run on a server that is not the live primary.

Upgrading a pair to 1.2#

From 1.2, the standby runs no Salt master until it is promoted, so restarting the primary never moves agents to it. Agents set up for failover by an earlier release need one more setting, so they keep trying both servers while neither answers (for example during a promotion). Add it once, from the install folder on the primary:

cd /opt/gardience
sudo docker compose exec salt-master salt -C 'G@gardience_groups:* or *' file.append /etc/salt/minion.d/gardience.conf "master_tries: -1"
sudo docker compose exec salt-master salt '*' service.restart salt-minion

On Windows the file is C:\ProgramData\Salt Project\Salt\conf\minion.d\gardience.conf. Agents enrolled with 1.2 already have this setting.

Then, on the standby, run this once after upgrading:

sudo /opt/gardience/gardience standby-mode

And on the primary, add the every-minute --keys-only cron line next to the regular standby-sync line.

Related#

• Install on one server
• Install without internet access
• Uninstall
• Requirements