Docs / Reference
Troubleshooting
Start here for almost any problem:
/opt/gardience/gardience doctorIt checks everything an operator should worry about and says which check failed. The sections below follow its checks, then list other messages you may see. If you still need help, build a support bundle (see Support bundle and logs).
gardience doctor reports FAIL or WARN
| Line | What to do |
|---|---|
N of M services running | Run gardience status to see which service is down, gardience logs SERVICE to see why, then gardience start. |
console answering on port N but not ready | A critical problem is listed on the next lines. Fix that. |
console not answering on port N | The web front end or the Gardience service is down. Check gardience status and gardience logs nginx. |
A component name and message (for example events: ..., siem: ...) | A background component reports a problem. See the matching section below. |
warm standby not synced successfully in the last hour | The line shows the last error from standby-sync. See Warm standby. |
Gardience cannot reach the Salt API | Check the Salt master with gardience logs salt-master. In attach mode, check that your salt-api is running and reachable at the URL given with --attach. |
data volume only N% free | Add disk space. The audit log and backups need room. There is no automatic clean-up of the audit log. |
TLS certificate expires in N days | Replace config/tls/gardience.crt and config/tls/gardience.key in the install directory, then run gardience restart. |
license in grace or license: ... | See Licensing. |
audit chain: ... | See The audit log failed verification. |
automatic pre-upgrade backups are NOT encrypted | Add GARDIENCE_BACKUP_PASS_FILE=/path/to/passphrase-file to .env. |
no backup from the last 7 days | Fine if you back up elsewhere. Otherwise schedule gardience backup. See Backup and restore. |
Installing
| Message | Fix |
|---|---|
Docker is required | Install Docker Engine. |
Docker Compose v2 is required (docker compose). | Install the Docker Compose plugin. |
Docker is installed but not running, or this user cannot use it (try sudo). | Start Docker, or run the installer with sudo. |
openssl is required. | Install OpenSSL. |
Gardience is already installed in DIR. | Manage the existing install with DIR/gardience, or install into another --dir. |
data from an earlier Gardience install named 'NAME' still exists | To bring the old install back, install and then run gardience restore BACKUP. To start fresh, remove the old volumes with the docker volume rm command shown in the message. This removes the old audit log, vault and agent keys for good. |
this offline bundle is for X but this server is Y | Use the offline bundle built for this server's processor type. |
Gardience did not come up. | Run docker compose logs in the install directory to see which service failed. |
Signing in and permissions
| Symptom or message | Cause and fix |
|---|---|
| Sign-in is refused with HTTP 429 | Too many sign-in attempts from one address: 10 a minute are allowed, with a burst of 5. Wait a minute. |
That user name is reserved. | The name is one Gardience uses for itself. See Reserved names. |
The Salt master is starting or unreachable. Try again in a few seconds. | The Salt master is restarting, or Gardience cannot reach it. If it persists, check gardience logs salt-master. |
Sign-in failed on the Salt master. | The Salt master returned an error. Check gardience logs salt-master. |
| A new role does not apply | Gardience reads a person's grants when they sign in. Sign out and back in. |
Your session was not started from the Gardience sign-in. Sign out and back in. | Changes, approvals and key creation need a session that started on the console's sign-in page. Sign out and back in. |
Only Gardience admins can change this (NAME is not one). | The action needs the admin role. |
NAME is not an approver. | Approving needs the admin or approver role. |
Two-person rule: you launched this run, so someone else must approve it. | Ask another admin or approver. |
That run was interrupted; launch it again. | The run was waiting for approval when the service restarted. Launch it again. |
| The console is read-only | No valid license, or a license past its grace period. See Licensing. |
Single sign-on
| Page title or message | Cause and fix |
|---|---|
| Single sign-on is off | Single sign-on is not turned on, or a different protocol is turned on. Check Settings > Single sign-on. |
| Identity provider unreachable | Gardience could not reach the issuer URL or metadata URL. Check the address and the server's outgoing network access. Use Test connection. |
| Sign-in expired | The sign-in took more than 10 minutes, was already used, or was started in another browser. Try again. |
| Sign-in failed: the identity provider refused the sign-in code | Wrong client ID or client secret, or the redirect URI registered with the provider does not match the one shown on the card. |
| Sign-in failed: the identity token or SAML response did not verify | Check the issuer URL and client ID, that the server's clock is correct (60 seconds of difference are allowed), and for SAML that the response or assertion is signed and the metadata is current. |
| No Gardience access | None of the person's groups is mapped. Check that the group is sent in the groups claim and that the mapping uses the value the provider sends (often a group ID, not a display name). |
| Sign-in refused: a user name Gardience cannot use | The user name claim is empty, reserved or has characters Gardience does not allow. Change the user name claim. |
| Sign-in refused: also a local Gardience account | Rename the local account or the single sign-on user. |
| Sign-in failed: the Salt master did not accept the sign-in | In attach mode, the single sign-on module is missing on your master, or its GARDIENCE_SSO_KEY does not match. Follow step 4 of ATTACH.md. |
Set the public URL before turning single sign-on on. | Fill in Public URL of this Gardience. |
Public URL must look like https://gardience.example.com (https, no path). | Remove any path, and use https://. |
Map at least one group to a role before turning single sign-on on. | Add a row under Groups to roles. |
API keys and gdn
| Message | Cause and fix |
|---|---|
Sign out and back in before creating API keys. | Sign out and back in, then create the key. |
Your role can only read, so your keys can only read. | Create a read key, or ask an admin for a role that can make changes. |
Only an admin can create a key whose runs skip approval. | Leave Require approval ticked, or ask an admin. |
Unknown or revoked API key. | Create a new key. |
This API key expired on DATE. | Create a new key in Settings > API keys. |
This key's owner (NAME) was removed from Gardience | Create a key owned by someone current. |
This key's owner (NAME) has not signed in for N days, so the key is paused. | The owner signs in to the console once; the key works again at once. |
This key is read-only. | Use an operate key. |
NAME is not allowed to run FUNCTION on NODES, so this key cannot either. | The key's owner lacks that permission on those nodes. Permissions are taken from the owner's last sign-in. |
No online node matched 'GROUP'. | The group is empty or its nodes are offline. Check with gdn groups. |
gdn: not logged in. Run: gdn login https://your-gardience | Run gdn login, or set GARDIENCE_URL and GARDIENCE_KEY. |
gdn: cannot reach URL: ... CERTIFICATE ... | The console uses a certificate your computer does not trust. Log in again with --ca and a copy of the server's certificate. |
The audit log failed verification
The console shows Audit log problem on the Reports page, gardience doctor shows audit chain: ..., and the GardienceAuditChainBroken alert fires.
- Do not ignore it. It means entries were changed, removed or replaced, or the signed checkpoint does not match.
- Compare with the copy in your SIEM if you forward the audit log.
- When the investigation is done, an admin writes what was found on the Reports page and clicks Acknowledge.
- A different admin or approver confirms. The alarm then clears, and both notes stay in the log.
If the message says the audit log cannot be written, the disk is usually full. Gardience refuses changes until it can write again. Free disk space.
SIEM forwarding
siem: cannot forward audit entries to TARGET: ...; they will be sent when it is reachable
The SIEM is down or unreachable, or for tls:// its certificate is not trusted or does not match the host name. Nothing is lost: forwarding resumes from the last entry the SIEM accepted. See Forward the audit log to a SIEM.
Audit forwarding target looks like udp://siem.example.com:514 or tls://siem:6514
The destination must be udp://, tcp:// or tls://, then a host name and a port.
Something needs attention
The banner at the top of every console page lists background problems. Each says what it means, what to do, and has a button to the page where it is fixed; Details shows the exact error. An admin can choose Hide for a day for a known problem (for example a log server that is down for maintenance). Hidden problems still show in gardience doctor and /health, and every hide is in the audit log. A problem that stops happening leaves the banner on its own after 10 minutes.
No nodes, or nodes not answering
- In attach mode, if the fleet stays empty, the console banner says what failed. The usual cause is the scanner account's permissions on your master (
ATTACH.mdstep 5). The message isthe scanner account cannot list enrolled nodes. events: lost the Salt event stream: live results and job auditing are paused. Check the Salt master.- Many stale nodes: check the network between the agents and the server's Salt ports (4505 and 4506 by default).
Backup and restore
| Message | Cause and fix |
|---|---|
snapshot failed: no backup written or archive failed: no backup written | Check free space where the backup is written, and that Docker is running. |
FILE does not match its checksum (FILE.sha256): nothing changed | The archive was damaged or altered. Use another copy. |
no FILE.sha256 next to the backup ... refusing | Copy the .sha256 file next to the backup. If it is truly lost, use --no-checksum. |
FILE has no FILE.mac, so it cannot be verified: refusing | Copy the .mac file next to the backup. Use --legacy only for backups made before 1.2. |
FILE failed its integrity check (wrong passphrase, or the file or its date was altered): refusing | Check you are using the right passphrase file. |
backup has no Gardience data in it: nothing changed | The file is not a Gardience backup, or was decrypted with the wrong passphrase. |
restore of VOLUME failed; the current data was left in place | Often Docker cannot see the install directory. Check free space and Docker's access to the install directory. |
Relative paths are taken from the install directory, not your current directory. Use full paths.
Upgrades
| Message | Cause and fix |
|---|---|
no BUNDLE.sha256 next to the bundle or no BUNDLE.sha256.sig | Download the .sha256 and .sha256.sig files with the bundle and keep them next to it. |
this install has no trusted release key | Check the release key's fingerprint on gardience.com/security and pass it with --trust-key. |
release signature does not verify or checksum mismatch | The bundle is damaged or not a genuine release. Download it again. |
VERSION is not newer than the installed VERSION: refusing | You are installing the same or an older version. Use --allow-downgrade only on purpose. |
the pre-upgrade backup failed: not upgrading (nothing changed) | Usually disk space. Run gardience upgrade BUNDLE --check to see. |
new version did not become healthy: rolling back | Gardience rolled back on its own. Check gardience logs gardience and contact support with a support bundle. |
Warm standby
| Message | Cause and fix |
|---|---|
not syncing: this server is the standby | Run standby-sync on the primary, not the standby. |
not syncing: this server is FENCED | The partner was promoted. Remove the cron job here and run gardience standby-mode to make this server the new standby. |
cannot ask HOST for its role over ssh or cannot reach HOST over ssh | Check that root on the primary can SSH to the standby with a key and no prompt. |
refusing: HOST is a standby of a NEWER generation | This server is out of date. Make it the standby. |
refusing: HOST says it is 'primary ...', not a standby | The standby was promoted, or standby-mode was never run there. |
copied, but the standby's Salt keys were not refreshed | Check that Gardience is installed on the standby at /opt/gardience (or set STANDBY_INSTALL_DIR in .env) and that the passphrase file exists at the same path there. |
ha-peer: cannot verify the standby's status | The partner cannot be reached, has not received the pair's keys yet, or the two clocks differ by more than 10 minutes. Sync both clocks with NTP. |
refusing: this server has no --peer-url (promote) | Reinstall or upgrade with --peer-url, or use --force once you are sure the old primary is down. |
refusing: the primary is still running and answering (promote) | Stop the old primary first with gardience stop on that server. |
refusing: BACKUP is older than the newest full backup this standby has received | Promote from the newest backup, or use --force. |
See High availability.