Monitoring

Gardience gives you four ways to watch it:

• Prometheus metrics at /gardience/api/v1/metrics.
• Alert rules for Prometheus, shipped with the product.
• Health and readiness checks at /gardience/health and /gardience/ready, for load balancers and scripts.
• gardience doctor, one command on the server that checks everything an operator should worry about.

Prometheus metrics#

The metrics endpoint needs an API key. Create a Read only key for your monitoring system under Settings > API keys (see API keys and the REST API).

Keys pause when their owner has not signed in for 30 days (7 days for single sign-on owners), and they expire after 90 days. Give the monitoring key an owner who signs in regularly, and replace the key before it expires. The GardienceMetricsDown alert below tells you if scraping stops.

An example Prometheus scrape job:

scrape_configs:
  - job_name: gardience
    scheme: https
    metrics_path: /gardience/api/v1/metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/gardience-key
    tls_config:
      ca_file: /etc/prometheus/gardience.crt
    static_configs:
      - targets: ["gardience.example.com"]

Keep job_name: gardience: the shipped GardienceMetricsDown rule looks for it. ca_file is needed only when the console uses its self-signed certificate (config/tls/gardience.crt in the install directory).

Metrics#

All metrics are gauges.

Grafana dashboard#

Open Administration > Automation analytics and choose Download the Grafana dashboard (.json). In Grafana, choose Dashboards > New > Import, upload the file, and pick the Prometheus data source that scrapes Gardience.

Automation analytics#

Administration > Automation analytics shows the same numbers in the console, for the last 7, 30, or 90 days: runs per day, success rate, median run time, each runbook and workflow with its success rate and average time, the machines with the most failed results, who ran the most, and the engineer time saved this month. It reads the run history Gardience already keeps, including runs moved to the archive.

| gardience_events_connected | 1 when the Salt event stream is connected, otherwise 0. |

Shipped alert rules#

The rules are in docs/enterprise/prometheus-alerts.yml in the release. Load the file into Prometheus with rule_files.

What to do when they fire:

• GardienceAuditChainBroken: open Reports in the console. Investigate before anyone acknowledges the alarm. Clearing it takes two people: an admin records what was found, then a different admin or approver confirms.
• GardienceComponentProblem: run gardience doctor. Signed-in users can also see each problem in the console banner.
• GardienceSaltEventsDisconnected: live results and job auditing are paused. Check the Salt master with gardience logs salt-master.
• GardienceManyNodesSilent: check the network between the server and the agents, and the agents themselves.
• GardienceMetricsDown: check that the console answers and that the scrape key still works.

Health and readiness checks#

Both need no sign-in and answer JSON.

/gardience/health#

Liveness. Always HTTP 200 while the Gardience service answers. It says whether the service is healthy, never why, because error text can name hosts and paths.

{"ok": true, "role": "primary", "generation": 0, "fenced": null}

• ok is false when a critical component reports a problem or the server is fenced.
• role and generation describe this server's place in a primary and standby pair. See High availability.

/gardience/ready#

Readiness, for load balancers, upgrades and gardience doctor. HTTP 200 only when this server is a working primary, connected to the Salt event stream, with no critical problem. Otherwise HTTP 503.

{"ready": false, "problems": ["events"]}

The critical components are:

Other problems (a failed scan, a SIEM that cannot be reached, and so on) show in the console and in gardience_errors, but do not make the server "not ready". A problem that has not happened again for 10 minutes stops counting, except audit, audit-chain and fenced, which last until fixed.

Signed-in users can see the full detail, including every component's message, at /gardience/api/health.

gardience doctor#

Run it on the server:

/opt/gardience/gardience doctor

It prints one line per check, marked PASS, WARN or FAIL, and exits with code 1 if anything failed, so you can run it from your monitoring system or from cron.

Related#

• Troubleshooting
• Support bundle and logs
• API keys and the REST API
• High availability