Drift checks and event rules

Drift is a difference between how a node is set up and how it should be set up. Gardience finds drift in two ways:

• Drift checks compare each node with the Gardience desired state on a timer and show what differs, line by line.
• Event rules act on what the machines report, in seconds: restart a watched service as soon as it stops, warn when a disk fills past a level, and configure a machine the moment it first joins.

The desired state#

The desired state is a Salt state named desired. It sets:

Several of these match baseline controls. See Security baseline controls.

Drift checks#

A drift check runs the desired state in test mode on every node. Test mode reports what would change without changing anything.

• Gardience runs a check 1 minute after the service starts, then every 30 minutes.
• To check now, open Security > Drift and choose Check now. The page reloads after about 25 seconds.
• A check waits up to 10 minutes for nodes to answer. It stops waiting 90 seconds after the last answer arrives, so offline nodes do not hold it up.

Read the drift report#

The Current drift table has one row per node:

• Matches baseline: nothing differs.
• N drifted: N settings differ. Each one is listed with Salt's comment and, where there is one, a diff. Lines starting with - are what is on the node now; lines starting with + are what the desired state would put back.

A setting Salt could not evaluate is listed with "(could not evaluate)". A node that returned an error shows an "error" item.

When drift first appears on a node that had none, Gardience posts a notification if a webhook is set in Administration > Settings. Nodes removed from the fleet drop out of the report.

Correct drift on one node#

1. On Security > Drift, find the node.
2. Choose Correct now.

Gardience applies the desired state to that node for real. The correction runs only if:

• Your role allows state.apply on the node.
• The node is not locked.
• The node's change window is open. Outside the window the request is refused, not held; correct it when the window opens.

The correction is recorded in the audit log as drift.corrected, with the settings that changed. It also appears under Corrections history, which shows what had changed before it was put back, and who corrected it.

Auto-correct#

When Auto-correct is ticked on the Drift report page, Gardience corrects drift by itself after each drift check. Nodes that are locked or outside their change window are skipped until a later check. Automatic corrections are recorded with "auto-correct" as the user.

Auto-correct has three limits, so one bad baseline or one conflict never goes fleet-wide:

• A few machines first. It corrects 5 machines first. If any of them fails or does not answer, the others wait, Gardience records drift.autocorrect_held and sends an alert. The next check tries again.
• Something else keeps changing it. If the same item changes back 3 times within a day on one machine, another tool or a person is setting it (a GPO, Intune, a script). Fighting it would change the setting back and forth forever, so auto-correct stops on that machine (drift.autocorrect_fight), sends an alert, and the Drift report says why. Decide which setting is right and correct the machine yourself with Correct now; auto-correct then takes it on again.
• Machines it never touches. Under How auto-correct protects you, list machines auto-correct must leave alone, for example a server whose owner manages it by hand. People can still correct them.

Only admins can turn auto-correct on or off and change its list. The change is recorded in the audit log as settings.changed.

Event rules#

Each machine's Salt agent reports events as they happen, over the connection it already keeps to the server, so Gardience reacts in seconds without polling and without connecting out to each machine. There are three kinds of rule:

Rules can be managed in the console, through the API with a configure key, or with Terraform. See Terraform.

Service rules#

A service rule says: if this service stops on these nodes, restart it.

To open event rules: choose Automation > Event rules.

Add or remove a rule#

Only Gardience admins can add or remove rules.

1. Under New rule, enter the service name, for example W3SVC. Names may use letters, digits, _, ., @ and -, up to 64 characters.
2. Pick Windows nodes or Linux nodes.
3. Choose Add rule.

Adding a rule for a service and operating system that already has one replaces it. To remove a rule, choose Remove. Both are recorded in the audit log (rule.saved and rule.deleted).

Rules are published to the agents each time a rule is added or removed. Gardience writes them to pillar as Salt service beacons and refreshes pillar on every node. A beacon is a check the agent runs on its own; here, the agent checks each watched service every 10 seconds and reports when its state changes.

Rules reach Windows nodes and Linux nodes that use systemd.

What happens when a watched service stops#

1. The node's agent reports that the service stopped.
2. The Salt master's drift reactor restarts the service on that node. A reactor is a Salt master rule that acts on events.
3. Gardience records the event in the audit log as drift.detected, with the message "service stopped; auto-restart dispatched".

The latest events appear under Recent rule firings on the Event rules page and under Drift auto-repair on the dashboard. Rule firings are recorded in the audit log only; they are not posted to the notification webhook.

The reactor acts only when all of these are true:

• The event comes from the node it names.
• The node name and service name are plain names.
• The service is one of the services published by your event rules. An event about any other service starts nothing, whoever sent it.

The restart goes to that one node only. Change windows do not delay these restarts. A locked node refuses them. See Change windows and locks.

When a service keeps stopping#

A restart cannot fix a service that crashes on every start (a bad configuration, a full disk, a missing file), and restarting it forever would hide the problem. So when a watched service stops 3 times within 30 minutes on one node, Gardience pauses its automatic restarts on that node:

1. It records drift.restart_paused in the audit log and sends an alert.
2. The service appears under Paused restarts on the Event rules page. Other nodes and other services are not affected.
3. Find the cause in the service's own log and fix it. Then start the service, or choose Start and resume on the Event rules page (admins).
4. As soon as the node's agent reports the service running again, restarts resume by themselves (drift.restart_resumed) and the count starts over. If it still cannot start, it stays paused.

The crash that reaches the limit may already have been restarted when the pause begins, so the service can be running for a moment while paused. That run does not end the pause: a report that it is running in the first 2 minutes ends it only after Start and resume. When it crashes again, it is not restarted.

The paused list is in srv/pillar/drift/paused.sls in the install folder, which the Salt master reads before every restart. Removing the service's rule also removes its pauses. If the Gardience service itself is down, the master keeps restarting services, using the last paused list.

Limits on audit entries#

Nodes send these events, so Gardience limits how many it writes to the audit log:

• At most one entry per node and service every 5 minutes. Reports in between are counted and noted in the next entry.
• At most 120 entries a minute across the fleet. Past that, Gardience writes one audit.rate_limited entry for the minute.

These limits apply to the audit log only. The reactor still restarts the service each time.

Disk rules#

A disk rule says: if this disk is more than N percent full on these nodes, warn. On Linux the disk is a mount point such as / or /var; on Windows a drive such as C:. The agents check every 5 minutes. When a disk is over its level:

1. Gardience records disk.nearly_full in the audit log with the node, the disk and how full it is.
2. It sends a notification if a webhook is set in Administration > Settings.

While the disk stays full, the agent keeps reporting it; Gardience writes and sends at most once an hour per node and disk, and counts the reports in between. One disk rule per operating system and disk: adding another for the same disk replaces it.

Clean up safe files#

Tick THEN clean up safe files on a disk rule and Gardience also frees space on that node when the disk passes its level. It cleans only that disk, removes only what is downloaded or rebuilt again on its own, and never touches logs (they are evidence) or anyone's files:

The result, with how much was freed and the free space before and after, is recorded as disk.cleaned (or disk.cleanup_failed) and sent as a notification. Cleanup runs at most once every 6 hours per node and disk: a disk that fills again that quickly needs a person. If none of these are on the full disk (a data disk, for example), or cleanup freed nothing, the notification says so and asks a person to make room. Locked nodes are skipped (disk.cleanup_skipped). Like service restarts, change windows do not delay it, because a full disk stops services.

Join rules#

A join rule says: when a machine joins the fleet for the first time, apply this state to it. The scope is any new machine, new Windows or Linux machines, or new machines in a group. The state is desired (the Gardience desired state above) or a saved state (see States).

What happens:

1. The machine connects for the first time and Gardience syncs its modules and records its groups.
2. Gardience checks each join rule. A locked machine is skipped and the skip is recorded (rule.join_skipped).
3. It applies the state as its automation identity and records rule.join_applied with the machines. Each machine's result is added to the audit log like any other change.

Join rules fire once per machine. A restart, an upgrade or a re-sync does nothing. Machines that were already in the fleet when Gardience 1.5.0 was installed never count as new. A machine removed from the fleet (its key deleted) counts as new if it joins again. A join rule may name a group that an open enrollment gives its machines, before the first one has joined.

Join rules pair with enrollments and first-boot scripts: a machine built by Terraform, a cloud template or a golden image joins and is configured within seconds of starting. See Enrollment tokens and Terraform.

Related#

• Security baseline controls
• Change windows and locks
• Reports, evidence and audit
• Patch and compliance dashboard
• Terraform