Held work
Gardience holds a change instead of failing it when the target node cannot take it yet. Held work waits in a queue and runs later.
Work is held for one of two reasons:
- offline: the node did not answer a quick check. The work runs when the node reconnects.
- change window: the node is in a node group whose change window is closed. The work runs when the window opens. See Change windows and locks.
Read-only checks are never held.
See held work
Open Automation > Change control and go to Held and queued work. The table lists the 30 most recent items:
| Column | Meaning |
|---|---|
| Queued (UTC) | When the work was held. |
| Work | What it does, with the Salt function and the target nodes. |
| Why | offline or change window. |
| Requested by | The person (or schedule author) the work runs for. |
| Status | Where the item is now. See the status list below. |
Each held item is recorded in the audit log as queue.held.
Automatic release
Gardience checks the queue every 30 seconds, and straight away when a node's agent reconnects. An item is released when all of these are true:
- At least one of its target nodes is online.
- No change window is closed for the online target nodes.
- The person who requested it is still allowed to run that Salt function on those nodes.
When an item is released:
- It runs on the target nodes that are online. Nodes still offline get a new held item labelled "(remaining offline nodes)".
- The audit log records
queue.released, and Gardience posts a notification if a webhook is set. - If the item came from a runbook or workflow run, the run's log shows the release and the job number.
If the requester has lost the right to run it, the item is not run. Its status becomes "refused (requester lacks permission)", the audit log records queue.refused, and Gardience posts a notification.
Items are released oldest first. Work held for the same node runs one item after another, in the order it was held, so a reboot never lands in the middle of the patch held before it. Up to 10 items for different nodes are released at the same time.
Override: run now
Admins can release an item before its window opens.
- Choose Run now (override) on the item.
- Enter the reason it must run now. A reason is required.
An override skips the change window check. It still needs at least one target node online, and it still checks the requester's rights. The audit log records queue.override under the admin's name, with the reason.
Cancel
To cancel an item, choose Cancel. The person who requested it and admins can cancel. The status becomes "cancelled by" and the name of the person who cancelled it.
Statuses
| Status | Meaning |
|---|---|
| queued | Waiting to be released. |
| releasing | Being released right now. |
| released | Sent to the online target nodes. |
| refused (requester lacks permission) | Not run because the requester's role no longer allows it. |
| cancelled by name | Cancelled. |
| interrupted: Gardience restarted while releasing this | Gardience stopped in the middle of a release. See below. |
| restored: release by hand after checking it has not already run | Restored from a backup. See below. |
Interrupted items
If Gardience restarts while it is releasing an item, it does not send the item again on its own, because Salt may already have started the job. The item is marked "interrupted" and a warning appears on the console. Check the job history in Patching > Jobs or the audit log. If the job did not run, request the work again.
Items restored after a backup
When you restore Gardience from a backup with gardience restore, the queue goes back to what it was when the backup was taken. Some of those items may already have run on the server the backup came from. Running them again could, for example, reboot a node twice.
So, when Gardience starts after a restore:
- Every item that was waiting is set to "restored: release by hand after checking it has not already run". None of them is released automatically.
- The audit log records
queue.restoredwith the number of items, and Gardience posts a notification. - Runs that were waiting for approval expire. Launch them again if they are still needed.
To handle each restored item:
- Check the job history and the audit log for the same change on the nodes listed under Work.
- If it already ran, choose Cancel.
- If it did not run and is still needed, an admin chooses Run now (override) and gives a reason. The item goes back in the queue and is released at once, skipping the change window check. If no target node is online, the console says so and the item waits as a normal queued item until a node reconnects.
Limits
- Up to 5,000 items can wait at once. When the queue is full, new work is not held; it is refused with a message asking you to release or cancel some items first. Waiting items are never dropped to make room.
- Gardience keeps the 300 most recent finished items as history. Waiting items are never trimmed.