Workflows and approvals

A workflow chains runbooks into steps. Each step has a path to follow when it succeeds and a path to follow when it fails. An approval step pauses the workflow until a second person approves it.

To open workflows: choose Automation > Workflows and approvals. Runs and approvals is at the top. Below it, each workflow card shows its steps in order as a short summary; Show the steps lists each one top to bottom, with the approval step highlighted, what happens if a step fails, and the steps that run only when something fails listed after the others.

Launch a workflow#

1. Find the workflow card.
2. Choose Start a new run.
3. Follow the run under Runs and approvals, or on its own page (/gardience/run?id= run id, the link API and AI answers give). The step tracker shows each step as done, running, waiting for approval, failed or not started. The log is written as it happens: each machine's result the moment it arrives, and a progress line every minute while machines are still working. The log keeps your place when you scroll up to read; scrolled to the bottom, it follows the newest lines.

On fleets of more than 200 machines, the log names every machine that failed and counts the ones that succeeded; every machine's result is in the audit log and on the job's page.

Each runbook step is checked against your role, just like running the runbook yourself. If your role does not allow a step's Salt function on its nodes, the step fails and the workflow follows that step's failure path.

To run a workflow on a timetable, choose Schedule… on its card. See Schedules.

How steps run#

• A step runs only on the nodes that answer a quick check when the step starts. It never waits for machines that are off.
• Changes for nodes that are offline are held and run when they reconnect. Read-only steps, such as Health check, skip offline nodes.
• Locked nodes are skipped, and the log says so.
• Nodes whose change window is closed are held until it opens, and the step runs on the rest. If every target node is outside its window, the step fails.
• If no target node is online, the step runs nothing and counts as a success. The log says "nothing ran".

Held work is listed in Automation > Change control. See Held work.

A run ends as success when it reaches the end of a success path, and halted when it stops anywhere else. A run stops after 50 steps, so steps that point at each other cannot loop forever. If a webhook is set in Administration > Settings, Gardience posts a message when a workflow finishes.

Built-in workflows#

Patch Tuesday: pilot ring, then broad ring#

The patch steps restart the machines that need a restart to finish their updates, wait until they are back, and patch them again, up to three passes, so a Windows cumulative update that installs only after a restart is installed in the same run. Change windows apply to every pass. After the run, the patched machines are checked again for vulnerabilities and pending updates, so fixed CVEs show at once instead of after the next daily check.

Built-in workflows follow each release: an upgrade brings their newest steps. A workflow an admin saved under a built-in workflow's name is theirs and is kept as it is.

Patch, reboot, wait, verify#

Every step targets Ring0-Pilot. The reboot step succeeds only when each node has restarted and reconnected (up to 20 minutes). The wait step then repeats the health check every 20 seconds, for up to 15 minutes, until it succeeds.

Approval steps and the two-person rule#

When a run reaches an approval step:

1. The run's status changes to "awaiting approval", and Gardience posts a notification if a webhook is set.
2. Everyone who may decide sees a count next to Workflows and approvals in the menu, and a bar at the top of every page naming what waits and who started it. Review opens the run.

3. The approver chooses Approve or Reject. Only people who may decide see these buttons; the person who started the run sees "Waiting for an approver. You started it, so someone else decides."

   • Approve follows the step's success path.
   • Reject follows the step's failure path.

The two-person rule: the person who launched the run cannot approve or reject it. Someone else must decide. The decision is recorded on the run and in the audit log as workflow.approved or workflow.rejected.

Who can approve: people with the admin or approver role. They must have signed in through the Gardience sign-in page.

An approval step in a workflow waits until someone decides. If the Gardience service restarts while a run waits for approval, the run picks up again at the same approval step. A run that was in the middle of a runbook step when the service restarted is marked "interrupted (service restarted)" and does not resume. After a restore from a backup, runs that were waiting for approval expire; launch them again if they are still needed.

The same rule applies in other places:

• Applying the baseline or rebooting many nodes from the dashboard. See Patching Linux and Windows.
• Runbook requests from API keys that require approval. They appear here as a run named "(requested via API)".
• Clearing an audit log alarm. See Reports, evidence and audit.

Changes from the console that wait for a second person#

Settings > Approvals > Changes from the console wait for a second person (admins):

• Off (default): people whose role allows a change run it directly. Changes that cannot be undone still need a second person, as always.
• Changes by anyone who is not an admin: a runbook, a command, a workflow, a Fix button or any other change started from the console by someone who is not an admin waits for an approver first.
• Every change, admins too: the same for everyone.

A held change becomes a run named "name (waits for an approver)" whose first step is the approval; once someone other than the person who started it approves, the change runs exactly as asked, still within change windows and locks. Whoever started it is told "Waiting for an approver." Reading never waits. AI agents and API keys follow their own key's Require approval setting. Each held change is recorded in the audit log as change.held_for_approval.

Change tickets#

To tie changes to your change-management process, open Settings > Change tickets (admins):

• Mode: Off (default), Record when given, or Required for every change. Runbooks, workflows, the dashboard's Fix buttons and API runs that change machines ask for a ticket number; read-only work never does.
• Ticket number pattern: a regular expression. Blank accepts numbers like CHG0012345, INC123 or OPS-123.
• Check URL (optional): Gardience calls it before starting the change, with {ticket} replaced by the number, and goes ahead only if it answers 200 and mentions the ticket. For ServiceNow, for example https://acme.service-now.com/api/now/table/change_request?number={ticket}; for Jira, https://acme.atlassian.net/rest/api/3/issue/{ticket}.
• Credential for the check (optional, write-only): either a whole header value such as Basic dXNlcjpwYXNz or a bare token (sent as Bearer). It is sealed and bound to the check URL it was saved with: change the URL and it is discarded, so a saved credential can never be sent anywhere new. It is never shown again, and saving one is recorded in the audit log.

The ticket is kept on the run, written in the run's log, and recorded in the audit log as change.ticket with the run it belongs to, so evidence packs show which change carried out which ticket. API callers pass it as "ticket" in the request body.

Patch rings#

A patch ring is a node group used to roll out updates in stages. Gardience has two built-in rings:

• Ring0-Pilot: nodes marked as pilot machines.
• Ring1-Broad: every node that is not a pilot machine.

To move a node between rings (admins only):

1. Open Machines > Inventory.
2. In the Patch ring column, choose Make pilot or Move to broad.

The node must be online to change its ring. Gardience stores the ring on the node itself (the grain gardience_ring=pilot). The change is recorded in the audit log.

If Ring0-Pilot has no nodes, a step that targets it fails with "the pilot ring is empty".

Create a workflow#

Admins, and anyone whose role lets them run runbooks (operators), can create workflows. A workflow an admin saves is active at once. One that someone else creates, or changes, waits for review: its card says "Waiting for an approver to review it", it cannot run or be scheduled, and approvers see it in their count and the bar at the top of each page. An approver who did not build it chooses Approve (it can run) or Send back (its author changes it and saves again). Every save and every review is in the audit log (workflow.saved, workflow.approved, workflow.rejected_definition). Built-in workflows keep their names; pick another name for your own.

1. Go to New workflow at the bottom of the page.
2. Enter a Name and a Description. Saving a workflow with the same name as an existing one replaces it.

3. Choose + Add step for each step and fill in:

   • Kind: Runbook or Approval.
   • Step name: what the step is called in the diagram and the log.
   • Runbook: the runbook to run (Runbook steps only).
   • Inputs: the runbook's form values as key=value pairs separated by semicolons, for example nodegroup=Windows; service=W3SVC. Inputs you leave out use the runbook's defaults.
   • On success: next (the default), end, or a step number.
   • On failure: halt (the default), end, or a step number.
4. Choose Save workflow.

Step numbers start at 1. next on the last step means end.

Related#

• Runbooks
• Patching Linux and Windows
• Schedules
• Change windows and locks