Sizing and scale results

What to give the Gardience server, and what we have and have not measured.

Recommended server#

Install with --fleet-size N so the Salt master is tuned for the fleet (worker threads, key cache, file and connection limits). Every row is an estimate until the real-agent numbers below are filled in for that size.

Linux host with Docker 24 or newer, amd64 or arm64. Disk grows mainly with the audit log and run history. Routine read-only work (scans every 5 minutes, drift checks every 30) is logged with a node count and a SHA-256 of the node list, about 1 KB per job whatever the fleet size: roughly 2 to 5 MB a day. Changes keep every node's name: about 25 bytes per node, so a patch run on 10,000 nodes adds about 250 KB. A 10,000-node fleet patched weekly grows the log by well under 1 GB a year. Segments are sealed at 50 MB; there is no automated retention.

What has been measured#

Gardience service, synthetic fleet (tools/tests/scale/run.sh 1000 5000 10000, 200,000-entry audit log):

Every console page and API read stays under 2 seconds at 10,000 nodes except the evidence export (under 5 s).

Real agents, lab (tools/tests/stress/stress.py): 23 real agents (Linux containers plus Windows 11 and Rocky 9 VMs). Full scan in 24 seconds; 15 new nodes joined in 5 seconds; 10 console runs at once all succeeded; agents reconnected within 27 seconds of a Salt master restart; Gardience used 52 MB of memory afterwards.

Real agents, larger runs (tools/tests/scale/agents.sh: separate, unmodified salt-minion daemons, each with its own keys and connection; every run keeps its own log under ~/.gardience-test-all/). Kept results:

• 30 agents (agents-30.log), including nodes rebuilt under old names: keys accepted, all answering, every node scored by Gardience 45 s after the scan started, one command answered by all 30 in 1 s, all 30 back after a Salt master restart in 104 s with no re-approval. PASS.
• 150 agents (agents-150-20261017-051914.log): all 150 keys accepted in 44 s and all answering in 130 s; the run then stalled because the laptop running it ran out of memory (about 45 MB per agent daemon on top of two VMs). Earlier 150, 200, 400 and 500 runs ended the same way. These show Salt accepting and reaching 150 agents; they say nothing either way about Gardience at that size.

Real agents, 1,002 nodes on AWS (2026-10-17, logs under ~/.gardience-test-all/aws/). Server: one m7g.xlarge (4 vCPU, 16 GB), installed from the release bundle. Agents, each an unmodified salt-minion daemon with its own keys and connection: 40 Ubuntu 24.04 VMs (5 on Salt 3006.28, 5 on 3007.15, 30 on 3008.3), 10 Windows Server 2022 VMs, and 952 more Ubuntu agents on 8 shared hosts. Results:

• Enrollment by token (autosign grains): all 1,002 accepted. A fleet scan scored every one of the 1,002 nodes (baseline and patch status, mixed Salt versions included); the full compliance data loads in 0.57 s (1.1 MB).
• Product end-to-end test against this server: 25 of 27 passed. The two that did not: the broad-ring patch of ~990 nodes was still running after 15 minutes (the 952 shared-host agents take turns on one package database per host, so this measures the test rig, not real machines), and an Ansible run was briefly held as "all nodes offline" while that patch kept the master busy (the run was still queued when the test ended; the offline check was fixed afterwards, see below).
• Salt master restart with 1,002 agents connected: every agent answering again 45 s later; console sessions survived the restart.
• Ansible playbooks through Gardience on the real VMs (controller mode, over the Salt connection): 40 Linux VMs across three Salt versions in 157 to 198 s; 10 Windows VMs with a vault secret in 71 s.
• Patch run on 10 Windows VMs: completed in 17 s, but each VM still listed its one pending item (a Defender definition update, which Microsoft republishes several times a day) afterwards, so this run did not show a change in patch state. The fresh Linux images had nothing to patch. Patching real machines at scale is not measured yet (see below). Security baseline on the 40 Linux VMs with a second person approving: the first run applied every control except the audit package (a Debian naming bug, then fixed) and failed on all 40; the run after the fix succeeded on all 40. Average score 43 before, 71 after the first run, 100 after the second.
• Console at 1,002 nodes: Minions page paged (20 per page by default); node-group page about 17 s; Groups page 0.3 s warm, 7.6 s cold.

Signed 1.2.4 on the same 1,002-agent fleet (2026-10-18, ~/.gardience-test-all/aws/change-campaign-0221.log and e2e-1.2.4-0224.log). The server was upgraded in place to the signed 1.2.4 bundle (doctor clean, 1,002 agents back).

• Fleet-wide change: an admin ran one runbook (set a grain) on every agent. Gardience reported the run finished in 72 s; asked directly on the Salt master afterwards, 1,002 of 1,002 agents held the new value.

• Product end-to-end test: 24 of 27 passed (sign-ins, governed permissions, scan of all 1,002, fix, runbook, pilot ring patch and verify, two-person approval, API, MCP, evidence packs, vault, audit chain). The three that did not, and why:

  • Broad-ring patch of 1,000 agents: 670 succeeded and 330 failed within the hour, and Gardience then re-applied the baseline on the failed nodes as the workflow says. The failures were the rig: one of the 8 shared hosts stopped answering (all 119 of its agents), and the 119 agents on each shared host run their package manager against one disk, so 45 failed with "No space left on device" and others on the package lock. Real machines each have their own disk and package database.
  • A health check by the approver: 873 of 992 answered; the 119 that did not were all on that dead host.
  • The Ansible controller run: it started on the online nodes and held the 119 offline ones, as designed, but when that host's agents came back the release overwrote the run's status with "released from queue" and never recorded a result, and a held playbook was released as a Salt call instead of through the controller. Both were bugs, fixed in 1.2.5 (a released part now gets its own result on the run and the run's final status counts every part).

Governed change on 4,977 real agents (2026-10-18, 1.2.5 plus the 1.3.0 service; ~/.gardience-test-all/aws/ governed-campaign-1510.log, scan5k-watch.log). Server: one m7g.2xlarge installed with --fleet-size 5000; agents: 4,977 unmodified salt-minion 3008.3 daemons, 625 per r7g.2xlarge host. The path a team or an AI agent uses, not an admin shortcut: a non-admin requester's operate key, narrowed to one runbook and one node group, previewed (4,977 nodes) and requested a hosts-file change; the request waited for approval; the requester's own key could not approve it; an admin's approve key did. Gardience ran it under its scoped automation identity and reported success after 166 s; asked directly on the Salt master afterwards, 4,977 of 4,977 agents held the change. The console answered in 134 ms (median) throughout. The job is in the audit log and the chain verifies. During the run salt-api's event stream dropped events and the job-cache reconciler added 186 jobs to the audit log that would otherwise be missing. A full fleet scan then listed all 4,977 nodes after 4 minutes, 4,960 fully scored and the rest filling in as their checks finished (625 agents per host share each host's CPU; real machines answer faster).

This run found and fixed four scale bugs before 1.3.0: AI tool results over 60,000 characters were cut mid-JSON (preview of 4,977 nodes); run status returned every node's result and no longer fit; scans gave up on answers the event stream had dropped instead of asking the master; and a first scan of a large fleet left late nodes invisible until the next scan.

Audit completeness under an event flood (2026-10-18, tools/tests/scale/audit-flood/). Job starts and returns fired on the master's own event bus at 4,000 to 4,600 events a second: 5 jobs of 20,000 nodes, then 4 jobs of 50,000 nodes (200,000 returns). The master's bus carried every event; salt-api's stream, which Gardience reads live, delivered as little as 1 of 5 job starts. With the job-cache reconciler every job was in the audit log within 49 s, no duplicates, audit chain intact, console answering in 5 ms.

Compatibility matrix on AWS (2026-10-18, ~/.gardience-test-all/aws/matrix-results*.json and matrix-run*.out). 42 VMs, one per operating system and Salt agent version: Windows Server 2016, 2019, 2022 and 2025; Ubuntu 20.04, 22.04 and 24.04; Debian 11 and 12; RHEL 9; Rocky 8 and 9; SUSE 15 SP6; Amazon Linux 2023; each with Salt 3006.28, 3007.15 and 3008.3. Every check through the product: health, scan, baseline applied, patch, reboot and reconnect, inventory. The final run (1.2.5 code) passed every check on all 42 (RHEL 9 VMs installed 275 real updates including a kernel, rebooted and came back; baseline 43 to 100). Earlier runs found six bugs, all fixed in 1.2.5: the SSH baseline check could not read settings on Ubuntu 24.04 (SSH started on demand) and the SSH fix did nothing on Rocky 8 (no drop-in folder); turning on the firewall cut the agent off for about 5 minutes on RHEL, Rocky, SUSE, Debian and Amazon Linux (the master's connection is now allowed before the firewall starts); a released held run lost its result and could look finished while patching continued (a reboot then interrupted a kernel install); Debian 11 could not be enrolled (Salt's bootstrap refuses end-of-life releases); and agent restarts and reboots Gardience schedules now wait for any running package manager (a restart mid-patch left two test VMs unbootable).

Real patching at scale: 1,020 independent Windows and Linux machines, governed (2026-10-25, Gardience 1.10.0 release candidate, tools/tests/aws/fleet-1000.sh and patch-campaign.py, evidence in docs/enterprise/evidence/fleet-1020-2026-10-25/). 1,000 Amazon Linux 2023 machines (t4g.nano) and 20 Windows Server 2022 machines (t3.medium), one server (c7g.2xlarge), installed from the bundle with the license. All 1,020 joined through one enrollment with no manual approval. Before, straight from the master: 1,020 answered, 64,000 updates pending (64 per Linux machine, including a kernel). A non-admin operator requested the patch; their own approval was refused; a second person approved it. Patch: 1,020 of 1,020 ok in 804 s, 0 failed, each machine's result in the audit chain. Reboot (requested and approved the same way): 1,020 of 1,020 answering again 237 s after the request. After: 1,020 fully patched, 0 updates pending, every Linux machine on the new kernel; the STIG/CIS-aligned baseline then passed on all 1,020. The signed evidence pack verifies (220 audit entries, 2,040 per-machine outcomes, all ok).

Earlier run: 600 independent machines, governed (2026-10-19, signed Gardience 1.3.0, tools/tests/aws/patch-fleet-launch.sh and patch-campaign.py, ~/.gardience-test-all/aws/patch600/). 600 separate EC2 machines (t4g.nano, 512 MB, each its own kernel, disk and package manager, spread over five availability zones, no public addresses) from a July 2026 Amazon Linux 2023 image, joined through an enrollment token. Before, straight from the master: 600 machines, 82,800 updates pending (138 each, including a kernel), all on kernel 6.1.176. A non-admin's operate key narrowed to the patch, reboot and health-check runbooks and one group requested the patch; its own approval was refused; a second person's approve key approved it. Patch: success after 168 s, 600 of 600 ok. Reboot (requested and approved the same way): success after 77 s, 600 of 600 restarted and reconnected. After, straight from the master: 600 of 600 fully patched, 0 updates pending, all 600 on kernel 6.1.188. The audit chain holds one job.results entry per change naming all 600 nodes as ok (requested by the non-admin), and verifies intact. Test-environment note: the machines had no internet, so the Salt package repository the agent installer had added was disabled on them before the run (updates came from Amazon's in-region repository), as on any private network.

Mixed fleet from the console: 634 machines, Windows and Linux (2026-10-19, the run behind the fleet images on gardience.com; evidence in docs/enterprise/evidence/fleet-634-2026-10-19/). 600 Linux machines (t4g.micro, 1 GB) and 34 Windows Server 2022 machines (t3.medium, 4 GB) on one server, about 70,000 updates pending. Patched from the console in about 15 minutes; the console's own counts, sampled each minute, reached 633 of 634 with 117 updates left, and the last machine was finished with the Fix action: 634 of 634 patched, 0 pending (console screenshot). Not kept: the server's audit log and run records (the server was deleted after the run); the 600-machine run above has them.

Real patching on machines with real updates waiting (2026-10-17, Gardience 1.2.2 candidate, ~/.gardience-test-all/aws/patch-real-.log). Ubuntu 24.04 VMs from a six-month-old image: 4 fresh VMs with 676 updates pending (169 each) were patched, rebooted and rescanned through Gardience in one round: patch 147 s, reboot and all back 43 s, then 4 of 4 fully patched with 0 updates pending. Windows Server 2022 VMs from a three-month-old image (4 VMs, 16 updates pending, including the October cumulative and .NET updates): every update installed through Gardience except one Windows Recovery Environment update (KB5131594), which Windows Update reported as installed successfully three times and still offered again after reboots, a known behavior of those updates when the recovery environment sits inside the OS partition; Gardience reports it as still pending, as Windows does. The cumulative update took about 33 minutes on a 2-vCPU VM. No kept log shows a Windows VM reaching fully patched: the 4 old-image VMs ended with 4 updates pending after 3 rounds (patch-real-windows-.log), and the baseline log records the 10 current-image VMs as 0 of 10 fully patched. These Windows runs overlapped a second test script by mistake, so their logs are not a clean single run.

Salt master restart with the whole fleet busy (every agent running a job, a full scan started, then the master restarted): all 1,018 agents (66 VMs, 952 shared-host agents) answering again within 30 s and through the next 9 minutes (restart-under-load-final.log). The agent hang seen once earlier (during a 990-node patch run that saturated the shared hosts' CPUs) did not recur, so the agents' self-recovery settings were not exercised by this run.

Problems this test found, all fixed. Tests cover the offline check, session handling and the hardening build check; the agent self-recovery and the Debian baseline fix were verified on the AWS machines only:

• Under load, a master restart left 136 of the 952 shared-host agents hung in Salt's own sign-in code (InvalidStateError in salt/crypt.py) and silent. Every agent Gardience configures now pings the master every 5 minutes and restarts itself through its service manager if that fails (ping_interval, auth_safemode).
• The offline check treated nodes that answered late under load as offline; it now keeps listening when most of a target is silent. Before this fix, the full end-to-end suite against the 1,000-agent fleet scored 25 passed, 2 failed (e2e-20261017-092100.log): a workflow was still running when the test ended, and an Ansible controller run was queued because 992 shared-host agents were judged offline. The suite was not re-run across the whole fleet after the fix; read operations, patching (on the real VMs) and restarts were. So sustained change work across ~1,000 agents at once is not yet shown end to end.
• The Linux baseline installed the audit package under its Red Hat name, failing on Ubuntu and Debian; the firewall fix now opens the ports a machine already serves before turning the firewall on.
• Built with Docker's legacy builder (no BuildKit), the Salt master image silently skipped part of its hardening; hardening is now a script that fails the build if anything does not apply, and gardience doctor checks it.
• Console sessions were lost on every master restart (now kept on disk) and were per browser tab (now shared).
• Real patching on Ubuntu: needrestart restarted the agent right after apt finished, so the patch result was lost and the run waited up to an hour; Gardience now suspends needrestart during the upgrade and restarts the agent itself afterwards, and any job whose agent restarts ends in seconds (unit-tested). A plain apt-get upgrade also kept back kernel updates, so no machine reached "fully patched"; Debian and Ubuntu now use dist-upgrade.
• Windows: Salt reports "Nothing to return" when a machine has no updates, which Gardience showed as a scan error, so a fully patched Windows machine could never show as patched. It now counts as patched once two scans agree (a brand-new machine's first search can come back empty before Windows Update is ready).
• An install made from a bundle in /tmp lost its trusted release key when /tmp was cleared; installs now keep the key in their own config folder.

What has not been measured yet#

• More than about 5,000 real agents on one master. Salt itself is widely run at tens of thousands of agents per master; Gardience's own layer is measured with 4,977 real agents (governed change, scan) and 50,000-node event floods (audit). Grow larger fleets in stages with the metrics in view.
• Patching thousands of independent machines in one run. Real patching is measured on 1,020 independent machines (above, limited by the test account's quota); the 4,977-agent runs share hosts, so they prove configuration changes and scans at that size, not independent patching.
• Windows per-task latency at scale. Windows agents answer more slowly than Linux (a few seconds per task, more on first patch scans). Fleet-wide Windows patch scans take minutes, not seconds, and that is normal.

Not supported yet#

• More than one Salt master (syndic or multi-master) under one Gardience.
• Two live consoles at once (active-active). A primary with a warm standby is supported, with failover by an operator or, with a witness, automatic (gardience autofailover); agents reconnect to the new primary on their own (see DR-RUNBOOK.md and site-docs/administration/high-availability.md).