Blog · 2026-10-19 · Gardience Engineering

Salt master high availability: multi-master and failover

Salt supports multi-master and failover setups, but key sync, split brain, and stale data catch teams out. The options, their trade-offs, and a tested warm-standby design with automatic failover.

The Salt master is a single point of control for the fleet. If it goes down, nothing breaks on the machines (the minions keep running), but nothing can be changed, patched, or checked until it is back. For most teams that is tolerable for minutes, not for days. These are the ways to make it shorter.

Salt's built-in options#

Salt's agents can be told about more than one master:

• Multi-master (hot-hot): each minion connects to several masters at once and accepts jobs from any of them. Simple to describe, but every master needs the same keys, states, and pillar, and two people can run conflicting jobs from two masters at the same moment.
• Failover (master_type: failover): each minion connects to the first master that answers and moves to the next if it goes away. Only one master is in charge at a time, which is easier to reason about.

Both need the masters to share the master key pair and the accepted minion keys, or the minions will not trust the standby, and both leave it to you to keep states, pillar, and any console data in sync.

What goes wrong in practice#

• Keys drift. A machine approved on the primary is unknown on the standby, so it is rejected after failover.
• Stale data wins. An old primary comes back after a failover with older data, and some minions return to it.
• Split brain. The primary is cut off from the standby but not from all the minions; both think they are in charge.
• Nobody notices the standby has been broken for weeks until the day it is needed.

A tested design: warm standby with a witness#

Gardience ships high availability as a feature rather than a recipe:

• A warm standby holds the master keys and is kept current with the primary, but runs no Salt master until it is promoted, so minions only ever find one master in charge.
• Keys-only sync between full backups, so a machine approved a minute ago is trusted after failover, and a revoked one stays revoked.
• Automatic failover through a witness. The primary records a signed heartbeat at a witness; the standby promotes itself only when that heartbeat has stopped for long enough (within about three minutes), never while the primary is alive.
• Fencing. A cut-off or returning old primary sees the promotion at the witness and stops its own Salt master, so there is never a second master with the same keys.
• Agents reconnect without re-approval, and failback is a deliberate step.

This is exercised in every release by an automated HA drill: primary failure, promotion, reconnection, fencing, network split and failback, with automatic failover under the three-minute limit.

Questions#

Do minions stop working if the Salt master goes down?#

No. Minions keep running and keep their configuration. You cannot send new jobs or collect results until a master is back.

Should I use multi-master or failover?#

Failover is easier to operate safely, because only one master is in charge at a time. Multi-master adds the risk of conflicting jobs from two masters.

How do I keep minion keys in sync between masters?#

Both masters need the same master key pair and the same accepted minion keys. Sync accepted and revoked keys often, not only in nightly backups, so a failover does not reject new machines or trust revoked ones.