Blog · 2026-10-23 · Gardience Engineering

Salt minion "No master could be resolved": the failover fix

With master_type failover, a Salt minion that can't resolve any master at the moment its master goes away logs "No master could be resolved" and never reconnects until restarted. What we found in Salt's code, how to reproduce it, and a small watchdog for Linux and Windows that fixes it.

We spent most of today on a bug that kept failing one step of our high availability test about one run in three. The setup was the standard Salt way to do it: each minion lists two masters, a primary and a standby, with master_type: failover. Kill the primary, promote the standby, and the minions should move over.

Most runs they did. Every so often one minion never came back. No error after that, no retries, nothing in the log for 18 minutes. Restarting the minion fixed it instantly.

If you've got a failover minion that's gone quiet and the last thing in its log is "No master could be resolved," this is probably what happened to you.

What the log looks like#

[INFO    ] Connection to master primary.example lost
[INFO    ] Trying to tune in to next master from master-list
[INFO    ] Moving possibly failed master primary.example to the end of the list of masters
[ERROR   ] DNS lookup or connection check of 'standby.example' failed.
[ERROR   ] DNS lookup or connection check of 'primary.example' failed.
[ERROR   ] No master could be resolved
[INFO    ] Starting up the Salt Minion

And then nothing. The minion process is still running. It just isn't connected to anything and isn't trying to be.

Why it happens#

We went through salt/minion.py in Salt 3008.3 to see what the minion does when it loses its master. Short version:

1. Every master_alive_interval seconds a scheduled job checks the master. When it finds it gone, the minion deletes that alive-check job and tries the next master on its list.
2. Before trying to connect, it resolves every master in the list once. If none of them resolves at that exact moment, it raises "No master could be resolved" right away. That happens before the retry loop that master_tries: -1 controls, so "retry forever" never kicks in.
3. The code that called it catches the error and moves on. The alive check is already gone, and the handler that would try again only runs if the minion thinks it's still connected. It doesn't anymore. Nothing ever schedules another attempt.

In our tests the minion then restarted itself and hung in its own startup, where even its scheduler stopped running. A minion started fresh by hand connected right away every time.

When does "none of them resolves" happen in real life? When both masters' names fail at the same instant: a DNS outage during the failover, both servers restarting at once, or cloud and container setups where a stopped machine's name disappears from DNS. It's a narrow window, which is why it only bites sometimes. It's also exactly the moment you need failover to work.

There are related reports on GitHub: #57660 (an unresolvable failover master quietly falls back to 127.0.0.1) and #66534 (only the last master keeps getting retried when names don't resolve).

Reproducing it#

We could only get this reliably by forcing the condition: point the minion at two names in /etc/hosts, remove both entries, then kill the primary's master process. Within 30 seconds the minion logs "No master could be resolved." Put the entries back and start the standby's master, and the minion still never connects. We watched it for 18 minutes before calling it.

The fix#

You can't fix this inside Salt's minion config: once the minion is stuck, nothing inside it runs. So the fix has to come from outside, the operating system restarting the minion when it's been without a master for a while.

Linux#

A minion always holds one TCP connection open to its master's publish port (4505 by default). This script checks for it and restarts salt-minion after 4 minutes without one. Save it as /usr/local/sbin/salt-agent-watchdog and chmod 755 it:

#!/bin/sh
# Every minute: if this Salt minion has had no connection to a master for 4 minutes, restart it.
S=/var/lib/salt-agent-watchdog; F=$S/since; mkdir -p $S
PORT=$(grep -hs '^publish_port:' /etc/salt/minion /etc/salt/minion.d/*.conf | tail -1 | awk '{print $2}')
HEX=$(printf '%04X' "${PORT:-4505}")
# state 01 in /proc/net/tcp is ESTABLISHED; the remote address column ends in the port, in hex
if awk -v p="$HEX" 'FNR>1 {split($3, a, ":"); if (a[2] == p && $4 == "01") f = 1} END {exit !f}' /proc/net/tcp /proc/net/tcp6 2>/dev/null; then
  rm -f "$F"; exit 0
fi
now=$(date +%s)
[ -f "$F" ] || { echo "$now" > "$F"; exit 0; }
[ $((now - $(cat "$F"))) -ge 240 ] || exit 0
echo "$now" > "$F"
logger -t salt-agent-watchdog "no master for 4 minutes: restarting salt-minion"
systemctl restart salt-minion

Run it every minute with a systemd timer. /etc/systemd/system/salt-agent-watchdog.service:

[Unit]
Description=Restart the Salt minion after 4 minutes without a master

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/salt-agent-watchdog

/etc/systemd/system/salt-agent-watchdog.timer:

[Unit]
Description=Check the Salt minion's master connection every minute

[Timer]
OnBootSec=2min
OnUnitActiveSec=60s
AccuracySec=10s

[Install]
WantedBy=timers.target

Then systemctl daemon-reload && systemctl enable --now salt-agent-watchdog.timer.

If both masters are down for a long time, it restarts the minion every 4 minutes until one comes back. That's on purpose. A fresh start is the one thing that reliably gets it reconnected.

Windows#

Same idea in PowerShell. Save as C:\ProgramData\salt-agent-watchdog.ps1:

$conf = 'C:\ProgramData\Salt Project\Salt\conf'
$port = 4505
$line = Get-ChildItem "$conf\minion", "$conf\minion.d\*.conf" -ErrorAction SilentlyContinue | Select-String '^publish_port:' | Select-Object -Last 1
if ($line) { $port = [int](($line.Line -split ':')[1].Trim()) }
$f = 'C:\ProgramData\salt-agent-watchdog-since'
if (Get-NetTCPConnection -RemotePort $port -State Established -ErrorAction SilentlyContinue) { Remove-Item $f -ErrorAction SilentlyContinue; exit 0 }
$now = [int][double]::Parse((Get-Date -UFormat %s))
if (-not (Test-Path $f)) { Set-Content $f $now; exit 0 }
if ($now - [int](Get-Content $f) -lt 240) { exit 0 }
Set-Content $f $now
Restart-Service salt-minion -Force

And run it every minute as SYSTEM:

$a = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\ProgramData\salt-agent-watchdog.ps1'
$t = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(1) -RepetitionInterval (New-TimeSpan -Minutes 1)
$p = New-ScheduledTaskPrincipal -UserId 'SYSTEM' -RunLevel Highest
Register-ScheduledTask -TaskName 'Salt agent watchdog' -Action $a -Trigger $t -Principal $p -Force

Make the names harder to lose#

Separately, you can make the trigger less likely. If your masters are listed by DNS name, either put them in each minion's hosts file or list them by IP address, so a DNS blip during a failover can't make both names fail at once. We'd still run the watchdog. It also covers anything else that leaves a minion hung.

What it looked like with the fix#

On Linux we ran the same forced failure with and without it. On Windows we tested the watchdog itself by stopping the minion service and waiting:

In Gardience#

We build Gardience on top of Salt, and it supports a primary and a standby server, so this one was ours to fix. As of Gardience 1.9.1 it installs this watchdog automatically on every agent that has a standby (the systemd timer on Linux, the scheduled task on Windows), and our release check runs this exact failure with and without the watchdog on every release.

Quick answers#

What does "No master could be resolved" mean in Salt?#

The minion tried to look up every master in its list and none resolved at that moment. With master_type: failover this can leave the minion disconnected for good until it's restarted.

Doesn't master_tries: -1 make the minion retry forever?#

Not in this case. The "No master could be resolved" error is raised before the retry loop that master_tries controls.

How do I fix a stuck Salt minion right now?#

Restart it: systemctl restart salt-minion on Linux, Restart-Service salt-minion on Windows. Then add the watchdog above so it doesn't need you next time.

Does this affect a single-master setup?#

We only saw it with master_type: failover and a list of masters. A minion with one master takes a different path when it reconnects.