Skip to content

WHM DNS cluster sync failures — diagnosing the silent drift

A WHM DNS cluster that stops syncing rarely throws a visible error. Here is the diagnostic order — logs, API tokens, network — and how to force a resync.

A cluster was healthy for months. Today a customer reports their TXT change isn’t visible from one of the nameservers. Looking at SOA serials, the master is at 2026051907 and the slave is at 2026051502 — five days of drift. WHM’s dashboard shows the cluster as active. The cluster status lies because nothing failed loudly; pushes have been refused individually and the daemon kept going.

What’s happening

dnsadmin synchronizes zones by calling the peer’s API. Each call is per-zone, per-change. When a call fails (HTTP 401, network timeout, peer in maintenance), dnsadmin logs the failure and moves on — it does not retry the same change later. The next change to the same zone triggers a new push attempt; if that succeeds, you might think the cluster is healthy when in fact one change was lost.

The cluster dashboard in WHM checks that the peer answers a heartbeat. A peer that answers heartbeats but refuses pushes (expired API token, permission revoked) shows green on the dashboard but drifts on every change.

How to verify

Compare SOA serials across all cluster nodes for several zones:

for zone in example.com test.com other.com; do
  for ns in master.example.com slave1.example.com slave2.example.com; do
    echo -n "$ns/$zone: "
    dig @$ns +short SOA $zone | awk '{print $3}'
  done
done

If serials differ for any zone, that zone has drifted. Pull the recent push log on the master:

tail -200 /usr/local/cpanel/logs/dnsadmin/log | grep -i error
grep -i 'fail\|401\|403\|timeout' /usr/local/cpanel/logs/dnsadmin/log | tail -50

Common failure messages:

  • 401 Unauthorized — the API token on the master is expired or revoked at the peer.
  • Connection refused — the peer’s WHM isn’t listening, or a firewall added between nodes.
  • 403 Forbidden — the API token doesn’t have DNS permissions; check the token’s permission set.
  • SSL_connect errors — the peer’s WHM SSL cert is expired or doesn’t match the configured hostname.

What to do

  1. Confirm the API tokens are still valid. On each peer, WHM > Development > Manage API Tokens. If the token used by another node is missing or shows no recent activity, regenerate and reapply.
  2. Test the peer’s WHM endpoint from the master:
curl -s -k -H "Authorization: whm root:TOKEN" \
  "https://peer.example.com:2087/json-api/version?api.version=1" | head

Should return JSON with the WHM version. If it doesn’t, the token, the network, or the peer’s WHM is the cause.

  1. Force a full resync on every zone:
/scripts/dnscluster reload
for zone in $(whmapi1 dumpzone --output=jsonpretty | jq -r '.data.zone[].name' 2>/dev/null); do
  /scripts/dnscluster synczone $zone
done

This runs sequentially and can take a long time on large fleets. Run it during a quiet window.

  1. Check named on the peer for zone load errors:
ssh peer "tail -100 /var/log/messages | grep -i named"
ssh peer "named-checkconf"
ssh peer "ls -la /var/named/example.com.db"

If named-checkconf shows an error, the cluster pushed a zone but named refused to load it. Common cause: a record with invalid syntax that the master’s named tolerates but the peer’s version rejects.

  1. Add monitoring that compares SOA serials weekly and alerts on drift. The WHM cluster dashboard is not a substitute for this check.

If two cluster nodes are drifting beyond what a resync can fix (different zones present on each), the cleanest fix is to declare one node the source of truth, export every zone with whmapi1 dumpzone, blow away the peer’s /var/named/*.db, and import. Stack Harbor monitors cluster sync as part of cPanel server administration.