HAProxy hot reload | Stack Harbor Knowledge Base

HAProxy hot reload swaps the running config with no dropped connections — old workers keep serving in-flight requests until their clients disconnect, while new workers immediately take new requests. This article explains the mechanism (master-worker, SIGUSR2, socket inheritance), the systemd unit semantics, what survives a reload and what does not, and the production checklist before every reload.

How to verify

For an existing install, the version flag and the process tree tell you whether you have master-worker mode wired correctly:

haproxy -vv | head -5
ps auxf | grep -E 'haproxy|wrapper' | head -10
systemctl cat haproxy | grep -E 'ExecReload|ExecStart|Type'
sudo journalctl -u haproxy -n 100 --no-pager | grep -iE 'reload|reexec|worker'

In master-worker mode (-Ws or the systemd-friendly equivalent), ps shows a single master with one or more workers as children. A reload produces a brief moment where the old workers are tagged [was: ...] while finishing their clients, then they exit.

What’s happening

HAProxy supports three reload modes:

Master-worker mode (-Ws) — recommended since 1.8, default since 2.0+. A small master process supervises one or more workers. On SIGUSR2, the master re-executes itself with the new config, forks new workers that inherit the listening sockets, and tells the old workers to stop accepting but keep serving in-flight requests. Old workers exit when their clients disconnect or hard-stop-after fires.
Daemon-only mode (-D) — older style. A new HAProxy is started with the new config and -sf <old_pids> to tell the new process which old ones to gracefully replace. The handover relies on SO_REUSEPORT to bind to the same port at the same time.
Foreground mode (-d) — debug only, no reload.

The systemd unit shipped by Ubuntu/Debian uses master-worker mode. systemctl reload haproxy sends SIGUSR2 to the master, which orchestrates everything. systemctl restart haproxy is the destructive path — it actually stops and restarts and drops connections.

What survives reload:

Listening sockets (passed from old worker to new worker via FD inheritance).
In-flight requests on existing connections (handled by old workers).
Stick tables, if peers is configured (table is pushed to a peer before reload).
SSL session tickets, if you have the ticket key file shared and persistent.

What does NOT survive a reload:

Stick tables without peers — gone.
In-memory counters and gauges (stats since startup resets per-worker).
TCP connections being slowly tarpitted — depending on hard-stop-after, they get force-closed.

hard-stop-after is the safety valve: it caps how long an old worker waits for its clients before being killed. Without it, a websocket or long-poll connection can keep an old worker alive for hours.

The procedure

Confirm master-worker mode is active. The systemd unit should use -W or -Ws:
```
systemctl cat haproxy | grep ExecStart
```
If the unit uses -D or no master flag, you are on the older reload path — modify the unit override (systemctl edit haproxy) to add -Ws and adapt the reload command.
Set hard-stop-after in global. A reasonable upper bound on how long old workers linger:
```
global
    hard-stop-after 30s
```
30 seconds is a typical safe value for HTTP traffic; raise to a few minutes for long-poll or WebSocket workloads, but understand that during a reload chain (multiple reloads in quick succession), workers stack up.
Configtest, then reload. Always in this order; the configtest catches typos that would crash the new workers on startup:
```
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
```
If configtest fails, the reload is a no-op — old workers keep running, the new config never loads. This is the safe failure mode.

Observe the reload effect. Worker count and the audit log:

ps auxf | grep haproxy
sudo journalctl -u haproxy -n 50 --no-pager
echo "show info" | sudo socat /run/haproxy/admin.sock - | grep -E 'Uptime|Process_num'

Process_num increments on each reload. Uptime resets when the new master takes over.

Persist stick tables across reloads. Configure peers in the global section and reference it from each stick-table you want to survive:
```
peers haproxy-peers
    peer haproxy1 10.0.10.5:1024
    peer haproxy2 10.0.10.6:1024

backend be_app
    stick-table type ip size 100k expire 30m peers haproxy-peers
    stick on src
```
See HAProxy peer protocol for the peer setup. On a single-instance HAProxy, the same peer name with one entry is acceptable — the table is pushed to its own future incarnation.
Frontend bind reuse. When you add a new bind line and reload, the master inherits the existing FDs and binds the new port. No SO_REUSEPORT magic needed — the master controls the listeners. Removing a bind is similarly graceful: old workers keep accepting on it until they drain, new workers ignore it.

Reload safety in automation. In CI/CD pipelines:

sudo cp new-haproxy.cfg /etc/haproxy/haproxy.cfg.tmp
sudo haproxy -c -f /etc/haproxy/haproxy.cfg.tmp \
  && sudo mv /etc/haproxy/haproxy.cfg.tmp /etc/haproxy/haproxy.cfg \
  && sudo systemctl reload haproxy

The atomic rename means that if the configtest fails, the live config is untouched. We bake this into the deployment script for every client.

Common pitfalls

systemctl restart haproxy looks similar to reload but drops connections. In a panicky on-call moment, the wrong command kills sessions. Train and document.
hard-stop-after not set + long-poll clients means workers from three reloads ago are still alive. Workers stack up; memory grows; eventually OOM.
Master-worker requires expose-fd listeners on the stats socket. Without it, the new master cannot inherit the listening FDs from the old, and the reload becomes a hard restart with a brief window of refused connections.
Stick tables without peers reset on every reload. If you rely on stick-tables for rate limiting, every reload gives abusers a fresh start.
Reloading rapidly (e.g., once a second during a config rollout) can leave many old worker processes around. pgrep -c haproxy should usually return 2-4; if it returns 20, you have stacking.

Stack Harbor scripts every reload behind configtest, atomic rename, and an audit log entry. Stick tables that matter are wired through peers. hard-stop-after is set deliberately per environment. For multi-instance HAProxy fleets, reloads are orchestrated one at a time with a brief wait so peer state propagates. This is part of how we operate Clustered Environments.