Skip to content

Safely testing and reloading Nginx — the never-down deploy pattern

Why nginx -t before systemctl reload is non-negotiable, what reload actually does to worker processes, and the upgrade pattern that swaps the master binary without dropping connections.

nginx -t && systemctl reload nginx is the single most-typed phrase in our Nginx runbook — every config change goes through it. The test catches syntax errors before they bring the site down; the reload swaps the running configuration without dropping a single in-flight connection. This article covers the workflow, the difference between reload and restart, and the binary-upgrade pattern that swaps Nginx itself with zero downtime.

How to verify

Confirm current state, configuration validity, and what is loaded right now:

sudo nginx -t
sudo nginx -T | head -50
systemctl status nginx --no-pager
ps -ef | grep -E 'nginx: (master|worker)' | grep -v grep
sudo journalctl -u nginx -n 50 --no-pager

nginx -T (capital) dumps the full rendered config. The process tree shows one master and N workers — N is worker_processes (typically auto, which uses one per CPU core).

What’s happening

Nginx runs a master process (root) and worker processes (www-data). The master holds the parsed configuration and the listening sockets; workers accept connections and serve requests. A reload (SIGHUP) tells the master to re-parse the config, start new workers with the new config, and signal old workers to shut down after they finish their current requests. In-flight connections are not interrupted. Workers shutting down still serve their existing connections until they finish.

A restart (systemctl restart nginx) stops the master and starts a new one. The listening sockets briefly close — anything mid-connect gets refused, anything mid-response gets cut. We do not restart in production unless we are upgrading the binary or recovering from a wedged state.

The third move is binary upgrade (USR2 + WINCH + QUIT): the master forks a new master with a new binary, both bind the same sockets, traffic gracefully moves over. Useful when upgrading Nginx itself.

The procedure

Standard config change:

  1. Edit the relevant file under /etc/nginx/.
  2. Validate before touching the running config:
    sudo nginx -t
    Two lines must appear: ... syntax is ok and ... test is successful. If either reports an error, fix it before going further.
  3. Reload:
    sudo systemctl reload nginx
    Exit code 0 means the master accepted the new config. Watch the journal:
    sudo journalctl -u nginx -n 20 --no-pager
  4. Smoke-test from outside immediately: curl -sI https://example.com/. If it errors, the new config produced a problem the syntax test did not catch (e.g. a referenced cert file is missing).
  5. Roll back fast: keep the previous version of changed files in git. Reverting and re-running steps 2–4 takes about ten seconds.

Binary upgrade pattern (apt upgrade of the nginx package without restarting):

  1. The apt postinst on Debian/Ubuntu does the right thing automatically — it sends USR2 to the running master. If you are upgrading manually, do it explicitly:
    sudo kill -USR2 $(cat /run/nginx.pid)
    This forks a new master with the new binary. Both masters now share the listening sockets.
  2. Tell the old workers to stop accepting new connections:
    sudo kill -WINCH $(cat /run/nginx.pid.oldbin)
  3. Once the old workers have drained (you can see them in ps), shut down the old master:
    sudo kill -QUIT $(cat /run/nginx.pid.oldbin)
  4. If the new binary turns out broken, you have not yet QUIT the old master — kill -HUP it to bring its workers back, then kill -QUIT $(cat /run/nginx.pid) to drop the new one.

Common pitfalls

  • nginx -t runs as root but reads files as the user that nginx will read them as — except it does not always; nginx -t -c /path/to/file re-checks under the right user. Mostly a non-issue, but a permission problem on a cert file can slip past the test.
  • systemctl reload nginx versus nginx -s reload: both work. We prefer systemctl because it returns an honest exit code and integrates with systemd’s restart policy. nginx -s reload returns 0 even if the reload was a no-op (because the master is dead).
  • Reloading after editing a file under /etc/letsencrypt/ directly — certbot’s renewal hook also reloads; concurrent reloads can race. Use one path or the other.
  • Forgetting to test BEFORE editing. If two operators are working in the box, the other operator’s mistake might already be in the config and your reload will surface it as if you caused it.
  • A reload that takes >5s usually means workers are stuck on long-lived connections (WebSocket, streaming). worker_shutdown_timeout defaults to no limit; we set it to 60s in production so a slow client cannot pin a worker forever.

In the engagements we run, the reload workflow is enforced by tooling — every config change goes through git, CI runs nginx -t against a local container, and the actual reload is triggered by a deployment script that captures journal output and rolls back on health-check failure. Stack Harbor operates this as part of managed operations so the workflow is the same on every host.