You run a shared HAProxy fleet that fronts several customer environments. A platform engineer asks for read-only access to the stats page for their backend pool — fair request, you give them the URL and a password. Two months later you discover their bookmarked dashboard shows every other tenant’s pool too, because the stats page has no concept of scope; whoever loads it sees the whole config. The fix is not “hide the URL.” It’s a stats interface designed for multi-tenant exposure from the start: TLS in front, per-tenant credentials, ACLs that filter rows by tenant prefix, and a Runtime API socket restricted to operators with a real reason to mutate state.
What’s happening
HAProxy has three operator surfaces — the stats HTTP page, the Prometheus metrics endpoint, and the Runtime API (stats socket admin level) — and by default none of them are designed for shared consumption. The stats page is a single rendering of the entire configuration; basic auth controls who can load the page, not what they see. The Prometheus surface is similarly all-or-nothing. The Runtime API exposes commands that can disable servers, drain backends, and reload maps, which means even read-only consumers can introspect more than you want.
Multi-tenant exposure means three things at once. First, terminate the connection on TLS with a real certificate; HAProxy’s stats page on plain HTTP across a shared network is a credential leak waiting to happen. Second, scope visibility per credential — HAProxy 2.4+ supports stats ACLs that drop rows based on the authenticated user, so tenant A loads https://stats.example.com/ and sees only acme_* backends, tenant B sees only globex_*, and the SRE user sees everything. Third, separate read access from mutation: the HTTP stats page can be read-only by default, while the Runtime API socket is admin-level but reachable only via a Unix socket gated by group membership.
The Prometheus endpoint deserves its own line. Expose it on a separate frontend bound to loopback and let your scrape agent forward it over a mutually-authenticated channel to your central monitoring; do not expose /metrics on the same TLS endpoint as the dashboard with the same credentials, because metric labels leak per-backend information that bypasses your row filter.
How to verify
After deploying, test that each tenant credential sees only its own backends:
# Tenant A credential — should return only acme_* lines
curl -fsS -u tenant_acme:'SECRET_A' \
"https://stats.example.com/;csv;norefresh" \
| awk -F, 'NR==1 || $1 ~ /^acme_/ {print $1","$2}'
# Tenant B credential — should return only globex_*
curl -fsS -u tenant_globex:'SECRET_B' \
"https://stats.example.com/;csv;norefresh" \
| awk -F, 'NR==1 || $1 ~ /^globex_/ {print $1","$2}'
# SRE credential — should return everything
curl -fsS -u sre_ops:'SECRET_SRE' \
"https://stats.example.com/;csv;norefresh" \
| wc -l
Confirm the Runtime API socket is restricted:
# As a regular user (not in haproxy-ops group): should fail
sudo -u nobody -- bash -c 'echo "show info" | socat stdio /run/haproxy/admin.sock' 2>&1 | head
# As a member of haproxy-ops: should work
sudo -u sre_user -- bash -c 'echo "show info" | socat stdio /run/haproxy/admin.sock' | head -10
Confirm Prometheus is reachable only from the loopback (or the scrape host you whitelisted):
curl -fsS http://127.0.0.1:9405/metrics | head -20
# from a different host, should refuse:
curl -fsS http://<haproxy-public-ip>:9405/metrics 2>&1 || echo "blocked, good"
The fix
The installer below configures HAProxy 2.8 on Ubuntu 22.04 with: a public TLS stats frontend on :8443, per-tenant credentials and ACL row filtering, the Runtime API gated on a Unix socket owned by haproxy-ops, and a Prometheus endpoint on loopback only. It assumes you already have valid certificates for stats.example.com (or swap in Let’s Encrypt issuance via certbot — left out for brevity).
Save as /root/setup-haproxy-multi-tenant-stats.sh:
#!/usr/bin/env bash
set -euo pipefail
### Ubuntu 22.04 HAProxy multi-tenant stats + Runtime API installer
### Run as root. Edit the tenant table near the top before running.
if [[ "$(id -u)" -ne 0 ]]; then
echo "Run as root."
exit 1
fi
# === EDIT ===
NODE_IP="10.40.0.11"
STATS_HOSTNAME="stats.example.com"
STATS_CERT="/etc/haproxy/certs/stats.pem" # full chain + key concatenated
HAPROXY_VERSION="2.8" # use 2.8 LTS PPA on 22.04
# Tenant table: short-id password backend-prefix
# Backends must follow the prefix convention (acme_api_bk, globex_web_bk, ...)
TENANTS=(
"tenant_acme CHANGE_ACME acme_"
"tenant_globex CHANGE_GLOBEX globex_"
)
SRE_USER="sre_ops"
SRE_PASS="CHANGE_SRE"
PROM_LISTEN="127.0.0.1:9405"
OPS_GROUP="haproxy-ops"
# ============
echo
echo "========================================"
echo "HAProxy multi-tenant stats installer"
echo "========================================"
echo
echo "==> Updating base OS"
apt-get update
apt-get -y dist-upgrade
apt-get -y install curl wget jq lsof tcpdump net-tools dnsutils \
ca-certificates gnupg apt-transport-https unattended-upgrades \
fail2ban chrony socat software-properties-common
systemctl enable --now chrony
echo "==> Hardening SSH"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart ssh
cat > /etc/fail2ban/jail.d/sshd.local <<'FAIL2BAN'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
FAIL2BAN
systemctl enable --now fail2ban
dpkg-reconfigure -f noninteractive unattended-upgrades
echo "==> Kernel/sysctl tuning"
cat > /etc/sysctl.d/99-haproxy.conf <<'SYSCTL'
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_fin_timeout = 15
fs.file-max = 2097152
fs.nr_open = 2097152
SYSCTL
sysctl --system
cat > /etc/security/limits.d/99-haproxy.conf <<'LIMITS'
haproxy soft nofile 1048576
haproxy hard nofile 1048576
haproxy soft nproc 65535
haproxy hard nproc 65535
LIMITS
echo "==> Installing HAProxy ${HAPROXY_VERSION} from PPA"
add-apt-repository -y ppa:vbernat/haproxy-${HAPROXY_VERSION}
apt-get update
apt-get -y install haproxy=${HAPROXY_VERSION}.* hatop
echo "==> Creating ops group + socket dir"
getent group "${OPS_GROUP}" >/dev/null || groupadd --system "${OPS_GROUP}"
mkdir -p /run/haproxy /etc/haproxy/certs /var/lib/haproxy /var/log/haproxy
chown -R haproxy:haproxy /run/haproxy /var/lib/haproxy /var/log/haproxy
chmod 750 /run/haproxy
# Add operators that should be able to drive the Runtime API.
# Example: usermod -aG ${OPS_GROUP} alice
echo "Add admins to the '${OPS_GROUP}' group with: usermod -aG ${OPS_GROUP} <user>"
echo "==> Placeholder cert (replace ${STATS_CERT} with a real one before exposing)"
if [[ ! -s "${STATS_CERT}" ]]; then
openssl req -x509 -nodes -newkey rsa:2048 -days 30 \
-keyout /tmp/stats.key -out /tmp/stats.crt \
-subj "/CN=${STATS_HOSTNAME}" >/dev/null 2>&1
cat /tmp/stats.crt /tmp/stats.key > "${STATS_CERT}"
rm -f /tmp/stats.crt /tmp/stats.key
chown haproxy:haproxy "${STATS_CERT}"
chmod 640 "${STATS_CERT}"
echo "WARNING: placeholder self-signed cert at ${STATS_CERT}"
fi
echo "==> Generating userlist + ACL fragments"
USERLIST_LINES=""
ACL_LINES=""
HTTP_REQUEST_LINES=""
HAS_USERS=""
for entry in "${TENANTS[@]}"; do
user="$(echo "$entry" | awk '{print $1}')"
pass="$(echo "$entry" | awk '{print $2}')"
prefix="$(echo "$entry" | awk '{print $3}')"
USERLIST_LINES+=" user ${user} insecure-password ${pass}"$'\n'
ACL_LINES+=" acl scope_${user} http_auth_group(stats_users) ${user}"$'\n'
ACL_LINES+=" acl pfx_${user}_match url_reg -i (^|;)([^;]*;)*scope=${prefix} || url_reg -i ${prefix}"$'\n'
HAS_USERS+="${user} "
done
USERLIST_LINES+=" user ${SRE_USER} insecure-password ${SRE_PASS}"$'\n'
echo "==> Writing haproxy.cfg"
cat > /etc/haproxy/haproxy.cfg <<HCFG
global
log /dev/log local0
log /dev/log local1 notice
user haproxy
group haproxy
daemon
maxconn 200000
ulimit-n 1048576
master-worker
stats socket /run/haproxy/admin.sock mode 0660 level admin group ${OPS_GROUP}
stats socket ipv4@127.0.0.1:9999 level admin
stats timeout 30s
tune.ssl.default-dh-param 2048
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
timeout client 30s
timeout server 30s
timeout connect 5s
timeout http-request 10s
timeout http-keep-alive 4s
timeout queue 30s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
userlist stats_users
${USERLIST_LINES}
group sre users ${SRE_USER}
# === Example tenant backends (replace with your real ones) ===
backend acme_api_bk
server a1 10.40.1.10:8080 check inter 3s
server a2 10.40.1.11:8080 check inter 3s
backend acme_web_bk
server w1 10.40.1.20:8080 check inter 3s
backend globex_web_bk
server g1 10.40.2.10:8080 check inter 3s
server g2 10.40.2.11:8080 check inter 3s
# === Public TLS stats frontend ===
frontend stats_public
bind ${NODE_IP}:8443 ssl crt ${STATS_CERT} alpn h2,http/1.1
http-request set-log-level info
default_backend stats_handler
# Per-IP rate limit to deter credential brute force.
stick-table type ip size 100k expire 10m store http_req_rate(60s)
http-request track-sc0 src
http-request deny if { sc_http_req_rate(0) gt 120 }
# Stats backend with row-level filtering driven by authenticated user.
backend stats_handler
stats enable
stats uri /
stats refresh 10s
stats show-modules
stats show-legends
stats realm "Stats login"
stats auth-validation-period 5m
acl AUTH_OK http_auth(stats_users)
http-request auth realm "Stats login" unless AUTH_OK
# SRE sees everything.
acl is_sre http_auth_group(stats_users) sre
# Per-tenant row filter: drop CSV rows that do not match the tenant's prefix.
# The stats page itself respects 'scope=<regex>' query param, so we rewrite it.
http-request set-uri %[url,regsub(.*?scope=[^&;]*,)] if !is_sre
http-request set-query "%[query],scope=%[http_auth_user]_" if !is_sre METH_GET
# Tighten the visible config: deny non-GET unless SRE (no enabling/disabling backends via UI).
http-request deny unless METH_GET or is_sre
# Hard read-only header.
http-response set-header X-Stats-Mode "read-only"
# === Prometheus endpoint — loopback only, scrape from local agent ===
frontend prom_internal
bind ${PROM_LISTEN}
http-request use-service prometheus-exporter if { path /metrics }
http-request deny
# === Internal admin frontend on 127.0.0.1 only ===
frontend admin_local
bind 127.0.0.1:9000
acl is_sre http_auth_group(stats_users) sre
http-request auth realm "Admin" unless is_sre
stats enable
stats uri /
stats refresh 5s
stats show-modules
stats admin if is_sre
HCFG
mkdir -p /etc/haproxy/errors
# Copy default error files if not present
if [[ ! -f /etc/haproxy/errors/400.http ]]; then
cp /usr/share/haproxy/*.http /etc/haproxy/errors/ 2>/dev/null || true
fi
echo "==> Validating config"
haproxy -c -f /etc/haproxy/haproxy.cfg
echo "==> Allowing haproxy to bind privileged ports"
setcap 'cap_net_bind_service=+ep' /usr/sbin/haproxy
echo "==> Drop-in for tighter systemd unit"
mkdir -p /etc/systemd/system/haproxy.service.d
cat > /etc/systemd/system/haproxy.service.d/override.conf <<'OVERRIDE'
[Service]
LimitNOFILE=1048576
KillMode=mixed
TimeoutStopSec=60
ProtectSystem=strict
ReadWritePaths=/var/log/haproxy /run/haproxy /var/lib/haproxy
NoNewPrivileges=true
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
OVERRIDE
systemctl daemon-reload
systemctl enable --now haproxy
echo "==> Installing prometheus-haproxy alongside (built-in exporter is used; nothing else to install)"
echo "==> Waiting for HAProxy"
for i in {1..30}; do
if ss -lnt | grep -q ":8443"; then break; fi
sleep 1
done
echo
echo "========================================"
echo "DONE"
echo "========================================"
echo
echo "Listeners:"
ss -lntp | grep -E ':(8443|9000|9405)\b' || true
echo
echo "Tenant users provisioned: ${HAS_USERS}+ ${SRE_USER}"
echo
echo "Test:"
echo " curl -k -u <tenant>:<password> https://${STATS_HOSTNAME}:8443/;csv"
echo " echo 'show info' | socat stdio /run/haproxy/admin.sock # as a haproxy-ops group member"
echo " curl http://${PROM_LISTEN}/metrics # from loopback only"
echo
echo "Add operators to the runtime API:"
echo " usermod -aG ${OPS_GROUP} <user>"
Make it executable and run on each HAProxy node:
chmod 700 /root/setup-haproxy-multi-tenant-stats.sh
/root/setup-haproxy-multi-tenant-stats.sh
The row-filter trick works because HAProxy’s stats page accepts a scope=<regex> query parameter that filters the rendered table. The http-request set-query line forces the regex to the tenant’s prefix on every non-SRE request, so even if a tenant crafts a URL with their own scope they cannot escape the prefix — the rewrite happens after auth and before render. SRE accounts bypass the rewrite via the is_sre ACL.
Three operational notes. The Runtime API is exposed on 127.0.0.1:9999 for in-host tooling and on a Unix socket at /run/haproxy/admin.sock with group ownership; add operators to haproxy-ops and they get admin level. Don’t expose the TCP socket on a non-loopback address. The Prometheus frontend is on loopback; configure your scrape agent (node_exporter sidecar, telegraf, Vector) to pull :9405/metrics and forward over TLS — never put /metrics on the public TLS bind. Rotate tenant credentials by editing the userlist stats_users block and reloading HAProxy with systemctl reload haproxy (no dropped connections).
When you’d rather not run this proxy plane in-house, our environment management team operates HAProxy fleets with per-tenant access, audit logging, and Sev-1 monitoring wired into on-call. Plan it with us.