Skip to content

HAProxy multi-tenant stats interface with TLS and per-tenant auth on Ubuntu 22.04

Expose the HAProxy stats + Runtime API safely on a shared proxy fleet — TLS, scoped basic-auth and ACL-filtered output so each tenant only sees their backends.

You run a shared HAProxy fleet that fronts several customer environments. A platform engineer asks for read-only access to the stats page for their backend pool — fair request, you give them the URL and a password. Two months later you discover their bookmarked dashboard shows every other tenant’s pool too, because the stats page has no concept of scope; whoever loads it sees the whole config. The fix is not “hide the URL.” It’s a stats interface designed for multi-tenant exposure from the start: TLS in front, per-tenant credentials, ACLs that filter rows by tenant prefix, and a Runtime API socket restricted to operators with a real reason to mutate state.

What’s happening

HAProxy has three operator surfaces — the stats HTTP page, the Prometheus metrics endpoint, and the Runtime API (stats socket admin level) — and by default none of them are designed for shared consumption. The stats page is a single rendering of the entire configuration; basic auth controls who can load the page, not what they see. The Prometheus surface is similarly all-or-nothing. The Runtime API exposes commands that can disable servers, drain backends, and reload maps, which means even read-only consumers can introspect more than you want.

Multi-tenant exposure means three things at once. First, terminate the connection on TLS with a real certificate; HAProxy’s stats page on plain HTTP across a shared network is a credential leak waiting to happen. Second, scope visibility per credential — HAProxy 2.4+ supports stats ACLs that drop rows based on the authenticated user, so tenant A loads https://stats.example.com/ and sees only acme_* backends, tenant B sees only globex_*, and the SRE user sees everything. Third, separate read access from mutation: the HTTP stats page can be read-only by default, while the Runtime API socket is admin-level but reachable only via a Unix socket gated by group membership.

The Prometheus endpoint deserves its own line. Expose it on a separate frontend bound to loopback and let your scrape agent forward it over a mutually-authenticated channel to your central monitoring; do not expose /metrics on the same TLS endpoint as the dashboard with the same credentials, because metric labels leak per-backend information that bypasses your row filter.

How to verify

After deploying, test that each tenant credential sees only its own backends:

# Tenant A credential — should return only acme_* lines
curl -fsS -u tenant_acme:'SECRET_A' \
  "https://stats.example.com/;csv;norefresh" \
  | awk -F, 'NR==1 || $1 ~ /^acme_/ {print $1","$2}'

# Tenant B credential — should return only globex_*
curl -fsS -u tenant_globex:'SECRET_B' \
  "https://stats.example.com/;csv;norefresh" \
  | awk -F, 'NR==1 || $1 ~ /^globex_/ {print $1","$2}'

# SRE credential — should return everything
curl -fsS -u sre_ops:'SECRET_SRE' \
  "https://stats.example.com/;csv;norefresh" \
  | wc -l

Confirm the Runtime API socket is restricted:

# As a regular user (not in haproxy-ops group): should fail
sudo -u nobody -- bash -c 'echo "show info" | socat stdio /run/haproxy/admin.sock' 2>&1 | head

# As a member of haproxy-ops: should work
sudo -u sre_user -- bash -c 'echo "show info" | socat stdio /run/haproxy/admin.sock' | head -10

Confirm Prometheus is reachable only from the loopback (or the scrape host you whitelisted):

curl -fsS http://127.0.0.1:9405/metrics | head -20
# from a different host, should refuse:
curl -fsS http://<haproxy-public-ip>:9405/metrics 2>&1 || echo "blocked, good"

The fix

The installer below configures HAProxy 2.8 on Ubuntu 22.04 with: a public TLS stats frontend on :8443, per-tenant credentials and ACL row filtering, the Runtime API gated on a Unix socket owned by haproxy-ops, and a Prometheus endpoint on loopback only. It assumes you already have valid certificates for stats.example.com (or swap in Let’s Encrypt issuance via certbot — left out for brevity).

Save as /root/setup-haproxy-multi-tenant-stats.sh:

#!/usr/bin/env bash
set -euo pipefail

### Ubuntu 22.04 HAProxy multi-tenant stats + Runtime API installer
### Run as root. Edit the tenant table near the top before running.

if [[ "$(id -u)" -ne 0 ]]; then
  echo "Run as root."
  exit 1
fi

# === EDIT ===
NODE_IP="10.40.0.11"
STATS_HOSTNAME="stats.example.com"
STATS_CERT="/etc/haproxy/certs/stats.pem"     # full chain + key concatenated
HAPROXY_VERSION="2.8"                          # use 2.8 LTS PPA on 22.04
# Tenant table: short-id  password  backend-prefix
# Backends must follow the prefix convention (acme_api_bk, globex_web_bk, ...)
TENANTS=(
  "tenant_acme   CHANGE_ACME    acme_"
  "tenant_globex CHANGE_GLOBEX  globex_"
)
SRE_USER="sre_ops"
SRE_PASS="CHANGE_SRE"
PROM_LISTEN="127.0.0.1:9405"
OPS_GROUP="haproxy-ops"
# ============

echo
echo "========================================"
echo "HAProxy multi-tenant stats installer"
echo "========================================"
echo

echo "==> Updating base OS"
apt-get update
apt-get -y dist-upgrade
apt-get -y install curl wget jq lsof tcpdump net-tools dnsutils \
  ca-certificates gnupg apt-transport-https unattended-upgrades \
  fail2ban chrony socat software-properties-common

systemctl enable --now chrony

echo "==> Hardening SSH"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart ssh

cat > /etc/fail2ban/jail.d/sshd.local <<'FAIL2BAN'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
FAIL2BAN
systemctl enable --now fail2ban

dpkg-reconfigure -f noninteractive unattended-upgrades

echo "==> Kernel/sysctl tuning"
cat > /etc/sysctl.d/99-haproxy.conf <<'SYSCTL'
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_fin_timeout = 15
fs.file-max = 2097152
fs.nr_open = 2097152
SYSCTL
sysctl --system

cat > /etc/security/limits.d/99-haproxy.conf <<'LIMITS'
haproxy soft nofile 1048576
haproxy hard nofile 1048576
haproxy soft nproc 65535
haproxy hard nproc 65535
LIMITS

echo "==> Installing HAProxy ${HAPROXY_VERSION} from PPA"
add-apt-repository -y ppa:vbernat/haproxy-${HAPROXY_VERSION}
apt-get update
apt-get -y install haproxy=${HAPROXY_VERSION}.* hatop

echo "==> Creating ops group + socket dir"
getent group "${OPS_GROUP}" >/dev/null || groupadd --system "${OPS_GROUP}"
mkdir -p /run/haproxy /etc/haproxy/certs /var/lib/haproxy /var/log/haproxy
chown -R haproxy:haproxy /run/haproxy /var/lib/haproxy /var/log/haproxy
chmod 750 /run/haproxy

# Add operators that should be able to drive the Runtime API.
# Example: usermod -aG ${OPS_GROUP} alice
echo "Add admins to the '${OPS_GROUP}' group with: usermod -aG ${OPS_GROUP} <user>"

echo "==> Placeholder cert (replace ${STATS_CERT} with a real one before exposing)"
if [[ ! -s "${STATS_CERT}" ]]; then
  openssl req -x509 -nodes -newkey rsa:2048 -days 30 \
    -keyout /tmp/stats.key -out /tmp/stats.crt \
    -subj "/CN=${STATS_HOSTNAME}" >/dev/null 2>&1
  cat /tmp/stats.crt /tmp/stats.key > "${STATS_CERT}"
  rm -f /tmp/stats.crt /tmp/stats.key
  chown haproxy:haproxy "${STATS_CERT}"
  chmod 640 "${STATS_CERT}"
  echo "WARNING: placeholder self-signed cert at ${STATS_CERT}"
fi

echo "==> Generating userlist + ACL fragments"
USERLIST_LINES=""
ACL_LINES=""
HTTP_REQUEST_LINES=""
HAS_USERS=""

for entry in "${TENANTS[@]}"; do
  user="$(echo "$entry" | awk '{print $1}')"
  pass="$(echo "$entry" | awk '{print $2}')"
  prefix="$(echo "$entry" | awk '{print $3}')"
  USERLIST_LINES+="  user ${user} insecure-password ${pass}"$'\n'
  ACL_LINES+="  acl scope_${user} http_auth_group(stats_users) ${user}"$'\n'
  ACL_LINES+="  acl pfx_${user}_match url_reg -i (^|;)([^;]*;)*scope=${prefix} || url_reg -i ${prefix}"$'\n'
  HAS_USERS+="${user} "
done

USERLIST_LINES+="  user ${SRE_USER} insecure-password ${SRE_PASS}"$'\n'

echo "==> Writing haproxy.cfg"
cat > /etc/haproxy/haproxy.cfg <<HCFG
global
  log /dev/log local0
  log /dev/log local1 notice
  user haproxy
  group haproxy
  daemon
  maxconn 200000
  ulimit-n 1048576
  master-worker
  stats socket /run/haproxy/admin.sock mode 0660 level admin group ${OPS_GROUP}
  stats socket ipv4@127.0.0.1:9999 level admin
  stats timeout 30s
  tune.ssl.default-dh-param 2048
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option forwardfor
  option http-server-close
  timeout client 30s
  timeout server 30s
  timeout connect 5s
  timeout http-request 10s
  timeout http-keep-alive 4s
  timeout queue 30s
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

userlist stats_users
${USERLIST_LINES}
  group sre users ${SRE_USER}

# === Example tenant backends (replace with your real ones) ===
backend acme_api_bk
  server a1 10.40.1.10:8080 check inter 3s
  server a2 10.40.1.11:8080 check inter 3s

backend acme_web_bk
  server w1 10.40.1.20:8080 check inter 3s

backend globex_web_bk
  server g1 10.40.2.10:8080 check inter 3s
  server g2 10.40.2.11:8080 check inter 3s

# === Public TLS stats frontend ===
frontend stats_public
  bind ${NODE_IP}:8443 ssl crt ${STATS_CERT} alpn h2,http/1.1
  http-request set-log-level info
  default_backend stats_handler

  # Per-IP rate limit to deter credential brute force.
  stick-table type ip size 100k expire 10m store http_req_rate(60s)
  http-request track-sc0 src
  http-request deny if { sc_http_req_rate(0) gt 120 }

# Stats backend with row-level filtering driven by authenticated user.
backend stats_handler
  stats enable
  stats uri /
  stats refresh 10s
  stats show-modules
  stats show-legends
  stats realm "Stats login"
  stats auth-validation-period 5m
  acl AUTH_OK http_auth(stats_users)
  http-request auth realm "Stats login" unless AUTH_OK

  # SRE sees everything.
  acl is_sre http_auth_group(stats_users) sre

  # Per-tenant row filter: drop CSV rows that do not match the tenant's prefix.
  # The stats page itself respects 'scope=<regex>' query param, so we rewrite it.
  http-request set-uri %[url,regsub(.*?scope=[^&;]*,)] if !is_sre
  http-request set-query "%[query],scope=%[http_auth_user]_" if !is_sre METH_GET

  # Tighten the visible config: deny non-GET unless SRE (no enabling/disabling backends via UI).
  http-request deny unless METH_GET or is_sre

  # Hard read-only header.
  http-response set-header X-Stats-Mode "read-only"

# === Prometheus endpoint — loopback only, scrape from local agent ===
frontend prom_internal
  bind ${PROM_LISTEN}
  http-request use-service prometheus-exporter if { path /metrics }
  http-request deny

# === Internal admin frontend on 127.0.0.1 only ===
frontend admin_local
  bind 127.0.0.1:9000
  acl is_sre http_auth_group(stats_users) sre
  http-request auth realm "Admin" unless is_sre
  stats enable
  stats uri /
  stats refresh 5s
  stats show-modules
  stats admin if is_sre
HCFG

mkdir -p /etc/haproxy/errors
# Copy default error files if not present
if [[ ! -f /etc/haproxy/errors/400.http ]]; then
  cp /usr/share/haproxy/*.http /etc/haproxy/errors/ 2>/dev/null || true
fi

echo "==> Validating config"
haproxy -c -f /etc/haproxy/haproxy.cfg

echo "==> Allowing haproxy to bind privileged ports"
setcap 'cap_net_bind_service=+ep' /usr/sbin/haproxy

echo "==> Drop-in for tighter systemd unit"
mkdir -p /etc/systemd/system/haproxy.service.d
cat > /etc/systemd/system/haproxy.service.d/override.conf <<'OVERRIDE'
[Service]
LimitNOFILE=1048576
KillMode=mixed
TimeoutStopSec=60
ProtectSystem=strict
ReadWritePaths=/var/log/haproxy /run/haproxy /var/lib/haproxy
NoNewPrivileges=true
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
OVERRIDE

systemctl daemon-reload
systemctl enable --now haproxy

echo "==> Installing prometheus-haproxy alongside (built-in exporter is used; nothing else to install)"

echo "==> Waiting for HAProxy"
for i in {1..30}; do
  if ss -lnt | grep -q ":8443"; then break; fi
  sleep 1
done

echo
echo "========================================"
echo "DONE"
echo "========================================"
echo
echo "Listeners:"
ss -lntp | grep -E ':(8443|9000|9405)\b' || true
echo
echo "Tenant users provisioned: ${HAS_USERS}+ ${SRE_USER}"
echo
echo "Test:"
echo "  curl -k -u <tenant>:<password> https://${STATS_HOSTNAME}:8443/;csv"
echo "  echo 'show info' | socat stdio /run/haproxy/admin.sock   # as a haproxy-ops group member"
echo "  curl http://${PROM_LISTEN}/metrics                       # from loopback only"
echo
echo "Add operators to the runtime API:"
echo "  usermod -aG ${OPS_GROUP} <user>"

Make it executable and run on each HAProxy node:

chmod 700 /root/setup-haproxy-multi-tenant-stats.sh
/root/setup-haproxy-multi-tenant-stats.sh

The row-filter trick works because HAProxy’s stats page accepts a scope=<regex> query parameter that filters the rendered table. The http-request set-query line forces the regex to the tenant’s prefix on every non-SRE request, so even if a tenant crafts a URL with their own scope they cannot escape the prefix — the rewrite happens after auth and before render. SRE accounts bypass the rewrite via the is_sre ACL.

Three operational notes. The Runtime API is exposed on 127.0.0.1:9999 for in-host tooling and on a Unix socket at /run/haproxy/admin.sock with group ownership; add operators to haproxy-ops and they get admin level. Don’t expose the TCP socket on a non-loopback address. The Prometheus frontend is on loopback; configure your scrape agent (node_exporter sidecar, telegraf, Vector) to pull :9405/metrics and forward over TLS — never put /metrics on the public TLS bind. Rotate tenant credentials by editing the userlist stats_users block and reloading HAProxy with systemctl reload haproxy (no dropped connections).

When you’d rather not run this proxy plane in-house, our environment management team operates HAProxy fleets with per-tenant access, audit logging, and Sev-1 monitoring wired into on-call. Plan it with us.