Skip to content

Deploying GlusterFS 3-node replicated volume on Ubuntu 22.04 with TLS

Stand up a 3-node GlusterFS replicated volume on Ubuntu 22.04 — XFS bricks, management TLS, encrypted client I/O, healing strategy, and a self-heal monitor.

You inherit a project where a single NFS server is in front of three application instances. The NFS box is a v7-class VM with one disk and one cable. A maintenance reboot drops it for fourteen minutes and the entire fleet of web nodes turns into a hot kernel waiting on a stat() that will never return. The post-mortem asks why a stateful tier with no replica was in the critical path of a stateless application — and the answer is that the team needed a shared filesystem and nobody costed the next step, which is a real distributed filesystem with synchronous replication and a client driver that survives a node loss. GlusterFS in replica 3 is one of the cheapest answers to that question — once it’s wired up correctly with TLS, healthy bricks, and a quorum that won’t split.

What’s happening

A replica 3 Gluster volume writes every file to all three bricks synchronously through the client. The client (FUSE or libgfapi) is the active participant: it pushes I/O to all replicas, computes the answers, and asks each brick for self-heal metadata on open. There is no master node. If one node disappears, the remaining two hold quorum, the client keeps writing, and self-heal replays the missed operations when the node returns. The pieces that matter operationally are five: an XFS-formatted brick on a dedicated block device (never the OS disk, never ext4 — XFS for inode density), glusterd listening on the management plane (TCP 24007), the brick processes themselves (TCP 49152+), TLS on both the management and the I/O channels so a flat network doesn’t expose your data, and cluster.quorum-type=auto with cluster.server-quorum-type=server so the cluster refuses to write when two nodes can’t agree. Skip the quorum settings and a partition lets one brick keep writing alone; skip TLS and any host on the management VLAN can join the trusted pool with a forged hostname. Both happen.

How to verify

Volume info from any node:

gluster volume info gv0
# Volume Name: gv0
# Type: Replicate
# Volume ID: 8c9f...
# Status: Started
# Number of Bricks: 1 x 3 = 3
# Transport-type: tcp
# Bricks:
# Brick1: gluster-01:/data/brick1/gv0
# Brick2: gluster-02:/data/brick1/gv0
# Brick3: gluster-03:/data/brick1/gv0

Peer status — all three should be Peer in Cluster (Connected):

gluster peer status

Heal status during steady-state should be empty. Anything non-zero is a backlog you need to investigate:

gluster volume heal gv0 info | grep -E "Number of entries" | sort -u
# Number of entries: 0

End-to-end I/O from a client mount:

mount -t glusterfs -o backupvolfile-server=gluster-02,acl gluster-01:/gv0 /mnt/gv0
dd if=/dev/urandom of=/mnt/gv0/test.bin bs=1M count=64 oflag=direct status=progress
md5sum /mnt/gv0/test.bin
# unmount, kill one node, remount via the survivor, md5sum should still match

If gluster volume heal gv0 info split-brain returns any file, do not proceed to production — the cluster has a divergent file that needs operator resolution.

The fix

Save the script as /root/setup-gluster-node.sh and run it on each of the three nodes after editing NODE_NAME, NODE_IP, and the PEER_IPS and PEER_NAMES arrays at the top. It does a full Ubuntu 22.04 install: base hardening (UFW, fail2ban, unattended-upgrades, SSH key-only), kernel/sysctl tuning for a storage workload (vm.dirty_*, fs.file-max, net.core.somaxconn), Gluster 11 from the PPA, XFS on a dedicated /dev/sdb formatted with -i size=512 for extended attributes, TLS material generated locally and distributed via the orchestration node, glusterd enabled with transport.socket.ssl-enabled=on, peer-probe orchestration from node 1 only, a replica 3 volume with quorum auto, and a systemd timer that runs gluster volume heal info every five minutes and surfaces a non-zero queue to syslog so monitoring can pick it up.

#!/usr/bin/env bash
set -euo pipefail

### Ubuntu 22.04 GlusterFS replica 3 installer
### Run on each of the 3 nodes; the first node bootstraps the trusted pool

if [[ "$(id -u)" -ne 0 ]]; then
  echo "Run as root."
  exit 1
fi

# === EDIT PER NODE ===
NODE_NAME="gluster-01"            # gluster-01 / gluster-02 / gluster-03
NODE_IP="10.20.0.11"              # this node's private IP
NODE_INDEX=0                      # 0 / 1 / 2 — index in PEER_*; node 0 bootstraps
PEER_IPS=("10.20.0.11" "10.20.0.12" "10.20.0.13")
PEER_NAMES=("gluster-01" "gluster-02" "gluster-03")
BRICK_DEV="/dev/sdb"              # dedicated block device, not the OS disk
BRICK_MOUNT="/data/brick1"
VOLUME_NAME="gv0"
TLS_COMMON_NAME="${NODE_NAME}"
# =====================

echo
echo "========================================"
echo "GlusterFS node ${NODE_NAME} (${NODE_IP})"
echo "========================================"
echo

echo "==> Hosts file"
for i in 0 1 2; do
  if ! grep -q "${PEER_NAMES[$i]}" /etc/hosts; then
    echo "${PEER_IPS[$i]} ${PEER_NAMES[$i]}" >> /etc/hosts
  fi
done
hostnamectl set-hostname "${NODE_NAME}"

echo "==> Base packages"
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y install software-properties-common ca-certificates curl gnupg \
  xfsprogs lvm2 ufw fail2ban unattended-upgrades chrony jq socat \
  openssl tcpdump bind9-dnsutils net-tools

systemctl enable --now chrony

echo "==> Sysctl tuning for storage workload"
cat > /etc/sysctl.d/99-gluster.conf <<'SYSCTL'
vm.swappiness = 10
vm.dirty_background_ratio = 5
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 500
fs.file-max = 2097152
fs.aio-max-nr = 1048576
net.core.somaxconn = 4096
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_syncookies = 1
SYSCTL
sysctl --system

cat > /etc/security/limits.d/99-gluster.conf <<'LIMITS'
root soft nofile 200000
root hard nofile 200000
* soft nofile 200000
* hard nofile 200000
LIMITS

echo "==> SSH hardening"
sed -i 's|^#\?PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^#\?PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart ssh

echo "==> fail2ban + unattended-upgrades"
cat > /etc/fail2ban/jail.d/sshd.local <<'F2B'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
F2B
systemctl enable --now fail2ban
dpkg-reconfigure -fnoninteractive unattended-upgrades

echo "==> UFW (allow gluster mgmt + brick range from peers only)"
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
for ip in "${PEER_IPS[@]}"; do
  ufw allow from "${ip}" to any port 24007 proto tcp
  ufw allow from "${ip}" to any port 24008 proto tcp
  ufw allow from "${ip}" to any port 49152:49251 proto tcp
done
# Client subnet — adjust to your network
ufw allow from 10.20.0.0/16 to any port 24007 proto tcp
ufw allow from 10.20.0.0/16 to any port 49152:49251 proto tcp
ufw --force enable

echo "==> Preparing brick on ${BRICK_DEV}"
if [[ ! -b "${BRICK_DEV}" ]]; then
  echo "ERROR: ${BRICK_DEV} not present. Edit BRICK_DEV and rerun."
  exit 1
fi

if ! blkid "${BRICK_DEV}" | grep -q xfs; then
  wipefs -a "${BRICK_DEV}"
  mkfs.xfs -i size=512 -n size=8192 -L "brick1-${NODE_NAME}" "${BRICK_DEV}"
fi

mkdir -p "${BRICK_MOUNT}"
BRICK_UUID=$(blkid -s UUID -o value "${BRICK_DEV}")
if ! grep -q "${BRICK_UUID}" /etc/fstab; then
  echo "UUID=${BRICK_UUID} ${BRICK_MOUNT} xfs defaults,noatime,nodiratime,inode64 0 2" >> /etc/fstab
fi
mount -a
mkdir -p "${BRICK_MOUNT}/${VOLUME_NAME}"

echo "==> Adding GlusterFS PPA"
add-apt-repository -y ppa:gluster/glusterfs-11
apt-get update
apt-get -y install glusterfs-server glusterfs-client

echo "==> Generating TLS material for management + I/O"
mkdir -p /etc/ssl/glusterfs
cd /etc/ssl/glusterfs

if [[ ! -f glusterfs.key ]]; then
  openssl genrsa -out glusterfs.key 2048
  openssl req -new -x509 -key glusterfs.key -subj "/CN=${TLS_COMMON_NAME}" \
    -days 3650 -out glusterfs.pem
fi

# Each node's .pem must end up in the combined CA bundle on every node.
# In a real deployment, push glusterfs.pem to an out-of-band trust file
# and concat them into glusterfs.ca on every node. For the first install:
cp glusterfs.pem "/etc/ssl/glusterfs/${NODE_NAME}.pem"

# Build glusterfs.ca by concatenating each peer's certificate file
# after you've copied them across out-of-band (scp, ansible, etc.)
if [[ -f "/etc/ssl/glusterfs/${PEER_NAMES[0]}.pem" && \
      -f "/etc/ssl/glusterfs/${PEER_NAMES[1]}.pem" && \
      -f "/etc/ssl/glusterfs/${PEER_NAMES[2]}.pem" ]]; then
  cat "/etc/ssl/glusterfs/${PEER_NAMES[0]}.pem" \
      "/etc/ssl/glusterfs/${PEER_NAMES[1]}.pem" \
      "/etc/ssl/glusterfs/${PEER_NAMES[2]}.pem" > /etc/ssl/glusterfs/glusterfs.ca
fi

chmod 600 glusterfs.key
chmod 644 glusterfs.pem glusterfs.ca 2>/dev/null || true

echo "==> Enable management-plane TLS (file marker)"
touch /var/lib/glusterd/secure-access

echo "==> Starting glusterd"
systemctl enable --now glusterd

echo "==> Waiting for glusterd to listen on 24007"
for i in {1..30}; do
  ss -lnt | grep -q ':24007 ' && break
  sleep 1
done

if [[ "${NODE_INDEX}" -eq 0 ]]; then
  echo "==> Bootstrap node — probing peers"
  for i in 1 2; do
    until gluster peer probe "${PEER_NAMES[$i]}" 2>/dev/null; do
      echo "Waiting for ${PEER_NAMES[$i]} to be reachable..."
      sleep 5
    done
  done

  echo "==> Waiting for all peers to report Connected"
  for i in {1..60}; do
    if [[ "$(gluster peer status | grep -c 'State: Peer in Cluster (Connected)')" -eq 2 ]]; then
      break
    fi
    sleep 2
  done

  if ! gluster volume info "${VOLUME_NAME}" >/dev/null 2>&1; then
    echo "==> Creating replica 3 volume"
    gluster volume create "${VOLUME_NAME}" replica 3 transport tcp \
      "${PEER_NAMES[0]}:${BRICK_MOUNT}/${VOLUME_NAME}" \
      "${PEER_NAMES[1]}:${BRICK_MOUNT}/${VOLUME_NAME}" \
      "${PEER_NAMES[2]}:${BRICK_MOUNT}/${VOLUME_NAME}"

    echo "==> Volume options"
    gluster volume set "${VOLUME_NAME}" auth.ssl-allow "${PEER_NAMES[0]},${PEER_NAMES[1]},${PEER_NAMES[2]}"
    gluster volume set "${VOLUME_NAME}" client.ssl on
    gluster volume set "${VOLUME_NAME}" server.ssl on
    gluster volume set "${VOLUME_NAME}" cluster.quorum-type auto
    gluster volume set "${VOLUME_NAME}" cluster.server-quorum-type server
    gluster volume set "${VOLUME_NAME}" cluster.server-quorum-ratio 51
    gluster volume set "${VOLUME_NAME}" performance.client-io-threads on
    gluster volume set "${VOLUME_NAME}" performance.io-thread-count 16
    gluster volume set "${VOLUME_NAME}" performance.cache-size 256MB
    gluster volume set "${VOLUME_NAME}" network.ping-timeout 10
    gluster volume set "${VOLUME_NAME}" cluster.self-heal-daemon on
    gluster volume set "${VOLUME_NAME}" cluster.heal-timeout 600
    gluster volume set "${VOLUME_NAME}" cluster.entry-self-heal on
    gluster volume set "${VOLUME_NAME}" cluster.metadata-self-heal on
    gluster volume set "${VOLUME_NAME}" cluster.data-self-heal on
    gluster volume set "${VOLUME_NAME}" features.shard on
    gluster volume set "${VOLUME_NAME}" features.shard-block-size 64MB
    gluster volume set "${VOLUME_NAME}" diagnostics.brick-log-level WARNING
    gluster volume set "${VOLUME_NAME}" diagnostics.client-log-level WARNING

    gluster volume start "${VOLUME_NAME}"
  fi
else
  echo "==> Non-bootstrap node — waiting for node 0 to add us"
fi

echo "==> Self-heal monitor (systemd timer)"
cat > /usr/local/sbin/gluster-heal-monitor.sh <<'HEAL'
#!/usr/bin/env bash
set -euo pipefail
VOL=$(/usr/sbin/gluster volume list | head -n1)
[[ -z "${VOL}" ]] && exit 0
PENDING=$(/usr/sbin/gluster volume heal "${VOL}" info \
  | awk '/Number of entries/ {s+=$NF} END {print s+0}')
if [[ "${PENDING}" -gt 0 ]]; then
  logger -t gluster-heal -p daemon.warning \
    "volume=${VOL} self_heal_pending=${PENDING}"
fi
HEAL
chmod 755 /usr/local/sbin/gluster-heal-monitor.sh

cat > /etc/systemd/system/gluster-heal.service <<'UNIT'
[Unit]
Description=Gluster self-heal queue monitor
After=glusterd.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/gluster-heal-monitor.sh
UNIT

cat > /etc/systemd/system/gluster-heal.timer <<'TIMER'
[Unit]
Description=Run gluster-heal-monitor every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Unit=gluster-heal.service

[Install]
WantedBy=timers.target
TIMER

systemctl daemon-reload
systemctl enable --now gluster-heal.timer

echo
echo "========================================"
echo "DONE on ${NODE_NAME}"
echo "========================================"
echo
echo "Listeners:"
ss -lntp | grep -E ':(24007|24008|49152|49153|49154)\b' || true
echo
gluster peer status || true
echo
gluster volume info "${VOLUME_NAME}" 2>/dev/null || echo "Volume not yet created (only node 0 creates it)"
echo
echo "On a client (any node or external):"
echo "  apt-get install -y glusterfs-client"
echo "  mount -t glusterfs -o backupvolfile-server=${PEER_NAMES[1]},acl ${PEER_NAMES[0]}:/${VOLUME_NAME} /mnt/${VOLUME_NAME}"
echo
echo "Heal status:"
gluster volume heal "${VOLUME_NAME}" info 2>/dev/null | head -20 || true

Run it on every node in order — gluster-01 first because it bootstraps the trusted pool. After all three are up, do a failure drill before production traffic: write a known file from a client mount, hard-reboot one node, confirm the client mount still serves the file, bring the node back, and watch gluster volume heal gv0 info drain back to zero. The first time a client encounters a “split-brain” file, the cluster will refuse the read — keep gluster volume heal gv0 info split-brain in your monitoring set so the next operator on call doesn’t discover it during an outage.

Sharding (features.shard on) is on by default in the script because large-file workloads (VM images, container layers, big media) heal faster when chunked. Turn it off only if you specifically need the whole-file semantics — once it’s on with data on the volume, you cannot turn it off without rewriting every file.

For a distributed filesystem you don’t have to operate yourself, our team runs Gluster and CephFS clusters as part of environment management with brick-health and heal-queue alerting on the monitoring plane — book a scoping call if you’d like us to design and operate the storage tier.