You inherit a project where a single NFS server is in front of three application instances. The NFS box is a v7-class VM with one disk and one cable. A maintenance reboot drops it for fourteen minutes and the entire fleet of web nodes turns into a hot kernel waiting on a stat() that will never return. The post-mortem asks why a stateful tier with no replica was in the critical path of a stateless application — and the answer is that the team needed a shared filesystem and nobody costed the next step, which is a real distributed filesystem with synchronous replication and a client driver that survives a node loss. GlusterFS in replica 3 is one of the cheapest answers to that question — once it’s wired up correctly with TLS, healthy bricks, and a quorum that won’t split.
What’s happening
A replica 3 Gluster volume writes every file to all three bricks synchronously through the client. The client (FUSE or libgfapi) is the active participant: it pushes I/O to all replicas, computes the answers, and asks each brick for self-heal metadata on open. There is no master node. If one node disappears, the remaining two hold quorum, the client keeps writing, and self-heal replays the missed operations when the node returns. The pieces that matter operationally are five: an XFS-formatted brick on a dedicated block device (never the OS disk, never ext4 — XFS for inode density), glusterd listening on the management plane (TCP 24007), the brick processes themselves (TCP 49152+), TLS on both the management and the I/O channels so a flat network doesn’t expose your data, and cluster.quorum-type=auto with cluster.server-quorum-type=server so the cluster refuses to write when two nodes can’t agree. Skip the quorum settings and a partition lets one brick keep writing alone; skip TLS and any host on the management VLAN can join the trusted pool with a forged hostname. Both happen.
How to verify
Volume info from any node:
gluster volume info gv0
# Volume Name: gv0
# Type: Replicate
# Volume ID: 8c9f...
# Status: Started
# Number of Bricks: 1 x 3 = 3
# Transport-type: tcp
# Bricks:
# Brick1: gluster-01:/data/brick1/gv0
# Brick2: gluster-02:/data/brick1/gv0
# Brick3: gluster-03:/data/brick1/gv0
Peer status — all three should be Peer in Cluster (Connected):
gluster peer status
Heal status during steady-state should be empty. Anything non-zero is a backlog you need to investigate:
gluster volume heal gv0 info | grep -E "Number of entries" | sort -u
# Number of entries: 0
End-to-end I/O from a client mount:
mount -t glusterfs -o backupvolfile-server=gluster-02,acl gluster-01:/gv0 /mnt/gv0
dd if=/dev/urandom of=/mnt/gv0/test.bin bs=1M count=64 oflag=direct status=progress
md5sum /mnt/gv0/test.bin
# unmount, kill one node, remount via the survivor, md5sum should still match
If gluster volume heal gv0 info split-brain returns any file, do not proceed to production — the cluster has a divergent file that needs operator resolution.
The fix
Save the script as /root/setup-gluster-node.sh and run it on each of the three nodes after editing NODE_NAME, NODE_IP, and the PEER_IPS and PEER_NAMES arrays at the top. It does a full Ubuntu 22.04 install: base hardening (UFW, fail2ban, unattended-upgrades, SSH key-only), kernel/sysctl tuning for a storage workload (vm.dirty_*, fs.file-max, net.core.somaxconn), Gluster 11 from the PPA, XFS on a dedicated /dev/sdb formatted with -i size=512 for extended attributes, TLS material generated locally and distributed via the orchestration node, glusterd enabled with transport.socket.ssl-enabled=on, peer-probe orchestration from node 1 only, a replica 3 volume with quorum auto, and a systemd timer that runs gluster volume heal info every five minutes and surfaces a non-zero queue to syslog so monitoring can pick it up.
#!/usr/bin/env bash
set -euo pipefail
### Ubuntu 22.04 GlusterFS replica 3 installer
### Run on each of the 3 nodes; the first node bootstraps the trusted pool
if [[ "$(id -u)" -ne 0 ]]; then
echo "Run as root."
exit 1
fi
# === EDIT PER NODE ===
NODE_NAME="gluster-01" # gluster-01 / gluster-02 / gluster-03
NODE_IP="10.20.0.11" # this node's private IP
NODE_INDEX=0 # 0 / 1 / 2 — index in PEER_*; node 0 bootstraps
PEER_IPS=("10.20.0.11" "10.20.0.12" "10.20.0.13")
PEER_NAMES=("gluster-01" "gluster-02" "gluster-03")
BRICK_DEV="/dev/sdb" # dedicated block device, not the OS disk
BRICK_MOUNT="/data/brick1"
VOLUME_NAME="gv0"
TLS_COMMON_NAME="${NODE_NAME}"
# =====================
echo
echo "========================================"
echo "GlusterFS node ${NODE_NAME} (${NODE_IP})"
echo "========================================"
echo
echo "==> Hosts file"
for i in 0 1 2; do
if ! grep -q "${PEER_NAMES[$i]}" /etc/hosts; then
echo "${PEER_IPS[$i]} ${PEER_NAMES[$i]}" >> /etc/hosts
fi
done
hostnamectl set-hostname "${NODE_NAME}"
echo "==> Base packages"
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y install software-properties-common ca-certificates curl gnupg \
xfsprogs lvm2 ufw fail2ban unattended-upgrades chrony jq socat \
openssl tcpdump bind9-dnsutils net-tools
systemctl enable --now chrony
echo "==> Sysctl tuning for storage workload"
cat > /etc/sysctl.d/99-gluster.conf <<'SYSCTL'
vm.swappiness = 10
vm.dirty_background_ratio = 5
vm.dirty_ratio = 15
vm.dirty_expire_centisecs = 6000
vm.dirty_writeback_centisecs = 500
fs.file-max = 2097152
fs.aio-max-nr = 1048576
net.core.somaxconn = 4096
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_syncookies = 1
SYSCTL
sysctl --system
cat > /etc/security/limits.d/99-gluster.conf <<'LIMITS'
root soft nofile 200000
root hard nofile 200000
* soft nofile 200000
* hard nofile 200000
LIMITS
echo "==> SSH hardening"
sed -i 's|^#\?PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^#\?PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart ssh
echo "==> fail2ban + unattended-upgrades"
cat > /etc/fail2ban/jail.d/sshd.local <<'F2B'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
F2B
systemctl enable --now fail2ban
dpkg-reconfigure -fnoninteractive unattended-upgrades
echo "==> UFW (allow gluster mgmt + brick range from peers only)"
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
for ip in "${PEER_IPS[@]}"; do
ufw allow from "${ip}" to any port 24007 proto tcp
ufw allow from "${ip}" to any port 24008 proto tcp
ufw allow from "${ip}" to any port 49152:49251 proto tcp
done
# Client subnet — adjust to your network
ufw allow from 10.20.0.0/16 to any port 24007 proto tcp
ufw allow from 10.20.0.0/16 to any port 49152:49251 proto tcp
ufw --force enable
echo "==> Preparing brick on ${BRICK_DEV}"
if [[ ! -b "${BRICK_DEV}" ]]; then
echo "ERROR: ${BRICK_DEV} not present. Edit BRICK_DEV and rerun."
exit 1
fi
if ! blkid "${BRICK_DEV}" | grep -q xfs; then
wipefs -a "${BRICK_DEV}"
mkfs.xfs -i size=512 -n size=8192 -L "brick1-${NODE_NAME}" "${BRICK_DEV}"
fi
mkdir -p "${BRICK_MOUNT}"
BRICK_UUID=$(blkid -s UUID -o value "${BRICK_DEV}")
if ! grep -q "${BRICK_UUID}" /etc/fstab; then
echo "UUID=${BRICK_UUID} ${BRICK_MOUNT} xfs defaults,noatime,nodiratime,inode64 0 2" >> /etc/fstab
fi
mount -a
mkdir -p "${BRICK_MOUNT}/${VOLUME_NAME}"
echo "==> Adding GlusterFS PPA"
add-apt-repository -y ppa:gluster/glusterfs-11
apt-get update
apt-get -y install glusterfs-server glusterfs-client
echo "==> Generating TLS material for management + I/O"
mkdir -p /etc/ssl/glusterfs
cd /etc/ssl/glusterfs
if [[ ! -f glusterfs.key ]]; then
openssl genrsa -out glusterfs.key 2048
openssl req -new -x509 -key glusterfs.key -subj "/CN=${TLS_COMMON_NAME}" \
-days 3650 -out glusterfs.pem
fi
# Each node's .pem must end up in the combined CA bundle on every node.
# In a real deployment, push glusterfs.pem to an out-of-band trust file
# and concat them into glusterfs.ca on every node. For the first install:
cp glusterfs.pem "/etc/ssl/glusterfs/${NODE_NAME}.pem"
# Build glusterfs.ca by concatenating each peer's certificate file
# after you've copied them across out-of-band (scp, ansible, etc.)
if [[ -f "/etc/ssl/glusterfs/${PEER_NAMES[0]}.pem" && \
-f "/etc/ssl/glusterfs/${PEER_NAMES[1]}.pem" && \
-f "/etc/ssl/glusterfs/${PEER_NAMES[2]}.pem" ]]; then
cat "/etc/ssl/glusterfs/${PEER_NAMES[0]}.pem" \
"/etc/ssl/glusterfs/${PEER_NAMES[1]}.pem" \
"/etc/ssl/glusterfs/${PEER_NAMES[2]}.pem" > /etc/ssl/glusterfs/glusterfs.ca
fi
chmod 600 glusterfs.key
chmod 644 glusterfs.pem glusterfs.ca 2>/dev/null || true
echo "==> Enable management-plane TLS (file marker)"
touch /var/lib/glusterd/secure-access
echo "==> Starting glusterd"
systemctl enable --now glusterd
echo "==> Waiting for glusterd to listen on 24007"
for i in {1..30}; do
ss -lnt | grep -q ':24007 ' && break
sleep 1
done
if [[ "${NODE_INDEX}" -eq 0 ]]; then
echo "==> Bootstrap node — probing peers"
for i in 1 2; do
until gluster peer probe "${PEER_NAMES[$i]}" 2>/dev/null; do
echo "Waiting for ${PEER_NAMES[$i]} to be reachable..."
sleep 5
done
done
echo "==> Waiting for all peers to report Connected"
for i in {1..60}; do
if [[ "$(gluster peer status | grep -c 'State: Peer in Cluster (Connected)')" -eq 2 ]]; then
break
fi
sleep 2
done
if ! gluster volume info "${VOLUME_NAME}" >/dev/null 2>&1; then
echo "==> Creating replica 3 volume"
gluster volume create "${VOLUME_NAME}" replica 3 transport tcp \
"${PEER_NAMES[0]}:${BRICK_MOUNT}/${VOLUME_NAME}" \
"${PEER_NAMES[1]}:${BRICK_MOUNT}/${VOLUME_NAME}" \
"${PEER_NAMES[2]}:${BRICK_MOUNT}/${VOLUME_NAME}"
echo "==> Volume options"
gluster volume set "${VOLUME_NAME}" auth.ssl-allow "${PEER_NAMES[0]},${PEER_NAMES[1]},${PEER_NAMES[2]}"
gluster volume set "${VOLUME_NAME}" client.ssl on
gluster volume set "${VOLUME_NAME}" server.ssl on
gluster volume set "${VOLUME_NAME}" cluster.quorum-type auto
gluster volume set "${VOLUME_NAME}" cluster.server-quorum-type server
gluster volume set "${VOLUME_NAME}" cluster.server-quorum-ratio 51
gluster volume set "${VOLUME_NAME}" performance.client-io-threads on
gluster volume set "${VOLUME_NAME}" performance.io-thread-count 16
gluster volume set "${VOLUME_NAME}" performance.cache-size 256MB
gluster volume set "${VOLUME_NAME}" network.ping-timeout 10
gluster volume set "${VOLUME_NAME}" cluster.self-heal-daemon on
gluster volume set "${VOLUME_NAME}" cluster.heal-timeout 600
gluster volume set "${VOLUME_NAME}" cluster.entry-self-heal on
gluster volume set "${VOLUME_NAME}" cluster.metadata-self-heal on
gluster volume set "${VOLUME_NAME}" cluster.data-self-heal on
gluster volume set "${VOLUME_NAME}" features.shard on
gluster volume set "${VOLUME_NAME}" features.shard-block-size 64MB
gluster volume set "${VOLUME_NAME}" diagnostics.brick-log-level WARNING
gluster volume set "${VOLUME_NAME}" diagnostics.client-log-level WARNING
gluster volume start "${VOLUME_NAME}"
fi
else
echo "==> Non-bootstrap node — waiting for node 0 to add us"
fi
echo "==> Self-heal monitor (systemd timer)"
cat > /usr/local/sbin/gluster-heal-monitor.sh <<'HEAL'
#!/usr/bin/env bash
set -euo pipefail
VOL=$(/usr/sbin/gluster volume list | head -n1)
[[ -z "${VOL}" ]] && exit 0
PENDING=$(/usr/sbin/gluster volume heal "${VOL}" info \
| awk '/Number of entries/ {s+=$NF} END {print s+0}')
if [[ "${PENDING}" -gt 0 ]]; then
logger -t gluster-heal -p daemon.warning \
"volume=${VOL} self_heal_pending=${PENDING}"
fi
HEAL
chmod 755 /usr/local/sbin/gluster-heal-monitor.sh
cat > /etc/systemd/system/gluster-heal.service <<'UNIT'
[Unit]
Description=Gluster self-heal queue monitor
After=glusterd.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/gluster-heal-monitor.sh
UNIT
cat > /etc/systemd/system/gluster-heal.timer <<'TIMER'
[Unit]
Description=Run gluster-heal-monitor every 5 minutes
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Unit=gluster-heal.service
[Install]
WantedBy=timers.target
TIMER
systemctl daemon-reload
systemctl enable --now gluster-heal.timer
echo
echo "========================================"
echo "DONE on ${NODE_NAME}"
echo "========================================"
echo
echo "Listeners:"
ss -lntp | grep -E ':(24007|24008|49152|49153|49154)\b' || true
echo
gluster peer status || true
echo
gluster volume info "${VOLUME_NAME}" 2>/dev/null || echo "Volume not yet created (only node 0 creates it)"
echo
echo "On a client (any node or external):"
echo " apt-get install -y glusterfs-client"
echo " mount -t glusterfs -o backupvolfile-server=${PEER_NAMES[1]},acl ${PEER_NAMES[0]}:/${VOLUME_NAME} /mnt/${VOLUME_NAME}"
echo
echo "Heal status:"
gluster volume heal "${VOLUME_NAME}" info 2>/dev/null | head -20 || true
Run it on every node in order — gluster-01 first because it bootstraps the trusted pool. After all three are up, do a failure drill before production traffic: write a known file from a client mount, hard-reboot one node, confirm the client mount still serves the file, bring the node back, and watch gluster volume heal gv0 info drain back to zero. The first time a client encounters a “split-brain” file, the cluster will refuse the read — keep gluster volume heal gv0 info split-brain in your monitoring set so the next operator on call doesn’t discover it during an outage.
Sharding (features.shard on) is on by default in the script because large-file workloads (VM images, container layers, big media) heal faster when chunked. Turn it off only if you specifically need the whole-file semantics — once it’s on with data on the volume, you cannot turn it off without rewriting every file.
For a distributed filesystem you don’t have to operate yourself, our team runs Gluster and CephFS clusters as part of environment management with brick-health and heal-queue alerting on the monitoring plane — book a scoping call if you’d like us to design and operate the storage tier.