Skip to content

Deploying Milvus 2.x standalone with etcd + MinIO on AlmaLinux 9

Production-grade single-node Milvus on AlmaLinux 9 — etcd for metadata, MinIO for object storage, systemd-managed Docker Compose, with backup and metrics wiring.

You evaluated Milvus, Qdrant and Weaviate for a recommendation engine that does hybrid scalar + vector search across a hundred million product embeddings. Milvus won the benchmark — its IVF and DiskANN indices held latency under load better than the alternatives for your shape of data. The proof of concept ran on docker run milvusdb/milvus:latest standalone on a developer laptop. Now it’s six weeks before launch, ops asks where the cluster is going to live, and you realize Milvus standalone is not a single binary — it’s Milvus plus etcd plus an S3-compatible object store, and the developer-laptop command silently used embedded etcd and local filesystem storage that won’t survive a host reboot in production. This is the install that actually works.

What’s happening

Milvus standalone is a misleading name. Even in single-node “standalone” mode Milvus depends on two external services to function: etcd as the metadata store (collection schemas, segment positions, index metadata, channel offsets) and an S3-compatible object store (MinIO is the bundled default) for the actual segment files — vector data, deletion records, statistics. The official docker-compose-for-development bundle ships all three in containers on a shared bridge network. That’s fine to test against; it’s not fine to run in production because the etcd container has no quorum (it’s a single member, so any restart is a full re-bootstrap), the MinIO container uses ephemeral storage by default, and there’s no resource isolation. A production-grade standalone deployment puts each component in its own container with persistent volumes, runs them under systemd so they survive a reboot, fronts MinIO with TLS, locks etcd to localhost, and lets you scale to distributed Milvus later by swapping etcd for a 3-node etcd cluster and MinIO for either a multi-node MinIO or AWS S3 directly.

How to verify

Start with the health check:

curl -fsS http://127.0.0.1:9091/healthz
# should print "OK"

Then connect via the Milvus CLI inside the container and list collections:

docker exec -it milvus-standalone /bin/bash -lc \
  "python3 -c \"from pymilvus import connections, utility; connections.connect('default', host='127.0.0.1', port='19530'); print(utility.list_collections())\""

Confirm etcd is reachable only from localhost and Milvus, and is healthy:

ss -lntp | grep ':2379'
# 127.0.0.1:2379 — should NOT bind 0.0.0.0
docker exec -it milvus-etcd etcdctl endpoint health
# 127.0.0.1:2379 is healthy: ...

Object store sanity:

docker exec -it milvus-minio mc alias set local http://127.0.0.1:9000 minioadmin "${MINIO_ROOT_PASSWORD}"
docker exec -it milvus-minio mc ls local/milvus-bucket --recursive | head -20
docker exec -it milvus-minio mc admin info local

If mc admin info reports Drive: Online (1/1) and Milvus is writing segment files under the bucket, the storage plane is working. If you see segments accumulating without a corresponding compaction cycle, your dataCoord.compaction interval is wrong — that’s a separate runbook.

The fix

Save the script as /root/setup-milvus-standalone.sh and run as root on a single AlmaLinux 9 VM with at least 16 GB RAM, 4 vCPU, and a dedicated data volume mounted at /var/lib/milvus. The script does base hardening, installs Docker CE from upstream, drops a docker-compose file with named volumes for each component, pins versions (Milvus 2.4, etcd 3.5, MinIO RELEASE.2024-08-17T01-24-54Z), generates a MinIO root password if not provided, exposes only the Milvus gRPC port (19530), web health (9091) and Prometheus metrics scrape on the management subnet, and registers a systemd unit so the stack starts on boot.

#!/usr/bin/env bash
set -euo pipefail

### AlmaLinux 9 Milvus 2.4 standalone installer (Milvus + etcd + MinIO)
### Run as root on a single VM
### Assumes a dedicated data volume mounted at /var/lib/milvus
### Treat etcd + MinIO as components of Milvus; do not point other services at them

if [[ "$(id -u)" -ne 0 ]]; then
  echo "Run as root."
  exit 1
fi

# === EDIT BEFORE RUN ===
HOST_FQDN="milvus-01.internal"        # hostname of this VM
NODE_IP="10.30.0.21"                  # this VM's private IP
MGMT_SUBNET="10.30.99.0/24"           # subnet allowed to scrape metrics
MINIO_ROOT_USER="minioadmin"
MINIO_ROOT_PASSWORD="$(openssl rand -hex 24)"   # or paste a fixed value
MILVUS_BUCKET="milvus-bucket"
# =======================

echo
echo "========================================"
echo "Milvus standalone on ${HOST_FQDN} (${NODE_IP})"
echo "========================================"
echo

echo "==> Base OS update + tooling"
dnf -y update
dnf -y install epel-release
dnf -y install vim curl wget chrony fail2ban dnf-automatic \
  policycoreutils-python-utils firewalld jq tar gzip ca-certificates

systemctl enable --now chronyd
hostnamectl set-hostname "${HOST_FQDN}"

echo "==> SELinux enforcing"
sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config
setenforce 1 || true

echo "==> SSH hardening"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart sshd

echo "==> fail2ban"
cat > /etc/fail2ban/jail.d/sshd.local <<'FAIL2BAN'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
FAIL2BAN
systemctl enable --now fail2ban

echo "==> dnf-automatic security updates"
cat > /etc/dnf/automatic.conf <<'DNFAUTO'
[commands]
upgrade_type = security
random_sleep = 900
download_updates = yes
apply_updates = yes
[emitters]
emit_via = stdio
DNFAUTO
systemctl enable --now dnf-automatic.timer

echo "==> firewalld — allow only what Milvus needs"
systemctl enable --now firewalld
firewall-cmd --permanent --remove-service=cockpit || true
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-port=19530/tcp   # Milvus gRPC
firewall-cmd --permanent --add-port=9091/tcp    # Milvus HTTP / health
firewall-cmd --permanent --new-zone=mgmt || true
firewall-cmd --permanent --zone=mgmt --add-source="${MGMT_SUBNET}"
firewall-cmd --permanent --zone=mgmt --add-port=9091/tcp   # metrics
firewall-cmd --reload

echo "==> Kernel tuning (vector workloads — large mmap, low swap)"
cat > /etc/sysctl.d/99-milvus.conf <<'SYSCTL'
vm.max_map_count = 1048576
vm.swappiness = 1
vm.overcommit_memory = 1
fs.file-max = 1048576
net.core.somaxconn = 4096
SYSCTL
sysctl --system

cat > /etc/security/limits.d/99-milvus.conf <<'LIMITS'
* soft nofile 1048576
* hard nofile 1048576
* soft memlock unlimited
* hard memlock unlimited
LIMITS

echo "==> Docker CE from upstream repo"
dnf -y install dnf-plugins-core
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
systemctl enable --now docker

echo "==> Data volume layout"
mkdir -p /var/lib/milvus/{etcd,minio,milvus}
chown -R 1000:1000 /var/lib/milvus/milvus   # Milvus runs as uid 1000

echo "==> docker-compose for Milvus standalone"
mkdir -p /opt/milvus
cat > /opt/milvus/.env <<EOF
MINIO_ROOT_USER=${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}
MILVUS_BUCKET=${MILVUS_BUCKET}
EOF
chmod 600 /opt/milvus/.env

cat > /opt/milvus/docker-compose.yml <<'COMPOSE'
networks:
  milvus:
    driver: bridge

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.16
    restart: unless-stopped
    networks: [milvus]
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - /var/lib/milvus/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls=http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2024-08-17T01-24-54Z
    restart: unless-stopped
    networks: [milvus]
    env_file: .env
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    volumes:
      - /var/lib/milvus/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  milvus-init-bucket:
    container_name: milvus-init-bucket
    image: minio/mc:RELEASE.2024-08-17T11-33-50Z
    depends_on:
      - minio
    networks: [milvus]
    env_file: .env
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      /usr/bin/mc alias set local http://minio:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD};
      /usr/bin/mc mb --ignore-existing local/${MILVUS_BUCKET};
      /usr/bin/mc version enable local/${MILVUS_BUCKET};
      exit 0
      "
    restart: "no"

  milvus:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.13
    restart: unless-stopped
    networks: [milvus]
    command: ["milvus", "run", "standalone"]
    env_file: .env
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
      MINIO_ACCESS_KEY_ID: ${MINIO_ROOT_USER}
      MINIO_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD}
      MINIO_USE_SSL: "false"
      MINIO_BUCKET_NAME: ${MILVUS_BUCKET}
      COMMON_STORAGETYPE: minio
      MQ_TYPE: rocksmq
    volumes:
      - /var/lib/milvus/milvus:/var/lib/milvus
    ports:
      - "127.0.0.1:19530:19530"   # gRPC, accessed via systemd-proxied port below
      - "127.0.0.1:9091:9091"     # health/metrics
    depends_on:
      - etcd
      - minio
      - milvus-init-bucket
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      timeout: 20s
      retries: 3
COMPOSE

echo "==> Expose Milvus on the node IP (gRPC + health)"
# Bind public listeners directly on the node IP. Re-publish via socat-style
# port forwarding to keep container binds on 127.0.0.1.
dnf -y install nftables
cat > /etc/nftables/milvus.nft <<EOF
table ip nat {
  chain prerouting {
    type nat hook prerouting priority -100;
    iif "lo" return
    tcp dport 19530 dnat to 127.0.0.1:19530
    tcp dport 9091  dnat to 127.0.0.1:9091
  }
}
EOF
# Easier path on AlmaLinux: keep containers on 127.0.0.1 and use a small nginx
# stream block for gRPC. nginx is more familiar to most operators than nft DNAT.
dnf -y install nginx
cat > /etc/nginx/conf.d/milvus-stream.conf <<EOF
# nginx stream module to expose Milvus on node IP
EOF
cat > /etc/nginx/nginx.conf.milvus-additions <<'NGX'
# add to nginx.conf at top level (outside http {})
stream {
  upstream milvus_grpc { server 127.0.0.1:19530; }
  upstream milvus_health { server 127.0.0.1:9091; }
  server { listen 19530; proxy_pass milvus_grpc; }
  server { listen 9091; proxy_pass milvus_health; }
}
NGX
# If your /etc/nginx/nginx.conf already has a stream block, merge by hand.
# For a fresh box, append:
if ! grep -q "milvus_grpc" /etc/nginx/nginx.conf; then
  cat /etc/nginx/nginx.conf.milvus-additions >> /etc/nginx/nginx.conf
fi
nginx -t
systemctl enable --now nginx

echo "==> systemd unit for the compose stack"
cat > /etc/systemd/system/milvus-stack.service <<'UNIT'
[Unit]
Description=Milvus standalone (Milvus + etcd + MinIO)
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/milvus
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target
UNIT

systemctl daemon-reload
systemctl enable --now milvus-stack.service

echo "==> Wait for Milvus health"
for i in {1..120}; do
  if curl -fsS http://127.0.0.1:9091/healthz | grep -q OK; then break; fi
  sleep 2
done

echo
echo "========================================"
echo "DONE"
echo "========================================"
echo
echo "MinIO root credentials saved at /opt/milvus/.env (0600)."
echo "  user=${MINIO_ROOT_USER}"
echo "  password=${MINIO_ROOT_PASSWORD}"
echo
echo "Listeners:"
ss -lntp | grep -E ':(19530|9091|9000|9001|2379)\b' || true
echo
echo "Milvus health:"
curl -fsS http://127.0.0.1:9091/healthz || true
echo
echo "Connect from a client:"
echo "  pymilvus: connections.connect('default', host='${NODE_IP}', port='19530')"
echo
echo "List collections (inside container):"
echo "  docker exec milvus-standalone /milvus/bin/milvus --help"

Make it executable and run:

chmod 700 /root/setup-milvus-standalone.sh
/root/setup-milvus-standalone.sh

A few operational notes that aren’t installer steps but matter once you have traffic. Milvus standalone uses rocksmq as its message queue — fine up to mid-single-digit-million-vectors workloads, but if you start pushing high write throughput you need to evaluate moving to Pulsar or Kafka, which is a hard migration. Plan the conversation to distributed Milvus before you hit the wall, not after. Schedule a daily MinIO mirror to a separate object store (mc mirror --watch local/milvus-bucket remote/milvus-mirror) and a daily etcdctl snapshot save to the same remote — those two artifacts together are what you restore from. Test a full restore into a sandbox once a quarter; the only thing worse than no backups is backups you’ve never validated.

For Milvus tiers operated under environment management — etcd quorum monitoring, MinIO disk health, segment compaction backlog alerts — talk to our monitoring team or start a scoping call.