You’ve been collecting traces with all-in-one Jaeger on a single VM for the last year. Retention is set to 24 hours because Badger fills the disk otherwise, the post-mortem you wrote last week kept saying “we don’t have the trace anymore,” and a vendor-hosted tracing tier is off the table because of data-residency commitments. The right answer is the production topology: Cassandra for span storage, Kafka as a write buffer in front of the storage tier, and Jaeger split into collector + ingester + query + agent so you can scale each independently. This is the installer for a three-Cassandra-node + three-Jaeger-component fleet on AlmaLinux 9.
What’s happening
Jaeger’s “all-in-one” binary is fine for local dev and never fine for production. The split-component topology that holds up under load is: OTLP/Jaeger receivers terminate on a jaeger-collector fleet (stateless, horizontally scalable), the collector writes spans to a Kafka topic, an jaeger-ingester fleet (also stateless) consumes from Kafka and writes to Cassandra, and a jaeger-query fleet serves the UI and API by reading from Cassandra. The Kafka step is the one operators skip first and regret first — without it, a slow Cassandra repair operation backs pressure all the way to your application SDKs and you start dropping spans during the exact incident you wrote tracing for.
Cassandra is chosen for span storage because the access pattern (write-heavy, time-partitioned, no joins, retention-driven TTL) matches what Cassandra is good at. The schema Jaeger creates uses TWCS (TimeWindowCompactionStrategy) and a TTL set per-tenant — once you tune the TTL down to fit your disk budget, compaction does the deletion for you and there are no “expire jobs” to run. Replication factor is RF=3 across three nodes with LOCAL_QUORUM consistency.
How to verify
Cluster health from any Cassandra node:
nodetool status
# Datacenter: dc1
# ===============
# Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
# -- Address Load Tokens Owns (effective) Host ID
# UN 10.20.0.31 1.42 GiB 16 100.0% ...
# UN 10.20.0.32 1.41 GiB 16 100.0% ...
# UN 10.20.0.33 1.43 GiB 16 100.0% ...
All three should be UN (Up, Normal). Then check the Jaeger keyspace exists:
cqlsh 10.20.0.31 -e "DESCRIBE KEYSPACE jaeger_v1_prod" | head -30
From the Jaeger collector host:
# Health
curl -fs http://127.0.0.1:14269/
# Receivers listening
ss -lntp | grep -E ':(4317|4318|14250|14268|14269)\b'
# Internal Prometheus metrics
curl -fs http://127.0.0.1:14269/metrics | grep -E '^jaeger_(collector|tracer)_' | head -20
End-to-end check: push a synthetic trace via otelcol-contrib or a small SDK script pointed at the collector’s OTLP port (4317), then query it back through the UI at http://<jaeger-query>:16686/. If spans land in the collector but the UI shows nothing, the ingester or Kafka is the gap — check jaeger_ingester_sarama_consumer_messages_total and jaeger_collector_kafka_writes_total.
The fix
This deployment has two scripts. setup-cassandra-node.sh runs on each of the three Cassandra nodes; setup-jaeger-node.sh runs on each of the three Jaeger hosts (which also each run a single-broker Kafka — for production, split Kafka onto its own three-node cluster, but the topology is identical).
Cassandra installer
Save as /root/setup-cassandra-node.sh on each of the three Cassandra nodes, edit the variables, run.
#!/usr/bin/env bash
set -euo pipefail
### AlmaLinux 9 Cassandra 4.1 installer for Jaeger span storage
### Three-node cluster, RF=3, vnodes=16, JVM heap = 8G on a 32G host
if [[ "$(id -u)" -ne 0 ]]; then
echo "Run as root."
exit 1
fi
# === EDIT PER NODE ===
NODE_NAME="cass01"
NODE_IP="10.20.0.31"
CLUSTER_NAME="jaeger-cass"
SEED_IPS="10.20.0.31,10.20.0.32,10.20.0.33"
DC_NAME="dc1"
RACK_NAME="rack1"
HEAP_SIZE="8G"
# =====================
echo "==> Updating base OS"
dnf -y update
dnf -y install epel-release
dnf -y install vim curl wget chrony fail2ban dnf-automatic \
policycoreutils-python-utils lsof tcpdump bind-utils tar gzip \
iproute net-tools jq socat java-17-openjdk-headless
systemctl enable --now chronyd
echo "==> SELinux enforcing"
sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config
setenforce 1 || true
echo "==> Kernel tuning for Cassandra"
cat > /etc/sysctl.d/99-cassandra.conf <<'SYSCTL'
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 10
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
vm.max_map_count = 1048575
vm.swappiness = 1
SYSCTL
sysctl --system
cat > /etc/security/limits.d/99-cassandra.conf <<'LIMITS'
cassandra soft nofile 200000
cassandra hard nofile 200000
cassandra soft nproc 32768
cassandra hard nproc 32768
cassandra soft memlock unlimited
cassandra hard memlock unlimited
cassandra soft as unlimited
cassandra hard as unlimited
LIMITS
echo "==> SSH hardening"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart sshd
systemctl enable --now fail2ban
cat > /etc/dnf/automatic.conf <<'DNFAUTO'
[commands]
upgrade_type = security
random_sleep = 900
download_updates = yes
apply_updates = yes
[emitters]
emit_via = stdio
DNFAUTO
systemctl enable --now dnf-automatic.timer
echo "==> Adding Apache Cassandra 4.1 repo"
cat > /etc/yum.repos.d/cassandra.repo <<'REPO'
[cassandra]
name=Apache Cassandra
baseurl=https://redhat.cassandra.apache.org/41x/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://downloads.apache.org/cassandra/KEYS
REPO
dnf -y install cassandra cassandra-tools
echo "==> Writing cassandra.yaml"
CASS_CONF=/etc/cassandra/conf/cassandra.yaml
cp -a "${CASS_CONF}" "${CASS_CONF}.orig"
sed -i "s|^cluster_name:.*|cluster_name: '${CLUSTER_NAME}'|" "${CASS_CONF}"
sed -i "s|^num_tokens:.*|num_tokens: 16|" "${CASS_CONF}"
sed -i "s|^listen_address:.*|listen_address: ${NODE_IP}|" "${CASS_CONF}"
sed -i "s|^rpc_address:.*|rpc_address: ${NODE_IP}|" "${CASS_CONF}"
sed -i "s|^broadcast_address:.*|broadcast_address: ${NODE_IP}|" "${CASS_CONF}"
sed -i "s|^broadcast_rpc_address:.*|broadcast_rpc_address: ${NODE_IP}|" "${CASS_CONF}"
sed -i "s|^endpoint_snitch:.*|endpoint_snitch: GossipingPropertyFileSnitch|" "${CASS_CONF}"
sed -i "s|^# seeds:.*|# seeds (filled below) |" "${CASS_CONF}"
sed -i "s|seeds:.*|seeds: \"${SEED_IPS}\"|" "${CASS_CONF}"
sed -i "s|^authenticator:.*|authenticator: PasswordAuthenticator|" "${CASS_CONF}"
sed -i "s|^authorizer:.*|authorizer: CassandraAuthorizer|" "${CASS_CONF}"
cat > /etc/cassandra/conf/cassandra-rackdc.properties <<EOF
dc=${DC_NAME}
rack=${RACK_NAME}
EOF
echo "==> JVM heap = ${HEAP_SIZE}"
sed -i "s|^#-Xms.*|-Xms${HEAP_SIZE}|" /etc/cassandra/conf/jvm-server.options || true
sed -i "s|^#-Xmx.*|-Xmx${HEAP_SIZE}|" /etc/cassandra/conf/jvm-server.options || true
sed -i "s|^-Xms.*|-Xms${HEAP_SIZE}|" /etc/cassandra/conf/jvm-server.options || true
sed -i "s|^-Xmx.*|-Xmx${HEAP_SIZE}|" /etc/cassandra/conf/jvm-server.options || true
echo "==> Starting Cassandra (this takes 60-120s for the first node)"
systemctl enable --now cassandra
echo "==> Waiting for CQL listener"
for i in {1..60}; do
if ss -lnt | grep -q ":9042"; then break; fi
sleep 5
done
echo
echo "========================================"
echo "Cassandra node ${NODE_NAME} (${NODE_IP}) up"
echo "========================================"
nodetool status
echo
echo "When all three are UN, run schema bootstrap from ONE node:"
echo " cqlsh ${NODE_IP} -u cassandra -p cassandra"
echo "Then on a single node only:"
echo " /root/jaeger-schema-bootstrap.sh"
After all three Cassandra nodes are up and nodetool status shows three UN entries, create the Jaeger keyspace from any one node. Save this as /root/jaeger-schema-bootstrap.sh and run it on one node only:
#!/usr/bin/env bash
set -euo pipefail
CASS_HOST="10.20.0.31"
KEYSPACE="jaeger_v1_prod"
DC_NAME="dc1"
TTL_HOURS=168 # 7 days
# Pull the Jaeger schema template
docker run --rm jaegertracing/jaeger-cassandra-schema:1.62.0 \
/cassandra-schema/v004.cql.tmpl > /tmp/jaeger-schema.cql.tmpl 2>/dev/null || \
curl -fsSL https://raw.githubusercontent.com/jaegertracing/jaeger/v1.62.0/plugin/storage/cassandra/schema/v004.cql.tmpl \
-o /tmp/jaeger-schema.cql.tmpl
sed -e "s|\${keyspace}|${KEYSPACE}|g" \
-e "s|\${replication}|{'class': 'NetworkTopologyStrategy', '${DC_NAME}': '3'}|g" \
-e "s|\${trace_ttl}|$(( TTL_HOURS * 3600 ))|g" \
-e "s|\${dependencies_ttl}|0|g" \
/tmp/jaeger-schema.cql.tmpl > /tmp/jaeger-schema.cql
cqlsh "${CASS_HOST}" -u cassandra -p cassandra -f /tmp/jaeger-schema.cql
cqlsh "${CASS_HOST}" -u cassandra -p cassandra -e "DESCRIBE KEYSPACE ${KEYSPACE}" | head -20
echo "Keyspace ${KEYSPACE} created with TTL=${TTL_HOURS}h"
Jaeger collector + ingester + query installer
Save as /root/setup-jaeger-node.sh on each of the three Jaeger hosts. Each runs collector + ingester + query + a single Kafka broker (collapse for tutorial brevity — separate Kafka in real production).
#!/usr/bin/env bash
set -euo pipefail
### AlmaLinux 9 Jaeger 1.62 installer (collector + ingester + query)
### Assumes Cassandra cluster already up (cass01/cass02/cass03)
### Kafka topology: 3 brokers co-located with Jaeger hosts in this guide
if [[ "$(id -u)" -ne 0 ]]; then
echo "Run as root."
exit 1
fi
# === EDIT PER NODE ===
NODE_NAME="jaeger01"
NODE_IP="10.20.0.41"
KAFKA_NODE_ID="1"
KAFKA_PEERS="10.20.0.41:9092,10.20.0.42:9092,10.20.0.43:9092"
KAFKA_QUORUM="1@10.20.0.41:9093,2@10.20.0.42:9093,3@10.20.0.43:9093"
CASS_CONTACT_POINTS="10.20.0.31,10.20.0.32,10.20.0.33"
CASS_KEYSPACE="jaeger_v1_prod"
CASS_USER="cassandra"
CASS_PASS="cassandra"
JAEGER_VERSION="1.62.0"
KAFKA_VERSION="3.7.1"
# =====================
echo "==> Base OS"
dnf -y update
dnf -y install epel-release vim curl wget chrony fail2ban dnf-automatic \
java-17-openjdk-headless tar gzip jq net-tools socat
systemctl enable --now chronyd
systemctl enable --now fail2ban
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart sshd
echo "==> Creating users"
useradd --system --no-create-home --shell /usr/sbin/nologin kafka || true
useradd --system --no-create-home --shell /usr/sbin/nologin jaeger || true
echo "==> Downloading Kafka ${KAFKA_VERSION}"
cd /opt
wget -q "https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/kafka_2.13-${KAFKA_VERSION}.tgz"
tar xzf "kafka_2.13-${KAFKA_VERSION}.tgz"
ln -sfn "/opt/kafka_2.13-${KAFKA_VERSION}" /opt/kafka
mkdir -p /var/lib/kafka /var/log/kafka
chown -R kafka:kafka /opt/kafka_2.13-${KAFKA_VERSION} /var/lib/kafka /var/log/kafka
cat > /opt/kafka/config/kraft/server.properties <<EOF
process.roles=broker,controller
node.id=${KAFKA_NODE_ID}
controller.quorum.voters=${KAFKA_QUORUM}
listeners=PLAINTEXT://${NODE_IP}:9092,CONTROLLER://${NODE_IP}:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://${NODE_IP}:9092
controller.listener.names=CONTROLLER
log.dirs=/var/lib/kafka
num.partitions=12
default.replication.factor=3
min.insync.replicas=2
log.retention.hours=24
log.segment.bytes=536870912
num.network.threads=4
num.io.threads=8
EOF
# Cluster UUID — generate once on node 1, copy to others
if [[ "${KAFKA_NODE_ID}" == "1" ]]; then
KAFKA_UUID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo "${KAFKA_UUID}" > /var/lib/kafka/cluster.uuid
fi
KAFKA_UUID="$(cat /var/lib/kafka/cluster.uuid 2>/dev/null || echo CHANGE_ME_CLUSTER_UUID)"
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format -t "${KAFKA_UUID}" -c /opt/kafka/config/kraft/server.properties --ignore-formatted
cat > /etc/systemd/system/kafka.service <<'KAFKAUNIT'
[Unit]
Description=Kafka broker (KRaft mode)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=kafka
Group=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
Restart=on-failure
LimitNOFILE=200000
Environment=KAFKA_HEAP_OPTS=-Xms2G -Xmx4G
[Install]
WantedBy=multi-user.target
KAFKAUNIT
systemctl daemon-reload
systemctl enable --now kafka
echo "==> Waiting for Kafka broker"
for i in {1..30}; do
if ss -lnt | grep -q ":9092"; then break; fi
sleep 3
done
# Create the spans topic from node 1 only
if [[ "${KAFKA_NODE_ID}" == "1" ]]; then
/opt/kafka/bin/kafka-topics.sh --bootstrap-server "${NODE_IP}:9092" \
--create --if-not-exists --topic jaeger-spans \
--partitions 24 --replication-factor 3 \
--config min.insync.replicas=2 --config retention.ms=86400000
fi
echo "==> Downloading Jaeger ${JAEGER_VERSION}"
cd /opt
wget -q "https://github.com/jaegertracing/jaeger/releases/download/v${JAEGER_VERSION}/jaeger-${JAEGER_VERSION}-linux-amd64.tar.gz"
tar xzf "jaeger-${JAEGER_VERSION}-linux-amd64.tar.gz"
ln -sfn "/opt/jaeger-${JAEGER_VERSION}-linux-amd64" /opt/jaeger
chown -R jaeger:jaeger /opt/jaeger-${JAEGER_VERSION}-linux-amd64
echo "==> jaeger-collector unit"
cat > /etc/systemd/system/jaeger-collector.service <<EOF
[Unit]
Description=Jaeger Collector
After=network-online.target kafka.service
Wants=network-online.target
[Service]
Type=simple
User=jaeger
Group=jaeger
Environment=SPAN_STORAGE_TYPE=kafka
Environment=KAFKA_PRODUCER_BROKERS=${KAFKA_PEERS}
Environment=KAFKA_PRODUCER_TOPIC=jaeger-spans
Environment=COLLECTOR_OTLP_ENABLED=true
Environment=COLLECTOR_GRPC_SERVER_HOST_PORT=:14250
Environment=COLLECTOR_HTTP_SERVER_HOST_PORT=:14268
Environment=COLLECTOR_OTLP_GRPC_HOST_PORT=:4317
Environment=COLLECTOR_OTLP_HTTP_HOST_PORT=:4318
Environment=METRICS_HTTP_ROUTE=/metrics
Environment=ADMIN_HTTP_HOST_PORT=:14269
ExecStart=/opt/jaeger/jaeger-collector
Restart=on-failure
LimitNOFILE=200000
[Install]
WantedBy=multi-user.target
EOF
echo "==> jaeger-ingester unit"
cat > /etc/systemd/system/jaeger-ingester.service <<EOF
[Unit]
Description=Jaeger Ingester
After=network-online.target kafka.service
Wants=network-online.target
[Service]
Type=simple
User=jaeger
Group=jaeger
Environment=SPAN_STORAGE_TYPE=cassandra
Environment=KAFKA_CONSUMER_BROKERS=${KAFKA_PEERS}
Environment=KAFKA_CONSUMER_TOPIC=jaeger-spans
Environment=KAFKA_CONSUMER_GROUP_ID=jaeger-ingester
Environment=CASSANDRA_SERVERS=${CASS_CONTACT_POINTS}
Environment=CASSANDRA_KEYSPACE=${CASS_KEYSPACE}
Environment=CASSANDRA_USERNAME=${CASS_USER}
Environment=CASSANDRA_PASSWORD=${CASS_PASS}
Environment=CASSANDRA_CONSISTENCY=LOCAL_QUORUM
Environment=ADMIN_HTTP_HOST_PORT=:14270
ExecStart=/opt/jaeger/jaeger-ingester
Restart=on-failure
LimitNOFILE=200000
[Install]
WantedBy=multi-user.target
EOF
echo "==> jaeger-query unit"
cat > /etc/systemd/system/jaeger-query.service <<EOF
[Unit]
Description=Jaeger Query UI
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=jaeger
Group=jaeger
Environment=SPAN_STORAGE_TYPE=cassandra
Environment=CASSANDRA_SERVERS=${CASS_CONTACT_POINTS}
Environment=CASSANDRA_KEYSPACE=${CASS_KEYSPACE}
Environment=CASSANDRA_USERNAME=${CASS_USER}
Environment=CASSANDRA_PASSWORD=${CASS_PASS}
Environment=CASSANDRA_CONSISTENCY=LOCAL_QUORUM
Environment=QUERY_HTTP_SERVER_HOST_PORT=:16686
Environment=QUERY_GRPC_SERVER_HOST_PORT=:16685
Environment=ADMIN_HTTP_HOST_PORT=:16687
ExecStart=/opt/jaeger/jaeger-query
Restart=on-failure
LimitNOFILE=200000
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now jaeger-collector jaeger-ingester jaeger-query
echo "==> Waiting for collector admin"
for i in {1..30}; do
if curl -fs http://127.0.0.1:14269/ >/dev/null 2>&1; then break; fi
sleep 2
done
echo
echo "========================================"
echo "Jaeger node ${NODE_NAME} (${NODE_IP}) up"
echo "========================================"
echo
echo "Listeners:"
ss -lntp | grep -E ':(4317|4318|9092|9093|14250|14268|14269|14270|16685|16686|16687)\b' || true
echo
echo "Health:"
curl -fs http://127.0.0.1:14269/ | head -20 || true
echo
echo "Send a test trace to the collector OTLP endpoint:"
echo " ${NODE_IP}:4317 (gRPC) or ${NODE_IP}:4318 (HTTP)"
echo
echo "Open the UI at http://${NODE_IP}:16686/"
After all three Jaeger nodes are up, point your OpenTelemetry gateway’s otlp/jaeger exporter at any of the Jaeger collectors (or put HAProxy in front of the three for client-side LB), and watch traces flow end-to-end: app SDK → OTel gateway → Jaeger collector → Kafka → Jaeger ingester → Cassandra → Jaeger query UI. Tune retention by lowering TTL_HOURS in the schema bootstrap script and re-running the relevant ALTER TABLE ... WITH default_time_to_live = ... statements; don’t try to fix a disk-full Cassandra by deleting span rows manually, the TTL machinery is what you want.
Stack Harbor operates self-hosted Jaeger + Cassandra stacks under Managed Cloud Ops — including Cassandra repair scheduling, Kafka lag alerting, and trace-volume capacity planning. Talk to us about your observability stack.