Skip to content

Deploying Grafana Pyroscope for continuous profiling on Ubuntu 22.04

Stand up a single-node Grafana Pyroscope server on Ubuntu 22.04 with object-storage backed retention, plus a sample agent config for Go, Java, and Python services.

The Go service has been at 75% CPU for three weeks. Nobody complained because P99 latency is still inside SLO. Then traffic doubles for a Black Friday simulation and the service starts returning 503s at 88% CPU — except the only thing in the dashboards is “CPU went up.” Metrics tell you something is consuming CPU. Logs tell you what URLs got hit. Neither tells you which function inside your service is burning the cycles. You spend a full day attaching pprof to one box at a time, comparing flamegraphs, and finally find that a JSON-decoding loop on a hot path is the culprit. Pyroscope makes that day five minutes — every Go, Java, Python, .NET, and Rust process emits a continuous profile, you query a time range in Grafana, and the flamegraph diff between yesterday and today highlights exactly which function is new.

What’s happening

Pyroscope is a continuous-profiling backend. Instead of running pprof -seconds=30 on demand against one process, Pyroscope collects CPU, allocation, lock-contention, and goroutine profiles from every instance every 10-15 seconds, in production, all the time, and stores them in object storage with retention measured in months. The agent (pull or push) talks to the language runtime’s profiling endpoint — /debug/pprof/profile for Go, JFR for Java, py-spy for Python — and ships compressed samples to the server.

The server stores samples as immutable Parquet-style blocks on object storage. Queries flatten samples within a time range into a flamegraph, and Grafana’s Pyroscope datasource exposes diff views (“this hour vs. last hour”, “before deploy vs. after deploy”) that highlight the function-level delta. On a single-node deployment, monolithic mode runs all the components (distributor, ingester, querier, store-gateway, compactor) in one process — adequate for hundreds of instrumented processes, low operational overhead.

How to verify

Server is healthy:

curl -s http://10.40.0.31:4040/ready
# expect: ready
curl -s http://10.40.0.31:4040/status/version

A profile is reaching the server — push a synthetic CPU profile from any host:

# Generate a tiny Go-style pprof, push it
curl -X POST 'http://10.40.0.31:4040/ingest?name=synthetic-test{}&from=now-30s&until=now&sampleRate=100&spyName=gospy&units=samples&aggregationType=sum' \
  -H 'Content-Type: application/octet-stream' \
  --data-binary @testdata/sample.pprof.gz

Then in the UI at /, search the app name synthetic-test — the flamegraph should render.

Confirm blocks are landing in S3:

aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://acme-pyroscope-profiles/anonymous/ \
  | head

The fix

Save as /root/setup-pyroscope.sh and run on the single VM. Installs Pyroscope 1.7, configures S3-backed long-term storage, NGINX for TLS, and a systemd unit. Then below the script is the agent-side configuration for the three most common runtimes.

#!/usr/bin/env bash
set -euo pipefail

### Ubuntu 22.04 Grafana Pyroscope (single-node monolithic) installer
### S3-backed, ingest port 4040, query through same port

if [[ "$(id -u)" -ne 0 ]]; then
  echo "Run as root."
  exit 1
fi

# === EDIT ===
NODE_IP="10.40.0.31"
PYROSCOPE_VERSION="1.7.1"
S3_ENDPOINT="s3.fr-par.scw.cloud"
S3_REGION="fr-par"
S3_BUCKET="acme-pyroscope-profiles"
S3_ACCESS_KEY="CHANGE_ME_ACCESS"
S3_SECRET_KEY="CHANGE_ME_SECRET"
PUBLIC_DOMAIN="pyroscope.example.com"
RETENTION_DAYS=90
# =============

echo
echo "========================================"
echo "Pyroscope ${NODE_IP}"
echo "========================================"

export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y dist-upgrade
apt-get -y install curl wget ca-certificates gnupg lsb-release \
  ufw fail2ban unattended-upgrades chrony jq tar gzip \
  iproute2 net-tools dnsutils socat nginx certbot python3-certbot-nginx \
  htop iotop

systemctl enable --now chrony

echo "==> UFW"
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
# Ingest from app subnet only — adjust source
ufw allow from 10.40.0.0/16 to any port 4040 proto tcp
ufw --force enable

echo "==> SSH hardening"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
systemctl restart ssh

cat > /etc/fail2ban/jail.d/sshd.local <<'FAIL2BAN'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
FAIL2BAN
systemctl enable --now fail2ban

cat > /etc/apt/apt.conf.d/20auto-upgrades <<'AUTO'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
AUTO

echo "==> Sysctl"
cat > /etc/sysctl.d/99-pyroscope.conf <<'SYSCTL'
fs.file-max = 2097152
net.core.somaxconn = 8192
vm.max_map_count = 262144
SYSCTL
sysctl --system

cat > /etc/security/limits.d/99-pyroscope.conf <<'LIMITS'
pyroscope soft nofile 200000
pyroscope hard nofile 200000
LIMITS

echo "==> Creating pyroscope user"
id -u pyroscope >/dev/null 2>&1 || useradd --system --no-create-home --shell /usr/sbin/nologin pyroscope
mkdir -p /var/lib/pyroscope /etc/pyroscope
chown -R pyroscope:pyroscope /var/lib/pyroscope /etc/pyroscope

echo "==> Downloading Pyroscope ${PYROSCOPE_VERSION}"
cd /tmp
wget -q "https://github.com/grafana/pyroscope/releases/download/v${PYROSCOPE_VERSION}/pyroscope_${PYROSCOPE_VERSION}_linux_amd64.tar.gz"
tar xzf "pyroscope_${PYROSCOPE_VERSION}_linux_amd64.tar.gz"
install -o root -g root -m 0755 pyroscope /usr/local/bin/pyroscope
/usr/local/bin/pyroscope --version

# Retention in days; compactor will keep this many days of compacted blocks
RETENTION_HOURS=$((RETENTION_DAYS * 24))

echo "==> Writing pyroscope.yml"
cat > /etc/pyroscope/pyroscope.yml <<EOF
target: all

server:
  http_listen_port: 4040
  grpc_listen_port: 9097
  log_level: info

multitenancy_enabled: false

memberlist:
  bind_port: 7946
  advertise_addr: ${NODE_IP}

distributor:
  ring:
    kvstore:
      store: memberlist

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
    join_after: 0s

querier:
  max_concurrent: 16

store_gateway:
  sharding_ring:
    kvstore:
      store: memberlist
    replication_factor: 1

compactor:
  data_dir: /var/lib/pyroscope/compactor
  sharding_ring:
    kvstore:
      store: memberlist
  retention_period: ${RETENTION_HOURS}h

storage:
  backend: s3
  s3:
    endpoint: ${S3_ENDPOINT}
    region: ${S3_REGION}
    bucket_name: ${S3_BUCKET}
    access_key_id: ${S3_ACCESS_KEY}
    secret_access_key: ${S3_SECRET_KEY}
    insecure: false

limits:
  max_global_series_per_tenant: 1000000
  ingestion_rate_mb: 100
  ingestion_burst_size_mb: 200

analytics:
  reporting_enabled: false
EOF
chown -R pyroscope:pyroscope /etc/pyroscope

cat > /etc/systemd/system/pyroscope.service <<'UNIT'
[Unit]
Description=Grafana Pyroscope
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=pyroscope
Group=pyroscope
ExecStart=/usr/local/bin/pyroscope -config.file=/etc/pyroscope/pyroscope.yml
Restart=on-failure
RestartSec=5s
LimitNOFILE=200000
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target
UNIT

systemctl daemon-reload
systemctl enable --now pyroscope

echo "==> NGINX TLS"
cat > /etc/nginx/sites-available/pyroscope <<EOF
upstream pyroscope_http { server 127.0.0.1:4040; keepalive 32; }

server {
  listen 80;
  server_name ${PUBLIC_DOMAIN};
  location /.well-known/acme-challenge/ { root /var/www/html; }
  location / { return 301 https://\$host\$request_uri; }
}

server {
  listen 443 ssl http2;
  server_name ${PUBLIC_DOMAIN};
  ssl_certificate /etc/letsencrypt/live/${PUBLIC_DOMAIN}/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/${PUBLIC_DOMAIN}/privkey.pem;
  ssl_protocols TLSv1.2 TLSv1.3;

  client_max_body_size 100m;
  proxy_read_timeout 120s;

  location / {
    proxy_set_header Host \$host;
    proxy_pass http://pyroscope_http;
  }
}
EOF
ln -sf /etc/nginx/sites-available/pyroscope /etc/nginx/sites-enabled/pyroscope
rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx

echo "==> Waiting for /ready"
for i in {1..60}; do
  if curl -fs "http://127.0.0.1:4040/ready" >/dev/null 2>&1; then break; fi
  sleep 2
done

echo
echo "========================================"
echo "DONE"
echo "========================================"
echo
ss -lntp | grep -E ':(80|443|4040|9097)\b' || true
echo
echo "Web UI: http://${NODE_IP}:4040"
echo "Ingest:  http://${NODE_IP}:4040/ingest"
echo "Run certbot --nginx -d ${PUBLIC_DOMAIN} once DNS is pointed."

Agent: Go application

Go programs ship with net/http/pprof. The Pyroscope Go SDK starts a background goroutine that pulls profiles from itself and pushes to the server every 15 seconds. Add to main.go:

import (
  "runtime"
  "github.com/grafana/pyroscope-go"
)

func init() {
  runtime.SetMutexProfileFraction(5)
  runtime.SetBlockProfileRate(5)
  pyroscope.Start(pyroscope.Config{
    ApplicationName: "checkout-api",
    ServerAddress:   "https://pyroscope.example.com",
    Logger:          pyroscope.StandardLogger,
    Tags:            map[string]string{"env": "prod", "region": "ca-east-1"},
    ProfileTypes: []pyroscope.ProfileType{
      pyroscope.ProfileCPU,
      pyroscope.ProfileAllocObjects,
      pyroscope.ProfileAllocSpace,
      pyroscope.ProfileInuseObjects,
      pyroscope.ProfileInuseSpace,
      pyroscope.ProfileGoroutines,
      pyroscope.ProfileMutexCount,
      pyroscope.ProfileBlockCount,
    },
  })
}

Agent: Java application (JFR)

Pyroscope’s Java agent uses Java Flight Recorder. Drop the agent JAR on the host and modify the JVM start command:

wget -q -O /opt/pyroscope-agent.jar \
  https://github.com/grafana/pyroscope-java/releases/latest/download/pyroscope.jar

# In your service unit, add:
# Environment=PYROSCOPE_APPLICATION_NAME=checkout-api
# Environment=PYROSCOPE_SERVER_ADDRESS=https://pyroscope.example.com
# Environment=PYROSCOPE_FORMAT=jfr
# Environment=PYROSCOPE_PROFILER_EVENT=cpu
# Environment=PYROSCOPE_PROFILER_ALLOC=512k
# Environment=PYROSCOPE_LABELS=env=prod,region=ca-east-1
# ExecStart=/usr/bin/java -javaagent:/opt/pyroscope-agent.jar -jar /opt/app.jar

Agent: Python application

For Django or Flask, install the Python SDK and start it before WSGI initializes:

# app/profiling.py
import pyroscope

pyroscope.configure(
    application_name="checkout-api",
    server_address="https://pyroscope.example.com",
    tags={
        "env": "prod",
        "region": "ca-east-1",
    },
    sample_rate=100,
    detect_subprocesses=True,
    oncpu=True,
    gil_only=True,
)

Then in wsgi.py:

import app.profiling  # noqa: F401  -- side effect: configure profiler
from app.wsgi import application

In Grafana, add the Pyroscope datasource:

# /etc/grafana/provisioning/datasources/pyroscope.yaml
apiVersion: 1
datasources:
  - name: Pyroscope
    type: grafana-pyroscope-datasource
    access: proxy
    url: https://pyroscope.example.com
    jsonData:
      keepCookies: []

The diff view is the killer feature — pick “last 1h vs. 1h before that”, filter by service=checkout-api, and any new function in the top of the flamegraph is your regression. Watch pyroscope_ingester_appenders_open and pyroscope_storage_blocks_loaded on the server; a sudden spike in open appenders without a corresponding rise in active services often means agents are restarting in a tight loop.

Run continuous profiling on Stack Harbor’s managed cloud compute with retention policy, agent rollout, and Grafana wiring under environment management — or book a scoping call if you want the observability tier delivered with on-call coverage included.