The Go service has been at 75% CPU for three weeks. Nobody complained because P99 latency is still inside SLO. Then traffic doubles for a Black Friday simulation and the service starts returning 503s at 88% CPU — except the only thing in the dashboards is “CPU went up.” Metrics tell you something is consuming CPU. Logs tell you what URLs got hit. Neither tells you which function inside your service is burning the cycles. You spend a full day attaching pprof to one box at a time, comparing flamegraphs, and finally find that a JSON-decoding loop on a hot path is the culprit. Pyroscope makes that day five minutes — every Go, Java, Python, .NET, and Rust process emits a continuous profile, you query a time range in Grafana, and the flamegraph diff between yesterday and today highlights exactly which function is new.
What’s happening
Pyroscope is a continuous-profiling backend. Instead of running pprof -seconds=30 on demand against one process, Pyroscope collects CPU, allocation, lock-contention, and goroutine profiles from every instance every 10-15 seconds, in production, all the time, and stores them in object storage with retention measured in months. The agent (pull or push) talks to the language runtime’s profiling endpoint — /debug/pprof/profile for Go, JFR for Java, py-spy for Python — and ships compressed samples to the server.
The server stores samples as immutable Parquet-style blocks on object storage. Queries flatten samples within a time range into a flamegraph, and Grafana’s Pyroscope datasource exposes diff views (“this hour vs. last hour”, “before deploy vs. after deploy”) that highlight the function-level delta. On a single-node deployment, monolithic mode runs all the components (distributor, ingester, querier, store-gateway, compactor) in one process — adequate for hundreds of instrumented processes, low operational overhead.
How to verify
Server is healthy:
curl -s http://10.40.0.31:4040/ready
# expect: ready
curl -s http://10.40.0.31:4040/status/version
A profile is reaching the server — push a synthetic CPU profile from any host:
# Generate a tiny Go-style pprof, push it
curl -X POST 'http://10.40.0.31:4040/ingest?name=synthetic-test{}&from=now-30s&until=now&sampleRate=100&spyName=gospy&units=samples&aggregationType=sum' \
-H 'Content-Type: application/octet-stream' \
--data-binary @testdata/sample.pprof.gz
Then in the UI at /, search the app name synthetic-test — the flamegraph should render.
Confirm blocks are landing in S3:
aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://acme-pyroscope-profiles/anonymous/ \
| head
The fix
Save as /root/setup-pyroscope.sh and run on the single VM. Installs Pyroscope 1.7, configures S3-backed long-term storage, NGINX for TLS, and a systemd unit. Then below the script is the agent-side configuration for the three most common runtimes.
#!/usr/bin/env bash
set -euo pipefail
### Ubuntu 22.04 Grafana Pyroscope (single-node monolithic) installer
### S3-backed, ingest port 4040, query through same port
if [[ "$(id -u)" -ne 0 ]]; then
echo "Run as root."
exit 1
fi
# === EDIT ===
NODE_IP="10.40.0.31"
PYROSCOPE_VERSION="1.7.1"
S3_ENDPOINT="s3.fr-par.scw.cloud"
S3_REGION="fr-par"
S3_BUCKET="acme-pyroscope-profiles"
S3_ACCESS_KEY="CHANGE_ME_ACCESS"
S3_SECRET_KEY="CHANGE_ME_SECRET"
PUBLIC_DOMAIN="pyroscope.example.com"
RETENTION_DAYS=90
# =============
echo
echo "========================================"
echo "Pyroscope ${NODE_IP}"
echo "========================================"
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y dist-upgrade
apt-get -y install curl wget ca-certificates gnupg lsb-release \
ufw fail2ban unattended-upgrades chrony jq tar gzip \
iproute2 net-tools dnsutils socat nginx certbot python3-certbot-nginx \
htop iotop
systemctl enable --now chrony
echo "==> UFW"
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
# Ingest from app subnet only — adjust source
ufw allow from 10.40.0.0/16 to any port 4040 proto tcp
ufw --force enable
echo "==> SSH hardening"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
systemctl restart ssh
cat > /etc/fail2ban/jail.d/sshd.local <<'FAIL2BAN'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
FAIL2BAN
systemctl enable --now fail2ban
cat > /etc/apt/apt.conf.d/20auto-upgrades <<'AUTO'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
AUTO
echo "==> Sysctl"
cat > /etc/sysctl.d/99-pyroscope.conf <<'SYSCTL'
fs.file-max = 2097152
net.core.somaxconn = 8192
vm.max_map_count = 262144
SYSCTL
sysctl --system
cat > /etc/security/limits.d/99-pyroscope.conf <<'LIMITS'
pyroscope soft nofile 200000
pyroscope hard nofile 200000
LIMITS
echo "==> Creating pyroscope user"
id -u pyroscope >/dev/null 2>&1 || useradd --system --no-create-home --shell /usr/sbin/nologin pyroscope
mkdir -p /var/lib/pyroscope /etc/pyroscope
chown -R pyroscope:pyroscope /var/lib/pyroscope /etc/pyroscope
echo "==> Downloading Pyroscope ${PYROSCOPE_VERSION}"
cd /tmp
wget -q "https://github.com/grafana/pyroscope/releases/download/v${PYROSCOPE_VERSION}/pyroscope_${PYROSCOPE_VERSION}_linux_amd64.tar.gz"
tar xzf "pyroscope_${PYROSCOPE_VERSION}_linux_amd64.tar.gz"
install -o root -g root -m 0755 pyroscope /usr/local/bin/pyroscope
/usr/local/bin/pyroscope --version
# Retention in days; compactor will keep this many days of compacted blocks
RETENTION_HOURS=$((RETENTION_DAYS * 24))
echo "==> Writing pyroscope.yml"
cat > /etc/pyroscope/pyroscope.yml <<EOF
target: all
server:
http_listen_port: 4040
grpc_listen_port: 9097
log_level: info
multitenancy_enabled: false
memberlist:
bind_port: 7946
advertise_addr: ${NODE_IP}
distributor:
ring:
kvstore:
store: memberlist
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
join_after: 0s
querier:
max_concurrent: 16
store_gateway:
sharding_ring:
kvstore:
store: memberlist
replication_factor: 1
compactor:
data_dir: /var/lib/pyroscope/compactor
sharding_ring:
kvstore:
store: memberlist
retention_period: ${RETENTION_HOURS}h
storage:
backend: s3
s3:
endpoint: ${S3_ENDPOINT}
region: ${S3_REGION}
bucket_name: ${S3_BUCKET}
access_key_id: ${S3_ACCESS_KEY}
secret_access_key: ${S3_SECRET_KEY}
insecure: false
limits:
max_global_series_per_tenant: 1000000
ingestion_rate_mb: 100
ingestion_burst_size_mb: 200
analytics:
reporting_enabled: false
EOF
chown -R pyroscope:pyroscope /etc/pyroscope
cat > /etc/systemd/system/pyroscope.service <<'UNIT'
[Unit]
Description=Grafana Pyroscope
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=pyroscope
Group=pyroscope
ExecStart=/usr/local/bin/pyroscope -config.file=/etc/pyroscope/pyroscope.yml
Restart=on-failure
RestartSec=5s
LimitNOFILE=200000
TimeoutStopSec=120
[Install]
WantedBy=multi-user.target
UNIT
systemctl daemon-reload
systemctl enable --now pyroscope
echo "==> NGINX TLS"
cat > /etc/nginx/sites-available/pyroscope <<EOF
upstream pyroscope_http { server 127.0.0.1:4040; keepalive 32; }
server {
listen 80;
server_name ${PUBLIC_DOMAIN};
location /.well-known/acme-challenge/ { root /var/www/html; }
location / { return 301 https://\$host\$request_uri; }
}
server {
listen 443 ssl http2;
server_name ${PUBLIC_DOMAIN};
ssl_certificate /etc/letsencrypt/live/${PUBLIC_DOMAIN}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/${PUBLIC_DOMAIN}/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
client_max_body_size 100m;
proxy_read_timeout 120s;
location / {
proxy_set_header Host \$host;
proxy_pass http://pyroscope_http;
}
}
EOF
ln -sf /etc/nginx/sites-available/pyroscope /etc/nginx/sites-enabled/pyroscope
rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx
echo "==> Waiting for /ready"
for i in {1..60}; do
if curl -fs "http://127.0.0.1:4040/ready" >/dev/null 2>&1; then break; fi
sleep 2
done
echo
echo "========================================"
echo "DONE"
echo "========================================"
echo
ss -lntp | grep -E ':(80|443|4040|9097)\b' || true
echo
echo "Web UI: http://${NODE_IP}:4040"
echo "Ingest: http://${NODE_IP}:4040/ingest"
echo "Run certbot --nginx -d ${PUBLIC_DOMAIN} once DNS is pointed."
Agent: Go application
Go programs ship with net/http/pprof. The Pyroscope Go SDK starts a background goroutine that pulls profiles from itself and pushes to the server every 15 seconds. Add to main.go:
import (
"runtime"
"github.com/grafana/pyroscope-go"
)
func init() {
runtime.SetMutexProfileFraction(5)
runtime.SetBlockProfileRate(5)
pyroscope.Start(pyroscope.Config{
ApplicationName: "checkout-api",
ServerAddress: "https://pyroscope.example.com",
Logger: pyroscope.StandardLogger,
Tags: map[string]string{"env": "prod", "region": "ca-east-1"},
ProfileTypes: []pyroscope.ProfileType{
pyroscope.ProfileCPU,
pyroscope.ProfileAllocObjects,
pyroscope.ProfileAllocSpace,
pyroscope.ProfileInuseObjects,
pyroscope.ProfileInuseSpace,
pyroscope.ProfileGoroutines,
pyroscope.ProfileMutexCount,
pyroscope.ProfileBlockCount,
},
})
}
Agent: Java application (JFR)
Pyroscope’s Java agent uses Java Flight Recorder. Drop the agent JAR on the host and modify the JVM start command:
wget -q -O /opt/pyroscope-agent.jar \
https://github.com/grafana/pyroscope-java/releases/latest/download/pyroscope.jar
# In your service unit, add:
# Environment=PYROSCOPE_APPLICATION_NAME=checkout-api
# Environment=PYROSCOPE_SERVER_ADDRESS=https://pyroscope.example.com
# Environment=PYROSCOPE_FORMAT=jfr
# Environment=PYROSCOPE_PROFILER_EVENT=cpu
# Environment=PYROSCOPE_PROFILER_ALLOC=512k
# Environment=PYROSCOPE_LABELS=env=prod,region=ca-east-1
# ExecStart=/usr/bin/java -javaagent:/opt/pyroscope-agent.jar -jar /opt/app.jar
Agent: Python application
For Django or Flask, install the Python SDK and start it before WSGI initializes:
# app/profiling.py
import pyroscope
pyroscope.configure(
application_name="checkout-api",
server_address="https://pyroscope.example.com",
tags={
"env": "prod",
"region": "ca-east-1",
},
sample_rate=100,
detect_subprocesses=True,
oncpu=True,
gil_only=True,
)
Then in wsgi.py:
import app.profiling # noqa: F401 -- side effect: configure profiler
from app.wsgi import application
In Grafana, add the Pyroscope datasource:
# /etc/grafana/provisioning/datasources/pyroscope.yaml
apiVersion: 1
datasources:
- name: Pyroscope
type: grafana-pyroscope-datasource
access: proxy
url: https://pyroscope.example.com
jsonData:
keepCookies: []
The diff view is the killer feature — pick “last 1h vs. 1h before that”, filter by service=checkout-api, and any new function in the top of the flamegraph is your regression. Watch pyroscope_ingester_appenders_open and pyroscope_storage_blocks_loaded on the server; a sudden spike in open appenders without a corresponding rise in active services often means agents are restarting in a tight loop.
Run continuous profiling on Stack Harbor’s managed cloud compute with retention policy, agent rollout, and Grafana wiring under environment management — or book a scoping call if you want the observability tier delivered with on-call coverage included.