Skip to content

Deploying Concourse CI on Ubuntu 22.04 with PostgreSQL backend and worker pool

Stand up a self-hosted Concourse CI on Ubuntu 22.04 — PostgreSQL backend, worker pool with garden-runC, TSA-signed worker keys, nginx + Let's Encrypt in front.

You inherit a platform team that lives by pipeline-as-code, the kind that wants every step to be a resource, every input declared, every output addressable — and the current CI is a Jenkins instance with 800 jobs nobody can map back to a repo. Concourse is the right shape for that team: explicit pipeline.yml, no plugins, no UI configuration that drifts. You pull the official tarball, run concourse web on one VM with the embedded backend, run concourse worker on the same VM, and the first pipeline ships. Three months later the embedded backend is full, two workers on two different VMs are stalling because the signing keys were regenerated on a redeploy, the auth is the default main team with a shared local password, and pipelines that mount Docker images burn through disk because the volume garbage collector never runs. A serious Concourse deployment is a different layout.

What’s happening

A production Concourse plane is three roles on three sets of hosts: a web node (HTTPS API, pipeline scheduler, auth, TSA bastion for workers), a PostgreSQL backend (separate VM or managed database, never embedded), and a pool of worker nodes (run garden-runC containers, hold ephemeral state). The web node signs worker registration with two key pairs — session_signing_key (signs ATC tokens) and tsa_host_key (TSA SSH host key the workers verify) — and accepts worker keys whose public counterpart is in authorized_worker_keys. Regenerating any of these keys breaks every worker that’s already attached. Auth is never the local user/password in production; wire it to your IdP via OIDC, GitHub team, or LDAP from day one. Worker state lives under /var/concourse/worker and grows fast — a docker-image resource caches every image layer, and without the volume GC the disk fills in days. Pipelines need explicit --paused defaults and resource check rate-limits; otherwise a misconfigured webhook resource hammers the upstream every two seconds.

How to verify

Web node should expose only :22, :80, :443, and TSA :2222:

ss -lntp | grep -E ':(22|80|443|2222|8080|5432)\b'
# 0.0.0.0:22   sshd
# 0.0.0.0:443  nginx
# 0.0.0.0:2222 concourse-tsa
# 127.0.0.1:8080 concourse-atc (behind nginx)

The fly CLI must authenticate against the web URL and show the workers:

fly -t prod login -c https://ci.example.com -u opsadmin
fly -t prod workers
# name        containers  platform  tags  team  state    version
# worker-01   3           linux     none  none  running  7.11.x
# worker-02   2           linux     none  none  running  7.11.x

Each worker should report a healthy garden daemon:

# On a worker host:
journalctl -u concourse-worker -n 50 | grep -i 'garden\|registered'
# garden-server.start
# worker.beacon.forward-conn.dial.start

If fly workers shows zero workers but journalctl on the worker shows key signature: invalid signature, the web node’s tsa_host_key or authorized_worker_keys was regenerated after the worker first registered — fix the key set and restart the worker.

The fix

This deployment uses two Ubuntu 22.04 hosts: one web + backend and one or more workers. The PostgreSQL backend can move to a managed database later. Save the web installer as /root/setup-concourse-web-ubuntu22.sh on the web host, the worker installer as /root/setup-concourse-worker-ubuntu22.sh on each worker. Both run as root.

Web installer

#!/usr/bin/env bash
set -euo pipefail

### Ubuntu 22.04 Concourse web + PostgreSQL clean VM installer
### nginx + Let's Encrypt; PostgreSQL local on internal port
### Generates TSA host key + session signing key + worker key pair
### Stores keys under /etc/concourse/keys — back these up before going live
### Local user auth disabled; configure your IdP before exposing /api/v1
### No local backups; assumes VM-level backup

if [[ "$(id -u)" -ne 0 ]]; then
  echo "Run as root."
  exit 1
fi

: "${CONCOURSE_DOMAIN:?Set CONCOURSE_DOMAIN=ci.example.com}"
: "${CONCOURSE_EMAIL:?Set CONCOURSE_EMAIL=ops@example.com for Let's Encrypt}"
: "${CONCOURSE_LOCAL_USER:?Set CONCOURSE_LOCAL_USER=opsadmin (bootstrap admin)}"
CONCOURSE_VERSION="${CONCOURSE_VERSION:-7.11.2}"
CONCOURSE_PG_PASSWORD="${CONCOURSE_PG_PASSWORD:-$(openssl rand -hex 24)}"
CONCOURSE_LOCAL_PASSWORD="${CONCOURSE_LOCAL_PASSWORD:-$(openssl rand -hex 16)}"

echo
echo "========================================"
echo "Concourse web Ubuntu 22.04 Setup"
echo "========================================"
echo

echo "==> Updating base OS"
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y upgrade

apt-get -y install \
  vim curl wget ca-certificates gnupg2 \
  chrony fail2ban unattended-upgrades \
  ufw lsof tcpdump dnsutils tar gzip \
  iproute2 net-tools openssl jq \
  software-properties-common

systemctl enable --now chrony

echo "==> Kernel/network tuning"
cat > /etc/sysctl.d/99-concourse.conf <<'SYSCTL'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_local_port_range = 10240 65535
kernel.dmesg_restrict = 1
fs.file-max = 2097152
vm.max_map_count = 262144
SYSCTL
sysctl --system

echo "==> UFW"
ufw --force reset
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 2222/tcp
ufw --force enable

echo "==> Hardening SSH"
sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart ssh

cat > /etc/fail2ban/jail.d/sshd.local <<'FAIL2BAN'
[sshd]
enabled = true
port = ssh
backend = systemd
maxretry = 4
findtime = 10m
bantime = 1h
FAIL2BAN
systemctl enable --now fail2ban

cat > /etc/apt/apt.conf.d/20auto-upgrades <<'AUTOUPG'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";
AUTOUPG

echo "==> Installing PostgreSQL 16"
install -d /usr/share/postgresql-common/pgdg
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
  | gpg --dearmor -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.gpg
echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.gpg] https://apt.postgresql.org/pub/repos/apt jammy-pgdg main" \
  > /etc/apt/sources.list.d/pgdg.list
apt-get update
apt-get -y install postgresql-16

systemctl enable --now postgresql

sudo -u postgres psql -v ON_ERROR_STOP=1 <<SQL
CREATE ROLE concourse WITH LOGIN PASSWORD '${CONCOURSE_PG_PASSWORD}';
CREATE DATABASE concourse OWNER concourse;
SQL

# Keep Postgres local; no listen on public interface
sed -i "s|^#listen_addresses.*|listen_addresses = 'localhost'|" /etc/postgresql/16/main/postgresql.conf
sed -i "s|^listen_addresses.*|listen_addresses = 'localhost'|" /etc/postgresql/16/main/postgresql.conf
systemctl restart postgresql

echo "==> Installing Concourse"
useradd --system --no-create-home --shell /usr/sbin/nologin concourse || true
cd /tmp
curl -fsSL -o concourse.tgz "https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/concourse-${CONCOURSE_VERSION}-linux-amd64.tgz"
tar -xzf concourse.tgz -C /opt
ln -sf /opt/concourse/bin/concourse /usr/local/bin/concourse
ln -sf /opt/concourse/fly-assets/fly-linux-amd64 /usr/local/bin/fly || true

echo "==> Generating signing keys"
mkdir -p /etc/concourse/keys/web
chown -R concourse:concourse /etc/concourse

ssh-keygen -t rsa -b 4096 -f /etc/concourse/keys/web/session_signing_key -N '' -q
ssh-keygen -t rsa -b 4096 -f /etc/concourse/keys/web/tsa_host_key -N '' -q
ssh-keygen -t rsa -b 4096 -f /etc/concourse/keys/web/worker_key -N '' -q

cp /etc/concourse/keys/web/worker_key.pub /etc/concourse/keys/web/authorized_worker_keys

chown -R concourse:concourse /etc/concourse
chmod 600 /etc/concourse/keys/web/session_signing_key /etc/concourse/keys/web/tsa_host_key /etc/concourse/keys/web/worker_key

echo "==> Writing /etc/default/concourse-web"
cat > /etc/default/concourse-web <<ENV
CONCOURSE_EXTERNAL_URL=https://${CONCOURSE_DOMAIN}
CONCOURSE_BIND_IP=127.0.0.1
CONCOURSE_BIND_PORT=8080
CONCOURSE_TSA_BIND_PORT=2222

CONCOURSE_POSTGRES_HOST=127.0.0.1
CONCOURSE_POSTGRES_PORT=5432
CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=${CONCOURSE_PG_PASSWORD}
CONCOURSE_POSTGRES_DATABASE=concourse
CONCOURSE_POSTGRES_SSLMODE=disable

CONCOURSE_SESSION_SIGNING_KEY=/etc/concourse/keys/web/session_signing_key
CONCOURSE_TSA_HOST_KEY=/etc/concourse/keys/web/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/etc/concourse/keys/web/authorized_worker_keys

CONCOURSE_ADD_LOCAL_USER=${CONCOURSE_LOCAL_USER}:${CONCOURSE_LOCAL_PASSWORD}
CONCOURSE_MAIN_TEAM_LOCAL_USER=${CONCOURSE_LOCAL_USER}

CONCOURSE_LOG_LEVEL=info
CONCOURSE_GC_INTERVAL=30s
CONCOURSE_GC_ONE_OFF_GRACE_PERIOD=5m
CONCOURSE_GC_FAILED_GRACE_PERIOD=1h
CONCOURSE_GC_HIJACK_GRACE_PERIOD=5m

CONCOURSE_DEFAULT_TASK_CPU_LIMIT=1024
CONCOURSE_DEFAULT_TASK_MEMORY_LIMIT=2GB

CONCOURSE_CLUSTER_NAME=production
CONCOURSE_ENABLE_GLOBAL_RESOURCES=true
CONCOURSE_ENABLE_PIPELINE_INSTANCES=true
ENV
chmod 600 /etc/default/concourse-web

cat > /etc/systemd/system/concourse-web.service <<'UNIT'
[Unit]
Description=Concourse CI web node
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=concourse
Group=concourse
EnvironmentFile=/etc/default/concourse-web
ExecStart=/usr/local/bin/concourse web
Restart=on-failure
LimitNOFILE=200000
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target
UNIT

systemctl daemon-reload
systemctl enable --now concourse-web

echo "==> Waiting for web on 127.0.0.1:8080"
for i in {1..120}; do
  if ss -lntp | grep -q '127.0.0.1:8080'; then break; fi
  sleep 2
done

echo "==> Installing nginx and certbot"
apt-get -y install nginx certbot python3-certbot-nginx

cat > /etc/nginx/sites-available/concourse <<NGINX
upstream concourse_backend {
    server 127.0.0.1:8080;
    keepalive 32;
}

server {
    listen 80;
    listen [::]:80;
    server_name ${CONCOURSE_DOMAIN};
    return 301 https://\$host\$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name ${CONCOURSE_DOMAIN};

    client_max_body_size 100M;
    proxy_read_timeout   600s;
    proxy_send_timeout   600s;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-Frame-Options SAMEORIGIN always;
    add_header Referrer-Policy strict-origin-when-cross-origin always;

    location / {
        proxy_set_header Host \$host;
        proxy_set_header X-Real-IP \$remote_addr;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto \$scheme;
        proxy_http_version 1.1;
        proxy_pass http://concourse_backend;
    }
}
NGINX

ln -sf /etc/nginx/sites-available/concourse /etc/nginx/sites-enabled/concourse
rm -f /etc/nginx/sites-enabled/default

nginx -t
systemctl enable --now nginx

certbot --nginx --non-interactive --agree-tos \
  -m "${CONCOURSE_EMAIL}" -d "${CONCOURSE_DOMAIN}" --redirect

systemctl enable --now certbot.timer

echo
echo "========================================"
echo "DONE — web"
echo "========================================"
echo
echo "Listeners:"
ss -lntp | grep -E ':(22|80|443|2222|8080)\b' || true
echo
echo "Local bootstrap admin: ${CONCOURSE_LOCAL_USER}"
echo "Bootstrap password:    ${CONCOURSE_LOCAL_PASSWORD}"
echo
echo "Worker key — distribute the PRIVATE half to each worker as /etc/concourse/keys/worker/worker_key"
echo "TSA host PUBLIC key — distribute to each worker as /etc/concourse/keys/worker/tsa_host_key.pub"
echo
echo "Web URL: https://${CONCOURSE_DOMAIN}/"
echo "fly login: fly -t prod login -c https://${CONCOURSE_DOMAIN} -u ${CONCOURSE_LOCAL_USER}"
echo
echo "NEXT: wire your IdP via CONCOURSE_OIDC_* or CONCOURSE_GITHUB_* in /etc/default/concourse-web, then disable local user."

Worker installer

Each worker needs the private half of the worker key (generated on the web node at /etc/concourse/keys/web/worker_key) and the public half of the TSA host key (generated at /etc/concourse/keys/web/tsa_host_key.pub). Copy these to the worker before running the script.

#!/usr/bin/env bash
set -euo pipefail

if [[ "$(id -u)" -ne 0 ]]; then
  echo "Run as root."
  exit 1
fi

: "${CONCOURSE_TSA_HOST:?Set CONCOURSE_TSA_HOST=ci.example.com:2222}"
: "${CONCOURSE_WORKER_NAME:?Set CONCOURSE_WORKER_NAME=worker-01}"
CONCOURSE_VERSION="${CONCOURSE_VERSION:-7.11.2}"

echo "==> Updating OS"
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y upgrade
apt-get -y install vim curl wget ca-certificates gnupg2 chrony fail2ban \
  unattended-upgrades ufw lsof iproute2 net-tools openssl jq

systemctl enable --now chrony

ufw --force reset
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw --force enable

sed -i 's|^[#[:space:]]*PermitRootLogin.*|PermitRootLogin prohibit-password|' /etc/ssh/sshd_config
sed -i 's|^[#[:space:]]*PasswordAuthentication.*|PasswordAuthentication no|' /etc/ssh/sshd_config
sshd -t && systemctl restart ssh

# Concourse worker needs cgroups v2 + iptables + apparmor disabled for garden-runC
apt-get -y install iptables btrfs-progs

# Disable apparmor on the worker only (garden-runC + apparmor on Ubuntu 22.04 = signal 11)
systemctl disable --now apparmor || true

useradd --system --no-create-home --shell /usr/sbin/nologin concourse || true

cd /tmp
curl -fsSL -o concourse.tgz "https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/concourse-${CONCOURSE_VERSION}-linux-amd64.tgz"
tar -xzf concourse.tgz -C /opt
ln -sf /opt/concourse/bin/concourse /usr/local/bin/concourse

mkdir -p /var/lib/concourse/worker /etc/concourse/keys/worker
chown -R concourse:concourse /var/lib/concourse /etc/concourse

# Sanity: keys must already be in /etc/concourse/keys/worker
for k in worker_key tsa_host_key.pub; do
  if [[ ! -s /etc/concourse/keys/worker/${k} ]]; then
    echo "Missing /etc/concourse/keys/worker/${k} — copy from the web host before running this script."
    exit 1
  fi
done
chmod 600 /etc/concourse/keys/worker/worker_key

cat > /etc/default/concourse-worker <<ENV
CONCOURSE_TSA_HOST=${CONCOURSE_TSA_HOST}
CONCOURSE_NAME=${CONCOURSE_WORKER_NAME}
CONCOURSE_WORK_DIR=/var/lib/concourse/worker
CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/keys/worker/tsa_host_key.pub
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/keys/worker/worker_key
CONCOURSE_RUNTIME=containerd
CONCOURSE_LOG_LEVEL=info
CONCOURSE_BAGGAGECLAIM_DRIVER=overlay
ENV
chmod 600 /etc/default/concourse-worker

cat > /etc/systemd/system/concourse-worker.service <<'UNIT'
[Unit]
Description=Concourse CI worker
After=network.target

[Service]
Type=simple
User=root
EnvironmentFile=/etc/default/concourse-worker
ExecStart=/usr/local/bin/concourse worker
Restart=on-failure
LimitNOFILE=200000
TimeoutStopSec=60
KillMode=process

[Install]
WantedBy=multi-user.target
UNIT

systemctl daemon-reload
systemctl enable --now concourse-worker

echo
echo "========================================"
echo "DONE — worker ${CONCOURSE_WORKER_NAME}"
echo "========================================"
echo
echo "Recent logs:"
journalctl -u concourse-worker -n 30 --no-pager
echo
echo "Confirm registration on the web node:"
echo "  fly -t prod workers"

Make both executable and run:

# On the web host:
chmod 700 /root/setup-concourse-web-ubuntu22.sh
CONCOURSE_DOMAIN=ci.example.com \
CONCOURSE_EMAIL=ops@example.com \
CONCOURSE_LOCAL_USER=opsadmin \
/root/setup-concourse-web-ubuntu22.sh

# Copy the keys to each worker host BEFORE running the worker installer:
scp /etc/concourse/keys/web/worker_key worker-01:/etc/concourse/keys/worker/worker_key
scp /etc/concourse/keys/web/tsa_host_key.pub worker-01:/etc/concourse/keys/worker/tsa_host_key.pub

# On each worker host:
chmod 700 /root/setup-concourse-worker-ubuntu22.sh
CONCOURSE_TSA_HOST=ci.example.com:2222 \
CONCOURSE_WORKER_NAME=worker-01 \
/root/setup-concourse-worker-ubuntu22.sh

Once both planes are up, log in with the bootstrap user printed by the web installer, run fly -t prod workers from your laptop, and confirm each worker reports running. Wire your IdP next — set CONCOURSE_OIDC_ISSUER, CONCOURSE_OIDC_CLIENT_ID, CONCOURSE_OIDC_CLIENT_SECRET, and CONCOURSE_MAIN_TEAM_OIDC_USER in /etc/default/concourse-web, restart the unit, log in via your IdP, and then remove the CONCOURSE_ADD_LOCAL_USER and CONCOURSE_MAIN_TEAM_LOCAL_USER lines. Pipelines should default to paused: true until reviewed; resources with webhooks must set check_every: 24h so polling doesn’t replace webhook delivery. Back up /etc/concourse/keys/web/ to your secrets manager — losing those keys means every worker has to be re-registered manually.

Run the CI plane on Stack Harbor’s managed cloud compute with patching, monitoring, and Sev-1 escalation in place, and let our deployment management team wire IdP, worker autoscaling, and pipeline failure routing. If you’d like to plan a Jenkins-to-Concourse cutover, start the conversation.