Friday afternoon you failed over a Postgres cluster from pg01 to pg02. The application reconnected, traffic is normal, you went home. Monday morning you start postgres on the old pg01 expecting it to rejoin as a replica. It crashes with requested starting point is ahead of the WAL flush position and the logs are full of record with incorrect prev-link. pg01 has WAL the new cluster never saw — writes that happened in the seconds between the dead-primary detection and the application giving up on it. The cluster moved to a new timeline. pg01 is on the old one. Neither can be a replica of the other without somebody choosing how to resolve the divergence.
What’s happening
Every PostgreSQL primary writes WAL segments tagged with a timeline ID. The first timeline a cluster runs on is 1. Each promotion increments it — a freshly promoted primary writes timeline 2, then 3, then 4 after subsequent failovers.
When a primary is demoted (gracefully or by death) and later wants to rejoin as a standby of the new primary, the standby needs to follow the new timeline’s WAL stream from a point where both diverged. If the old primary committed writes after the divergence point — the gap between when it stopped being the leader and when the cluster declared it dead — those writes exist on the old timeline but not on the new one. The new primary’s WAL doesn’t reference them.
Two options exist:
-
pg_rewind— rewinds the old primary back to the divergence point by reading both WAL streams, identifies which data files were touched after the divergence, and copies only those blocks from the new primary. Fast (minutes for a 1 TB database), but only works if:- The old primary was cleanly shut down (or its WAL is intact and readable).
- WAL exists from before the divergence point on both sides (in archive or on disk).
- The cluster has
wal_log_hints = onordata_checksumsenabled. - Any writes on the old timeline that pg_rewind discards are acceptable losses.
-
pg_basebackup— wipe the old primary’s data directory and rebuild from scratch by streaming a full base backup from the new primary. Always works. Slow (hours for a 1 TB database, gated by network bandwidth and disk write speed). No risk of subtle corruption from a partial rewind.
The decision is rarely “which is faster” — it’s “do I trust pg_rewind for this specific case.” On clusters with frequent failovers (drills, weekly switchovers) pg_rewind is the workhorse. On clusters where the old primary has been off for days or had an ungraceful shutdown, pg_basebackup is the safer default.
How to verify
Check what timeline the new primary is on:
sudo -u postgres psql -h <new-primary> -c "
SELECT pg_walfile_name(pg_current_wal_lsn()) AS current_wal_file,
pg_current_wal_lsn() AS current_lsn;"
# Output like: 000000040000000B000000A1
# ^^^^^^^^ timeline ID = 4
Check what timeline the old primary’s data dir is on:
# On the old primary host (Postgres NOT running yet)
sudo -u postgres /usr/pgsql-16/bin/pg_controldata /var/lib/pgsql/16/data \
| grep -E "TimeLineID|Latest checkpoint|REDO location"
# Output:
# Latest checkpoint's TimeLineID: 3
# Latest checkpoint location: B/9CFA0000
If the timelines differ (here, new=4 and old=3), you have divergence and you need either pg_rewind or pg_basebackup.
Verify the divergence is real by reading the new primary’s timeline-history file:
sudo -u postgres psql -h <new-primary> -c "
SELECT * FROM pg_read_file('pg_wal/00000004.history') AS history;"
# Output like:
# history
# ---------
# 1 0/300000A0 no recovery target specified
# 2 0/5FA00000 no recovery target specified
# 3 B/9CFA0150 no recovery target specified
Each line shows when that timeline ended and what LSN was the divergence point. The line for the old primary’s timeline (here, 3) tells you the LSN at which the new primary forked. If the old primary’s Latest checkpoint location is past that LSN, the old primary has writes the new one doesn’t.
Now check if pg_rewind is viable:
# 1. wal_log_hints (or data_checksums) must be on
sudo -u postgres psql -h <new-primary> -c "SHOW wal_log_hints;"
sudo -u postgres psql -h <new-primary> -c "SHOW data_checksums;"
# At least one must be 'on'
# 2. WAL covering the divergence point must be available on the new primary
sudo -u postgres psql -h <new-primary> -c "
SELECT pg_last_wal_replay_lsn(),
pg_walfile_name(pg_last_wal_replay_lsn());"
# AND check that WAL going back to the divergence LSN still exists in pg_wal/ or restorable from archive
# 3. The old primary must have a clean shutdown OR have its WAL recoverable
sudo -u postgres /usr/pgsql-16/bin/pg_controldata /var/lib/pgsql/16/data \
| grep "Database cluster state"
# Want: "shut down" or "shut down in recovery"
# If it says "in production", mysqld... uh, postgres crashed without writing a clean checkpoint
If any of those checks fails, skip pg_rewind and go straight to pg_basebackup.
The fix
Option A — pg_rewind (fast path)
This is the right choice when:
- The old primary was cleanly shut down (or its WAL is intact).
- WAL covering the divergence point is on both sides.
wal_log_hints = onordata_checksumswas on at initdb.- The cluster’s failover history is short (hours, not days).
Step 1 — Make sure the old primary is fully stopped:
sudo systemctl stop patroni # if Patroni is in the picture
sudo systemctl stop postgresql-16
ps aux | grep postgres # confirm nothing is running
Step 2 — Connect from the old primary to the new primary and run pg_rewind:
# Use a superuser with REPLICATION privilege (the rewind user, or postgres)
# Set the password in PGPASSFILE or in the connection string
export PGPASSWORD='<superuser-password>'
sudo -u postgres /usr/pgsql-16/bin/pg_rewind \
--target-pgdata=/var/lib/pgsql/16/data \
--source-server="host=<new-primary-ip> port=5432 user=postgres dbname=postgres sslmode=require" \
--progress \
--restore-target-wal
The --restore-target-wal flag tells pg_rewind to fetch needed WAL via the restore_command configured on the target — useful when on-disk WAL was rotated away.
Watch the output. You should see:
servers diverged at WAL location 0/B9CFA0150 on timeline 3
rewinding from last common checkpoint at 0/B9CF80E0 on timeline 3
reading source file list
reading target file list
reading WAL in target
need to copy 234 MB (total source directory size is 1248 MB)
234/234 kB (100%) copied
creating backup label and updating control file
syncing target data directory
Done!
If you see target server needs to use either data checksums or wal_log_hints or could not find previous WAL record, pg_rewind cannot proceed. Fall through to Option B.
Step 3 — Configure as a standby and start:
PostgreSQL 12+ uses standby.signal and connection info in postgresql.auto.conf. pg_rewind doesn’t set this up — you need to:
# Create the standby.signal file
sudo -u postgres touch /var/lib/pgsql/16/data/standby.signal
# Add primary_conninfo to postgresql.auto.conf (or use ALTER SYSTEM later)
sudo -u postgres tee -a /var/lib/pgsql/16/data/postgresql.auto.conf <<EOF
primary_conninfo = 'host=<new-primary-ip> port=5432 user=replicator password=<repl-pass> sslmode=require application_name=pg01'
restore_command = 'cp /var/lib/pgsql/16/archive/%f %p'
EOF
If you use Patroni, Patroni will rewrite these files when it starts — you just need to make sure the data directory’s state is correct so Patroni recognizes the node as a valid standby candidate.
Step 4 — Start the service and watch it catch up:
sudo systemctl start patroni # or postgresql-16 if no Patroni
# Watch the log
sudo tail -F /var/lib/pgsql/16/data/log/postgresql-*.log
You should see:
LOG: database system was interrupted; last known up at ...
LOG: entering standby mode
LOG: redo starts at B/9CFA0150
LOG: consistent recovery state reached at B/...
LOG: database system is ready to accept read-only connections
LOG: started streaming WAL from primary at B/... on timeline 4
The key line is started streaming WAL ... on timeline 4. That’s confirmation the rewind worked and the node is now following the new primary.
Confirm from the new primary’s side:
SELECT application_name, state, sync_state,
pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes
FROM pg_stat_replication;
-- pg01 should show state=streaming, lag dropping to 0
Option B — pg_basebackup (safe path)
This is the right choice when:
- The old primary’s data dir is suspect (crash, corruption, partial recovery already attempted).
- WAL coverage of the divergence point is gone.
- The old primary has been off for days and the disk is wildly out of date.
- You don’t have
wal_log_hintsordata_checksumsenabled. - Time-to-rejoin matters less than confidence the rejoin is clean.
Step 1 — Stop and wipe the old data directory:
sudo systemctl stop patroni
sudo systemctl stop postgresql-16
# Move the old data dir aside (don't delete until you're sure the new clone works)
sudo mv /var/lib/pgsql/16/data /var/lib/pgsql/16/data.old.$(date +%F)
# Recreate the empty data dir with the right ownership
sudo -u postgres mkdir -p /var/lib/pgsql/16/data
sudo chmod 700 /var/lib/pgsql/16/data
Step 2 — Stream a base backup from the new primary:
sudo -u postgres /usr/pgsql-16/bin/pg_basebackup \
--host=<new-primary-ip> \
--port=5432 \
--username=replicator \
--pgdata=/var/lib/pgsql/16/data \
--format=plain \
--write-recovery-conf \
--checkpoint=fast \
--progress \
--verbose
--write-recovery-conf makes pg_basebackup create standby.signal and write primary_conninfo into postgresql.auto.conf for you.
On a 1 TB database over 1 Gbps, expect 2-3 hours. On 10 Gbps with NVMe on both ends, 20-30 minutes. The bottleneck is whichever is slower: source disk read, network, or target disk write.
Step 3 — Start the service:
sudo systemctl start patroni
sudo tail -F /var/lib/pgsql/16/data/log/postgresql-*.log
Look for the same started streaming WAL ... on timeline 4 message as in Option A.
Step 4 — Delete the old data directory after verification:
# Confirm the new replica is healthy, streaming, and zero-lag
patronictl -c /etc/patroni/patroni.yml list
# Then reclaim disk
sudo rm -rf /var/lib/pgsql/16/data.old.*
Avoiding the next one
Three things every cluster should have, drawn from this runbook:
wal_log_hints = onat initdb time. Without it, you don’t have the option ofpg_rewind— your only option is the slow path. Cheap insurance.- WAL archiving to durable, network-accessible storage. Object storage (S3, MinIO, Garage) works. Cluster-local disk does not.
pg_rewindandpg_basebackupboth benefit from being able to fetch old WAL on demand. - A documented failback runbook with the decision tree. When you’re staring at “requested starting point is ahead of the WAL flush position” at 3 a.m., you want the answer on a wiki page, not in someone’s head.
For Postgres clusters where you’d rather not pick rewind-vs-basebackup at 3 a.m., we operate the full HA stack on Cloud Compute with WAL archiving, drill cadence, and failback procedure as part of Managed Cloud Ops. Start a scoping call if you’d like us to take the cluster over.