Skip to content

Disaster recovery drill — the quarterly that tells you if your backups work

Plan and execute a credible DR drill — scope, RTO/RPO targets, the rehearsal script, the failure modes you actually find, and how the report should read.

The backups have been running for six months. The status emails are green. The retention policy looks right. None of this tells you whether you can actually restore the system on the day everything is on fire. A disaster recovery drill is the only honest test — pick a scenario, restore into an isolated environment, and time it. This article is the format we use across client engagements: scope, schedule, the rehearsal script, what counts as success, and the report shape that turns the drill into an action item list rather than a self-congratulatory PDF.

How to verify

Before the drill, confirm the inputs are ready. Half the failed drills fail in this step, not in the restore.

# Are all backup tiers reporting healthy?
restic -r "$RESTIC_REPOSITORY" snapshots --latest 1
borg list "$BORG_REPO" --last 1
aws s3 ls s3://acme-offsite-backups/ --summarize | tail -5
# Is the runbook the team will follow up-to-date?
ls -la docs/runbooks/dr-*
git -C docs log --since='3 months ago' -- runbooks/
# Is the target environment provisioned?
terraform -chdir=infra/dr-target plan -refresh-only

If the latest snapshot is older than the RPO, stop and fix that before drilling. If the runbook hasn’t been touched in three months while the production infra has, the drill will fail on stale assumptions.

What’s happening

A DR drill is a rehearsal. The point is to surface latent failures: missing IAM permissions on the restore role, an undocumented manual step, a backup that decrypts but contains the wrong dataset, a network ACL that prevents the restored host from reaching its dependencies. None of these are detectable by looking at backup completion alerts.

The drill is scoped to a scenario — “primary region unreachable”, “primary database corrupted”, “ransomware on the application host” — and to a target environment. The target should be production-like enough that the result is meaningful and isolated enough that the drill can’t damage prod. In practice this is a dedicated DR account or a separate region, with no inbound paths to or from the live environment.

Two numbers are the contract: RTO (Recovery Time Objective) — how long until service is restored; RPO (Recovery Point Objective) — how much data loss is tolerable. The drill measures both. If the actual recovery takes 2 RTO, the drill failed even if the data is correct. If the most recent snapshot loses 4 RPO of data, the drill failed even if it restored in time.

The procedure

  1. Pre-drill: scope and freeze. A week before, write down the scenario, the target environment, the participants, the RTO/RPO targets, and what counts as success. Freeze runbook changes during this window. Notify the support desk so they don’t react to drill telemetry as a real incident.

  2. T-0: declare the scenario. “Primary region is unreachable as of 09:00.” Start the clock.

  3. Provision the target environment. Terraform stands up the DR account, networks, IAM, and the empty resource skeleton.

    cd infra/dr-target
    terraform apply -var "drill_date=2026-06-02"
  4. Restore the data tier first. Object storage, databases, persistent volumes. Order matters; the application tier cannot come up without its state.

    # restic into a sandbox volume
    restic -r "$RESTIC_REPOSITORY" restore \
      --target /mnt/dr-restore \
      --include /var/lib/postgresql/15/main \
      latest
    
    # then load into the DR postgres
    sudo -u postgres pg_basebackup -D /var/lib/postgresql/15/main \
      -h dr-pg-1 -U replicator -P
  5. Restore the application tier. Container images from the registry, secrets from the secret store, configuration from Git. If any of these need manual edits, that is a finding to record.

  6. Cut over and validate. Update DNS or load balancer weights to point at the DR environment. Run the smoke tests. Confirm the application is serving real traffic shape (synthetic transactions, login, write, read).

    curl -fsS https://dr.example.com/api/health
    ./scripts/smoke-test.sh dr.example.com
  7. Measure. Wall-clock from T-0 to “users could be served” is RTO. The timestamp of the most recent successfully-restored data versus T-0 is RPO. Write both down with the time, not “about an hour”.

  8. Tear down. The DR environment should not exist after the drill — Terraform destroy, S3 bucket emptied, IAM roles archived. A long-lived “DR account that’s already set up” is a security and cost liability.

  9. Write the report within 48 hours. Three sections: (a) actual RTO/RPO vs target, (b) what worked, (c) findings — each with an owner and a due date. The report goes to engineering AND to the customer’s executive sponsor.

Common pitfalls

  • A drill that “passes” because the team improvised around five undocumented steps is not a passing drill; the steps need to be added to the runbook before the next drill.
  • “Restored to a fresh VPC” is not the same as “restored to a VPC that talks to other services” — the network and IAM dependencies are often what break first.
  • DNS TTLs longer than the RTO target make the cutover slower than your engineers; check this before the drill, not during.
  • A drill on a quiet Sunday morning is a different beast than a drill at 14:00 on a Tuesday — vary the time across drills to test team availability.
  • The vendor that backs up your data also needs a DR plan for itself — verify their restore SLA covers your RTO.

In the engagements we run, DR drills are scheduled quarterly per customer environment, the scenarios rotate, and the report is part of the operations review. The runbook is the artifact the customer keeps even if they part with us. See /en/services/managed-operations/ and /en/solutions/multi-region-environments/.