A boardroom slide promises “RTO 1 hour, RPO 5 minutes” for a platform whose last full restore drill took six hours. The numbers were chosen by the executive who wrote the slide, not by the engineers who would have to deliver them, and not by the finance function that would have to fund the architecture making them true. This article frames RTO and RPO as a trade-off, not a wishlist — what each cut costs, what the realistic targets look like at each spend tier, and how to negotiate from the architecture upward rather than from the slide downward.
How to verify
# Time the actual restore for a representative dataset
time pg_restore --jobs=4 -d test_restore /backups/prod-snapshot.dump
# Measure incremental snapshot lag (RPO upper bound)
aws ec2 describe-snapshots --filters Name=volume-id,Values=vol-xxx \
--query 'Snapshots[*].[StartTime]' --output text | sort | tail -2
# Pilot light → warm standby promotion drill timer
time ./scripts/promote-dr-region.sh us-east-2
# Sanity check on backup cadence vs declared RPO
ls -la /backups/db/ | tail -10
If time on the restore exceeds the declared RTO by 50% or more, the RTO is fiction until either the restore process gets faster or the target relaxes. If snapshot lag drifts above the declared RPO during normal operation, the cadence is wrong.
What’s happening
RTO (Recovery Time Objective) and RPO (Recovery Point Objective) measure two different costs:
- RTO is the duration of downtime acceptable — measured forward from incident declaration to service restored. It is a property of tooling, runbook quality, and drill-derived muscle memory.
- RPO is the volume of data loss acceptable — measured backward from incident to the last consistent recoverable state. It is a property of backup cadence, replication, and write durability.
Cutting either number costs money, and the curve is non-linear. Going from RPO 24h to RPO 1h is a different conversation from going from RPO 5m to RPO 30s; the first is “schedule more snapshots”, the second is “stream WAL to a hot replica continuously and pay for that bandwidth and the standby compute forever.” The same is true for RTO: going from “best effort overnight” to “4h” is a process and runbook conversation; going from “4h” to “15min” is a hot-standby architecture conversation.
The four canonical recovery shapes — backup-and-restore, pilot light, warm standby, hot standby (or multi-region active-active) — map onto rough RPO/RTO bands. The trick is that the bands overlap, and you can move within a band by tooling improvements alone. You only need to climb to a more expensive band when the target falls below the floor of your current band.
The decision
-
Define the business cost of each hour of downtime and each unit of lost data, separately. The finance team owns these numbers. “We lose roughly X per hour” and “Each lost transaction costs us Y plus reputational impact.” Without these, the conversation has no anchor and the targets will get squeezed against zero by whoever shouts loudest.
-
Read the targets off the cost curve, not off ambition. If downtime costs roughly modest amounts per hour, an RTO of 4 hours may be fine; the engineering and recurring cost of cutting it to 15 minutes outweighs the expected loss. If a single hour of downtime is catastrophic, the math reverses.
-
Pick the recovery shape that places the target in the middle of its band, not at its floor. Operating at the floor of a band means every restore drill is a near-miss and any process drift pushes you out of compliance. The bands look roughly like this:
- Backup-and-restore. RPO hours to a day, RTO hours to days. Cheapest. Suitable for cold environments, low-volume internal apps, or systems that can tolerate a full rebuild from infrastructure-as-code plus latest backup.
- Pilot light. RPO minutes to an hour, RTO 1-4 hours. A minimal copy of the architecture runs in the DR region with data replicated; compute scales up on declaration. Cost: data replication + a small standing footprint.
- Warm standby. RPO seconds to minutes, RTO 15-60 minutes. A scaled-down but functional copy of the system runs continuously, ready to take traffic with a capacity scale-up. Cost: half-ish the production footprint, always running.
- Hot standby / active-active. RPO near-zero, RTO seconds. Two regions take traffic simultaneously with synchronous-ish replication. Cost: full duplicate footprint plus the consistency complexity of distributed writes.
-
Negotiate target asymmetry. RTO and RPO rarely need the same architectural shape. A read-heavy CMS may tolerate RTO 1h but require RPO 1m (lost edits are visible to authors). A batch ETL system may tolerate RPO 4h (rerun the batch) but require RTO 15m (downstream consumers expect fresh data on schedule). Build the architecture for whichever target is tighter; the looser one comes free.
-
Validate the chosen numbers with a real drill. The numbers on the slide are credible only after a quarterly DR drill produces an MTTR (Mean Time To Recovery) less than the RTO. Tracking MTTR over time tells you whether process drift is eroding the margin. If MTTR creeps up over consecutive drills, you have an operational debt building and the next real incident will exceed the RTO.
Common pitfalls
- Confusing “we have backups” with “we can restore in N hours.” A backup that has never been restored to a fresh system is a hope, not a recovery plan. The restore time is the relevant number, not the backup completion time.
- Promising the same RTO across all services. A frontend stateless tier and a sharded database tier have very different recovery shapes. Lumping them into one number forces overspend on the easy services or underspec on the hard ones.
- Setting RPO equal to backup cadence and stopping there. RPO is measured from incident time, not from backup time. A daily backup at 02:00 implies RPO up to 24h if the incident strikes at 01:55. Adjust the cadence (or add log shipping) to match the declared RPO at worst-case alignment.
- Treating “multi-AZ” as DR. A multi-AZ deployment protects against single-AZ failure within a region. A regional event (control plane outage, networking event, account compromise) takes down all AZs. If RTO/RPO need to survive regional events, the DR shape must cross regions.
- Picking the architecture before the targets. Engineering chooses warm standby because “it feels right,” then the business is told they have RTO 30 minutes. If the business doesn’t need 30 minutes, the warm standby cost is overspend. Targets first, shape second.
Stack Harbor runs the quarterly drill, measures MTTR, and reports the gap between declared RTO/RPO and observed reality — so the architecture stays sized to what the business actually needs, not to a number on a slide. See clustered environments for the operating model behind warm-standby and active-active builds.