Skip to content

Route 53 health checks and DNS failover: a setup that works when a region does not

How to configure Route 53 health checks and failover routing so traffic actually moves when your primary endpoint dies — including the TTL and alias details that decide whether it works.

DNS failover on Route 53 is three pieces working together: a health check that watches the primary endpoint from outside, a pair of records with the failover routing policy (primary and secondary), and TTLs low enough that resolvers notice the switch. Miss any of the three and you get the classic failure: the health check goes red, the record flips, and traffic keeps hammering the dead primary for an hour because the TTL was 3600.

What’s happening

Route 53 health checkers run from many locations around the world, outside your infrastructure. Each checker probes your endpoint on its own schedule — every 30 seconds at the standard interval, every 10 on the fast interval, which costs more — and the endpoint is considered healthy while more than 18% of checkers report success. That distributed design means one flaky transit path does not fail you over, but it also means the checkers must be able to reach the endpoint: a health check against a private IP or a security group that does not admit the Route 53 checker ranges will sit unhealthy forever.

Three health check types exist. Endpoint checks probe an IP or hostname over HTTP, HTTPS, or TCP, optionally matching a string in the first 5,120 bytes of the response body. Calculated checks combine other checks with boolean logic. CloudWatch alarm checks track the state of an alarm, which is the escape hatch for private resources — the checkers never touch the endpoint; they watch the metric instead.

For alias records there is a fourth mechanism that often replaces an explicit health check entirely: Evaluate target health. An alias to an ALB with that flag inherits the load balancer’s own target health, so if every target behind the ALB fails, Route 53 treats the record as unhealthy without a separate checker.

The failover routing policy then does something simple: answer with the primary record while it is healthy, answer with the secondary when it is not. The secondary can be another region, a static maintenance page on S3/CloudFront, or a reduced-capacity standby.

What to check first

  • Can the checkers reach the endpoint? It must be publicly reachable, and any firewall must admit the published Route 53 health checker IP ranges.
  • Is the health check attached to the record? A health check that exists but is not associated with the primary failover record changes nothing.
  • What is the TTL? Failover speed is health check detection time plus TTL. Sixty seconds is the usual choice for failover records.
  • Does the secondary actually work? A standby nobody has tested is a second outage waiting behind the first.

How to verify

Create the pieces, then interrogate them:

# Health check status as seen by every checker location
aws route53 get-health-check-status --health-check-id 11111111-2222-3333-4444-555555555555

# What Route 53 will answer right now for the failover record
aws route53 test-dns-answer --hosted-zone-id Z0123456789ABC \
  --record-name app.example.com --record-type A

In the console, Route 53 > Health checks shows the check’s status and a Monitoring tab with the health percentage over time; the HealthCheckStatus metric also lands in CloudWatch (us-east-1, regardless of where your workload runs), so you can alarm on it — wire that into the same notification path as the rest of your CloudWatch alarm baseline.

Then run the only test that counts, in a maintenance window: kill the primary (stop the service, or block the checker range at the firewall) and time how long until dig +short app.example.com @8.8.8.8 returns the secondary’s answer. That number — detection plus TTL — is your real failover time.

The runbook

  1. Create the health check against the primary. HTTPS type, the real hostname in the Host name field so SNI and virtual hosts work, a path like /healthz that exercises the application rather than just the TCP port, and string matching on a known body token if the endpoint can return a 200 with an error page.
  2. Point it at something meaningful. A check on / that returns 200 from a CDN cache while the origin is down tells you nothing. Check the layer you intend to fail over.
  3. Create the primary record. Alias to the primary ALB or CloudFront distribution, routing policy Failover, failover record type Primary, and either the health check ID attached or Evaluate target health enabled — one of the two, deliberately chosen.
  4. Create the secondary record. Same name and type, failover record type Secondary. No health check is required on the secondary, but adding one means you find out the standby is broken before the day you need it.
  5. Set TTLs for failover, not for cache efficiency. Sixty seconds on non-alias failover records. Alias records inherit the target’s TTL, which is already short for ALB and CloudFront targets.
  6. Test the flip both ways. Fail over, verify, fail back. Route 53 returns to the primary automatically once its check recovers — make sure the application tolerates that flap, or gate recovery behind a longer failure threshold.

When this is the wrong move

DNS failover is the right tool for region-level or site-level failure, and the wrong tool for anything faster. Resolvers and operating systems cache answers beyond your TTL more often than the standards say they should, so plan for minutes of drain, not seconds. If you need connection-level failover inside one region, that is the load balancer’s job, not DNS. If the database is the thing failing over, DNS is only the last step — the data layer decision comes first, and that is a Multi-AZ versus multi-region design question. And for a workload that cannot tolerate any manual verification before failback, an active-active weighted setup may fit better than primary/secondary.

Common mistakes

  • Health-checking a private endpoint directly instead of using a CloudWatch alarm check, then trusting a check that can never succeed — or never fail.
  • Leaving the default TTL on the failover record, which turns a 90-second detection into an hour-long outage.
  • Pointing the check at a static page that stays up while the application behind it is down.
  • Never rehearsing failover, so the secondary has stale configuration, missing secrets, or an expired certificate when it finally receives traffic.

Prevention

  • Alarm on the health check’s own HealthCheckStatus metric so a red check pages a human even when failover works.
  • Rehearse the failover quarterly and record the measured end-to-end time.
  • Keep the secondary deployed from the same pipeline as the primary so it cannot drift.
  • Review checker reachability after any firewall or WAF change — blocking the checkers is a self-inflicted failover.
  • Document the failback decision: automatic when the check recovers, or manual after verification.

Health check pricing is per check per month, with fast interval, HTTPS, and string matching billed as optional features — the details are in the Route 53 documentation. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.