Skip to content

AWS Updated 2026-07-04

RDS unexpected failover: what triggers Multi-AZ failover and how to read the events

An RDS Multi-AZ instance failed over and nobody asked it to. Here is what actually triggers a failover, how to read the RDS event log to find the cause, and why some apps stay broken after it.

An RDS Multi-AZ failover you did not initiate is triggered by a short list of things: the primary host or its storage failed, the Availability Zone had a problem, the OS or instance was patched, the instance class was changed, or the primary became unresponsive long enough for RDS health checks to give up. The authoritative record is the RDS event log — aws rds describe-events — which names the trigger in plain text. The failover itself is the system working as designed; the investigation is about whether the trigger is benign and why your application noticed more than it should have.

What’s happening

In a Multi-AZ DB instance deployment, RDS keeps a synchronously replicated standby in another Availability Zone. When RDS decides the primary is unhealthy — or an operation requires it — it promotes the standby and repoints the instance’s DNS endpoint at it. The endpoint hostname never changes; the IP behind it does. Failover typically completes in about 60-120 seconds for Multi-AZ instance deployments; Multi-AZ DB cluster deployments (the two-readable-standbys flavour) typically fail over faster, usually under 35 seconds.

The documented triggers: an Availability Zone outage or network degradation; primary host hardware failure; storage failure on the primary; the database process crashing or hanging past health-check tolerance (an OS-level out-of-memory kill is a classic — the engine died, RDS failed over, and the cause was memory pressure, not AWS); operations you or automation initiated — instance class modification, OS maintenance, a reboot-with-failover; and manual reboot-db-instance --force-failover.

Two things make a clean failover feel like an outage. First, DNS: clients that cached the old IP — JVMs with default resolver caching, connection pools that never recycle, anything that resolved once at boot — keep hammering a dead address long after the standby is live. Second, writes in flight at the moment of failover are rolled back, and applications without retry logic surface those as user-facing errors.

What to check first

  • The RDS event log — RDS > Events, or the CLI below. The failover events state the reason; this is the single source of truth.
  • When it happened vs your maintenance window — a failover inside the window during a pending-maintenance week is almost certainly patching, not failure.
  • FreeableMemory and SwapUsage before the event — a memory death spiral preceding the failover means the trigger was your workload, and it will happen again.
  • Whether the app recovered on its own — if the database was back in two minutes but errors lasted forty, your real problem is client-side DNS caching or missing retries.
  • AWS Health Dashboard at health.aws.amazon.com for AZ-level events that would explain host or network failures.

How to verify

Pull the events around the incident:

aws rds describe-events \
  --source-identifier prod-db \
  --source-type db-instance \
  --start-time 2026-07-03T00:00:00Z \
  --end-time 2026-07-04T12:00:00Z \
  --query 'Events[].[Date,Message]' --output table

You are looking for pairs like Multi-AZ instance failover started / Multi-AZ instance failover completed, and the messages around them: host replacement, storage failure, “Recovery of the DB instance has started”, or an applied maintenance action. Confirm which AZ is primary now:

aws rds describe-db-instances --db-instance-identifier prod-db \
  --query 'DBInstances[0].[AvailabilityZone,SecondaryAvailabilityZone,MultiAZ]'

Cross-check the memory story in CloudWatch:

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS --metric-name FreeableMemory \
  --dimensions Name=DBInstanceIdentifier,Value=prod-db \
  --start-time 2026-07-03T00:00:00Z --end-time 2026-07-04T00:00:00Z \
  --period 300 --statistics Minimum

The runbook

  1. Read the event log before theorizing. The trigger is usually stated outright. Host replacement and AZ issues are AWS-side and benign in isolation; engine crashes and memory pressure are yours to fix.
  2. Classify the trigger. One-off infrastructure failure: note it and move on — this is precisely what Multi-AZ is for. Maintenance-driven: your window and notification settings need attention, covered in RDS maintenance windows. Workload-driven (OOM, hang): open a performance investigation — see RDS out of memory.
  3. Fix DNS handling in clients. Set JVM networkaddress.cache.ttl to 30 or less, ensure pools validate connections on checkout and recycle on failure, and never resolve the endpoint once at startup. This is the highest-value fix on this page.
  4. Add retry with backoff for writes. Transactions interrupted by failover roll back cleanly; an app that retries idempotently turns a failover into a latency blip instead of an error page.
  5. Subscribe to failover events. aws rds create-event-subscription --subscription-name prod-db-failover --sns-topic-arn arn:aws:sns:ca-central-1:123456789012:ops --source-type db-instance --event-categories failover failure — so the next one pages you before customers do.
  6. Drill it. Run aws rds reboot-db-instance --db-instance-identifier staging-db --force-failover against staging quarterly and measure application recovery time. If staging takes ten minutes to recover from a two-minute failover, production will too.

When this is the wrong move

Do not treat a single failover with a clean AWS-side explanation as a reliability crisis — chasing an AZ hardware replacement with architecture changes is effort spent on a solved problem. Conversely, do not normalize repeated failovers: two or more in a month with “recovery started” messages and no AWS Health correlation means the primary keeps dying, and the cause is almost always in your workload or parameters. And if your recovery-time requirements are stricter than 60-120 seconds of write unavailability, the honest conversation is architecture — Multi-AZ cluster deployments or Aurora — which we compared in Multi-AZ vs Multi-Region. Engine-specific failover behaviour is documented in the Amazon RDS documentation.

Common mistakes

  • Blaming AWS by default and never reading describe-events, where the actual reason has been sitting the whole time.
  • Leaving JVM or OS DNS caching at defaults, turning every 90-second failover into a 30-minute application outage.
  • Testing failover never, then discovering during a real one that the app holds connections open until they rot.
  • Assuming Multi-AZ means zero interruption — it means fast recovery with a brief write outage, and in-flight transactions still roll back.

Prevention

  • Event subscriptions on failover and failure categories wired to the on-call channel.
  • Client-side: short DNS TTLs, connection validation on checkout, retry-with-backoff on writes.
  • Watch FreeableMemory, SwapUsage, and CPU trends — most “mystery” failovers are preceded by weeks of visible pressure.
  • Quarterly forced-failover drills in staging with a measured application recovery target.
  • Keep maintenance windows scheduled in a real off-peak hour so planned failovers land when nobody is watching dashboards in anger.

A Multi-AZ failover is the platform keeping a promise; the event log tells you why it had to, and the client-side fixes decide whether anyone notices next time. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.