This guide explains what happens during an automatic failover in Amazon ElastiCache for Redis, and it is written for engineers who run Redis there in production. ElastiCache is the managed Redis service from AWS. With Multi-AZ turned on, it keeps a replica (a live copy of your data) in a second availability zone, which is a separate data center in the same region. When the primary node fails, ElastiCache promotes that replica to take its place.
Picture a real night. At 03:14, the primary in us-east-1b has a hardware failure, and automatic failover promotes the replica in us-east-1c. The alert clears before the on-call engineer is fully awake. The next morning, the dashboard shows a 12-second blip where about 0.4% of requests returned a 500 error. The team calls it a successful failover and moves on. Those 12 seconds held rejected writes, client reconnects and a change of primary. The sections below walk through each step, how your clients should respond, why failover sometimes does not trigger even when the primary is degraded, and what to do when that happens.
What’s happening
ElastiCache’s automatic failover is a managed wrapper around Redis Sentinel-style health checks plus AWS infrastructure events. The flow looks like this.
Health detection (1–3 seconds). ElastiCache continuously monitors every node via the management plane. Failure signals include: TCP connection refused, response timeout on PING, sustained CPU near 100%, or an EC2-level event indicating the underlying host has failed. The default detection threshold is conservative — a single missed ping does not trigger failover. A handful of consecutive failures or a clear infrastructure signal does.
Promotion decision (1–2 seconds). Once the primary is judged dead, ElastiCache picks a replica to promote. The decision considers replication lag (the most-current replica wins), AZ placement (prefers staying multi-AZ), and node health. If the cluster has Multi-AZ enabled with automatic failover, the promotion is fully automatic. If it does not, the cluster stalls in a degraded state until manual intervention.
DNS update (1–5 seconds). The cluster’s primary endpoint is a Route 53 record pointing at the current primary. On promotion, ElastiCache updates the DNS record to point at the new primary. TTL is short (typically 5–10 seconds), but every client must respect it. Clients with long-lived cached DNS will keep hitting the old endpoint.
Replica reconfiguration (5–10 seconds). The remaining replicas (and the demoted old primary if it comes back) must reset replication and point at the new primary. During this window the cluster is multi-primary-ish in appearance to clients that connect to specific nodes — the old primary may respond READONLY to writes, the new primary accepts them, and replicas may have a brief replication catch-up before they serve consistent reads.
Client reconnection (variable, 0–30 seconds). Each client library handles the failover differently. Some detect the disconnect immediately and reconnect via the primary endpoint DNS, which now resolves to the new node. Some hold the dead TCP connection until the OS times it out, which is much longer. Clients that resolved the primary endpoint once at startup and cached the IP will never see the new primary without an explicit refresh.
The total observable downtime is dominated by the longest of these phases. With healthy clients and Multi-AZ enabled, 5–15 seconds is typical. With clients that cache DNS or do not retry, the same failover can look like a 60-second outage from a single app server’s perspective even though the cluster recovered in 8 seconds.
The rare case where automatic failover does not trigger: ElastiCache distinguishes “hard” failures (the node is unreachable, the host is down) from “soft” issues (the node responds slowly, returns errors but is reachable). A read-only event on the primary — for example, an out-of-memory condition triggering eviction storms but the node is still responding to TCP — may not trigger failover even when the primary is effectively useless. The replica is healthy, but the cluster never promotes it because the primary technically still answers PING.
How to verify
Three observability checks tell you what the cluster did and how the clients handled it.
CloudWatch events around the failover. ElastiCache publishes events for every state transition.
aws elasticache describe-events \
--source-identifier saas-prod-001 \
--source-type cache-cluster \
--start-time $(date -u -v-1d +%Y-%m-%dT%H:%M:%S) \
--duration 60 \
--max-records 50The events you want to see in a successful failover: Replication group ... primary location changed from <az> to <az>, Failover from primary node ... completed. Time between the first event and the completion event is the cluster-side recovery time.
Per-client error rate during the window. From the app side, what matters is the request error rate during the failover window. Filter access logs for the failover window and count 5xx responses by upstream error code.
grep "23:14:" /var/log/app/access.log \
| awk '$9 >= 500 { print $9, $NF }' \
| sort | uniq -c | sort -rnA healthy distribution shows a brief spike of 502 or 504 from the load balancer, peaking for a few seconds and clearing. A bad distribution shows the error rate not clearing for 30+ seconds, which means clients are not reconnecting — that is a client-config problem, not an ElastiCache problem.
Replication lag right before the failover. If automatic failover was forced by a degraded primary rather than a hard failure, the promoted replica may have been seconds behind. Any writes accepted in those final seconds are lost.
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name ReplicationLag \
--dimensions Name=CacheClusterId,Value=saas-prod-002 \
--start-time $(date -u -v-1h +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 60 \
--statistics Maximum \
--query 'Datapoints[?Maximum > `1.0`]' \
--output textReplication lag should normally be measured in milliseconds (under 1 second). Sustained lag of several seconds points to write throughput saturating replication or a network constraint between AZs. Lag matters most at the moment of failover.
For planned testing, ElastiCache exposes a TestFailover API. This is what you use to exercise the failover path in staging or to validate that production clients handle it.
aws elasticache test-failover \
--replication-group-id saas-prod \
--node-group-id 0001The call is rate-limited — you can trigger it once per node group per five minutes, and only five times per node group per 24 hours. The call returns immediately; the actual failover takes the usual 5–15 seconds. Watching error rates during the call gives you a real measurement of how the clients behave, not a simulation.
The fix
When automatic failover did its job but the clients did not catch up smoothly, the fix is on the client side, not the cluster.
Set short DNS TTL respect on the client. Java apps with the default JVM DNS cache of “forever” will not see the new primary. Set networkaddress.cache.ttl to 30 or 60 seconds in java.security, or as a JVM flag:
-Dsun.net.inetaddr.ttl=30Node and Python clients generally respect the OS DNS resolver, which respects record TTL. But long-lived connections established before the TTL expires will not move until the connection drops. Force the drop with the client retry strategy.
Configure client retry with READONLY handling. The pattern in the session-store article applies here: ioredis’s reconnectOnError returning true on READONLY makes the client drop and reconnect. phpredis benefits from a ping health check in the pool reset path. lettuce (Java) handles this natively via ClusterTopologyRefreshOptions.
Drop connections aggressively on idle. Long-lived TCP connections survive failover only if the OS detects the dead remote and sends FIN/RST. TCP keepalive on a 60-second interval makes this happen within roughly 60 seconds. Without keepalive, a dead connection can persist for hours from the OS perspective.
new Redis({
// ... other options
family: 4,
keepAlive: 60000,
});When automatic failover does not trigger but the cluster is degraded, the response is manual.
Manually invoke failover with TestFailover. This is the same API used for planned tests. If the primary is responding to PING but actually misbehaving (eviction storm, high latency, memory exhausted with noeviction), the call promotes the healthier replica.
Check Multi-AZ is actually enabled. A surprising number of “automatic failover did not happen” tickets resolve to “automatic failover was disabled in the cluster config.” Single-AZ clusters do not failover automatically. Production clusters should always have Multi-AZ enabled with automatic failover on.
aws elasticache describe-replication-groups \
--replication-group-id saas-prod \
--query 'ReplicationGroups[].[ReplicationGroupId,AutomaticFailover,MultiAZ]' \
--output tableBoth AutomaticFailover and MultiAZ should read enabled.
The frame to carry: a failover is not a “thing that happens instantly.” It is a 5–15-second state transition in which clients must cooperate. The cluster’s job is to detect, promote, and update DNS. The client’s job is to drop dead connections, retry on READONLY, and respect short DNS TTL. Most “ElastiCache had an outage” tickets are really “the client did not handle a 10-second failover gracefully” tickets. Fix the client behavior, and the cluster’s natural recovery time becomes the operational reality.
Stack Harbor designs Redis topologies as part of cluster-tier work in Managed AWS and the HA Clusters operating layer. If a primary degradation never triggered failover and your team had to step in, start with an infrastructure consult.