Skip to content

AWS Updated 2026-07-04

High availability on AWS: multi-AZ patterns that survive real failures

An AWS high availability setup that works when a zone actually degrades: multi-AZ load balancing, autoscaling with capacity to lose, Multi-AZ databases, NAT per zone, and the failure tests that prove it.

A working AWS high availability setup is one pattern applied consistently: every tier either runs in at least two availability zones with capacity to lose one, or fails over to another zone automatically without a human in the loop. Load balancer across AZs, autoscaling group spanning the same AZs with headroom, Multi-AZ database, one NAT gateway per AZ, and no state trapped on any single instance. Most “highly available” environments fail this test at exactly one tier — and that tier sets the availability of everything above it.

What’s happening

Availability zones are physically separate data center groups within a region, connected by low-latency links; the practical contract is that they fail independently. Multi-AZ design converts “an AZ had a bad day” from an outage into a capacity reduction. But the contract only pays out if every layer honors it, and real AZ events are rarely clean power-offs — they are partial: elevated error rates, one zone’s network degrading, EBS slowdowns in a single AZ. Architectures survive these “gray failures” when each layer can detect and route around a sick zone on its own — load balancer health checks ejecting bad targets, autoscaling replacing failed instances in the surviving zones, the database promoting its standby.

The recurring blind spots are the singletons. RDS without Multi-AZ enabled. A single NAT gateway, which makes every private subnet’s egress depend on one AZ. A cron box or queue worker that exists exactly once. Session state on instance disk, so losing a zone logs out half the users even though “nothing went down.” Each singleton silently caps the whole system’s availability regardless of how redundant everything else is. Multi-AZ also has a modest, real cost: standby database instances bill like primaries, extra NAT gateways bill hourly plus data processing, and cross-AZ data transfer is metered — which is why each singleton usually has a defender pointing at the bill.

What to check first

  • AZ spread of running capacity. Not the diagram — the actual instances. Two AZs on paper and 90% of capacity in one of them is a single-AZ system with extra steps.
  • Database Multi-AZ flag. One field, checked in seconds, and the most common gap.
  • NAT gateway count and placement. One NAT total, or NATs in each AZ but route tables all pointing at one of them — both are the same singleton.
  • State on instances. Sessions, uploads, local caches that matter. If an instance’s disk is precious, the tier is not redundant.
  • The capacity math. Two instances at 80% load each do not survive losing one. N+1 means the surviving zone(s) can carry peak.

How to verify

# Where capacity actually lives, by AZ
aws ec2 describe-instances --filters Name=instance-state-name,Values=running \
  --query 'Reservations[].Instances[].Placement.AvailabilityZone' | sort | uniq -c

# Is the database really Multi-AZ?
aws rds describe-db-instances \
  --query 'DBInstances[].[DBInstanceIdentifier,MultiAZ,AvailabilityZone,SecondaryAvailabilityZone]' \
  --output table

# ASG: zones and healthy capacity
aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].[AutoScalingGroupName,AvailabilityZones,DesiredCapacity,MinSize]'

# Are targets healthy in every zone the LB serves?
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:ca-central-1:123456789012:targetgroup/prod/abc123 \
  --query 'TargetHealthDescriptions[].[Target.Id,Target.AvailabilityZone,TargetHealth.State]' \
  --output table

# One NAT or several, and where
aws ec2 describe-nat-gateways \
  --query 'NatGateways[].[NatGatewayId,SubnetId,State]' --output table

In the console, EC2 > Load Balancers > (select) > Network mapping shows which AZs the load balancer serves, and RDS > Databases > (select) > Configuration shows Multi-AZ state and the standby’s zone. Elastic Load Balancing behavior — including how cross-zone load balancing differs between ALB (on by default) and NLB (off by default at the load balancer level) — is documented at https://docs.aws.amazon.com/elasticloadbalancing/; that default difference matters when one zone is down to a single healthy target.

The approach

  1. Fix the subnet foundation first. Public and private subnets in at least two AZs, symmetric routing per AZ — the layout in multi-AZ subnet design. Everything else builds on this.
  2. Make the database Multi-AZ. For RDS, enabling Multi-AZ is a modification with a brief failover interruption at apply time — schedule it. Failover is automatic and typically completes in tens of seconds to a couple of minutes. When someone asks whether this should be multi-region instead, the answer framework is in Multi-AZ vs multi-region.
  3. Spread and buffer the app tier. ASG across the AZs with MinSize giving N+1, ELB health checks on the application path (not just TCP), and health check settings tuned so ejection is faster than your patience — see health check tuning.
  4. Kill the state singletons. Sessions to Redis or DynamoDB, uploads to S3, caches rebuildable. The test: terminating any one instance mid-day should be a non-event.
  5. One NAT gateway per AZ, with per-AZ routing. Each private subnet’s route table points at the NAT in its own AZ, so a zone failure takes down only its own egress.
  6. Find the hidden singletons. The cron instance, the lone queue worker, the bastion everyone depends on, the self-managed Redis on one box. Each one gets redundancy, failover, or a written risk acceptance.
  7. Prove it with a failure test. In a maintenance window: terminate an instance, force an RDS failover (aws rds reboot-db-instance --db-instance-identifier prod-db --force-failover), and watch what users would see. An untested HA design is a hypothesis.

When this is the wrong move

Full multi-AZ is over-engineering for workloads that tolerate hours of downtime — an internal reporting tool restored from snapshot in two hours may not justify a standby database billing around the clock. Multi-AZ also does nothing for regional failures, bad deploys, data corruption, or deleted tables; those need backups and a DR plan, not more zones. And if the application itself is a monolith that holds state in memory by design, buy the redundancy at the data tier and be honest that the app tier fails over with a restart, not transparently.

Common mistakes

  • Multi-AZ subnets, single-AZ capacity. The VPC spans three zones; the ASG desired count is 1.
  • Health checks that lie. A / returning 200 from a process whose database connection is dead keeps a sick target in service.
  • Two of everything, at 80% load each. Redundancy without headroom fails exactly when it is needed.
  • Testing failover never, or only at launch. Drift breaks failover silently; the first real AZ event becomes the test.

Keeping it under control

  • Alarm on AZ imbalance and on unhealthy-host count per target group, not just on total capacity.
  • Re-run the failure test quarterly: one instance kill, one forced database failover, timed and observed.
  • Gate new services on the singleton question in review: “what happens when the AZ this runs in degrades?”
  • Watch cross-AZ data transfer in the bill; it is the tax that tempts people to quietly re-concentrate into one zone.
  • Keep the capacity math written down: peak load, per-instance capacity, and the N+1 target per tier.

Multi-AZ is the cheapest availability tier AWS offers precisely because the zones already exist — the work is refusing to let any tier opt out of using them. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.