Multi-AZ subnet design for a real 3-tier VPC — the CIDR plan that survives year three

The first VPC for a 3-tier SaaS gets carved up in an afternoon. Someone picks 10.0.0.0/16, slices it into six /24 subnets — public-a, public-b, private-a, private-b, db-a, db-b — and ships. Two years later the platform has expanded to three AZs because us-east-1c keeps going dark, the application tier needs ECS tasks plus EKS nodes plus the legacy EC2 fleet running in parallel during the EKS migration, and the /24 private subnets are at 240 used IPs out of 251 usable. There is no room left for a green ASG to scale into, and the team is debating whether to rebuild the VPC or fan out to a second one and live with cross-VPC routing. Both options take a quarter. The mistake was the original /24. Here is the layout that does not back you into that corner.

What’s happening

A VPC /16 gives you 65,536 addresses. AWS reserves five per subnet — the first four (network, VPC router, DNS, future use) and the last (broadcast equivalent). A /24 subnet has 256 addresses minus five reserved, so 251 usable. That sounds like a lot until you remember that every ENI on an EC2 instance consumes one, every Fargate task consumes one, every Lambda in a VPC consumes one per concurrent invocation, every interface VPC endpoint consumes one per AZ, every ALB and NLB node consumes one per AZ, and an EKS cluster with the VPC CNI consumes one per pod.

The numbers stack up fast. An EKS node running m6i.2xlarge can host roughly 58 pods on the VPC CNI, each holding a /32. Three nodes per AZ at peak is 174 pod IPs per AZ before you have counted the kubelet or any other workload on that subnet. A /24 does not fit that. You need a /22 minimum for any AZ that runs EKS with the default CNI, and /22 does not leave room for the ALB nodes, the Lambda elastic network interfaces, the interface endpoints, and the future workload you have not designed yet.

The shape of the plan is: pick AZs first, allocate generously per AZ, never re-use the same CIDR in two environments. A /16 VPC split across three AZs at /20 per AZ per tier (public, private, database) is the working default for a SaaS that will hire its first SRE before year three.

How to verify

The worked example for us-east-1, VPC 10.20.0.0/16:

The reserved range matters. When the application team decides they want a fourth subnet tier for ECS-on-EC2 isolated from the EKS subnets, or a fifth tier for the analytics fleet that must not see customer database traffic, those subnets come out of the reserved range and use the same /20 size, in the same AZ order. The pattern is consistent across environments and predictable when you read the routing table.

Route tables follow the tier, not the AZ:

• One public route table, attached to all three public subnets. Default route 0.0.0.0/0 to the internet gateway. Local route for the VPC /16.
• Three private route tables, one per AZ, attached to that AZ’s private subnet. Default route 0.0.0.0/0 to the AZ-local NAT gateway. Local route for the VPC /16. The reason for per-AZ private route tables instead of one shared table is that egress traffic should stay in-AZ — sending us-east-1a egress through a us-east-1b NAT gateway costs cross-AZ data transfer and adds a failure mode for us-east-1b outages.
• Database route table — no internet route, only the local VPC route. Attached to all three database subnets. Database subnets never reach the internet directly.

Inspect with:

aws ec2 describe-subnets \
  --filters Name=vpc-id,Values=vpc-0123456789abcdef0 \
  --query 'Subnets[].[CidrBlock,AvailabilityZone,Tags[?Key==`Name`]|[0].Value]' \
  --output table

aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=vpc-0123456789abcdef0 \
  --query 'RouteTables[].[Tags[?Key==`Name`]|[0].Value,Associations[].SubnetId,Routes[?DestinationCidrBlock==`0.0.0.0/0`].GatewayId]' \
  --output table

The output should show three subnets per tier, the public route table with one entry, and three private route tables each with a different NAT gateway ID.

The fix

The procedure for laying out a new VPC, in order:

1. Pick the CIDR before anything else. Reserve a contiguous /14 or /12 at the org level for AWS VPCs, then carve /16 per VPC. Production, staging, and development each get their own /16 with no overlap, even if they live in different accounts. You will peer them, or attach them to a Transit Gateway, and the moment two environments share a CIDR the peering fails. The convention we hold is 10.0.0.0/8 for AWS, partitioned by account and environment — 10.10.0.0/16 for the shared services account, 10.20.0.0/16 for production, 10.30.0.0/16 for staging, etc.

2. Pick the number of AZs. Three for production. Two for staging or development where the cost of the extra NAT gateway and cross-AZ data transfer outweighs the redundancy. Never one — even a development VPC in one AZ is one AZ outage from a half-day of work lost.

3. Allocate by AZ, then by tier. Use the /20 per-tier-per-AZ default. If the workload is known small (a single low-traffic web service), /22 per tier per AZ is the floor. Below that you are buying back the problem in eighteen months.

4. Tag every subnet with the tier name. Name=prod-private-1a, not Name=subnet-2. The name is what every Terraform module, every IaC review, and every incident pager reads. A subnet named subnet-2 is the one that gets deleted by mistake during cleanup.

5. Reserve /20 blocks for future tiers. Document the reserved range in the VPC’s tag set and in the team’s runbook. The next team to add a subnet tier should not have to do CIDR archaeology.

The anti-pattern we see most: the /24 private subnet that hit 240 used IPs and the team’s response was to create a second VPC and peer it. Now they have two route tables to maintain, two NAT gateway sets to pay for, and the application traffic crosses VPCs for no reason other than the original CIDR choice. The cost of redoing the CIDR plan in year one is a weekend. The cost in year three is a quarter.

The other rule we hold: do not size a subnet based on what you currently run. Size it based on what an EKS cluster, an interface endpoint per service, an ECS Fargate fleet, and a fleet of Lambda concurrencies would consume if you adopted them tomorrow. That number is roughly (nodes × max-pods-per-node) + (services × 3 endpoints) + (lambda-concurrency) + 20% headroom. For a mid-size SaaS the number is in the low thousands, which is why /22 is the floor and /20 is the working default.

For the multi-VPC story once you outgrow a single one, see Transit Gateway hub-and-spoke. For the egress side of the design, see NAT Gateway vs NAT instance and VPC endpoints.

Stack Harbor designs and audits production VPC topologies as part of Managed AWS and ongoing Managed Cloud Ops. If your VPC was carved up before the company hired its first SRE, start with an infrastructure consult.