Skip to content

AWS Updated 2026-07-04

NAT Gateway errors and port allocation exhaustion: symptoms and fixes

Intermittent outbound timeouts through a NAT Gateway usually mean ErrorPortAllocation: too many concurrent connections to the same destination. How to confirm it and the four ways to fix it.

When outbound connections through a NAT Gateway start failing intermittently under load — timeouts, not refusals, often to one specific API or database — the first metric to pull is ErrorPortAllocation. A NAT Gateway can hold roughly 55,000 simultaneous connections to each unique destination (IP, port, and protocol combination); past that, new connections to that destination get no source port and die silently. The fixes are more source IPs on the gateway, more gateways, fewer idle connections, or removing the traffic from NAT entirely with VPC endpoints.

What’s happening

A NAT Gateway rewrites each outbound connection to come from its own Elastic IP, picking a free source port per connection. Ports are a finite resource per destination tuple: for one destination IP and port, one NAT source IP offers about 55,000 usable ports. Your fleet fanning out to thousands of different destinations rarely hits this; five hundred application pods hammering a single third-party API endpoint or one external database absolutely can. When no port is free, the gateway increments ErrorPortAllocation and drops the connection attempt — the client just sees a connect timeout.

Two more failure modes share the symptom space. IdleTimeoutCount rises when connections sit quiet past 350 seconds and the gateway drops them — applications with long-lived, mostly idle connections (database pools, message consumers) see “connection reset by peer” after idle periods unless they send keepalives more often than that. And PacketsDropCount climbing without port errors points at the gateway’s bandwidth ceiling or an unhealthy gateway rather than exhaustion.

Capacity-wise, a NAT Gateway scales automatically up to 100 Gbps and millions of packets per second, so raw throughput is rarely the constraint — concurrency to a single destination is. AWS’s lever for that is secondary IPv4 addresses: a gateway can carry multiple EIPs (up to eight), and each additional address multiplies the port pool per destination. Cost mechanics matter here too: you pay per gateway-hour and per GB processed, so the same traffic through three AZ-local gateways costs the same processing but three hourly charges — and cross-AZ traffic to a single gateway adds inter-AZ transfer on top; the trade-offs are laid out in NAT Gateway vs NAT instance.

What to check first

  • ErrorPortAllocation metric — any non-zero value during the failure window is your confirmation.
  • IdleTimeoutCount — if resets happen after quiet periods rather than under load, it is the 350-second idle timer, not exhaustion.
  • Destination concentration — is the failing traffic mostly toward one IP:port? Flow logs answer this quickly.
  • How many EIPs the gateway has — one address means one 55k pool per destination.
  • Which AZs route through which gateway — a single gateway serving three AZs concentrates all load and adds cross-AZ charges.

How to verify

Console: VPC > NAT gateways > select gateway > Monitoring. CLI:

# Port allocation errors over the incident window
aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value=nat-0abc123 \
  --start-time 2026-07-03T12:00:00Z --end-time 2026-07-03T18:00:00Z \
  --period 300 --statistics Sum

# Idle timeouts and active connections tell the rest of the story
aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name ActiveConnectionCount \
  --dimensions Name=NatGatewayId,Value=nat-0abc123 \
  --start-time 2026-07-03T12:00:00Z --end-time 2026-07-03T18:00:00Z \
  --period 300 --statistics Maximum

To find the hot destination, query flow logs for the NAT ENI and group by destination address and port — top talkers by connection count, not bytes. The metric definitions live in the Amazon VPC documentation.

The fix

  1. Add secondary IPv4 addresses to the gateway. VPC > NAT gateways > Actions > Edit secondary IP address associations, or aws ec2 assign-private-nat-gateway-address / associate additional EIPs for a public gateway. Each extra address adds another ~55k ports per destination — the fastest relief when one destination is hot.
  2. Split load across one gateway per AZ. If a single gateway serves the whole VPC, deploy one per AZ and point each private route table at its local gateway. This divides per-destination concurrency, removes cross-AZ data charges, and drops the blast radius of an AZ failure.
  3. Fix the connection behaviour that creates the pressure. Reuse connections (HTTP keep-alive, gRPC channels, database pools sized deliberately) instead of opening one per request; close what you open; enable TCP keepalives under 350 seconds for long-lived idle connections so the gateway does not reap them.
  4. Take AWS-bound traffic off the NAT path entirely. S3 and DynamoDB via gateway endpoints are free and unmetered; ECR, CloudWatch, SQS and friends via interface endpoints remove both the port pressure and the per-GB processing charge — the selection logic is in VPC endpoints: gateway vs interface.
  5. For massive single-destination fan-out, shard by destination. If one partner API is the sink, ask for multiple endpoint IPs or use their regional hostnames so the per-tuple limit applies per IP rather than once.

When this is the wrong move

Do not add gateways or EIPs when the metrics say idle timeout — that is an application keepalive problem, and more ports will not fix resets after quiet periods. Do not reach for a NAT instance to “get control of ports” unless you genuinely need packet-level tuning and accept becoming the operator of that box. And if the failing traffic is inbound, stop: NAT Gateways are egress-only, and no amount of NAT debugging fixes an inbound path that should be an ALB or an EIP.

Common mistakes

  • Diagnosing from application logs alone — “timeout” strings look identical for security group blocks, DNS failures, and port exhaustion; only the gateway metrics separate them.
  • Scaling instance counts during the incident, which multiplies connection attempts and deepens the exhaustion.
  • Putting all three AZs behind one gateway to save two hourly charges, then paying more in cross-AZ transfer and sharing one failure domain.
  • Forgetting that stopped-then-restarted long-idle connections were reaped at 350 seconds, and blaming the destination service.

Prevention

  • Alarm on ErrorPortAllocation > 0 and on IdleTimeoutCount trending — both are early, unambiguous signals.
  • Baseline ActiveConnectionCount per gateway so growth toward the ceiling is visible quarters in advance.
  • Route AWS service traffic through VPC endpoints as a default posture, not as an incident response.
  • Set connection-pool maximums and keepalive intervals as deployment standards for anything that talks through NAT.
  • Review NAT topology (one per AZ, secondary IPs where hot) whenever fleet size doubles.

Port exhaustion is one of the few AWS problems with a perfectly named metric — the discipline is looking at it before restarting things. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.