You created a VPC endpoint for S3 or another AWS service, but the NAT Gateway data-processing charge did not drop and flow logs still show traffic leaving through public IPs. For a gateway endpoint (S3, DynamoDB), the cause is almost always that the endpoint is not associated with the route tables of the subnets doing the talking, or the traffic is cross-region. For an interface endpoint, it is almost always that private DNS is off — clients keep resolving the service’s public IPs and route out through NAT exactly as before.
What’s happening
The two endpoint types intercept traffic by completely different mechanisms, and each has one classic failure.
A gateway endpoint works through routing: when you associate it with a route table, AWS inserts a route whose destination is a managed prefix list (pl-...) covering the service’s public IP ranges in that region, targeting the endpoint (vpce-...). Any subnet whose route table lacks that entry keeps sending S3 traffic to its default route — the NAT Gateway. Two corollaries follow. Added subnets or new route tables do not inherit the association. And the prefix list covers only the endpoint’s own region: a request to a bucket in another region resolves to IPs outside the prefix list and rides NAT no matter what.
An interface endpoint works through DNS: it creates ENIs with private IPs in your subnets, and — if private DNS is enabled and the VPC’s enableDnsSupport and enableDnsHostnames attributes are on — the service’s normal hostname (sqs.ca-central-1.amazonaws.com) resolves to those private IPs inside the VPC. With private DNS off, the hostname keeps resolving publicly, and only clients explicitly using the endpoint-specific DNS name (vpce-...amazonaws.com) use the endpoint. Clients that hardcode public IPs, pin global endpoints, or sit behind their own resolver bypass it the same way. The endpoint’s security group must also allow 443 from the clients, or traffic will resolve privately and then hang — a different failure with a different signature.
Endpoint policies are the third, quieter layer: a restrictive policy can deny the specific bucket or API action, producing 403s that teams misread as “the endpoint does not work” and route around via NAT.
What to check first
- Gateway endpoint: is the
pl-route in the source subnet’s route table? Look for the prefix-list route targetingvpce-.... - Interface endpoint: what does the hostname resolve to from the instance? Private 10.x/172.x/192.168.x answers mean the endpoint is in play; public answers mean it is not.
- Same region? Cross-region S3/DynamoDB calls never use a gateway endpoint.
- Endpoint security group allows TCP 443 from the client subnets or security groups.
- VPC DNS attributes on, since interface endpoint private DNS depends on both.
How to verify
# Gateway endpoint: which route tables is it associated with?
aws ec2 describe-vpc-endpoints \
--filters "Name=vpc-id,Values=vpc-0abc123" \
--query 'VpcEndpoints[].{Id:VpcEndpointId,Svc:ServiceName,Type:VpcEndpointType,RTs:RouteTableIds,PrivDns:PrivateDnsEnabled}'
# Does the source subnet's table carry the prefix-list route?
aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=subnet-0abc123" \
--query 'RouteTables[].Routes[?starts_with(DestinationPrefixListId||``,`pl-`)]'
From an instance in the affected subnet:
# Interface endpoint test: private IPs = endpoint in use
dig sqs.ca-central-1.amazonaws.com +short
# Gateway endpoint test for S3: this still shows public IPs (normal!) —
# the interception is at the ROUTE layer, so check the route, not DNS
dig s3.ca-central-1.amazonaws.com +short
The definitive proof is the money and the packets: VPC > NAT gateways > Monitoring for BytesOutToDestination trending down after the fix, and flow logs on the NAT ENI showing S3 prefix-list ranges disappearing. Endpoint mechanics are documented in the Amazon VPC documentation, and the gateway-vs-interface decision itself is covered in VPC endpoints for S3 and DynamoDB.
The fix
- Associate the gateway endpoint with every relevant route table. VPC > Endpoints > select endpoint > Route tables > Manage route tables, or
aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0abc123 --add-route-table-ids rtb-0aaa rtb-0bbb. The prefix-list route appears automatically; you never hand-write it. - Enable private DNS on the interface endpoint.
aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0abc123 --private-dns-enabled, after confirming both VPC DNS attributes are on. Existing long-lived connections keep their old resolution until they reconnect — restart stubborn clients. - Open the endpoint security group. Allow TCP 443 from the application subnets or their security groups. This is the fix when DNS already returns private IPs but connections hang.
- Point cross-region traffic at same-region resources or accept NAT. Replicate the bucket into the workload’s region, or use the specific regional endpoint hostname and budget the NAT path consciously for what remains.
- Fix the endpoint policy, not the route, on 403s. If requests reach the service and get denied, widen the endpoint policy (or the bucket policy’s
aws:sourceVpcecondition) — rerouting through NAT to dodge a policy is how audit findings are born. - Restart or re-resolve clients after the change. JVMs with infinite DNS cache TTL and pooled connections will happily use the old public path for hours.
When this is the wrong move
If the traffic volume to the service is trivial, an interface endpoint’s per-AZ hourly charge plus per-GB processing can exceed the NAT charge it displaces — endpoints for everything is a posture, not automatically a saving; do the arithmetic per service. Cross-region and cross-partition traffic is not fixable with gateway endpoints at all. And if what you actually need is private access from on-premises over VPN or Direct Connect, gateway endpoints do not serve that path — that is interface endpoint territory by design.
Common mistakes
- Running
digagainst S3, seeing public IPs, and concluding the gateway endpoint is broken — gateway endpoints do not change DNS; the route does the work. - Creating the endpoint in the right VPC but associating only the main route table while workloads use per-AZ tables.
- Leaving private DNS disabled because the console warning about the DNS attributes was dismissed during creation.
- Forgetting new subnets: the endpoint association list is static, and the subnet added last quarter routes through NAT again.
Prevention
- Make endpoint associations part of the subnet module in Terraform/CloudFormation so new route tables are attached at birth.
- Alarm on NAT
BytesOutToDestinationper gateway; a step change upward is often an endpoint association silently lost or bypassed. - Query flow logs monthly for NAT traffic matching AWS service prefix lists — that is the list of endpoints you should have.
- Standardize on private DNS enabled for every interface endpoint unless a documented exception exists.
- Test from a real workload subnet after every endpoint change, not from the bastion that lives somewhere else.
An endpoint that exists but carries nothing is pure cost with no benefit — five minutes of dig, route inspection, and one NAT metric tell you which side of the mechanism failed. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.