A security team reads about defense in depth and decides the VPC needs Network ACLs configured “properly” — not the default allow-all, but explicit deny rules layered on top of the security groups. The first NACL rule blocks an obvious malicious port. The second rule is more specific. By the fifth rule, an apt-get update from one of the private-subnet instances starts hanging. The on-call gets paged. The cause turns out to be the NACL blocking the return traffic on an ephemeral port — NACLs are stateless, and the team forgot that allowing outbound on port 443 also needs an inbound rule for the ephemeral port range. The instinct is to argue that NACLs are broken. They are not — they are stateless and behave exactly as documented. The lesson is that NACLs are the wrong tool for most workloads, and the right place for that defense in depth is somewhere else.
What’s happening
Security groups (SGs) operate at the ENI level. Each EC2 instance, RDS database, Lambda function in a VPC, ALB target, and interface VPC endpoint has one or more SGs attached. An SG is a set of rules that say “allow inbound from this source on this port” or “allow outbound to this destination on this port.” SGs are stateful — if you allow inbound on port 443, the return traffic flows back without an explicit outbound rule. SGs are allow-only — there is no deny rule. The default outbound is allow-all; the default inbound is deny-all (allow nothing unless explicitly permitted). SGs can reference other SGs by ID as a source — allow inbound from sg-app-tier on port 5432 rather than from a CIDR. This is the property that makes SGs the right primitive for application-tier rules: the rule reads “the app servers can talk to the database,” not “the IP range 10.20.16.0/20 can talk to the database.”
Network ACLs (NACLs) operate at the subnet level. Every subnet is associated with exactly one NACL (the default NACL allows all, in and out). A NACL is an ordered list of rules — rule 100 evaluates before rule 200 — and the first matching rule wins. NACLs are stateless — if you allow inbound on port 443, you must also allow outbound on the ephemeral port range (32768-65535 on Linux, 49152-65535 on Windows) for the response. NACLs can deny — you can write a rule that denies traffic from a specific CIDR. NACLs cannot reference other NACLs or SGs as sources — only CIDR ranges. This is the property that makes NACLs awkward for most application-tier rules: you have to map every source to a CIDR range, and the rules get stale every time the source changes.
The order of evaluation:
- Outbound from instance A: SG outbound rule on A’s ENI → NACL outbound rule on A’s subnet → routing → NACL inbound rule on B’s subnet → SG inbound rule on B’s ENI.
- Inbound to A on the return path: NACL inbound on A’s subnet (must allow ephemeral port if NACL is restrictive) → SG inbound rule on A’s ENI (stateful, automatically allows if outbound was allowed) → instance receives.
The ephemeral-port gotcha is in step 2. The SG knows the conversation; it lets the response in automatically. The NACL does not; it sees a fresh inbound packet on a high-numbered port and asks “do I have a rule for this?” If the NACL inbound rules only allow port 443 and you forgot the ephemeral range, the response is dropped.
How to verify
The decision matrix for which tool to use for which control:
| Control | SG | NACL |
|---|---|---|
| App servers can talk to the database | yes (reference SG) | wrong tool |
| Block all traffic from a specific malicious CIDR | possible with explicit revoke, but better to deny at the edge | yes (deny rule) |
| Allow only inbound HTTPS to the ALB | yes | yes but redundant with SG |
| Block outbound to a specific risky CIDR | revoke or do not add | yes (deny rule) |
| Isolate two subnets so they cannot talk to each other at all | possible with SG rules but the SG would have to be on every instance | yes (subnet-level deny) |
| Quarantine a subnet during incident response | possible by modifying every SG | yes (one NACL change quarantines the subnet) |
| Allow inbound only from a known partner CIDR | yes | yes |
The pattern that emerges: SGs handle the application-tier rules (which services can talk to which), NACLs handle the subnet-tier coarse controls (which subnet ranges can talk to which subnets at all, plus the deny-list rules SGs cannot express).
Inspect existing SG rules:
# SGs on a specific instance
aws ec2 describe-instances \
--instance-ids i-0123456789abcdef0 \
--query 'Reservations[].Instances[].SecurityGroups[].[GroupId,GroupName]' \
--output table
# What an SG allows
aws ec2 describe-security-groups \
--group-ids sg-app-tier \
--query 'SecurityGroups[].IpPermissions[].[FromPort,ToPort,IpProtocol,IpRanges,UserIdGroupPairs]' \
--output jsonInspect existing NACLs:
aws ec2 describe-network-acls \
--filters Name=vpc-id,Values=vpc-0123456789abcdef0 \
--query 'NetworkAcls[].[NetworkAclId,IsDefault,Entries[].[RuleNumber,Egress,Protocol,RuleAction,CidrBlock,PortRange]]' \
--output jsonThe default NACL allows all in and all out — both rule 100 (allow all) and rule * (default deny). Most production VPCs leave NACLs at default and do all the work with SGs. The exception is when a specific compliance regime requires subnet-level deny rules, or when an incident needs subnet quarantine.
The fix
The procedure for designing SG and NACL together:
-
Default to SGs for everything that is an application-tier rule. “App can talk to database,” “ALB can talk to app,” “bastion can talk to app on SSH.” SG-to-SG references make these rules read naturally and stay correct as the source instances change.
-
Name SGs by role, not by service.
sg-app-tier,sg-db-tier,sg-bastion,sg-alb. Notsg-i-0123456789abcdef0orsg-app-1. The names appear in every flow log analysis and every incident review; readable names cut investigation time. -
Use NACLs for two things only:
- Deny-list rules. If a specific CIDR needs to be blocked organization-wide (a known-bad range, an old peering partner that should not have any reach), add it to the NACL at a low rule number (e.g. 50) so it evaluates first.
- Subnet quarantine. During an incident, the fastest way to isolate a subnet is a NACL change — modify the subnet’s NACL to deny all and the entire subnet is offline in seconds. Restoring is one rule revert. The same isolation via SGs requires modifying every instance’s SG, which is slow and risky.
-
Leave the default NACL alone. Or, if your security baseline requires non-default NACLs, write them with the ephemeral-port allow rules in mind:
# Create a non-default NACL for private subnets aws ec2 create-network-acl --vpc-id vpc-0123456789abcdef0 # Allow inbound HTTPS for return traffic from outbound calls # (the SG already handles the conversation; this just permits the packet at the subnet) aws ec2 create-network-acl-entry \ --network-acl-id acl-private \ --rule-number 100 \ --protocol tcp \ --rule-action allow \ --ingress \ --cidr-block 0.0.0.0/0 \ --port-range From=443,To=443 # CRITICAL: allow the ephemeral port range for return traffic aws ec2 create-network-acl-entry \ --network-acl-id acl-private \ --rule-number 110 \ --protocol tcp \ --rule-action allow \ --ingress \ --cidr-block 0.0.0.0/0 \ --port-range From=32768,To=65535 # Allow outbound on whatever the apps need aws ec2 create-network-acl-entry \ --network-acl-id acl-private \ --rule-number 100 \ --protocol tcp \ --rule-action allow \ --egress \ --cidr-block 0.0.0.0/0 \ --port-range From=443,To=443 -
Test the NACL changes from a real instance before flipping in production. Run
curl https://example.comfrom an EC2 instance in the subnet; if it hangs, the NACL is blocking the ephemeral return traffic. The failure mode is always silent timeouts, not error messages.
The anti-pattern we see most: a security audit recommendation that says “harden NACLs” gets implemented as a non-default NACL with rules that mirror the SGs, the ephemeral-port range gets missed, and apt-get update and DNS queries fail intermittently. The right response to that audit finding is to push back: SGs already enforce the application-tier rules; NACLs adding a stateless second layer mostly creates operational pain and rarely catches a real attacker that the SGs missed.
The other rule we hold: SG rules are infrastructure-as-code, NACL rules even more so. A production VPC where SGs and NACLs are edited from the console is a VPC where the next incident takes an extra two hours because nobody knows the current state.
For the subnet plan that defines what the NACLs apply to, see Multi-AZ subnet design. For the flow log analysis that surfaces which rules are firing, see VPC Flow Logs analysis.
Stack Harbor designs SG-and-NACL topologies for compliance and security baselines as part of Managed AWS and ongoing Managed Cloud Ops. If a security audit has handed you a NACL hardening project, start with an infrastructure consult.