The security team forwards a list of indicator IPs from a threat-intel feed and asks “have any of our hosts talked to these?” The intuitive answer is to grep the application logs, but the application logs only see HTTP — they miss the DNS queries, the outbound S3 calls, the SSH attempts, the anything-not-HTTP. The right log source is VPC Flow Logs. The team enables them on the production VPC, points them at CloudWatch Logs, runs the search, finds two hosts that did talk to a flagged IP, and then gets a bill for the CloudWatch ingestion at the end of the month that is bigger than the EC2 fleet for the same period. The mistake is not flow logs — it is sending them to CloudWatch on a VPC moving terabytes of traffic. The right architecture is S3 plus Athena, with filtering. Here is the working setup.

What’s happening

VPC Flow Logs capture metadata about every IP-level conversation inside the VPC. Each log record is a fixed-format line with source IP, destination IP, source port, destination port, protocol, packet count, byte count, start and end timestamp, action (ACCEPT or REJECT), and the network interface. Flow logs do not capture packet contents — they are connection metadata, not packet capture. The record format is extensible; you can add fields like the VPC ID, subnet ID, instance ID, TGW attachment, and TCP flags. The custom format is worth using; the default format is missing the fields you need for “which instance talked to this IP.”

Flow logs can be configured at three scopes:

VPC-level — every ENI in the VPC. This is what you want for security analysis.
Subnet-level — every ENI in a subnet. Useful for targeting a specific tier.
ENI-level — a single network interface. Useful for debugging a specific instance.

Flow logs can be sent to three destinations:

CloudWatch Logs — convenient for ad-hoc Insights queries, expensive at volume. The per-GB ingestion cost is the line item that surprises teams. A VPC pulling 50 TB/month of traffic with flow logs at default settings can generate a CloudWatch ingestion line larger than the rest of the AWS bill combined.
S3 — cheap (S3 storage rate, no per-GB ingestion charge for flow logs). The catch is that S3-stored flow logs are not directly queryable; you need Athena or a similar tool on top.
Kinesis Data Firehose — for streaming to a SIEM or a third-party log analytics platform. Useful for organizations that have an existing log pipeline; the in-AWS cost is the Firehose throughput.

The cost gotcha that bites once: enabling flow logs to CloudWatch on a high-traffic VPC without filtering produces hundreds of millions of log records per day. The right architecture is flow logs to S3, partitioned by date and account, with Athena tables ready for the queries you actually run.

How to verify

The setup, for a VPC that does not yet have flow logs:

# 1. Create the S3 bucket for flow logs (with lifecycle policy)
aws s3api create-bucket --bucket org-flow-logs-us-east-1 --region us-east-1
aws s3api put-bucket-lifecycle-configuration --bucket org-flow-logs-us-east-1 \
  --lifecycle-configuration file://lifecycle.json

# lifecycle.json:
# Transition to S3 Standard-IA at 30 days, Glacier at 90, expire at 365.

# 2. Enable flow logs on the VPC, with custom format
aws ec2 create-flow-logs \
  --resource-type VPC \
  --resource-ids vpc-0123456789abcdef0 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::org-flow-logs-us-east-1/vpc/ \
  --log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${flow-direction}'

After ~10 minutes the first flow log batches land in S3, partitioned by AWSLogs/<account>/vpcflowlogs/<region>/<year>/<month>/<day>/.

Set up Athena to query:

CREATE EXTERNAL TABLE vpc_flow_logs (
  version INT,
  account_id STRING,
  interface_id STRING,
  srcaddr STRING,
  dstaddr STRING,
  srcport INT,
  dstport INT,
  protocol INT,
  packets BIGINT,
  bytes BIGINT,
  start_ts BIGINT,
  end_ts BIGINT,
  action STRING,
  log_status STRING,
  vpc_id STRING,
  subnet_id STRING,
  instance_id STRING,
  tcp_flags INT,
  type STRING,
  pkt_srcaddr STRING,
  pkt_dstaddr STRING,
  region STRING,
  az_id STRING,
  flow_direction STRING
)
PARTITIONED BY (year STRING, month STRING, day STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '
LOCATION 's3://org-flow-logs-us-east-1/AWSLogs/111122223333/vpcflowlogs/us-east-1/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.year.type' = 'integer',
  'projection.year.range' = '2024,2030',
  'projection.month.type' = 'integer',
  'projection.month.range' = '1,12',
  'projection.day.type' = 'integer',
  'projection.day.range' = '1,31',
  'storage.location.template' = 's3://org-flow-logs-us-east-1/AWSLogs/111122223333/vpcflowlogs/us-east-1/${year}/${month}/${day}/'
);

The partition projection (no MSCK REPAIR needed) keeps Athena scans cheap — a query scoped to one day reads one day’s partitions, not the whole table.

The bread-and-butter query, “who talked to a leaked IP in the last 7 days”:

SELECT 
  from_unixtime(start_ts) AS conn_time,
  srcaddr,
  dstaddr,
  srcport,
  dstport,
  instance_id,
  bytes,
  action
FROM vpc_flow_logs
WHERE year = '2026' AND month = '5'
  AND day BETWEEN '20' AND '26'
  AND (dstaddr = '203.0.113.42' OR srcaddr = '203.0.113.42')
ORDER BY conn_time DESC
LIMIT 100;

For a list of leaked IPs in a file, use IN:

WHERE dstaddr IN ('203.0.113.42', '198.51.100.7', '192.0.2.99')

The second high-value query, “top outbound destinations by bytes”:

SELECT 
  dstaddr,
  SUM(bytes) AS total_bytes,
  COUNT(*) AS flow_count
FROM vpc_flow_logs
WHERE year = '2026' AND month = '5' AND day = '25'
  AND flow_direction = 'egress'
  AND NOT regexp_like(dstaddr, '^10\.')
GROUP BY dstaddr
ORDER BY total_bytes DESC
LIMIT 50;

This is the query that diagnoses an unexpected egress bill. Resolve the top IPs with dig -x to identify the destination service.

The third query, “REJECTed traffic by destination port”:

SELECT 
  dstport,
  COUNT(*) AS reject_count,
  COUNT(DISTINCT srcaddr) AS distinct_sources
FROM vpc_flow_logs
WHERE year = '2026' AND month = '5' AND day = '25'
  AND action = 'REJECT'
  AND flow_direction = 'ingress'
GROUP BY dstport
ORDER BY reject_count DESC
LIMIT 50;

REJECTed ingress on weird ports is the noise of the internet — bots probing for open services. Dense REJECTed ingress on a port your services do listen on is a tell that a security group is blocking traffic you might want.

The fix

The procedure for setting up flow logs in a way that scales:

Choose S3 first, CloudWatch only for short-window debugging. S3 is the cheap, durable, queryable destination for the long tail. CloudWatch Logs Insights is the fast, expensive destination for the next 24 hours. If you must use CloudWatch, scope to a single subnet or ENI and use a short retention window.
Use the custom log format with instance_id and tcp_flags. Without instance_id, every flow log row references an ENI ID, and you have to join ENI-to-instance separately. With it, the queries are self-contained. tcp_flags lets you see whether the conversation was a real connection (SYN+ACK present) or a probe (SYN-only, no response).
Partition by date with projection. A flow log table without partitions, queried over a month, scans tens of GB and costs real money per query. Partitioned with projection, the same query scans 1 GB. The difference is one schema change.
Set an S3 lifecycle policy. Flow logs are append-only. After 30 days they should be on Standard-IA; after 90 days on Glacier; after a year, expire (or longer, if compliance requires). The lifecycle policy is the only thing keeping the bucket from becoming the largest S3 spender in the org.
Decide what to log. The default is “all” — both ACCEPT and REJECT. For most production VPCs, “all” is correct because the REJECT records are the security signal. For VPCs with very high traffic volume where logging cost is a real concern, “ACCEPT only” or “REJECT only” can cut the volume in half. Be aware that “ACCEPT only” loses the security-finding side.
For multi-account orgs, centralize the flow log bucket. Every account sends to one bucket in the security/logging account, partitioned by source account. The Athena table is created once in the security account; the security team queries across all accounts without needing per-account credentials.

The mistake to avoid: enabling flow logs to CloudWatch Logs at VPC scope on a VPC pulling tens of TB per month. The CloudWatch ingestion line will arrive larger than expected. Switch to S3 first, or scope the CloudWatch flow logs to a single subnet, or filter to REJECT-only if security is the only use case.

The other rule we hold: flow logs are valuable for the queries you do not know you need yet. The bias is toward enabling them on every production VPC, at S3, with retention. The first time a leaked IP comes up, you want the data from a month ago, not “let me enable flow logs and check back in two weeks.”

For the underlying subnet design that flow logs operate on, see Multi-AZ subnet design. For the egress visibility flow logs reveal, see NAT Gateway vs NAT instance.

Stack Harbor sets up flow log architectures and the Athena queries that operate on them as part of Managed AWS and ongoing Managed Cloud Ops. If your VPC has no flow logs or has them in the expensive place, start with an infrastructure consult.