A few months ago someone asked me a question that sounds simple: "can you tell me everything that talked to this database in the last 48 hours?" We run a 22-account AWS Organization for a court system, so questions like that aren't curiosity. They're audit requests, and "I think it was the app servers" is not an acceptable answer.
The console's Flow Logs viewer gets you nowhere with that question. CloudWatch Logs Insights can do it, but at CloudWatch ingestion prices, keeping flow logs there across a whole Organization is the kind of line item that shows up in your FinOps review with a red circle around it.
The setup that actually works, and costs a fraction: Flow Logs delivered to S3 in Parquet, a lifecycle policy so old logs age out of your bill, and an Athena table on top so you can answer real questions with SQL. Enable it once, and you have a queryable record of every network conversation in your VPC.
This article is the full walkthrough: enabling it properly (there are three checkboxes people miss that make Athena 10x cheaper), the table DDL, and ten copy-paste queries I actually use. Grab a coffee.
First, what flow logs are — and what they are not
A flow log record is metadata about a network flow: source and destination IP, ports, protocol, bytes, packets, whether the traffic was accepted or rejected, and (in newer versions) very useful extras like which AWS service the traffic went to and which path it took to leave the VPC.
What it is not: packet capture. There's no payload, no HTTP headers, no DNS names. If you need to know what was said, you need tcpdump or Traffic Mirroring. Flow logs tell you who talked to whom, when, how much, and whether the firewall let it happen. In my experience that answers about 80% of the questions that reach a network team.
One more thing worth knowing: records are aggregated over an interval (1 or 10 minutes), so a "flow" is a summary, not individual packets. For forensics and cost analysis that's exactly what you want.
Why S3 instead of CloudWatch Logs
Short version: delivery to S3 costs roughly half of CloudWatch ingestion per GB, and then storage is S3 storage — pennies, and you control it with lifecycle rules. CloudWatch keeps charging you ingestion for every GB, and flow logs are chatty. On a busy VPC the difference is not academic; it's the difference between "always on everywhere" and "we only enable it during incidents", and flow logs you didn't enable before the incident are worth exactly nothing.
The trade-off is losing Logs Insights and near-real-time alerting on the raw stream. For alerting, GuardDuty already analyzes flow log data internally without you enabling anything. For investigation — which is what you actually do with flow logs — Athena on S3 is better anyway: real SQL, joins, and you pay per query instead of per GB stored.
Check the current numbers on the pricing page for your region before quoting them to your boss, but in every account I've run the math for, S3 wins by a wide margin.
Enabling it right: the three options that matter
You can enable flow logs at the VPC, subnet, or ENI level. Enable at the VPC level — it covers every ENI in the VPC, including the ones that get created tomorrow.
When you create the flow log (Console: VPC → your VPC → Flow logs → Create flow log), the defaults will work, and they will also quietly cost you money later. Set these:
1. Format: Parquet, not plain text. Under "Log file format", pick Parquet. Athena scans columnar Parquet dramatically more efficiently than gzipped text — AWS's own docs cite queries running faster and cheaper, and in practice I've seen scans drop to a tenth of the bytes. This single checkbox is most of your future Athena bill.
2. Hive-compatible prefixes + partition by hour. Two more checkboxes ("Hive-compatible S3 prefix", "Partition logs by time"). They organize the S3 layout so Athena can skip everything outside the time window you're querying. Without partitions, every query scans your entire history; with them, "yesterday, 2pm to 4pm" scans two hours of data.
3. Custom format with all the fields. The default format is fine, but the v3–v7 fields are where the good stuff lives. Select all fields — storage is cheap and you can't add fields retroactively. The ones you'll thank yourself for later: flow-direction, traffic-path, pkt-src-aws-service, pkt-dst-aws-service, tcp-flags, az-id, and pkt-srcaddr/pkt-dstaddr (the original addresses when traffic goes through a NAT gateway — without these, everything behind NAT looks like the NAT).
In Terraform, the whole thing looks like this:
resource "aws_flow_log" "vpc" {
vpc_id = aws_vpc.main.id
traffic_type = "ALL"
log_destination_type = "s3"
log_destination = aws_s3_bucket.flow_logs.arn
max_aggregation_interval = 600 # 10 min; use 60 for finer forensics, ~more records
destination_options {
file_format = "parquet"
hive_compatible_partitions = true
per_hour_partition = true
}
# All fields up to v7 — order matters, it defines your table schema
log_format = "$${version} $${account-id} $${interface-id} $${srcaddr} $${dstaddr} $${srcport} $${dstport} $${protocol} $${packets} $${bytes} $${start} $${end} $${action} $${log-status} $${vpc-id} $${subnet-id} $${instance-id} $${tcp-flags} $${type} $${pkt-srcaddr} $${pkt-dstaddr} $${region} $${az-id} $${pkt-src-aws-service} $${pkt-dst-aws-service} $${flow-direction} $${traffic-path}"
}
The bucket: policy and lifecycle
The delivery service needs permission to write. If you create the flow log through the console pointing at your bucket, AWS adds this policy for you; in Terraform you own it:
resource "aws_s3_bucket" "flow_logs" {
bucket = "myorg-vpc-flow-logs-prd"
}
data "aws_iam_policy_document" "flow_logs_delivery" {
statement {
principals {
type = "Service"
identifiers = ["delivery.logs.amazonaws.com"]
}
actions = ["s3:PutObject"]
resources = ["${aws_s3_bucket.flow_logs.arn}/*"]
condition {
test = "StringEquals"
variable = "s3:x-amz-acl"
values = ["bucket-owner-full-control"]
}
}
statement {
principals {
type = "Service"
identifiers = ["delivery.logs.amazonaws.com"]
}
actions = ["s3:GetBucketAcl"]
resources = [aws_s3_bucket.flow_logs.arn]
}
}
resource "aws_s3_bucket_policy" "flow_logs" {
bucket = aws_s3_bucket.flow_logs.id
policy = data.aws_iam_policy_document.flow_logs_delivery.json
}
Now the part everyone forgets until the storage line starts growing: lifecycle. Decide how long flow logs are useful to you. My rule of thumb: hot for 30 days (that covers almost every investigation), cold for audit retention, gone after that. Adjust the archive tier and the expiration to your compliance requirements — in a regulated environment, that number comes from your retention policy, not from a blog post:
resource "aws_s3_bucket_lifecycle_configuration" "flow_logs" {
bucket = aws_s3_bucket.flow_logs.id
rule {
id = "flow-logs-aging"
status = "Enabled"
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 90
storage_class = "GLACIER"
}
expiration {
days = 365
}
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
}
}
One caveat: Athena can't query objects that already moved to Glacier. Your queryable window is whatever you keep in Standard/IA. That's another reason the 30/90 split works well.
The Athena table
Create a workgroup first if you don't have one, set a query result location, and — do this, seriously — set a per-query data scan limit. It turns a careless SELECT * over a year of logs from a bill into an error message.
Then the table. Because we enabled Hive-compatible prefixes and hourly partitions, we can use partition projection and never think about MSCK REPAIR TABLE or Glue crawlers again — Athena computes the partitions from the template:
CREATE EXTERNAL TABLE IF NOT EXISTS vpc_flow_logs (
version int,
account_id string,
interface_id string,
srcaddr string,
dstaddr string,
srcport int,
dstport int,
protocol bigint,
packets bigint,
bytes bigint,
start bigint,
`end` bigint,
action string,
log_status string,
vpc_id string,
subnet_id string,
instance_id string,
tcp_flags int,
type string,
pkt_srcaddr string,
pkt_dstaddr string,
region string,
az_id string,
pkt_src_aws_service string,
pkt_dst_aws_service string,
flow_direction string,
traffic_path int
)
PARTITIONED BY (
`aws-account-id` string,
`aws-region` string,
`year` string,
`month` string,
`day` string,
`hour` string
)
STORED AS PARQUET
LOCATION 's3://myorg-vpc-flow-logs-prd/AWSLogs/'
TBLPROPERTIES (
"projection.enabled" = "true",
"projection.aws-account-id.type" = "injected",
"projection.aws-region.type" = "enum",
"projection.aws-region.values" = "us-east-1,sa-east-1",
"projection.year.type" = "integer",
"projection.year.range" = "2024,2030",
"projection.month.type" = "integer",
"projection.month.range" = "1,12",
"projection.month.digits" = "2",
"projection.day.type" = "integer",
"projection.day.range" = "1,31",
"projection.day.digits" = "2",
"projection.hour.type" = "integer",
"projection.hour.range" = "0,23",
"projection.hour.digits" = "2",
"storage.location.template" = "s3://myorg-vpc-flow-logs-prd/AWSLogs/aws-account-id=${aws-account-id}/aws-service=vpcflowlogs/aws-region=${aws-region}/year=${year}/month=${month}/day=${day}/hour=${hour}"
)
Adjust the bucket name, the region list, and the year range. The column names match the custom format we defined — dashes become underscores in Parquet. If you used a different field order, your table must match it: the format string is your schema contract.
Two habits that keep queries fast and cheap: always filter on the partition columns (year, month, day, hour — and aws-account-id if you centralize the Organization's logs in one bucket), and select only the columns you need. Parquet only reads what you ask for.
The 10 queries
All of these assume you replace the date filters with your window. I'm using a WHERE block like this everywhere — copy it once:
-- reusable time filter: adjust and paste into each query
WHERE year = '2026' AND month = '07' AND day = '03'
1. Who talked to this IP? (the audit question)
SELECT srcaddr, dstaddr, dstport, protocol, action,
SUM(bytes) AS total_bytes, COUNT(*) AS flows
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND (srcaddr = '10.20.5.44' OR dstaddr = '10.20.5.44')
GROUP BY srcaddr, dstaddr, dstport, protocol, action
ORDER BY total_bytes DESC;
2. What are my security groups and NACLs rejecting, and who's knocking?
SELECT srcaddr, dstaddr, dstport, COUNT(*) AS rejected_flows
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND action = 'REJECT'
AND flow_direction = 'ingress'
GROUP BY srcaddr, dstaddr, dstport
ORDER BY rejected_flows DESC
LIMIT 50;
3. Exposure check: is anything from the internet actually reaching SSH/RDP?
SELECT srcaddr, dstaddr, dstport, COUNT(*) AS flows, SUM(bytes) AS total_bytes
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND dstport IN (22, 3389)
AND action = 'ACCEPT'
AND flow_direction = 'ingress'
AND NOT regexp_like(srcaddr, '^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)')
GROUP BY srcaddr, dstaddr, dstport
ORDER BY flows DESC;
If this returns rows you can't explain, stop reading and go fix a security group.
4. Top talkers by volume (start of every data transfer cost investigation)
SELECT srcaddr, dstaddr, pkt_dst_aws_service,
SUM(bytes) AS total_bytes
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND flow_direction = 'egress'
GROUP BY srcaddr, dstaddr, pkt_dst_aws_service
ORDER BY total_bytes DESC
LIMIT 25;
5. Who is burning the NAT Gateway? (NAT processing charges are a classic FinOps finding — this names the culprits)
-- interface_id = the NAT Gateway's ENI (find it in the console or via CLI)
SELECT pkt_srcaddr AS original_source, dstaddr, dstport,
SUM(bytes) AS total_bytes
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND interface_id = 'eni-0abc123def456'
AND flow_direction = 'egress'
GROUP BY pkt_srcaddr, dstaddr, dstport
ORDER BY total_bytes DESC
LIMIT 25;
Note pkt_srcaddr — that's the field that survives NAT and shows you the actual pod or instance.
6. East-west heavy hitters (candidates for same-AZ placement or a cost conversation)
SELECT srcaddr, dstaddr, az_id, SUM(bytes) AS total_bytes
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND regexp_like(srcaddr, '^10\.')
AND regexp_like(dstaddr, '^10\.')
GROUP BY srcaddr, dstaddr, az_id
ORDER BY total_bytes DESC
LIMIT 25;
Cross-reference the top pairs with your subnet-to-AZ map: high-volume pairs crossing AZs are paying the cross-AZ tax on every byte, twice.
7. Hybrid debugging: everything crossing to on-prem
-- replace with your on-prem CIDR
SELECT srcaddr, dstaddr, dstport, action,
SUM(bytes) AS total_bytes, COUNT(*) AS flows
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND (regexp_like(dstaddr, '^192\.168\.') OR regexp_like(srcaddr, '^192\.168\.'))
GROUP BY srcaddr, dstaddr, dstport, action
ORDER BY total_bytes DESC;
When "the application can't reach the mainframe", this tells you in one query whether traffic is leaving, being rejected, or never arriving back.
8. The asymmetric NACL bug, caught red-handed
Stateless NACLs need the return path open on ephemeral ports. When someone forgets, the request goes out and the response gets rejected — which looks exactly like a timeout from the application side:
SELECT srcaddr, srcport, dstaddr, dstport, COUNT(*) AS rejected_returns
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND action = 'REJECT'
AND srcport IN (80, 443, 5432, 3306, 1433) -- responses FROM service ports
AND dstport BETWEEN 1024 AND 65535 -- TO ephemeral ports
GROUP BY srcaddr, srcport, dstaddr, dstport
ORDER BY rejected_returns DESC;
Rows here = a NACL missing its ephemeral return rule. This query has ended multi-hour debugging sessions in five minutes.
9. Port scan detection (one source, many ports)
SELECT srcaddr, COUNT(DISTINCT dstport) AS distinct_ports,
COUNT(*) AS attempts
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND flow_direction = 'ingress'
GROUP BY srcaddr
HAVING COUNT(DISTINCT dstport) > 100
ORDER BY distinct_ports DESC;
10. Connections attempted but never established (SYN with no reply)
tcp_flags in a flow record is the OR of the flags seen in the window. A flow showing only SYN (value 2) means somebody tried and nobody answered — great for spotting half-open scans, dead dependencies, or a service that stopped listening:
SELECT srcaddr, dstaddr, dstport, COUNT(*) AS syn_only_flows
FROM vpc_flow_logs
WHERE year = '2026' AND month = '07' AND day = '03'
AND tcp_flags = 2
AND protocol = 6
GROUP BY srcaddr, dstaddr, dstport
ORDER BY syn_only_flows DESC
LIMIT 50;
What this setup won't do
Flow logs won't give you DNS names (pair them with Route 53 Resolver query logs and you can join IPs to the names that were resolved — that's a future article), won't show payloads, and won't capture traffic to some AWS-internal interfaces (the docs keep a list of exclusions — instance metadata and Amazon DNS among them). And remember the Glacier caveat: your Athena window is what's still in Standard/IA.
None of that reduces the value here. For the daily reality of a network team — "who talked to X", "why is this timing out", "where is the NAT bill coming from", "is anything from outside reaching port 22" — this is the highest ratio of answers-per-dollar I know on AWS.
Enable it before you need it. Flow logs are like backups that way: the ones you start collecting after the incident tell you nothing about the incident.
If this was useful, I write about AWS networking from the packet up — VPC, hybrid DNS, EKS networking. Follow along here or on LinkedIn.
Top comments (1)
This is a great reference, bookmarking it. One small thing on the bucket policy: the delivery.logs.amazonaws.com principal is org-wide by default, so if you're centralizing flow logs from a 22-account Organization into one bucket like this, it's worth adding an aws:SourceAccount (or aws:SourceArn) condition alongside the bucket-owner-full-control one. Otherwise any account's flow-log delivery service could theoretically write into paths it shouldn't, not just yours. Same confused-deputy pattern that shows up with cross-account log delivery generally. (Disclosure: I work on Shieldly, an AWS IAM/resource-policy analysis tool, so I'm primed to look for this specific thing.)