A customer came to me frustrated about their AWS bill. After reviewing the billing dashboard, we found that over 50% of their costs were coming from CloudWatch vended logs — specifically VPC Flow Logs. Here's how we cut that bill in half with two simple changes.
Why CloudWatch gets so expensive for VPC Flow Logs
(Vended logs are logs AWS services generate for you automatically — VPC Flow Logs, Route 53 query logs, CloudFront access logs.)
VPC Flow Logs are incredibly useful, but storing everything in CloudWatch Logs gets pricey fast:
- Data ingestion charges per GB
- Storage costs that accumulate over time
- No automatic retention policies by default
- Vended logs piling up quietly
The fix: move VPC Flow Logs to S3
They didn't need real-time querying for most flow logs — mainly weekly security reviews and occasional troubleshooting. Perfect candidates for S3.
Cost comparison (1 TB of logs):
| CloudWatch Logs | S3 + Parquet | |
|---|---|---|
| Ingestion | $0.50/GB | — |
| Storage | $0.03/GB/mo | $0.023/GB/mo |
| Compression | none | 80–90% smaller |
| Monthly total | ~$530 | ~$2.30 + query cost |
How I did it (4 steps)
1. Send VPC Flow Logs to S3 in Parquet format
In the VPC console, create a flow log with destination S3 and log file format Parquet. Parquet auto-compresses by 80–90% — massive storage savings vs plain text.
2. Set up S3 lifecycle policies
Don't keep everything in S3 Standard forever. A lifecycle rule:
- Day 0–30: S3 Standard (immediate Athena analysis)
- Day 30+: S3 Glacier Instant Retrieval (cheaper, still queryable)
- Day 90+: S3 Glacier Deep Archive (~$0.00099/GB, compliance)
- Day 365: expire
3. Set retention on CloudWatch log groups
The biggest quick win — many log groups had no retention policy, so logs were kept forever. Set sane retention: 7 days for debug, 30 for app logs, 90+ for audit/compliance. Never leave it as "Never expire."
4. Query with Amazon Athena when needed
Logs are now in S3, so query them on demand with Athena — you only pay for what you query. Parquet makes those queries fast and cheap.
The results
- CloudWatch costs dropped ~50% in the first billing cycle
- Parquet compression cut storage ~85%
- Query performance actually improved with Athena + Parquet
- Retention policies stopped future cost creep
Pro tip: Don't wait for costs to become a problem. Set billing alerts and review CloudWatch usage monthly — many teams are shocked when they finally check the detailed bill.
Takeaway
Moving VPC Flow Logs from CloudWatch to S3 with Parquet was one of the easiest cost-optimization wins I've done. Direct S3 delivery + Parquet compression + proper retention delivered immediate results. If your AWS bill looks high, start in Cost Explorer and look for vended logs and log groups with no retention.
Originally published on srinun.in. I write about DevOps, AWS, and Kubernetes — connect with me on LinkedIn.
Top comments (0)