Centralizing AWS WAF Logs with Kinesis Firehose and S3

When it comes to cloud security, logs are very important. Having them centralized and well-structured is key for auditing, detecting incidents, and generating meaningful reports.

In this article, we’ll walk through how to configure AWS WAF logging to a centralized S3 bucket using Amazon Kinesis Data Firehose , leveraging dynamic partitioning to keep logs organized by account, Web ACL, and date.

Why Firehose Instead of Direct S3 Logging?

At first glance, sending logs directly from WAF to S3 seems like the simplest solution. However, costs and flexibility tell a different story.

Direct S3 logging from WAF :

• AmazonCloudWatch USE1-S3-Egress-Bytes costs $0.25 per GB (first 10TB).

Via Firehose :

• Amazon Kinesis Firehose Dynamic Partitioning → $0.02 per GB
• Amazon Kinesis Firehose PutRecordBatch → $0.029 per GB
• Total: $0.049 per GB → ~80% cheaper.

On top of that, Firehose offers extra benefits:

• Process logs in-transit with Lambda.
• Dynamically partition logs in S3.
• Custom compress data before storing.

Architecture

This solution works whether you manage Web ACLs through Firewall Manager or deploy them individually across multiple AWS accounts.

Step-by-Step Setup

1. Create the Lambda Function

The following Lambda extracts account ID and WebACL name from the WAF log and sets partition keys. You have to create this function in the same account where Firewall Manager is configured, or in the same account where the Web ACL is created:

import json
import base64
from datetime import datetime

def lambda_handler(event, context):
    output = []
    for record in event['records']:
        payload = base64.b64decode(record['data'])
        log = json.loads(payload)
        webacl_arn = log.get("webaclId", "")
        account_id = "UnknownAccount"
        webacl_name = "UnknownWebACL"
        try:
            arn_parts = webacl_arn.split(":")
            if len(arn_parts) > 4:
                account_id = arn_parts[4] # Account ID
            resource_parts = webacl_arn.split("/")
            if len(resource_parts) > 2:
                webacl_name = resource_parts[-2] # WebACL name
        except Exception:
            pass
        now = datetime.utcnow()
        year = str(now.year)
        month = f"{now.month:02d}"
        day = f"{now.day:02d}"
        record['result'] = 'Ok'
        record['metadata'] = {
            "partitionKeys": {
                "accountId": account_id,
                "webaclName": webacl_name,
                "year": year,
                "month": month,
                "day": day
            }
        }
        output.append(record)
    return {'records': output}

With this setup, logs will be stored in S3 like:

s3://aws-waf-logs-your-bucket///year=2025/month=09/day=09/

2. Allow Firehose to Invoke the Lambda

Update the Lambda resource policy so that firehose.amazonaws.com can invoke it.

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "Inovke",
      "Effect": "Allow",
      "Principal": {
        "Service": "kinesis.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-1:<AccountId>:function:<your-lambda-name>",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:firehose:us-east-1:<AccountId>:deliverystream/<your-firehose>"
        }
      }
    },
    {
      "Sid": "GetConfig",
      "Effect": "Allow",
      "Principal": {
        "Service": "kinesis.amazonaws.com"
      },
      "Action": "lambda:GetFunctionConfiguration",
      "Resource": "arn:aws:lambda:us-east-1:<AccountId>:function:<your-lambda-name>",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:firehose:us-east-1:<AccountId>:deliverystream/<your-firehose>"
        }
      }
    }
  ]
}

Note that the ”AWS:SourceArn” refers to a resource that does not exist yet, but it will be created in the next steps

3. Create the IAM Role for Firehose

The role needs:

• Trust policy for Firehose.
• Permissions for the S3 bucket.
• Permissions to invoke the Lambda function.

Trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket",
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket/*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "lambda:InvokeFunction",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:us-east-1:<AccountID>:function:<your-lambda-name>:*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:<AccountID>:log-group:/aws/kinesisfirehose/waflogs:log-stream:*"
      ],
      "Effect": "Allow"
    }
  ]
}

4. Create the Destination S3 Bucket in the logs account

The bucket must start with the prefix aws-waf-logs- , and have the following cross-account bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountFirehosePut",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<AccountID>:role/<Role created previously>"
        ]
      },
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket",
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<AccountID>:role/<Role created previously>"
        ]
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::aws-waf-logs-your-destination-bucket/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

5. Create the Firehose Delivery Stream

This step can only be done via CLI or SDK , as the console doesn’t allow choosing a bucket from another account. The delivery stream name must start with the prefix “aws-waf-logs-”

Example cli command:

aws firehose create-delivery-stream \
  --delivery-stream-name aws-waf-logs-<your-name>\
  --delivery-stream-type DirectPut \
  --extended-s3-destination-configuration '{
    "RoleARN": "arn:aws:iam::<AccountId>:role/<Role created previously>",
    "BucketARN": "arn:aws:s3:::aws-waf-logs-your-destination-bucket",
    "Prefix": "!{partitionKeyFromLambda:accountId}/!{partitionKeyFromLambda:webaclName}/!{partitionKeyFromLambda:year}/!{partitionKeyFromLambda:month}/!{partitionKeyFromLambda:day}/",
    "ErrorOutputPrefix": "processing-failed/",
    "BufferingHints": {
      "SizeInMBs": 128,
      "IntervalInSeconds": 300
    },
    "CompressionFormat": "GZIP",
    "ProcessingConfiguration": {
      "Enabled": true,
      "Processors": [
        {
          "Type": "Lambda",
          "Parameters": [
            {
              "ParameterName": "LambdaArn",
              "ParameterValue": "arn:aws:lambda:us-east-1:<AccountId>:function:<your-lambda-name>:$LATEST"
            },
            {
              "ParameterName": "NumberOfRetries",
              "ParameterValue": "3"
            },
            {
              "ParameterName": "RoleArn",
              "ParameterValue": "arn:aws:iam::<AccountId>:role/<Role created previously>"
            },
            {
              "ParameterName": "BufferSizeInMBs", //Change this base on your need
              "ParameterValue": "1"
            },
            {
              "ParameterName": "BufferIntervalInSeconds",
              "ParameterValue": "300"
            }
          ]
        }
      ]
    },
    "DynamicPartitioningConfiguration": {
      "RetryOptions": {
        "DurationInSeconds": 300
      },
      "Enabled": true
    },
    "CustomTimeZone": "UTC"
  }'

6. Configure WAF or Firewall Manager

Finally, update your Web ACLs (or Firewall Manager policies) to use the delivery stream as the log destination.

In a Web ACL:

setting this up you’ll:

• Save up to 80% in logging costs compared to direct S3 delivery.
• Keep logs centralized and neatly organized by account and WebACL.
• Simplify analysis with Athena, OpenSearch, or SIEMs.

This approach is scalable and works seamlessly across multi-account environments managed with Firewall Manager, as well as standalone Web ACLs.