Anil KUMAR

Posted on Dec 14

Day 18: Image Processing Serverless Project using AWS Lambda

#lambda #terraform #aws

Today marks the Day 18 of 30 Days of Terraform challenge by Piyush Sachdeva. In this Blog, we will deep dive into a project of Images Processing Serverless Project using AWS Lambda entirely using Terraform. We’ll walk through an end-to-end image processing project i.e. from uploading a file to S3, to automatically processing it using a Lambda function, all orchestrated through Terraform.

Before diving deep into the project, lets first understand what exactly is AWS Lambda and why it is used and what is the significance of that.

AWS Lambda:

At its core, AWS Lambda is a serverless function. What is a serverless, you would imagine that for any service to be up and running we would need a service. yeah that's true.

When you want to host any app or server, you need to set up a service and deploy that app on that server like EC2 Instance.

But What do you mean by Serverless, There are no servers at all, which isn’t really true. There are servers involved, but the difference is we don’t manage them.

With Lambda, we don’t provision servers at all. Instead of thinking about machines, we think about functions. We simply write our application code, package it, and upload it as a Lambda function.

AWS takes care of everything else.

The servers still exist, but AWS manages them for us. We don’t worry about operating systems, scaling, or uptime. Our responsibility ends with the code.

And here’s the key difference:
a Lambda function does not run all the time and runs only when something triggers it.

An event could be:

A file being uploaded to an S3 bucket
A scheduled time (for example, every Monday at 7 AM)
A real-time system event

Project Architecture:

┌─────────────────┐
│  Upload Image   │  You upload image via AWS CLI or SDK
│   to S3 Bucket  │
└────────┬────────┘
         │ s3:ObjectCreated:* event
         ↓
┌─────────────────┐
│ Lambda Function │  Automatically triggered
│ Image Processor │  - Compresses JPEG (quality 85)
└────────┬────────┘  - Low quality JPEG (quality 60)
         │            - WebP format
         │            - PNG format
         │            - Thumbnail (200x200)
         ↓
┌─────────────────┐
│ Processed S3    │  5 variants saved automatically
│    Bucket       │
└─────────────────┘

We’ll have two S3 buckets:

One bucket where we upload the original image

Another bucket where the processed images will be stored

The first bucket is our source bucket. Whenever we upload an image to this bucket, that upload creates an S3 event.

And remember what we discussed earlier, events are exactly what serverless functions like Lambda are waiting for.

So as soon as an image is uploaded, that S3 event will trigger our Lambda function.

This Lambda function is where all the image processing logic lives. It will take the original image and automatically generate:

A JPEG image with 85% quality

Another JPEG image with 60% quality

A WebP version

A PNG version

And a thumbnail image resized to 200 by 200

All of this happens without us clicking any extra buttons or running any manual commands.

Components:

Upload S3 Bucket: Source bucket for original images
Processed S3 Bucket: Destination bucket for processed variants
Lambda Function: Image processor with Pillow library
Lambda Layer: Pillow 10.4.0 for image manipulation
S3 Event Trigger: Automatically invokes Lambda on upload

Terraform Code:

Now we will go through the code for the project execution.

The first step is to clone the repository and move into the Day 18 directory.

The repository lives here

Once we clone it, navigate into the day-18 folder and then into the terraform directory. This is where all the Terraform files for today’s project live.

1. Making unique resource names:

resource "random_id" "suffix" {
  byte_length = 4
}

locals {
  bucket_prefix         = "${var.project_name}-${var.environment}"
  upload_bucket_name    = "${local.bucket_prefix}-upload-${random_id.suffix.hex}"
  processed_bucket_name = "${local.bucket_prefix}-processed-${random_id.suffix.hex}"
  lambda_function_name  = "${var.project_name}-${var.environment}-processor"
}

As we know the names of the S3 bucket should be unique, we need to name the S3 buckets carefully making sure those names do not exist earlier. For this we use a resource of random_id in Terraform which generates random characters and we append them before and after our bucket name.

We build a common bucket prefix using the project name and environment
We create two bucket names i.e. one for uploads and one for processed images
We append the random suffix so the names stay unique
We also define a clear name for our Lambda function

2. Creating the Source S3 Bucket

This project begins with an S3 bucket that acts as the source bucket. We will be uploading image which will start the entire image processing workflow.

# S3 Bucket for uploading original images (SOURCE)
resource "aws_s3_bucket" "upload_bucket" {
  bucket = local.upload_bucket_name
}

We’re simply creating an S3 bucket and giving it the name we already prepared using locals.

3. Enabling Versioning:

resource "aws_s3_bucket_versioning" "upload_bucket" {
  bucket = aws_s3_bucket.upload_bucket.id

  versioning_configuration {
    status = "Enabled"
  }
}

Versioning helps us keep track of changes. If the same file name is uploaded again, S3 doesn’t overwrite the old object, it stores a new version instead. Eventhough it is not needed in this project, we will keep it as it is best industry standard.

4. Enabling Server-Side Encryption:

Next, we enable server-side encryption.

resource "aws_s3_bucket_server_side_encryption_configuration" "upload_bucket" {
  bucket = aws_s3_bucket.upload_bucket.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

Here we are enabling the server_side_encryption on the bucket so the files in the bucket will be encrypted. This ensures that any image uploaded to the bucket is encrypted at rest using AES-256. We don’t need to manage encryption keys manually, AWS takes care of that for us.

5. Making Bucket Private:

We are making the source bucket private so that no one else will be accessing this source bucket.

resource "aws_s3_bucket_public_access_block" "upload_bucket" {
  bucket = aws_s3_bucket.upload_bucket.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

By blocking public ACLs and policies, we make sure the bucket isn’t accidentally exposed. In real production systems, public access is usually handled through controlled layers in front of S3, not directly on the bucket itself.

6. Creating Destination S3 Bucket:

Now we will be doing the same above steps for Destination Bucket too.

resource "aws_s3_bucket" "processed_bucket" {
  bucket = local.processed_bucket_name
}

resource "aws_s3_bucket_versioning" "processed_bucket" {
  bucket = aws_s3_bucket.processed_bucket.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "processed_bucket" {
  bucket = aws_s3_bucket.processed_bucket.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "processed_bucket" {
  bucket = aws_s3_bucket.processed_bucket.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

We have done creating the bucket, enabling versioning on that, enabling server-side encryption and making that bucket access private.

IAM Roles and Policies:

This is an important section and we need to be very careful about what access does a Lambda role needs for this project.

Instead of hard-coding permissions or credentials, AWS uses roles to define what a service is allowed to do. In our case, we want the Lambda function to:

Write logs to cloudwatch so we can see what’s happening
Read images from the source bucket
Write processed images to the destination bucket

1. Creating the IAM Role for Lambda:

We start by creating an IAM role that Lambda can assume.

resource "aws_iam_role" "lambda_role" {
  name = "${local.lambda_function_name}-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

This role doesn’t give any permissions yet. It simply says:
This role can be assumed by AWS Lambda.
AWS even provides a policy generator to help create these documents, which makes life easier when you’re starting out.

2. Defining the Permissions with an IAM Policy:

Next, we create a policy that tells AWS exactly what this Lambda function is allowed to do.

resource "aws_iam_role_policy" "lambda_policy" {
  name = "${local.lambda_function_name}-policy"
  role = aws_iam_role.lambda_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:${var.aws_region}:*:*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:GetObjectVersion"
        ]
        Resource = "${aws_s3_bucket.upload_bucket.arn}/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject",
          "s3:PutObjectAcl"
        ]
        Resource = "${aws_s3_bucket.processed_bucket.arn}/*"
      }
    ]
  })
}

There are 3 blocks in the above IAM JSON Policy:

The first block allows the Lambda function to create log groups, log streams, and write logs. Without this, we’d have no visibility into what the function is doing, especially if something goes wrong.

The second block allows Lambda to read objects from the source bucket. This is how it gets access to the uploaded image.

The third block allows Lambda to write objects to the destination bucket. This is where all the processed images will be stored.

We can also give S3 full access but it is not recommended for Best practices.

With this IAM role and policy in place, our Lambda function will be able to:

Read images from S3 bucket
Process them using Pillow Libraries
Store the results to Destination Bucket
Write logs to Cloudwatch to inspect

3. LAMBDA LAYER (Pillow):

A Lambda layer is a way to package external libraries and dependencies separately from our function code. Instead of bundling everything inside the function zip, we place shared or heavy dependencies into a layer and then attach that layer to the Lambda function.

This keeps the function code clean and makes dependencies easier to manage.

resource "aws_lambda_layer_version" "pillow_layer" {
  filename            = "${path.module}/pillow_layer.zip"
  layer_name          = "${var.project_name}-pillow-layer"
  compatible_runtimes = ["python3.12"]
  description         = "Pillow library for image processing"
}

Here’s what’s happening:

filename points to a zip file that contains the Pillow library
layer_name gives the layer a clear, readable name
compatible_runtimes ensures this layer works with Python 3.12

How do we create the pillow_layer.zip file in the first place?

Because AWS Lambda runs on Linux, the dependencies inside the layer must also be built for a Linux environment. This is important, especially if you’re working on macOS or Windows.

To solve this, we use Docker.

4. LAMBDA FUNCTION (Image Processor):

Our Lambda function is written in Python and lives inside the repository. To package it correctly, we use a Terraform data source.

# Data source for Lambda function zip
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/lambda_function.py"
  output_path = "${path.module}/lambda_function.zip"
}

This data source takes the Python file, compresses it into a zip archive, and makes it ready for deployment.

Even though we’re working with a local file here, Terraform treats this as data it needs to reference during deployment which is exactly what data sources are designed for.

5. Defining the Lambda Function:

Now we define the Lambda function itself.

resource "aws_lambda_function" "image_processor" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = local.lambda_function_name
  role             = aws_iam_role.lambda_role.arn
  handler          = "lambda_function.lambda_handler"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  runtime          = "python3.12"
  timeout          = 60
  memory_size      = 1024

  layers = [aws_lambda_layer_version.pillow_layer.arn]

  environment {
    variables = {
      PROCESSED_BUCKET = aws_s3_bucket.processed_bucket.id
      LOG_LEVEL        = "INFO"
    }
  }
}

In the above block:

filename points to the zip file created earlier
function_name gives the Lambda function a clear identity
role attaches the IAM role we created, allowing the function to access S3 and logs
handler tells Lambda where execution begins in the Python file
runtime specifies Python 3.12
timeout is set to 60 seconds, which is more than enough for image processing
memory_size is set to 1024 MB to give the function enough resources

6. CloudWatch Logs:

We will create a cloudwatch log group to make sure logs are retained in a predictable way.

resource "aws_cloudwatch_log_group" "lambda_processor" {
  name              = "/aws/lambda/${local.lambda_function_name}"
  retention_in_days = 7
}

7. S3 EVENT TRIGGER:

Now, we will give S3 permission to invoke our Lambda function.

# Lambda permission to be invoked by S3
resource "aws_lambda_permission" "allow_s3" {
  statement_id  = "AllowExecutionFromS3"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.image_processor.function_name
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.upload_bucket.arn
}

# S3 bucket notification to trigger Lambda
resource "aws_s3_bucket_notification" "upload_bucket_notification" {
  bucket = aws_s3_bucket.upload_bucket.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.image_processor.arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.allow_s3]
}

Without the above permission, S3 events would never be able to trigger the function, even if everything else was configured correctly.

Whenever an object is created, in any way, invoke this Lambda function. As long as an object is created in the bucket, the event fires.

Deployment:

Now everything is set, now all that is left is to deploy them. We will be deploying this entire project using a shell script named deploy.sh

When we run this script, here’s what it does.

First, it performs a few basic checks. It makes sure:

AWS CLI is installed
Terraform is installed

If either of these is missing, the script stops and tells us exactly what’s wrong. This saves time and avoids confusion later.

Next, the script builds the Lambda layer.

This is an important step. Remember, the Pillow library needs to be compiled in a Linux environment to work correctly with AWS Lambda. Instead of doing this manually, the script calls another helper script that uses Docker to:

Spin up a Linux-based Python environment
Install Pillow in the correct directory structure
Package everything into a pillow_layer.zip file

Once that’s done, the script moves into the Terraform directory and runs the familiar commands:

terraform init
terraform plan
terraform apply

Terraform then takes over and creates every AWS resource we discussed:

Both S3 buckets
IAM roles and policies
Lambda layer
Lambda function
CloudWatch log group
S3 event trigger

When the deployment finishes, the script prints out something very useful information in the terraform output console.

Testing / Verification:

Setup is all done, Now all we need to do is to just upload an image to the upload bucket.

It can be any JPG or JPEG file. We can upload it using:

The AWS Console or AWS CLI

Any method that creates an object in the bucket

The moment the file is uploaded, the event is triggered.

Behind the scenes:

Lambda starts
The image is processed
Five new images are generated
All processed files appear in the destination bucket
If we open CloudWatch, we can also see the logs generated by the Lambda function, helpful for understanding what happened and for troubleshooting if something goes wrong.

Conclusion:

And with that, we’ve completed Day 18 of the 30 Days of Terraform Challenge.

We have gone a deep dive into a serverless project of Image processing using AWS Lambda involving S3 Buckets and CloudWatch.

DEV Community