Marvelous Olaoluwa

Posted on May 22

Building a Serverless Image Processing Pipeline with AWS Lambda and S3

#automation #aws #serverless #tutorial

Serverless computing has transformed the way developers build scalable applications without managing infrastructure. One practical and highly efficient project is creating an automated image processing pipeline using Amazon S3 and AWS Lambda.

In this project, whenever a user uploads an image into an S3 bucket, AWS Lambda automatically resizes the image and stores the optimized version into another bucket. This approach is commonly used in social media platforms, e-commerce websites, blogs, and content delivery systems.

Why Build This Project?

This project demonstrates:

Event-driven architecture
Serverless automation
Real-world cloud workflows
Cost-efficient computing
Scalable media processing

It is an excellent beginner-to-intermediate AWS project for cloud engineers, DevOps engineers, and backend developers.

Project Architecture

Workflow

User uploads an image to an S3 bucket
S3 triggers a Lambda function automatically
Lambda processes and resizes the image
The optimized image is stored in another S3 bucket
Users can access optimized images globally

Services Used

AWS Service	Purpose
Amazon S3	Store uploaded and processed images
AWS Lambda	Automatically process uploaded files
IAM	Secure permissions management
CloudWatch	Logging and monitoring

Step 1: Create the Upload Bucket

Go to the AWS Management Console and navigate to Amazon S3

Create a bucket:

Bucket Name: image-upload-bucket-yourname
Region: Choose nearest region
Keep default configurations

This bucket stores original uploaded images.

Step 2: Create the Processed Images Bucket

Create another bucket named:

optimized-images-bucket-yourname

This bucket stores resized and optimized images generated by Lambda.

Step 3: Create IAM Role for Lambda

Navigate to AWS IAM

Create a role with:

Lambda permissions
S3 read/write access
CloudWatch logging permissions

Attach policies:

AmazonS3FullAccess
AWSLambdaBasicExecutionRole

Step 4: Create the Lambda Function

Navigate to AWS Lambda

Create a function:

Runtime: Python 3.12
Function Name: ImageResizeFunction

Lambda Function Code

import boto3
from PIL import Image
import io

s3 = boto3.client('s3')

DEST_BUCKET = 'optimized-images-bucket-yourname'

def lambda_handler(event, context):

    source_bucket = event['Records'][0]['s3']['bucket']['name']
    source_key = event['Records'][0]['s3']['object']['key']

    response = s3.get_object(Bucket=source_bucket, Key=source_key)
    image_content = response['Body'].read()

    image = Image.open(io.BytesIO(image_content))

    image.thumbnail((300, 300))

    buffer = io.BytesIO()
    image.save(buffer, 'JPEG')
    buffer.seek(0)

    s3.put_object(
        Bucket=DEST_BUCKET,
        Key=f"resized-{source_key}",
        Body=buffer,
        ContentType='image/jpeg'
    )

    return {
        'statusCode': 200,
        'body': 'Image processed successfully'
    }

Step 5: Configure S3 Trigger

Inside the upload bucket:

Go to Properties
Open Event Notifications
Create Event Notification
Select:

PUT Object
1. Choose destination:
Lambda Function
1. Select ImageResizeFunction

Now every upload automatically triggers processing.

Step 6: Test the Application

Upload any JPG or PNG image into the upload bucket.

Expected result:

Lambda automatically processes the image
Resized image appears in the optimized bucket
Logs are visible in CloudWatch

Monitoring with CloudWatch

Navigate to Amazon CloudWatch

CloudWatch helps you:

Monitor Lambda executions
Debug application issues
Track failures and performance
Analyze invocation metrics

Real-World Use Cases

This architecture is used in:

E-commerce platforms
Social media apps
Portfolio websites
Blogging platforms
Media streaming services
AI-powered image analysis systems

Key Benefits

Scalability

AWS automatically scales based on uploads.

Cost Optimization

You only pay when Lambda executes.

Automation

No manual image resizing needed.

High Availability

S3 provides durable and highly available storage.

Security

IAM roles secure access between services.

Challenges Faced During Deployment

While implementing this project, some practical issues appeared:

Incorrect IAM permissions blocked S3 access
Large image uploads exceeded Lambda timeout
Missing dependencies caused deployment failures
Event notification misconfiguration stopped triggers

These are common real-world DevOps and cloud engineering challenges that improve troubleshooting skills.

Improvements You Can Add

Future enhancements:

Convert images to WebP format
Add AI-based image moderation
Integrate Amazon CloudFront CDN
Create a frontend upload portal
Add DynamoDB metadata storage
Implement authentication with Cognito

Conclusion

Building a serverless image processing pipeline using AWS Lambda and Amazon S3 is one of the best hands-on projects for understanding event-driven cloud architecture.

This project provides practical exposure to:

Cloud automation
Infrastructure scalability
Monitoring and observability
Security best practices
Real-world backend workflows

As businesses increasingly adopt serverless technologies, projects like this help developers gain production-level cloud engineering experience while building scalable and efficient applications.

DEV Community