How to Build a Serverless Video Processing Pipeline with AWS Lambda and S3

🎬 Introduction

Processing videos at scale can be a real headache—servers get overloaded, costs skyrocket, and things break when you least expect them to. But what if you could process videos efficiently without worrying about infrastructure? That’s where AWS Lambda and S3 come in! With a fully serverless approach, you can automate video processing with minimal effort and cost.

In this post, I'll walk you through how to build a serverless video processing pipeline that automatically handles video uploads, processes them with FFmpeg, and stores the results in S3—without you lifting a finger after setup. Let’s dive in! 🎥

🛠️ Architecture Overview

Here’s how it works:

Upload: A user uploads a video file to an S3 bucket.
Trigger Lambda: S3 event notifications trigger an AWS Lambda function.
Process Video: The Lambda function runs FFmpeg to process the video (convert format, extract thumbnails, or transcode).
Store Output: The processed video is stored back in S3.

📋 Prerequisites

Before we get started, you’ll need:

An AWS account
Basic knowledge of AWS Lambda, S3, and FFmpeg
AWS CLI and Serverless Framework (optional for deployment)

🏗️ Step 1: Setting Up S3 Buckets

Let’s start by creating two S3 buckets—one for input videos and another for processed outputs.

aws s3 mb s3://my-video-input-bucket
aws s3 mb s3://my-video-output-bucket

🖥️ Step 2: Create a Lambda Function

AWS Lambda doesn’t come with FFmpeg by default, so we’ll use a Lambda Layer to include a compiled FFmpeg binary.

📌 1. Create the FFmpeg Layer

Download a precompiled FFmpeg binary and package it as a Lambda layer:

mkdir -p ffmpeg-layer/bin
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz | tar -xJ --strip-components=1 -C ffmpeg-layer/bin
zip -r ffmpeg-layer.zip ffmpeg-layer
aws lambda publish-layer-version --layer-name ffmpeg --zip-file fileb://ffmpeg-layer.zip --compatible-runtimes python3.8

📌 2. Write the Lambda Function

Now, let’s write the Lambda function that will process videos automatically.

import os
import boto3
import subprocess

s3_client = boto3.client("s3")
FFMPEG_PATH = "/opt/bin/ffmpeg"

def lambda_handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = record["s3"]["object"]["key"]
        local_input = "/tmp/input.mp4"
        local_output = "/tmp/output.mp4"
        output_bucket = "my-video-output-bucket"

        print(f"📥 Downloading {key} from {bucket}...")
        s3_client.download_file(bucket, key, local_input)

        print("🎞️ Processing video with FFmpeg...")
        cmd = [FFMPEG_PATH, "-i", local_input, "-vf", "scale=1280:720", local_output]
        subprocess.run(cmd, check=True)

        print("📤 Uploading processed video...")
        s3_client.upload_file(local_output, output_bucket, key.replace(".mp4", "_processed.mp4"))

        return {"status": "✅ Processing Complete", "output_file": key.replace(".mp4", "_processed.mp4")}

🔄 Step 3: Configure S3 Event Trigger

Next, we set up an S3 event notification to trigger our Lambda function whenever a video is uploaded.

{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "arn:aws:lambda:REGION:ACCOUNT_ID:function:VideoProcessor",
      "Events": ["s3:ObjectCreated:*"]
    }
  ]
}

🚀 Step 4: Deploy and Test

Now it’s time to deploy and test our setup! You can use AWS CLI or the Serverless Framework. Once deployed, upload a test video:

aws s3 cp test-video.mp4 s3://my-video-input-bucket/

Lambda will process the video and save the output in my-video-output-bucket. Pretty cool, right? 😃

🎯 Conclusion

And there you have it—a fully serverless video processing pipeline that scales effortlessly! This setup allows you to process videos without managing any servers, making it cost-efficient and hassle-free.