On-Demand Video Streaming! AWS Elemental MediaConvert and AWS Lambda

“Challenges faced to find the solution of how to convert the video file format from mp4 to hls or mp4 in compressed file size within some seconds”. I have checked different ways so that I can get my video file format conversion to the same or different format with compressed file size within a few seconds and within my budget. I got the solution to use an amazon elastic transcoder service or aws elemental mediaconvert. I have chosen aws elemental mediaconvert, as it is cheaper in cost as compared to transcoder and includes additional features with high security[DRM]. In aws elemental mediaconvert, i am able to stream the video file on On-demand basis. Also the job can be automatically able to create whenever there is a video file in source bucket and s3 trigger on lambda function which transcodes the file and stores it in s3 destination bucket as per output group settings. Also can be able to get notified for complete and error status of job. In terms of cost, it is a very cheaper one for transcoding a file in other formats.

AWS Elemental MediaConvert is a file-based video transcoding service with broadcast-grade features. It allows you to easily create video-on-demand (VOD) content for broadcast and multiscreen delivery at scale. The service combines advanced video and audio capabilities with a simple web services interface and pay-as-you-go pricing. With AWS Elemental MediaConvert, you can focus on delivering compelling media experiences without having to worry about the complexity of building and operating your own video processing infrastructure. To learn more, read the AWS Elemental MediaConvert.

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code as a ZIP file or container image, and Lambda automatically and precisely allocates compute execution power and runs your code based on the incoming request or event, for any scale of traffic. You can set up your code to automatically trigger from 140 AWS services or call it directly from any web or mobile app. You can write Lambda functions in your favorite language (Node.js, Python, Go, Java, and more) and use both serverless and container tools, such as AWS SAM or Docker CLI, to build, test, and deploy your functions. To learn more, read the AWS Lambda.

In this post, you will get to know how to convert any on-demand video file to HLS or MP4 format using MediaConvert and Lambda. Also get notified via email regarding job complete or error status. Here I have taken two S3 buckets and then created a lambda function for convert of formats with IAM role and set a S3 trigger on it which will automatically generate the output video files of same or different formats in the s3 destination bucket. Setup of cloudwatch event on mediaconvert for complete and error job status with notification service.

Prerequisites

You’ll need an Amazon Simple Storage Service for this post. Getting started with amazon Simple Storage Service provides instructions on how to create a bucket in simple storage service. For this blog, I assume that I have two buckets for input and output of videos.

Architecture Overview

Diagram

The architectural diagrams show the overall deployment architecture with data flow, AWS S3, AWS Elemental MediaConvert, AWS Lambda, Cloudwatch event and Amazon SNS.

Solution overview

The blog post consists of the following phases:

1. Create IAM role and folder in source bucket
2. Create Lambda function with S3 trigger
3. Get notification via email for Job status using Simple Notification Service and Cloudwatch Event
4. Testing on-demand video file streaming
5. Check of cloudwatch log groups and metrics for media convert

I have two S3 bucket for input and output of video as below →

Phase 1: Create IAM role and folder in source bucket

1. In AWS IAM console, create a role named as MediaConvertRole with policy as s3 full access and api gateway invoke full access.
2. Create a role named as VODLambdaRole with policy as lambda basic execution role and inline policy as VODLambdaPolicy for mediaconvert.
3. Create a folder in the source bucket named as inputs for incoming video files as on demand basis.

Phase 2: Create Lambda function with S3 trigger

1. In AWS Lambda Console, create a function named as VODLambdaConvert with python3.8 and VODLambdaRole. Set runtime setting of handler as convert.handler.
   
   

2. Import zip file in function which contains convert.py and job.json. Convert.py consist of code for convert of video format and job.json consist of output group setting format.
   
   
   
   
   
   
   
   
   

3. In the configuration tab, set the timeout as 2min 0sec, environment variable as shown in screenshot and monitoring enabled.
   
   
   

4. Set S3 trigger on lambda function as choose source bucket, event type as object created and prefix as inputs/.
   

Phase 3: Get notification via email for Job status using Simple Notification Service and Cloudwatch Event

1. In the Amazon Simple Notification Service console, create a topic named as VODNotification with default settings. And create a subscription for email notification basis and do subscription confirmation in subscribed email id. Update the access policy of SNS with topic arn as shown in screenshot.
   
   
   
   
   
   
   
   
   

2. Goto Cloudwatch console, Create a rule named as VODNotifyEventRule with complete and error policy. And set of target as SNS topic and input transformer pattern.
   
   
   

Phase 4: Testing on-demand video file streaming

1. Input of video file in source bucket named as vod-inputfolder-bucket. The S3 trigger on the lambda function starts the job process in media convert and the event gets notification on subscribed email once the running job gets finished with complete or error status.
   
   
   
   

2. Check the job detailed summary in screenshot such as duration of job, start and completion of job with input and output details.
   
   
   
   
   

3. Checkout the destination bucket named as vod-outputmediafolder-bucket with convert formats of input video file in hls, mp4 and thumbnails. Also check the streaming of video in VLC Player.
   
   
   
   
   
   

Phase 5: Check of cloudwatch log groups and metrics for mediaconvert

1. You can get the logs of jobs in the cloudwatch log group.
   
   

2. You can also check the metrics of mediaconvert in cloudwatch.
   
   

Clean-up

Delete the environment as: S3 buckets, IAM roles, Lambda function and cloudwatch log group. Note: Mediaconvert jobs created cannot be deleted.

Pricing

I review the pricing and estimated cost of this example -

For AWS Elemental MediaConvert →
Cost for BOM-P-AVC-HD-MHQ-30 of 12.307 minutes = $0.52
Cost for BOM-P-AVC-SD-MHQ-30 of 36.090 minutes = $0.76
Cost for BOM-B-AVC-HD-S-30 of 383.913 minutes = $5.76
Cost for BOM-B-AVC-SD-S-30 of 383.913 minutes = $2.88
Cost of mediaconvert = $(0.52+0.76+5.76+2.88) = $9.92
For Lambda →
Cost of request and lambda-GB per second = $0.00
For Simple Storage Service →
Cost of APS3-Requests = $0.04
For cloudwatch, data transfer and simple notification (under free tier) →

Cost = $0.00
Total Cost = $(9.92+0.00+0.04+0.00) = $9.96

Summary

In this post, I have shown you how to convert any video file to HLS or MP4 format using MediaConvert and Lambda.

For more details on mediaconvert and lambda service, Checkout Get started with AWS Elemental MediaConvert and AWS Lambda, open the AWS Elemental MediaConvert console and AWS Lambda console. To learn more, read the AWS Elemental MediaConvert documentation and AWS Lambda documentation.

Comments

[Vinit K]

Can you share the lambda source code. Unable to see from image

[GargeeBhatnagar]

Sure, please mail your ID..

[devanand73]

GargeeBhatnagar , excellent article. can you kindly share convert.py file to [e-mail] as i want to try out.

[GargeeBhatnagar]

Thanks😀 ! Please share your mail ID..

[Vinit K]

[e-mail] Thank you so much

[Everton Wolff]

Is it possible to send the lambda code and the JSON of the works by email?
email: [e-mail]
@evertonwolff

[S.Shah]

Hello gargee, Your tutorial is great, but code is readable from the pictures, would you please share the code on mail [e-mail] .
Thanks

[진호 최]

@bhatnagargargee
Great post.
Can you share the lambda code and the JSON of the works by email?
email : [e-mail]

[Quốc Tú]

Can I get your code Python Lambda? My email is [e-mail]

[Mohamed Fayaz]

Great read, just an immediate thought, your account number and arns are usually be treated as sensitive information I believe so you could obfuscate In your screenshots 😊

[GargeeBhatnagar]

Hi Mohamed,
Thanks for suggestion! Will surely take care of it😀

[yashwantvarma]

This is Greatfull stuff!!, Thanks for sharing

[GargeeBhatnagar]

ThankYou😀

[Kevin N. Murphy]

Great post, thanks for making it!

[GargeeBhatnagar]

ThankYou😀

[sumit kumar bighaniya]

Can You share convert.py file?