Introduction :
Hello everyone,
Recently I had the opportunity to work on a music application, where I have developed a REST API to download the audio file given a video ID of the source youtube video. In this blog, I will be covering my experience while building the project like common pitfalls, design decisions I had to undergo while working on the project and maximum performance I could achieve given the constraint so let's dive into the project.
Youtube_dl open-source library :
The initial challenge while working on downloading content from youtube, is to find an open-source library, fortunately, I found a library called youtube-dl open-source library from GitHub. After going through the library and experimenting with python SDK I was comfortable working with the library, the interesting part is the SDK gives an option to download the directly best quality mp3 file makes its execution locally easy and not demanding much attention, problems araised when I tried to implement in the cloud.
Lambda and its limitation :
As my code started working locally I packaged my lambda function file and its dependent libraries into a zip file and uploaded it into lambda configured with python environment and tried to download the file into the temporary repository in lambda. To my surprise, the lambda function produced an error which is due to the fact that mp4 to mp3 conversion requires FFmpeg which is Linux kernel depended, unfortunately, we don't have access to the operating system of lambda so it was like a dead-end for conversion, but later I got to know about lambda layers and we could initialize a layer on top of the function with FFmpeg library, it was a great way to get introduced to lambda layers and I have implemented it.
The lambda function started running and given the appropriate output, but the heavy load of downloading a file and later converting it into mp3 has been pushing the limits of lambda, especially when given a large video file, making the implementation not much reliable and also due to limitation of the size of tmp repository to 512Mb handling a video larger than it lead to errors. To overcome these limitations I have decided the following steps:
- It's time to Decouple everything
- Avoiding lambda /tmp repository
- Elastic transcoder to the rescue!
- What does my architecture now look like?
1. It's time to Decouple everything
Since discussed above the computing and storage limitations of lambda, I tried to break the process into multiple small processes hence decoupling the application. The first small process would be handling the API gateway request and revert back the response of the s3-presigned URL if the file present or else send file is processing. Allocating the lambda function just to handle API gateway requests helps us satisfy the criteria to send a response within 29 seconds a request being made from API gateway or else it produces an error message, Adding any additional load on lambda can hinder the time constraints at hand.
So the functionality handled by lambda is to check if the requested file is processed or not from dynamoDB and send a response based on the status of the file being processed.
2. Avoiding lambda /tmp repository
The way youtube-dl works is, initially it downloads the mp4 file to the path location mentioned in its sdk, where once the file download is completed it starts processing audio conversion, here we try to decouple the processes as discussed above, where we create one lambda function for transcoding and other lambda function for downloading, since we are using 2 different lambda function it doesn't make sense to download directly to /tmp repository instead, we need to figure out a way to divert the incoming data to another more elastic storage which in our case is S3.
there are few implementations I found using node js, popular ones were from the source cloudonaut.io named Download youtube videos using lambda and store them in s3 but since many of the implementations found were on node js, it made it difficult to implement in python, anyways from stack overflow i could be able to establish establish the dedicated connection, the code of the python version could be found in my github repository under youtube downloader lambda function.
3. Elastic transcoder to the rescue!
Performing transcoding of media files from mp4 to the desired quality of mp3 puts a strain on lambda especially for long media files, especially in our case where we are planning to perform transcoding of media files of length 4 hours, hence using AWS service elastic transcoder comes to rescue!.
While configuring elastic transcoder, we usually set up a data pipeline with source and target S3 buckets while selecting presetID as "1351620000001-300010" which is 320kpbs audio quality. once, everything is configured writing lambda function for passing the s3 mp4 objects into the pipeline and make sure the lambda function gets triggered by the S3 event notification for .mp4 format.
The above operation makes sure that the transcoding operation is performed immediately after the mp4 file is uploaded to s3 Next, we need to initialize another lambda function which performs operations such as updating data in dynamodb, deleting mp4 file that was generated to activate this lambda function we use similar s3 event notification but, instead we trigger for .mp3 file format being uploaded.
4. What does my architecture now look like?
The github code consist of source code for 4 lambda functions we used, soon ill be updating a cloud formation template making it easy to deploy.
Top comments (0)