Good question. 🤔
Introduction
I have been playing around with a YouTube clone I call FooTube. I had set up video uploads to be sent fro...
For further actions, you may consider blocking this person and/or reporting abuse
Hey ben,
Its nice to see someone refactored my code, especially with the newish layer system, but theres a couple of things you can do that will fit your use case better.
First in my original I use jpg and -ss because I only needed one screenshot and it had to be jpg because of the file system we were using at the time.
You would be better to change to output codec to png - you can just remove the vcodec (theres also another line in there I left in which doesnt do anything for your use case).
For permissions you can move the ffmpeg/ffprobe to /tmp and then run chmod there but you need to update you ffmpeg path.
Actually, in your use case its actually better not to use any of this, it would be better to use something like aws elemental media convert to generate thumbnails/gifs etc.
Though if you wanted to use this function, its better to use SNS to push to your application when its completed. So send to SNS after completing upload - SNS pushes to your application/sends email w/e.
You could also be crazy and use AWS Rekognition to autogenerate tags for the videos.
With your current setup you may run into memory problems seeking 75% on larger files, whats your current largest test files? I assume in your current setup you also use ffprobe to output JSON to get the metadata right?
Its good to see a refactored version of my code for a fake youtube. Never though of using it like that.
Regarding the memory problem:
I wanted to implement this for one of my use cases and was worried about the memory issues too.
But this article here explains how the FFmpeg works with URL and seek. wistia.com/learn/marketing/faster-...
And looks like we are good?
Thanks for your article, it was helpful and the best out there on this topic. I added a Resources section to give credit where credit's due.
I did some testing with PNG and I was surprised the PNGs were larger. I've been under the general impression that PNG for graphics, JPG for photos.
JPG ~ 12KB
PNG ~ 61KB
Thanks for the tip on SNS, it looks very useful. I would have to do more testing and use some larger files, I think I largest video I tested with so far was only 15MB.
I figured trying to recreate what I could of YouTube would be a good way to learn stuff. So far it has. Obviously, their code is complex, but its fun to come with a function to determine what video to play next for example.
Hey Derek:
For the output codec, why do you suggest using png?
Hey Ben
Nice guide - really useful for what i am coding now.
I pretty much copy-pasted the code after fitting variables to my needs.
I have an issue though. When i test the function the reports all well and green lights all the way, and it does generate the images, but the images are 0 byte size and empty.
Do you have any idea what could be wrong?
Cheers
hey, did you ever figure out the solution to this? it's uploading an empty file for me and not showing any error :/
Hi Ben,
it was a great article.
I reproduced it using docker and S3. No lambda.
I will use your post to write an article on how to do this in a highly available manner.
Depending on the number of images requested, I generate the requested screenshots.
If the duration of the video = 19 seconds and you request 3 images, that makes (19-1) / 3 = 7
image 1 = 7,
image 2 = 7 + 7 = 14,
image 3 = 7 + 7 + 7 = 21
Image 3 is 21 seconds and 21 seconds longer than the duration of the video, it will be ignored and therefore we only generate frame 1 and frame 2.
To avoid image name conflicts generated on disk, I generate my own file names.
const fileNameOnDisk = Math.floor (1000000000000 + Math.random () * 9000000000000);
When the image is generated and uploaded to AWS, I delete the files from disk asynchronously.
I added time markers to detect excessively long executions. We have three important stages:
1- Read the file with ffprobe to get the details of the file, mainly the duration of the video. (maximum 30 seconds)
2- Use ffmpeg to generate the screenshots (maximum 3 seconds per image)
3- Upload the files to s3 (maximum 5 seconds per image)
You should put markers on these three steps and therefore trigger the email log when it exceeds your maximum time.
This allows you to be proactive on issues.
Hey Ben,
Thanks for the post. I get 3 jpg files in my bucket's 'cache' folder which are black frames with the following striped across the top.
The image “myapp.s3-us-west-1.amazonaws.com/c... cannot be displayed because it contains errors."
No thumbnail folder is created.
Any thoughts?
Thanks!
Thx!
I'm not sure. I guess the first thing i might try is logging the srcKey and dstKey variables since the dst key should be creating the thumbnails folder when an image is uploaded. Put the console.log statements in a then you can check them in the cloud watch reports. I copied my exact code to a gist just i case there was some kind of typo or something.
Thanks for the response. I ended up using my web server & ffmpeg to create the thumbs and upload them. You have inspired me to learn more about Lambda, however! Thx
Hi Ben,
Thank you for the article.
I keep facing the same issue with the permission for the binary file.
Instead of linux distribution I used windows zip to zip the file for creating layers.
START RequestId: f1840de8-2d36-4453-98ff-d0d6b66b88db Version: $LATEST
2020-04-07T13:56:38.652Z f1840de8-2d36-4453-98ff-d0d6b66b88db INFO ffmpegPath /opt/nodejs/ffmpeg
2020-04-07T13:56:38.773Z f1840de8-2d36-4453-98ff-d0d6b66b88db INFO Error: spawn /opt/nodejs/ffmpeg EACCES
at Process.ChildProcess._handle.onexit (internal/child_process.js:267:19)
at onErrorNT (internal/child_process.js:469:16)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
errno: 'EACCES',
code: 'EACCES',
syscall: 'spawn /opt/nodejs/ffmpeg',
path: '/opt/nodejs/ffmpeg',
spawnargs: [
'-y',
'-ss',
5,
'-t',
3,
'-i',
'thumbnail-test-zuku.s3.amazonaws.c...
'fps=10,scale=320👎flags=lanczos,split [o1] [o2];[o1] palettegen [p]; [o2] fifo [o3];[o3] [p] paletteuse',
'-f',
'gif',
'pipe:1'
]
}
I had similar problems using the Windows zip. I ended up using the Windows Subsystem For Linux environment to zip any files. There is a command line zip utility. This resolved the errors.
I was also having the problem of the images being written coming out at 0 bytes. Also, regarding creating the layer, you don't need to package up ffmpeg yourself as you can just deploy a pre-made layer from the AWS SAR.
My solution ended up being quite different from that outlined here but your article did get me started; for others still having issues I have written a tutorial on my own blog which documents how I got it working.
Hi Ben,
it was a great article.
I reproduced it using docker and S3. No lambda.
I will use your post to write an article on how to do this in a highly available manner.
Depending on the number of images requested, I generate the requested screenshots.
If the duration of the video = 19 seconds and you request 3 images, that makes (19) / 3 = 7
image 1 = 7,
image 2 = 7 + 7 = 14,
image 3 = 7 + 7 + 7 = 21
Frame 3 is 21 seconds and 21 seconds longer than the video length, it will be skipped and so we generate only frame 1 and frame 2.
To avoid image name conflicts generated on disk, I generate my own file names.
const fileNameOnDisk = Math.floor (1000000000000 + Math.random () * 9000000000000);
When the image is generated and uploaded to AWS, I delete the files from disk asynchronously.
I added time markers to detect excessively long executions. We have three important stages:
1- Read the file with ffprobe to get the details of the file, mainly the duration of the video. (maximum 30 seconds)
2- Use ffmpeg to generate the screenshots (maximum 3 seconds per image)
3- Upload the files to s3 (maximum 5 seconds per image)
You should put markers on these three steps and therefore trigger the email log when it exceeds your maximum time.
This allows you to be proactive on all issues.
can you give me your aws lambda ffmpeg ARN number ??