DEV Community

Discussion on: How do you merge millions of small files in a S3 bucket to larger single files to a separate bucket daily?

talr98 profile image

Why do you need the entire data in your lambda? You should use streaming & multi-part uploading to S3

whynotmarryj profile image
just a martian Author

1 min of files exceeds 500mb breaking tmp