DEV Community

Discussion on: How do you merge millions of small files in a S3 bucket to larger single files to a separate bucket daily?

Collapse
talr98 profile image
TalR98

Why do you need the entire data in your lambda? You should use streaming & multi-part uploading to S3

Collapse
whynotmarryj profile image
just a martian Author

1 min of files exceeds 500mb breaking tmp