DEV Community

Discussion on: How do you merge millions of small files in a S3 bucket to larger single files to a separate bucket daily?

Collapse
 
talr98 profile image
Tal Rofe

Why do you need the entire data in your lambda? You should use streaming & multi-part uploading to S3

Collapse
 
whynotmarryj profile image
just a martian

1 min of files exceeds 500mb breaking tmp