DEV Community

Discussion on: How do you merge millions of small files in a S3 bucket to larger single files to a separate bucket daily?

Collapse
leouofa profile image
Leonid Medovyy

Assuming you need to do it daily. You could write a script to download the json from the buckets you need to hard drive, merge them into one file and upload them to elsewhere. Ruby has gems that make working with AWS easier, but you could use any other language.