DEV Community

just a martian
just a martian

Posted on

How do you merge millions of small files in a S3 bucket to larger single files to a separate bucket daily?

We have a situation where we're currently uploading events to s3 in real-time. The result is roughly 30 million tiny json files (<1kb) per day. These files sit in a raw layer bucket with the following folder format "#{folder_name}/#{year}/#{month}/#{day}/#{hour}/#{minute}/#{System.os_time()}-#{file_name}.#{file_ext}". We want to send this to a data warehouse for analytics but need the files to be much larger (150-200mb). What solutions are there for merging json files from a s3 bucket back to a separate s3 bucket. I have tried developing a lambda to tackle this problem but it was not enough since all the files must be downloaded in /tmp and lambda ran out of memory. Please help :)

Discussion (5)

tiguchi profile image
Thomas Iguchi

I think it mostly depends what the expected target output format is. If you simply need to concatenate all events as a list (JSON array), then it could be probably done by opening an output stream for a file in the target S3 bucket, and writing each JSON file to it one after the other. Memory consumption should be constant, given that all input JSON files are the same size. You only need to make sure that the list of event file paths / handles is not loaded into a collection all at once, so you don't run out of memory.

But it sounds like you need to apply more complicated merge logic? What's an example for an event file and what's the expected result format?

peterb profile image
peterb • Edited

Redshift Spectrum does an excellent job of this, you can read from S3 and write back to S3 (parquet etc) in one command as a stream

e.g. take lots of jsonl event files and make some 1 GB parquet files
create external table mytable (....)
row format serde ''
stored as inputformat 'org.apache.hadoop.mapred.TextInputFormat'
outputformat ''
location 's3://bucket/folderforjson/path/yesr/month/day ...'

upload ('select columns from mytable where ...')
to 's3://bucket/folderforparquet/year/month/day...'
iam_role 'arn:aws:iam::123456789:role/prod....-role'
format parquet
partition by (year, month, day)

You can buy Redshift by the hour, and Redshift Spectrum is $5 per TB

leouofa profile image
Leonid Medovyy

Assuming you need to do it daily. You could write a script to download the json from the buckets you need to hard drive, merge them into one file and upload them to elsewhere. Ruby has gems that make working with AWS easier, but you could use any other language.

talr98 profile image

Why do you need the entire data in your lambda? You should use streaming & multi-part uploading to S3

whynotmarryj profile image
just a martian Author

1 min of files exceeds 500mb breaking tmp