I was able to process a large AWS S3 file (million records) parallely with S3 Select 😋
Thought of sharing my appraoch with DEV Community 😉
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink.
Hide child comments as well
Confirm
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I was able to process a large AWS S3 file (million records) parallely with S3 Select 😋
Thought of sharing my appraoch with DEV Community 😉
Parallelize Processing a Large AWS S3 File
Idris Rampurawala ・ Jun 25 ・ 6 min read