loading...

Discussion on: Coding Sidekiq Workers the Right Way

Collapse
amit_savani profile image
Amit Patel

I need to process large(size and number) CSVs (couple of MBs having more than 0.1 million records each). Each record in the CSV has to be processed sequentially.

There are lot many relations comes into picture and so I have use memoization a lot to reduce db calls.

So in such cases, I cannot use the technique mentioned in point no 2.

Collapse
mxvanzant profile image
Michael VanZant

Load raw-ish data into a work table and then process in the database?