DEV Community

Cover image for Memory efficient way of reading and downloading a large file in Ruby
Kartikey Tanna
Kartikey Tanna

Posted on • Originally published at kartikey.dev on

10 4

Memory efficient way of reading and downloading a large file in Ruby

To read a large file from the disk

File.foreach method reads the file line by line; that is why it is safe to use for large files.

It can accept the block to execute each line of a file.

Example:

File.foreach('example.jsonl') { |line| JSON.parse(line) }
Enter fullscreen mode Exit fullscreen mode

However, in the Ruby documentation, you will not find this method defined on the File class. It is defined on IO, which is a superclass of File.

To download the large files from the Internet

We can use IO.copy_stream to download the large files from the Internet. It will create a stream instead of loading the whole file into memory before writing.

Example:

IO.copy_stream(open('http://example.com/file.jsonl'), 'file.jsonl')
Enter fullscreen mode Exit fullscreen mode

To sum it up…

Memory friendly methods:

File.foreach, IO.copy_stream

Other methods to read a file:

File.read, File.readlines

Top comments (0)

Image of Docusign

🛠️ Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more