DEV Community

Cover image for Memory efficient way of reading and downloading a large file in Ruby
Kartikey Tanna
Kartikey Tanna

Posted on • Originally published at kartikey.dev on

10 4

Memory efficient way of reading and downloading a large file in Ruby

To read a large file from the disk

File.foreach method reads the file line by line; that is why it is safe to use for large files.

It can accept the block to execute each line of a file.

Example:

File.foreach('example.jsonl') { |line| JSON.parse(line) }
Enter fullscreen mode Exit fullscreen mode

However, in the Ruby documentation, you will not find this method defined on the File class. It is defined on IO, which is a superclass of File.

To download the large files from the Internet

We can use IO.copy_stream to download the large files from the Internet. It will create a stream instead of loading the whole file into memory before writing.

Example:

IO.copy_stream(open('http://example.com/file.jsonl'), 'file.jsonl')
Enter fullscreen mode Exit fullscreen mode

To sum it up…

Memory friendly methods:

File.foreach, IO.copy_stream

Other methods to read a file:

File.read, File.readlines

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more