HowTo: Working with large files in Ruby efficiently.

How can we read files in Ruby?

* Testing file generated by running a following command:

❯ openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem

It has a clearly defined the beginning and the end of the file which fill be useful while reading the files.

1. File.read() which is actually IO.read():

> file = File.read("cert.pem")
=> "-----BEGIN CERTIFICATE-----\nMIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\nTDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\nzqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\nTgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\naigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\nraNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\nGNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\naeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\nlybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n-----END CERTIFICATE-----\n"
> file.bytesize
=> 956
> file.class
=> String

read method reads the entire file's content and assigns it to the variable as single String.

2. File.new() and its synonym File.open():

> file = File.new("cert.pem")
=> #<File:cert.pem>
> lines = file.readlines
=> ["-----BEGIN CERTIFICATE-----\n",
 "MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\n",
 "TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\n",
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\n",
 "zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n",
 "1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\n",
 "TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n",
 "7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\n",
 "aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n",
 "4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\n",
 "raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n",
 "9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\n",
 "GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\n",
 "aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\n",
 "lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n",
 "-----END CERTIFICATE-----\n"]
> lines.class
=> Array

new or open methods returns an instance of the File class on which we can call readlines method which reads the entire file's content, splits it line by line and returns an Array of Strings where one element is one line from the file.

3. File.readlines() which is actually IO.readlines():

> lines = File.readlines("cert.pem")
=> ["-----BEGIN CERTIFICATE-----\n",
 "MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\n",
 "TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\n",
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\n",
 "zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n",
 "1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\n",
 "TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n",
 "7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\n",
 "aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n",
 "4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\n",
 "raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n",
 "9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\n",
 "GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\n",
 "aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\n",
 "lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n",
 "-----END CERTIFICATE-----\n"]
> lines.class
=> Array

Here, we have the same output as in the previous example by calling just class method readlines on File class.

4. File.foreach() which is actually IO.foreach():

> file = File.foreach("./cert.pem")
=> #<Enumerator: ...>
> file.entries
=> ["-----BEGIN CERTIFICATE-----\n",
 "MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\n",
 "TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\n",
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\n",
 "zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n",
 "1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\n",
 "TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n",
 "7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\n",
 "aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n",
 "4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\n",
 "raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n",
 "9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\n",
 "GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\n",
 "aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\n",
 "lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n",
 "-----END CERTIFICATE-----\n"]
> lines.class
=> Array

foreach method returns an Enumerator instance on which we call entries which returns an Array of String, again each element is a line from the file.

As we can see above there are many methods that allow us to read the file. However which one should we use and why? Let's create a large file and check those methods again!

Which methods should we use to read large files?

Generating our test file

At first, let's generate a large file with randomized data inside:

require 'securerandom'
one_megabyte = 1024 * 1024

name = "large_1G"
size = 1000

File.open("./#{name}.txt", 'wb') do |file|
  size.times do
    file.write(SecureRandom.random_bytes(one_megabyte))
  end
end

• w - Write-only, truncates existing file to zero length or creates a new file for writing.
• b - Binary file mode. Suppresses EOL <-> CRLF conversion on Windows. And sets external encoding to ASCII-BIT unless explicitly specified.

As the result we generated 1GB file:

❯ ls -lah
...
-rw-r--r--   1 user  user   1.0G Aug 31 22:10 large_1G.txt

Defining our metrics and profilers

There are probably 2 the most important metrics that we would like to track in our experiment:

• Time - How long does it take to open and read the file?
• Memory - How much memory does it take to open and read the file?

Also there will be one additional metric describing how many objects were freed by Garbage Collector.

We can prepare simple profiling methods:

# ./helpers.rb
require 'benchmark'

def profile_memory
  memory_usage_before = `ps -o rss= -p #{Process.pid}`.to_i
  yield
  memory_usage_after = `ps -o rss= -p #{Process.pid}`.to_i

  used_memory = ((memory_usage_after - memory_usage_before) / 1024.0).round(2)
  puts "Memory usage: #{used_memory} MB"
end

def profile_time
  time_elapsed = Benchmark.realtime do
    yield
  end

  puts "Time: #{time_elapsed.round(2)} seconds"
end

def profile_gc
  GC.start
  before = GC.stat(:total_freed_objects)
  yield
  GC.start
  after = GC.stat(:total_freed_objects)

  puts "Objects Freed: #{after - before}"
end

def profile
  profile_memory do 
    profile_time do 
      profile_gc do
        yield
      end
    end 
  end 
end

Testing our methods for reading files

• .read

file = nil
profile do
  file = File.read("large_1G.txt")
end

Objects Freed: 39
Time: 0.52 seconds
Memory usage: 1000.05 MB

• .new + #readlines

file = nil
profile do
  file = File.new("large_1G.txt").readlines
end

Objects Freed: 39
Time: 4.19 seconds
Memory usage: 1298.4 MB

• .readlines

file = nil
profile do
  file = File.readlines("large_1G.txt")
end

Objects Freed: 39
Time: 4.24 seconds
Memory usage: 1284.61 MB

• .foreach

file = nil
profile do
  file = File.foreach("large_1G.txt").to_a
end

Objects Freed: 40
Time: 4.42 seconds
Memory usage: 1284.31 MB

The examples we can see above allowed us to read the whole file and store it in local memory as one String or as an Array of Strings (each line from the file as one element in the Array).

As we can see, it requires at least as much memory as the size of the file:

• one String - 1GB file requires 1GB of memory.
• an Array of Strings - 1GB memory for file's content + additional memory for an Array (+- 300MB here). This approach has one advantage, we can access whichever line of the file we want as long as we know which line is it.

At this point we can see that the methods that we tested are not really efficient. The bigger the file, the more memory we need. In longer term this approach might lead to some serious consequences, even killing the application.

Now, we need to us ourselves a question. Can we process our files line by line? If so, then we can read our files in a different way:

• .new + #each

file = nil

profile do
  file = File.new("large_1G.txt")
  file.each { |line| line }
end

Objects Freed: 4100808
Time: 2.08 seconds
Memory usage: 57.68 MB

• .new + #advise + #each

file = nil

profile do
  file = File.new("large_1G.txt")
  file.advise(:sequential)
  file.each { |line| line }
end

Objects Freed: 4100808
Time: 2.22 seconds
Memory usage: 55.71 MB

Calling #advise method announces an intention to access data from the current file in a specific pattern. No major improvement here with using #advise method.

• .new + #read - reading chunk by chunk

file = nil
chunk_size = 4096
buf = ""

profile do
  file = File.new("large_1G.txt")
  while buf = file.read(chunk_size)
    buf.tap { |buf| buf }
  end
end

Objects Freed: 256037
Time: 1.27 seconds
Memory usage: 131.64 MB

We defined the chunk as 4096 bytes and we read our file chunk by chunk. Depending on the structure of your file this approach might be useful.

• .foreach + #each_entry

file = nil

profile do
  file = File.foreach("large_1G.txt")
  file.each_entry { |line| line }
end

Objects Freed: 4100809
Time: 2.22 seconds
Memory usage: 53.02 MB

Creating an Enumerator instance as file and reading file line by line using each_entry method.

First thing we can notice is that memory usage is way lower. Main reason for that is that we read the file line by line and when the line is processed then it's garbage collected. We can see that by the size of the Objects Freed, it's quite high.

We also tried to use here an #advise method which we can tell how we want to process our file. More about IO#advise can be found in the documentation. Unfortunately, it didn't help us out here.

Except IO#each method we have also similar methods like IO#each_byte (reading byte by byte),IO#each_char (reading char by char) and IO#each_codepoint.

In the example with reading by chunks (IO#read) the memory usage will vary depending on the chunk size. If you find this way useful you can experiment with the chunk size.

When using IO.foreach we operate on Enumerator which gives us a few more methods like: IO#each_entry, IO#each_slice, IO#each_cons. There is also lazy method which returns a Enumerator::Lazy. Lazy Enumerator has a few additional methods which enumerate values only on an as-needed basis. If you don't need to read the entire file but, for example, looking for a particular line containing given expression then it might be worth to check it out.

I could finish the article at this point, but what if before we even start reading the file we need to decrypt it? Let's move further to the example.

Decrypting large file and processing it line by line

Prerequisites

Before we decrypt the file we need to encrypt our generated large file. We are going to use AES with 256 bits key length with Cipher Block Chaining (CBC) as mode.

cipher = OpenSSL::Cipher::AES256.new(:CBC)
cipher.encrypt
KEY = cipher.random_key
IV = cipher.random_iv

Now, let's encrypt out file:

cipher = OpenSSL::Cipher::AES256.new(:CBC)
cipher.encrypt
cipher.key = KEY
cipher.iv = IV

file = nil
enc_file = nil

profile do
  file = File.read("large_1G.txt")
  enc_file = File.open("large_1G.txt.enc", "wb")
  enc_file << cipher.update(file)
  enc_file << cipher.final
end

file.close
enc_file.close

Objects Freed: 12
Time: 3.6 seconds
Memory usage: 1000.02 MB

Seems like encrypting is also a quite memory consuming task. Let's adjust the algorithm a little bit:

cipher = OpenSSL::Cipher::AES256.new(:CBC)
cipher.encrypt
cipher.key = KEY
cipher.iv = IV

file = nil
enc_file = nil

profile do 
  buf = ""
  file = File.open("large_1G.txt", "rb")
  enc_file = File.open("large_1G.txt.enc", "wb")
  while buf = file.read(4096)
    enc_file << cipher.update(buf)
  end
  enc_file << cipher.final
end

file.close
enc_file.close

Objects Freed: 768048
Time: 5.05 seconds
Memory usage: 145.93 MB

By changing the algorithm to read and cipher the file by chunks made the task much less memory consuming.

Decrypt

All right, let's try to decrypt it now:

decipher = OpenSSL::Cipher::AES256.new(:CBC)
decipher.decrypt
decipher.key = KEY
decipher.iv = IV

dec_file = nil
enc_file = nil

profile do 
  buf = ""
  enc_file = File.open("large_1G.txt.enc", "rb")
  dec_file = File.open("large_1G.txt.dec", "wb")
  while buf = enc_file.read(4096)
    dec_file << decipher.update(buf)
  end
  dec_file << decipher.final
end

dec_file.close
enc_file.close

Objects Freed: 768050
Time: 3.5 seconds
Memory usage: 152.12 MB

Now, let's compare our files whether we properly encrypted and decrypted it:

❯ diff large_1G.txt large_1G.txt.dec

No differences were found. We are good here!

We managed to lower the memory usage quite significantly. That's great!

Treat this article as a toolset that you can use in your specific case.

This article was originally posted on my personal dev blog: https://tjay.dev/

Photo by Erwan Hesry on Unsplash