All software has to write data to the file system at some point. This is also true for programs written in Rust. Log messages need to be persisted (as long as they are not just written to stdout), and data needs to be saved for later, or else. Naturally, writing data is the opposite of reading data.
There are multiple ways to deal with writing data to files in Rust. Some of these ways are easier than others, but all have their justification. Sometimes you directly want to write text into a file, and other times you write some kind of binary format. No matter your use case, Rust probably has a way to deal with it.
In this article, you will learn about common ways of how to write files in Rust.
Just in case you enjoy YouTube videos more than an article, you can also watch the video above instead!
Writing All Data to a File at Once
Regularly, you only want to persist data that you already have in memory. In this case, this method is one of the most straightforward ways to deal with it.
It allows you to write all data at once, and it doesn’t matter whether your data is a String or already in binary format. Additionally, a file is conveniently created for you if it does not exist yet.
Its advantages include:
- It is convenient to use
- It is merely one line of code
- fs::write takes care of everything for you
As usual, this method also has a few drawbacks, which are:
- It always overwrites the file
- Its performance highly depends on the size of your data – the more data you write, the longer it takes
If this is what you are looking for, then take a look at the following code as an example of how to write all data to a file at once:
use std::fs;
fn write_string_to_file(path: &str, data: &str) -> Result<(), Box<dyn std::error::Error>> {
fs::write(path, data)?;
Ok(())
}
fn write_data_to_file(path: &str, data: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
fs::write(path, data)?;
Ok(())
}
Writing All Data to a File at Once With the fs::File API
The method above is just a convenient wrapper around this method, which gives you a bit more control over how you deal with a file and its contents.
The only difference here is that there is no convenient under-the-hood conversion of Strings into binary. This is something you need to do yourself, in this case.
The advantage of this method is the following:
- It is still convenient to use
- It only takes a few lines of code, depending on how you do it
- It’s flexible to extend and alter
And its drawbacks include these:
- It always overwrites the file – no appending
- Its performance highly depends on the size of your data – the more data you write, the longer it takes
If that’s okay for you, you can take the following code as a starting point for your own implementation:
use std::{fs, io::Write};
fn write_string_to_file(path: &str, data: &str) -> Result<(), Box<dyn std::error::Error>> {
let mut file = fs::File::create(path)?;
// You need to take care of the conversion yourself
// and can either try to write all data at once
file.write_all(&data.as_bytes())?;
// Or try to write as much as possible, but need
// to take care of the remaining bytes yourself
let remaining = file.write(&data.as_bytes())?;
if remaining > 0 {
// You need to handle the remaining bytes
}
Ok(())
}
fn write_data_to_file(path: &str, data: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
let mut file = fs::File::create(path)?;
// You can either try to write all data at once
file.write_all(&data)?;
// Or try to write as much as possible, but need
// to take care of the remaining bytes yourself
let remaining = file.write(&data)?;
if remaining > 0 {
// You need to handle the remaining bytes
}
Ok(())
}
Appending Data to a File
You don’t always want to overwrite your files. Sometimes, you process things in a loop and don’t want to create a huge String or byte array before you dump your data. Such methods also help to keep the memory profile of your Rust binary low.
If you want to append data to an existing file, there is a convenient method for it. It uses fs::OpenOptions which allows for granular control over how the file is opened.
The advantages of this method include the following:
- You gain full control over how the file is handled
- It still only takes a few lines of code, depending on how you do it
- The performance of this method depends on you size the data you write
As usual, this method has some drawbacks, as well:
- Every call is a potential system call
- Every call writes to the file system, which can be costly if the data size is not optimal
If this method seems like something you want to use, you can see an example below:
use std::{fs, io::Write};
fn append_string_to_file(path: &str, data: &str) -> Result<(), Box<dyn std::error::Error>> {
let mut file = fs::OpenOptions::new().create(true).append(true).open(&path)?;
// You need to take care of the conversion yourself
// and can either try to write all data at once
file.write_all(&data.as_bytes())?;
// Or try to write as much as possible, but need
// to take care of the remaining bytes yourself
let remaining = file.write(&data.as_bytes())?;
if remaining > 0 {
// You need to handle the remaining bytes
}
Ok(())
}
fn append_data_to_file(path: &str, data: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
let mut file = fs::OpenOptions::new().create(true).append(true).open(&path)?;
// You can either try to write all data at once
file.write_all(&data)?;
// Or try to write as much as possible, but need
// to take care of the remaining bytes yourself
let remaining = file.write(&data)?;
if remaining > 0 {
// You need to handle the remaining bytes
}
Ok(())
}
Writing and Appending Data to a File with BufWriter
All previously presented methods have one thing in common: They always try to write everything you give them at once, and they offer no flexibility in this regard.
If you are looking for a way to write large chunks of data consistently without impacting the performance of your program, you can use a BufWriter. Internally, a BufWriter buffers the data it receives and tries to optimize the way it writes it to the file system.
This method requires not many new concepts and even uses the APIs you have already learned about above. Additionally, it also has a few advantages:
- You have full control over how the file is handled
- It gives you a lot of flexibility
- A BufWriter optimizes access to the file system under the hood
Gladly, there are only two drawbacks to this method:
- It can be a little more difficult to use
- It takes a little more code to implement
If you want to use the most flexible way to write data to files, check the code below for an example:
use std::fs;
use std::io::{BufWriter, Write};
fn append_string_to_file(path: &str, data: &str) -> Result<(), Box<dyn std::error::Error>> {
let file = fs::OpenOptions::new().create(true).append(true).open(&path)?;
let mut file = BufWriter::new(file);
// You need to take care of the conversion yourself
// and can either try to write all data at once
file.write_all(&data.as_bytes())?;
// Or try to write as much as possible, but need
// to take care of the remaining bytes yourself
let remaining = file.write(&data.as_bytes())?;
if remaining != 0 {
// handle...
}
// You definitely need to flush a BufWriter
// as it cannot guarantee emptying its buffer
// when it goes out of scope
file.flush()?;
Ok(())
}
fn append_data_to_file(path: &str, data: &[u8]) -> Result<(), Box<dyn std::error::Error>> {
let file = fs::OpenOptions::new().create(true).append(true).open(&path)?;
let mut file = BufWriter::new(file);
// You can either try to write all data at once
file.write_all(&data)?;
// Or try to write as much as possible, but need
// to take care of the remaining bytes yourself
let remaining = file.write(&data)?;
if remaining != 0 {
// handle...
}
// You definitely need to flush a BufWriter
// as it cannot guarantee emptying its buffer
// when it goes out of scope
file.flush()?;
Ok(())
}
Summary
While developing software, you often have to write data to files. As usual, Rust is no different than other languages when it offers multiple ways to perform these kinds of operations. In this article, you learned about four different ways to write files in Rust.
Like always in life, everything you do can have both advantages and disadvantages; thus, the concept is no different in this regard. You always have to spend a few thoughts and think about which way is the best for your specific use case.
Writing all data at once is a great way to perform the action with as little code as possible. Additionally, you don’t have to worry too much about the format of the data. This convenience comes at a cost, though. The more data you need to write, the larger your memory grows, and there is no control over preventing an existing file from being overridden.
Writing all data at once with the File API is another way to perform the action of dumping all your data onto the file system. Unlike its wrapper API, it offers you more flexibility in handling this process. You also lose the flexibility of a convenient API that does conversion if you deal with Strings. Additionally, the problem with large data blobs remains. A lot of data means a lot of memory.
Appending data to a file with the OpenOptions API brings even more flexibility than all previous methods. Here, you don’t have to worry about the size of your data. You append data to a file as you process it and can keep a lower memory profile. This method needs some fine-tuning, though. Appending data often leads to system calls that are sometimes pretty costly, so there is still room for improvement.
Finally, using a BufWriter gives you the most flexibility, but it’s also a relatively low-level approach to performing write operations. The BufWriter itself does at least buffer the data for you, but its effectiveness still depends on your usage. If you don’t take care while implementing your write operations with a BufWriter, you can also quickly sabotage all your efforts at improving the performance of your program.
Top comments (0)