(This article is cross-posted from my blog.)
The FlatBuffers project is an extremely efficient schema-versioned serialization library. In this tutorial, you'll learn how to use it in Rust.
To learn more about why we need yet another way to encode data, go read my post Why FlatBuffers.
FlatBuffers is a serialization format from Google. It's really fast at reading and writing your data: much quicker than JSON or XML, and often faster than Google's other format, Protocol Buffers. It's schema-versioned, which means your data has integrity (like in a relational database). FlatBuffers supports thirteen programming languages: C++, C#, C, Dart, Go, Java, JavaScript, Lobster, Lua, PHP, Python, Rust, and TypeScript.
This post will show you how to set up FlatBuffers and then use it in a demo Rust program.
(Full disclosure: I maintain the Golang, Python, and Rust FlatBuffers codebases.)
This tutorial has seven short parts:
- Install the FlatBuffers compiler
- Create a new Cargo project (if needed)
- Write a FlatBuffers schema definition
- Generate Rust accessor code from the schema
- Install the FlatBuffers Rust runtime library
- Write a demo Rust program to encode and decode example data
- Learn more and get involved
If you'd like to see all of the code in one place, I've put the project up at a GitHub repository.
1. Install the FlatBuffers compiler
First things first: let's install the compiler.
The compiler is used only in development. That means you have no new system dependencies to worry about in production environments!
Installation with Homebrew on OSX
On my OSX system, I use Homebrew to manage packages. To update the Homebrew library and install FlatBuffers, run:
$ brew update
$ brew install flatbuffers
(As is usual on my blog, I indicate CLI input with the prefix $.)
Personally, I like to install the latest development version from the official Git repository:
$ brew update
$ brew install flatbuffers --HEAD
If successful, you will have the flatc program accessible from your shell. To verify it's installed, execute flatc:
$ flatc
flatc: missing input files
...
Other installation methods
If you'd like to install from source, install a Windows executable, or build for Visual Studio, head over to my post Installing FlatBuffers for more.
2. Create a new Cargo project (if needed)
(If you're adding FlatBuffers to an existing Rust project, you can skip this step.)
Create a basic Cargo configuration with the following command:
$ cargo new rust_flatbuffers_example
Created binary (application) `rust_flatbuffers_example` package
There will now be a directory called rust_flatbuffers_example. Change the current working directory to that:
$ cd rust_flatbuffers_example
Now, note that the directory contains the following files:
$ tree
.
|-- Cargo.toml
`-- src
`-- main.rs
Finally, check that the Cargo package is properly configured. Do this by running the example program that Cargo automatically generated:
$ cargo run --quiet
Hello, world!
If you do not see this output, then please have a look at the official documentation on setting up Rust and Cargo to troubleshoot your configuration.
3. Write a FlatBuffers schema definition
All data in FlatBuffers are defined by schemas. Schemas in FlatBuffers are plain text files, and they are similar in purpose to schemas in databases like Postgres.
We'll work with data that make up user details for a website. It's a trivial example, but good for an introduction. Here's the schema:
// myschema.fbs
namespace users;
table User {
name:string;
id:ulong;
}
root_type User;
Place the above code in a file called myschema.fbs, in the root of your Cargo project.
This schema defines User, which holds one user's name and id. The namespace for these types is users (which will be the generated Rust package name). The topmost type in our object hierarchy is the root type User.
Schemas are a core part of FlatBuffers, and we're barely scratching the surface with this one. It's possible to have default values, vectors, objects-within-objects, enums, and more. If you're curious, go read the documentation on the schema format.
4. Generate Rust accessor code from the schema
The next step is to use the flatc compiler to generate Rust code for us. It takes a schema file as input, and outputs ready-to-use Rust code.
In the directory with the myschema.fbs file, run the following command:
$ flatc --rust -o src myschema.fbs
This will generate Rust code in a new file called myschema_generated.rs in the pre-existing src directory. Here's what our project looks like afterwards:
$ tree
.
|-- Cargo.lock
|-- Cargo.toml
|-- myschema.fbs
`-- src
|-- main.rs
`-- myschema_generated.rs
1 directory, 6 files
Note that one file is generated for each schema file.
A quick browse of src/myschema_generated.rs shows that there are three sections to the generated file. Here's how to think about the different function groups:
- Type definition and initializer for reading
Userdata
pub struct User { ... }
pub fn get_root_as_user(buf: &[u8]) -> User { ... }
pub fn get_size_prefixed_root_as_user(buf: &[u8]) -> User { ... }
pub fn init_from_table(table: flatbuffers::Table) -> Self { ... }
- Instance methods providing read access to
Userdata
pub fn name(&self) -> Option<&'a str> { ... }
pub fn id(&self) -> u64 { ... }
- Functions used to create new
Userobjects
pub fn create(_fbb: &mut flatbuffers::FlatBufferBuilder, args: &UserArgs) -> flatbuffers::WIPOffset { ... }
(Note that I've elided some of the lifetime annotations in the above code.)
We'll use these functions when we write the demo program.
5. Install the FlatBuffers Rust runtime library
The official FlatBuffers Rust runtime package is hosted on crates.io: Official FlatBuffers Runtime Rust Library.
To use this in your project, add flatbuffers to your dependencies manifest in Cargo.toml. The file should now look similar to this:
[package]
name = "rust_flatbuffers_example"
version = "0.1.0"
authors = ["rw <me@rwinslow.com>"]
edition = "2018"
[dependencies]
flatbuffers = "*"
I use * to fetch the latest package version. In general, you'll want to pick a specific version. You can learn more about this in the documentation for the Cargo dependencies format.
6. Write a demo Rust program to encode and decode example data
Now, we'll overwrite the default Cargo "Hello World" program with code to write and read our FlatBuffers data.
(We do this for the sake of simplicity, so that I can avoid explaining the Cargo build system here. To learn more about Cargo projects, head over to the official documentation on Cargo project file layouts.)
Imports
To begin, we will import the generated module using the mod statement. Place the following code in src/main.rs:
// src/main.rs part 1 of 4: imports
extern crate flatbuffers;
mod myschema_generated;
use flatbuffers::FlatBufferBuilder;
use myschema_generated::users::{User, UserArgs, finish_user_buffer, get_root_as_user};
This usage of the mod keyword instructs the Rust build system to make the items in the file called myschema_generated.rs accessible to our program. The use statement makes two generated types, User and UserArgs, accessible to our code with convenient names.
Writing
FlatBuffer objects are stored directly in byte slices. Each Flatbuffers object is constructed using the generated functions we made with the flatc compiler.
Append the following snippet to your src/main.rs:
// src/main.rs part 2 of 4: make_user function
pub fn make_user(bldr: &mut FlatBufferBuilder, dest: &mut Vec<u8>, name: &str, id: u64) {
// Reset the `bytes` Vec to a clean state.
dest.clear();
// Reset the `FlatBufferBuilder` to a clean state.
bldr.reset();
// Create a temporary `UserArgs` object to build a `User` object.
// (Note how we call `bldr.create_string` to create the UTF-8 string
// ergonomically.)
let args = UserArgs{
name: Some(bldr.create_string(name)),
id: id,
};
// Call the `User::create` function with the `FlatBufferBuilder` and our
// UserArgs object, to serialize the data to the FlatBuffer. The returned
// value is an offset used to track the location of this serializaed data.
let user_offset = User::create(bldr, &args);
// Finish the write operation by calling the generated function
// `finish_user_buffer` with the `user_offset` created by `User::create`.
finish_user_buffer(bldr, user_offset);
// Copy the serialized FlatBuffers data to our own byte buffer.
let finished_data = bldr.finished_data();
dest.extend_from_slice(finished_data);
}
This function takes a FlatBuffers Builder object and uses generated methods to write the user's name and ID.
Note that the name string is created with the bldr.create_string function. We do this because, in FlatBuffers, variable-length data like strings need to be created outside the object that references them. In the code above, this is still ergonomic because we can call bldr.create_string inline from the UserArgs object.
Reading
FlatBuffer objects are stored as byte slices, and we access the data inside using the generated functions (that the flatc compiler made for us in myschema_generated.rs).
Append the following code to your src/main.rs:
// src/main.rs part 3 of 4: read_user function
pub fn read_user(buf: &[u8]) -> (&str, u64) {
let u = get_root_as_user(buf);
let name = u.name().unwrap();
let id = u.id();
(name, id)
}
This function takes a byte slice as input, and initializes a FlatBuffer reader for the User type. It then gives us access to the name and ID values in the byte slice.
The main function
Now we tie it all together. This is the main function:
// src/main.rs part 4 of 4: main function
fn main() {
let mut bldr = FlatBufferBuilder::new();
let mut bytes: Vec<u8> = Vec::new();
// Write the provided `name` and `id` into the `bytes` Vec using the
// FlatBufferBuilder `bldr`:
make_user(&mut bldr, &mut bytes, "Arthur Dent", 42);
// Now, `bytes` contains the serialized representation of our User object.
// To read the serialized data, call our `read_user` function to decode
// the `user` and `id`:
let (name, id) = read_user(&bytes[..]);
// Show the decoded information:
println!("{} has id {}. The encoded data is {} bytes long.", name, id, bytes.len());
}
This function writes, reads, then prints our data. Note that bytes is the byte vector with encoded data. This is the serialized data you could send over the network, or save to a file.
Running it
$ cargo run
Arthur Dent has id 42. The buffer is 48 bytes long.
To recap, what we've done here is write a short program that uses generated code to write, then read, a byte slice in which we encoded data for an example User. This User has name "Arthur Dent" and ID 42.
7. Learn more and get involved
FlatBuffers is an active open-source project, with backing from Google. It's Apache-licensed, and available for C++, C#, C, Dart, Go, Java, JavaScript, Lobster, Lua, PHP, Python, Rust, and TypeScript (with more languages on the way!).
Here are some resources to get you started:
Top comments (1)
Well written and nice that you put links to cargo and other rust specific good practices in there.
I’m curious how the ergonomics are in comparison to Protobuf and tonic for example.
Do you have a benchmark comparison at hand that illustrates flatbuffers in action vs protobuf?