In the previous post, we covered the scope of the project and we wrote the CLI frontend using StructOpt which we'll later use to package the implementation of our server and client.
It is recommended that you follow in order since each post builds off the progress of the previous post.
This is the second post of a 4 part series. If you would like to view this post in a single page, you can follow along on my blog.
Protocol Buffers
What are Protocol Buffers?
Protocol Buffers (protobufs) are a way to define a data schema for how your data is structured as well as how to define how programs interface with each other w/ respect to your data in a language-independent manner.
This is achieved by writing your data in the protobuf format and compiling it into a supported language of your choice as implemented as gRPC.
The result of the compilation generates a lot of boilerplate code.
Not just data structures with the same shape and naming conventions for your language’s native data types. But also generates the gRPC network code for the client that sends or the server that receives these generated data structures.
For what it’s worth, an added bonus are servers and clients having the possibility to be implemented in different languages and inter-operate without issue due to. But we’re going to continue to work entirely in Rust for this example
Where should protobuf live in the codebase?
Before jumping into the protobuf, I wanted to mention my practice for where to keep the file itself.
$ tree
.
├── Cargo.lock
├── Cargo.toml
├── proto
│ └── cli.proto
└── src
└── main.rs
I like to keep the protobuf in a directory named proto typically at the same level as the Cargo.toml because as we’ll see soon, the build script will need to reference a path to the protobuf for compilation. The file name itself is arbitrary and naming things is hard so do your best to support your future self with meaningful names.
The example protobuf
cli.proto
syntax = "proto3";
package remotecli;
// Command input
message CommandInput {
string command = 1;
repeated string args = 2;
}
// Command output
message CommandOutput {
string output = 1;
}
// Service definition
service RemoteCLI {
rpc Shell(CommandInput) returns (CommandOutput);
}
We start the file off by declaring the particular version of syntax we’re using. proto3.
We need to provide a package name.
The proto3 docs say this is optional, but our protobuf Rust code generator Prost requires it to be defined for module namespacing and naming the resulting file.
Defined are 2 data structures, called messages.
The order of the fields are numbered and are important for identifying fields in the wire protocol when they are serialized/deserialized for gRPC communication.
The numbers in the message must be unique and the best practice is to not change the numbers once in use.
For more details, read more about Field numbers in the docs.
The CommandInput message has 2 string fields - one singular and the other repeated.
The main executable, which we refer to as command the first word of the user input.
The rest of the user input is reserved for args.
The separation is meant to provide structure for the way a command interpreter like Bash defines commands.
The CommandOutput message doesn’t need quite as much structure. After a command is run, the Standard Output will be returned as a single block of text.
Finally, we define a service RemoteCLI with a single endpoint Shell. Shell takes a CommandInput and returns a CommandOutput.
Compile the protobuf with Tonic
Now that we have a protobuf, how do we use it in our Rust program when we need to use the generated code?
Well, we need to configure the build to compile the protobuf into Rust first.
The way we accomplish that is by using a build script (Surprise! Written in Rust) but is compiled and executed before the rest of the compilation occurs.
Cargo will run your build script if you have a file named build.rs in your project root.
$ tree
.
├── build.rs
├── Cargo.toml
├── proto
│ └── cli.proto
└── src
└── main.rs
build.rs
fn main() {
tonic_build::compile_protos("proto/cli.proto").unwrap();
}
The build script is just a small Rust program with a main() function.
We’re using tonic_build to compile our proto into Rust. We’ll see more tonic soon for the rest of our gRPC journey.
But for now we only need to add this crate into our Cargo.toml as a build dependency.
Cargo.toml
[package]
name = "cli-grpc-tonic-blocking"
version = "0.1.0"
authors = ["T.J. Telan <t.telan@gmail.com>"]
edition = "2018"
[dependencies]
# CLI
structopt = "0.3"
[build-dependencies]
# protobuf->Rust compiler
tonic-build = "0.3.0"
Build dependencies are listed under its own section [build-dependencies]. If you didn’t know, your build scripts can only use crates listed in this section, and vice versa with the main package.
You can look at the resulting Rust code in your target directory when you cargo build.
You’ll have more than one directory with your package name plus extra generated characters due to build script output. So you may need to look through multiple directories.
$ tree target/debug/build/cli-grpc-tonic-blocking-aa0556a3d0cd89ff/
target/debug/build/cli-grpc-tonic-blocking-aa0556a3d0cd89ff/
├── invoked.timestamp
├── out
│ └── remotecli.rs
├── output
├── root-output
└── stderr
I’ll leave the contents of the generated code to those following along, since there’s a lot of it and the relevant info is either from the proto or will be covered in the server and client implementation.
This code will only generate once. Or unless you make changes to build.rs. So if you make changes to your proto and you want to regenerate code, you can force a code regen by using touch.
$ touch build.rs
$ cargo build
We just covered the creation of our data schema in the Protocol Buffer format and using Tonic to compile the protobufs into Rust code with Rust build scripts.
In the next post we'll cover using our generated Rust code, the implementation of the gRPC server, and plugging in the code into our CLI frontend.
I hope you'll follow along!
Top comments (0)