DEV Community: Matt Davies

Rust #8: Strings

Matt Davies — Tue, 10 Aug 2021 08:51:09 +0000

Strings, at first glance in Rust, may seem very complicated. First of all, there's the existence of two string types: String and str. Then, there's the fact you can't easily index them. On top of that, as you dig deeper in the language, there are more string types such as CString, OsString, PathBuf, CStr, OsStr and Path. And then, if that wasn't enough, there are different ways to construct them and to convert between them because different APIs use different string types.

No wonder people complain about Rust's learning curve, which I will personally argue is not as steep as C++.

But, the truth is, that when you understand a few concepts like Unicode, UTF-8, ownership and slices, it is not that complicated. In fact, necessary, and can also increase performance.

In this blog article, I will talk about the whys of strings and how to convert between the various types. I will not talk about some of the operations you can do on strings as that is a huge topic. I do recommend you look at the Rust Standard API documentation for more information.

ASCII

In the far annals of time, when computers only talked in English, there needed to be a method to map byte values in computer memory to characters that included digits, letters (upper and lower case), and punctuation. Computers only see bytes; they do not see digits, letters and punctuation. This meant that if you wanted to show text on printed paper or a screen, there needed to be a standard way of saying "whenever you see this byte value, print the letter A".

You could, for example, say that a byte of value 0 means A, a value 1 means B and so on. You could invent anything you wanted as long as your software on the computer and the hardware that printed the letter agreed. They had to agree, because if they did not, sending HELLO could come out as JONNY or $£&&!.

Two major standards appeared in the 1960s: IBM's EBCDIC and ASCII's committee's ASCII code. EBCDIC was the abbreviation for Extended Binary Coded Decimal Interchange Code. ASCII stood for American Standard Code for Information Interchange. But ASCII won over, probably because it was easier to remember its name!

ASCII uses 7 of the 8 bits in a byte to encode a character of code. Values from 0-31 and 127 were character codes and used to give meta-data to printers and terminals: for example, a new line or backspace. Values 32-126 represented all the printable characters in the English language as well as many punctuation characters. It had some nice properties such as to convert from upper case to lower case you only needed to add 32 or set bit 5.

The unused 8th bit was used for error detection when sending those bytes over networks. But later when computers started appearing in countries other than English speaking languages, more characters needed to be added and that 8th bit was given up for 128 more characters. Then new competing standards turned up that matched ASCII for the first 128 values, but differed in the second 128 values. You had the concept of code pages that defined how byte values 128-255 were mapped.

But eventually even 256 separate values were not enough to contain all the characters that were printable in the world. Things got complicated very fast. So enter the Unicode Consortium.

Unicode

It turns out there are many, MANY different characters in the world. To support that many characters a new standard had to be created and this was run by the Unicode Consortium. They manage the Unicode standard that, according to Wikipedia, "defines 143,859 characters covering 154 modern and historic scripts, as well as symbols, emojis and non-visual control and formatting codes.". But the consortium doesn't just maintain those characters and codes, but how to display them, normalise, decompose, collate and render them. It is a very, very, and dare I add one more 'very', complicated system.

For example, é, the accented letter 'e' that can appear in French and many other languages, can be represented by a single value, or by two values: the value of the letter e and the value meaning put an acute accent on the previous character. Normalisation is the process of making sure all the characters follow one representation or the other. Composition is the process of reducing it to a single value, decomposition is the process of expanding it to the longer representation.

Each value, or code point, is represented as a 32-bit value. The first 128 values match ASCII exactly, which means converting ASCII to Unicode trivial, and converting Unicode to ASCII trivial as well as long as all code points lie between the values of 0 and 127. The various values are divided into planes that represent a range of 65,536 values:

Plane	Value range in hex	Name
0	0000-FFFF	Basic Multilingual Plane
1	10000-1FFFF	Supplementary Multilingual Plane
2	20000-2FFFF	Supplementary Ideographic Plane
3	30000-3FFFF	Tertiary Ideographic Plane
4-13	40000-DFFFF	Unused
14	E0000-EFFFF	Supplementary Special-purpose Plane
15-16	F0000-10FFFF	Supplementary Private Use Area Planes

As you can see from the table above, not all value ranges are used within the 32-bit space. Only 21 bits are used. I guess we need more bits when we discover aliens and learn their languages.

But the downside of Unicode representation is that now all text takes 4 times as much space, irregardless or language and how common each symbol is. Thanks to David Prosser and Ken Thompson (yes, that Ken Thompson, inventor and co-inventor of Unix, the B programming language, the grep utility and the modern programming language Go), we have a better representation.

UTF-8

In 1992, Dave Prosser, working for the Unix System Laboratories, submitted a proposal for a new representation of Unicode that mapped 31 bits to 1-5 bytes in such a way that lower Unicode values could be represented with fewer bytes. ASCII itself could be represented as normal single byte ASCII. It worked by using bit 7 to indicate that a multibyte sequence was being used. In the first byte of the sequence, the number of 1 bits following bit 7 represented how many bytes would follow. The encoding worked like this:

Number of bytes	Value range	Byte 1	Byte 2	Byte 3	Byte 4	Byte 5
1	0000000-0000007F	0xxxxxxx
2	0000080-0000207F	10xxxxxx	1xxxxxxx
3	0002080-0008207F	110xxxxx	1xxxxxxx	1xxxxxxx
4	0082080-0208207F	1110xxxx	1xxxxxxx	1xxxxxxx	1xxxxxxx
5	2082080-7FFFFFFF	11110xxx	1xxxxxxx	1xxxxxxx	1xxxxxxx	1xxxxxxx

Plane 0 of Unicode, which contained the most common characters around the world, could be encoded 1-3 bytes. All the current Unicode characters could be encoded, at most, within 4 bytes.

Ken Thompson improved it at the cost of adding a 6th byte, but allowed a program to tell the difference between an initial byte that started a sequence and an intermediate byte. He did this by reserving 10 for only the following bytes:

Number of bytes	Value range	Byte 1	Byte 2	Byte 3	Byte 4	Byte 5	Byte 6
1	0000000-0000007F	0xxxxxxx
2	0000080-000007FF	110xxxxx	10xxxxxx
3	0000800-0000FFFF	1110xxxx	10xxxxxx	10xxxxxx
4	0010000-001FFFFF	11110xxx	10xxxxxx	10xxxxxx	10xxxxxx
5	0200000-03FFFFFF	111110xx	10xxxxxx	10xxxxxx	10xxxxxx	10xxxxxx
6	4000000-7FFFFFFF	1111110x	10xxxxxx	10xxxxxx	10xxxxxx	10xxxxxx	10xxxxxx

Plane 0 can still be encoded in 1-3 bytes, ASCII is still represented and all Unicode characters can still be encoded in up to 4 bytes. UTF-8 is such an amazingly clever standard. Well done David and Ken!

Rust uses the UTF-8 standard to encode its strings in the String struct. So any character in a string will be encoded using between 1 and 4 bytes.

Ownership and slices

To recap, Rust strictly defines the concept of ownership. This is where a variable, parameter etc owns the memory it requires to store its value, and it is the ONLY variable to own that memory. Rust's compiler guarantees that there is only a single owner of a piece of memory at any single time. This also means that when the owner goes out of scope, the compiler can safely destroy the memory and give it back to the operating system.

Secondly, the owner can dish out references to its memory and lend them to other parts of the program. It can do so either mutably or immutably. Rust enforces that you can only give out a single mutable reference (i.e. the reference holder can mutate the memory the owner owns) at any time. The owner itself cannot even mutate itself while there's a mutable reference. But, if there are no mutable references, the owner can give out any number of immutable references, which also means that the owner itself cannot mutate itself while they still exist.

It is also important to note that if there are any references live when the owner goes out of scope, that is a compiler error. Rust will just not allow you to do that. To enforce that, there is a concept of lifetimes, but I do not need to go into that for the topic of strings.

So to summarise, you can only have these 3 states in Rust:

Only an owner. If the variable is marked mut, it can mutate itself.
Only an owner and any number of immutable references. Everything is read-only. The owner cannot mutate itself and neither can the references mutate the owned memory as they are immutable.
Only an owner and a single mutable reference. Only the mutable reference can mutate the owned memory. The owner can only read itself.

One type of reference, mutable or immutable, is called a slice. It is a special reference to an array of memory or a contiguous sequence of values. The array that contains elements of type T is written as [T] and the slice, because it's just a reference to an array, is written as either &[T] or &mut [T]. Simply put, a slice can be thought of as a view into some contiguous memory owned by something else. A slice, due to being a reference, does not own any memory and so therefore requires something else to own that memory to have a view on to it.

Now a slice doesn't have to view all of the owned memory. It can view a subset of it. And Rust guarantees both at compile-time and run-time that a slice cannot view outside the bounds of the owned memory. This characteristic of slices is crucial to having performant operations on strings as you will see.

A slice has two pieces of information. Firstly, a pointer to the start of the memory it views, and secondly, the size of the memory it can view.

And now we can talk about what the string types above are all about. Some are owners, types that own the memory that contain the string characters, and some are array types that with the reference operator & implement the slice.

Owner type	Slice type
String	&str
OsString	&OsStr
PathBuf	&Path

`String`

If we look at the definition of std::string::String we see this:

struct String {
    vec: Vec<u8>
}

This means a String, internally, is an array of bytes. However, there is one important aspect to those array of bytes. Rust guarantees through its standard APIs that those array of bytes can only contain a valid sequence of UTF-8 characters. Likewise, &str guarantees that it is a view on the whole or part of a String that only contains UTF-8 characters. The APIs guarantee that the start pointer of a &str and its implied end pointer (the start pointer plus its length) fall exactly on UTF-8 character boundaries.

The fact that the data is UTF-8 means that when you ask for a character from the string, you can get back 1, 2, 3 or even 4 bytes of data due to the encoded nature of UTF-8. This is why when you extract a character from a string you get back a char type, which is 4 bytes in size. Rust's string APIs will convert the UTF-8 sequence of bytes to a single Unicode code point or code points.

It is this very complexity of how characters or codes in UTF-8 can be multiple bytes (or not) that makes it very cumbersome to use when processing these strings. It is important to Rust that no matter what you do the UTF-8 is valid, otherwise it will panic.

One more aspect to talk about is string literals. For example:

let greeting = "Hello";

All literal strings are embedded in the executable. And because that memory is unmoving and frozen in time, you can only grab an immutable slice to that string. Therefore, all literal strings are of types &str. They can never be mutated as the memory containing all the literals in an executable are always read-only on all operating systems.

Technically speaking, the type is really &'static str because the literal is available for the entire lifetime of the program. If you don't understand lifetimes at time of reading this article then don't worry about this detail right now.

`OsString`

Different operating systems use different techniques for encoding their strings. This is important to know when interacting with the operating system directly because UTF-8 encodings may not cut it. Therefore, Rust introduces std::ffi::OsString to abstract away the differences between operating systems. It is defined in the FFI (Foreign Function Interface) module because it is often required when talking to C code and other languages.

On Unix systems, strings are sequences of non-zero bytes, often in UTF-8 encoding. On Windows, strings can be encoded in either 8-bit ASCII or a 16-bit representation called UTF-16 (another Unicode encoding system using 1 or 2 16-bit values). Rust strings are always UTF-8 that may contain zeroes. So Rust provides a new type with fallible conversion functions.

Like String, OsString has a companion slice type called std::ffi::OsStr.

`PathBuf`

std::path::PathBuf is a OsString with a particular purpose. It is used to represent filenames. And because filenames are used by the operating system, they are required to be OS-specifically encoded. The definition of PathBuf is:

struct PathBuf {
    inner: OsString,
}

Similar to the other string representations, there is a companion slice type called std::fs::Path. Along with this type are many useful methods for manipulating paths, like producing slices on the extension part, or the filename part.

`CString`

Finally, out of the string types that I wish to talk about in this article, there is the std::ffi::CString type. This is used to communicate with C functions because C stores strings very differently to Rust. For starters, there is no length parameter as C uses a terminating zero byte, called the null-terminator, to mark the end of the string. This means that C strings can never contain zero bytes and finding the length requires stepping through all the characters. Also, C strings have no implied encoding. Due to this, CString is usually constructed from a byte array or vector.

Constructing a CString will make sure that there are no zero bytes in the middle of an array and will return an error if it finds them.

CStrings are a vector of the type c_char which is an unsigned byte to match the char type of the C language. The companion type of CString is CStr which contains an array of c_chars.

When working with C code, you usually find yourself converting a String to a CString, which can then be passed to the function that takes a const char* using CStr's as_ptr() method.

Conversions

Another source of confusion when starting with Rust is how do you convert from one type to another. I have often found myself rushing to Stack Overflow to find code that shows me how. First, I will cover conversion between owner types and their slice types. Then, I will visit each string type and talk about how to convert from any other different string type.

From owned types to slices

Converting from an owned type to a slice type is as simple as putting a reference operator in front:

let s = String::from("Hello");
let os_s = OsString::from("Foo");
let path_s = PathBuf::from("/home/matt/.bashrc");
let c_s = CString::from("C sucks!");

let ref_s: &str = &s;
let ref_os_s: &OsStr = &os_s;
let ref_path_s: &Path = &path_s;
let ref_c_s: &CStr = &c_s;

This magic works because of the Deref trait. For example, String implements Deref<Target=str> and can inherit all of str's methods. This also means that you can pass a String to a function that expects a &str or &mut str is the owner is mutable. This is why many functions that accept strings use &str as the parameter type. But when returning a string, functions often return an owned type so its lifetime isn't linked to the input parameters. So, for example, you can often see a function like:

fn concat(s1: &str, s2: &str) -> String {
    let s = String::with_capacity(s1.len() + s2.len());
    s.push(s1);
    s.push(s2);
    s
}

...
let greet = String::from("Hello ");
let place = String::from("World");

let my_greeting = concat(&greet, &place);

Because converting an owned type to a slice is just constructing a pointer and a length, there is no allocation. Also, by accepting a &str, the function doesn't care whether you pass a &String or a string literal.

From slices to owned types

This is a little more complicated but not much. To construct an owned type from a slice, you have to allocate memory and copy the characters from the memory the slice is referencing. This is because someone else owns that memory and you can't have two owners. There are multiple ways of doing it.

One way is via the From trait, which declares the from method. All the owned string types described here implement From for their companion slice type. You've already seen its use in the examples above.

Another way is using a method on the slice type. Unfortunately, the different string types use different methods names:

Owned type	Method to convert from slice type
`String`	`&str.to_owned()`
`String`	`&str.to_string()`
`OsString`	`&OsStr.to_os_string()`
`PathBuf`	`&Path.to_path_buf()`
`CString`	no method exists

You will notice that str has two methods: to_owned() and to_string(). to_string() actually comes from the trait ToString. The default version of this trait using Rust's formatting functions and ends up being slower than the to_owned() method. So I highly recommend you use to_owned() or the From trait methods.

Unfortunately, none of the other string types implement the ToString type as the conversion can fail. to_string() does not return a Result.

Conversions between string types

There are many different functions with different naming conventions to convert between string types, sometimes the owned type, sometimes the slice type. I thought about how I would represent this information. I considered tables for each string type I was converting to. But sometimes there isn't a single function you can call. In the end I decided on graph representations.

The first graph shows conversions between different types using their methods. Where a ? appears after the method name, it means it is fallible and either returns an Option or Result. You will need to handle that. Red nodes are the owned types.

The second graph shows conversion using the From trait. If an arrow travels from type T to type U, it means that U can be created using the From<T> trait. Again red nodes are owned types.

I hope these graphs help you convert from any string type to another.

Holy Cow!

There is one more string type in Rust I want to discuss and that is the std::borrow::Cow<T> type where T is a slice type that implements the ToOwned trait. All the string slice types mentioned in this article implement the ToOwned trait.

COW is an acronym for Copy On Write. It can either hold an immutable slice or an owned type. It is commonly used for parameters and return types and is constructed from the multiple From traits it implements.

If you construct it from a slice (such as a string literal or &str), it will hold an immutable reference to that slice. If you construct it from an owned type, it will take ownership:

let s = String::from("Hello");
let c1 = Cow::from(s); // c1 = Owned("Hello")
let c2 = Cow::from("World"); // c2 = Borrowed("Hello")

This makes Cow a useful return type for a function that can return a static string or a generated string:

fn main() {
    let g1 = greet(Some("Matt"));   // g1 = Owned("Hello Matt")
    let g2 = greet(None);           // g2 = Borrowed("Hello human!")
}

fn greet(name: Option<&str>) -> Cow<'static, str> {
    match name {
        Some(name) => Cow::from(format!("Hello {}", name)),
        None => Cow::from("Hello human!"),
    }
}

You can notice that the generic parameters passed to Cow are a lifetime parameter and the slice type. The slice type is required if Cow is ever borrowed. The owned type is derived from the slice type's ToOwned trait. The lifetime parameter is required for in the case of borrowing a reference. If the Cow owns data, a lifetime is no required.

In the example above, g1 owns the data and g2 doesn't because it references a static string literal.

Cow types can be easily cloned too. If it owns a type, it returns a Cow that borrows. You can also obtain a mutable reference to the contents by calling to_mut and that will clone if necessary.

Summary

That concludes my article for this week. I hope you find strings easier to understand now and realise that the reasons they are complicated are justified. We talked about owned types and their associated slice types. We talked about why there are different types due to differences in UTF-8 and operating system representations. We also talked about how you can convert between them using a combination of their methods and From traits. Finally, we discussed the Cow type that can represent both an owned type or a referenced type.

Until next time!

Rust #7: Command-Line interfaces

Matt Davies — Sun, 01 Aug 2021 13:00:47 +0000

I have discovered that Rust has great support for writing Command-Line Interface (CLI) tools, or tools that you interact with on the command-line.

The basic requirement of CLI tools is to be able to read the arguments that the user types after the command on the terminal and process them.

You may have noticed that most CLI tools are written in Rust look the same when interacting with them. For example, invalid use of them shows some text on how you should use them. Typing --help after the command will list all sub-commands (if any) and flags (commands that start with a hyphen or double hyphen) that you can use. Even typing --version after the command will show version information. I guarantee that none of the CLI tool developers worked hard to provide this functionality.

So to provide an introduction on how to write CLI tools, let me introduce you to my demo CLI app called greeter:

// in src/main.rs
use std::env;

fn main() {
    let args: Vec<String> = env::args().collect();

    enum Greet {
        Unknown,
        Hello,
        Goodbye,
    }

    let mut greet = Greet::Unknown;

    let mut name = String::from("World");
    let mut i = 1;
    while i < args.len() {
        if (args[i] == "-n" || args[i] == "--name") && i < args.len() - 1 {
            name = args[i + 1].clone();
            i += 1;
        } else if args[i] == "hello" {
            greet = Greet::Hello;
        } else if args[i] == "bye" {
            greet = Greet::Goodbye;
        }
        i += 1;
    }

    println!(
        "{}",
        match greet {
            Greet::Unknown => String::from("No idea what to do!"),
            Greet::Hello => format!("Hello {}", name),
            Greet::Goodbye => format!("Goodbye {}", name),
        }
    );
}

This program takes a sub-command (a command that follows after the program's name) and an optional flag to provide a name to use in the greeting. Let's put it through its paces:

$ greet
No idea what to do!
$ greet hello
Hello World
$ greet bye
Goodbye World
$ greet --name Matt
No idea what to do!
$ greet hello --name Matt
Hello Matt
$ greet bye -n Matt
Goodbye Matt

But there are problems with this program. For example, greet hello bye --name Matt --name Bob is valid and will result in Goodbye Bob. We'd probably want to restrict that kind of confusing use. Processing command-line parameters is tricky and full of edge cases.

There are even more issues. There is no help, no support for version information and as mentioned before, no good error handling.

So the code above is not idiomatic at all and is implemented similar to how a C programmer might do so. Processing command-lines arguments is error-prone and is code that you usually incrementally increase as you come back to it time and time again to implement more features. C programmers can use a library called Getopt to manage this and is very common in Unix CLI tools.

Rust provides us with the arguments using std::env::args() that provides an iterator (in true Rust fashion) that can iterate through all the commands that were given. The very first iteration provides the name of the program, which is why the loop starts at 1 and not 0.

Enter the Clap!

`clap` crate

Clap is a crate that was written to manage command-line options and provide a familiar interface via --help and --version. Clap uses the builder pattern to declaratively describe how your tool should interact with the command-line. So let's start rewriting our amazing CLI tool using Clap.

For starters, it can provide application information:

use clap::App;

fn main() {
    let matches = App::new("The Amazing Greeter")
        .version("1.0")
        .author("Matt Davies")
        .about("The best greeter in town!")
        .get_matches();
}

Don't forget to add clap = "2.33" to Cargo.toml in the [dependencies] section.

This short program already gives us plenty of functionality. For example:

$ cargo run -q -- --help
The Amazing Greeter 1.0
Matt Davies
The best greeter in town!

USAGE:
    greet

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

The standard help information! The -- in the command cargo run -- --help separates the arguments that go to Cargo from the arguments that go to our program. So --help is sent to our program and Clap dutifully follows it out. The -q flag sent to Cargo stops it from outputting build information ('q' is for quiet). Let's try the version information:

$ cargo run -q -- -V
The Amazing Greeter 1.0

Now we need to add support for the name flag:

use clap::{App, Arg};

fn main() {
    let matches = App::new("The Amazing Greeter")
        .version("1.0")
        .author("Matt Davies")
        .about("The best greeter in town!")
        .arg(
            Arg::with_name("name")
                .short("n")
                .long("name")
                .value_name("NAME")
                .help("Provides a name to use in the greeting")
                .takes_value(true)
                .default_value("World"),
        )
        .get_matches();

    let name = matches.value_of("name").unwrap();
    println!("NAME: {}", name);
}

The .arg() takes a builder that is created with Arg::with_name(). From this builder, we can set all the properties for the option. First, we give it a short name (a single character) and a long name. This allows the option to be provided with -n or --name. value_name() and help() provide extra information for the help information:

$ cargo run -q -- --help
The Amazing Greeter 1.0
Matt Davies
The best greeter in town!

USAGE:
    greet [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -n, --name <NAME>    Provides a name to use in the greeting [default: World]

value_name() gives the name between the angled brackets and help() provides the text afterwards. Of course, the information from short() and long() are used too.

takes_value() tells clap that --name requires a value to follow after it and default_value() provides a value if the option is omitted. You may also have noticed that hyphened names are called flags unless they have values following them. In that case, they are called options. You may have additionally noticed that the value you passed to default_value() is also shown in the help information.

The intent for how this option is interacted with is much, much clearer than the original naive C-like code we used before. Let's test it further:

$ cargo run -q
NAME: World
$ cargo run -q --name Matt
NAME: Matt

Let's continue and add our sub-commands hello and bye:

use clap::{App, Arg, SubCommand};

fn main() {
    let matches = App::new("The Amazing Greeter")
        .version("1.0")
        .author("Matt Davies")
        .about("The best greeter in town!")
        .arg(
            Arg::with_name("name")
                .short("n")
                .long("name")
                .value_name("NAME")
                .help("Provides a name to use in the greeting")
                .takes_value(true)
                .default_value("World"),
        )
        .subcommand(
            SubCommand::with_name("hello")
                .about("Let's meet!")
                .version("1.0")
                .author("Matt Davies (again!)"),
        )
        .subcommand(
            SubCommand::with_name("bye")
                .about("We part ways")
                .version("1.0")
                .author("Matt Davies (again!)"),
        )
        .get_matches();

    let name = matches.value_of("name").unwrap();

    match matches.subcommand_name() {
        Some("hello") => println!("Hello {}", name),
        Some("bye") => println!("Goodbye {}", name),
        _ => println!("No idea what to do!"),
    }
}

interestingly, the sub-commands have their own about(), version() and author() calls in the builder generated by SubCommand::with_name(). So let's try some help:

$ cargo r --q -- -h
The Amazing Greeter 1.0
Matt Davies
The best greeter in town!

USAGE:
    greet [OPTIONS] [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -n, --name <NAME>    Provides a name to use in the greeting [default: World]

SUBCOMMANDS:
    bye      We part ways
    hello    Let's meet!
    help     Prints this message or the help of the given subcommand(s)

Now, we have a subcommands section with the added subcommand help added:

cargo r -q -- help hello
greet-hello 1.0
Matt Davies (again!)
Let's meet!

USAGE:
    greet hello

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

Now the name of the application is the concatenation of the application name and the subcommand: greet-hello. I find this interesting as this implies you can add plugins by creating programs that are called greet-<command> providing that greet generates the filename if the subcommand is unknown, and calls it. This is exactly how Cargo works. I wonder if Cargo uses clap? Looking at its dependencies we can see that yes, indeed, it does. Will Clap execute plugins automatically? Let's try it:

$ cargo r -q -- test
error: Found argument 'test' which wasn't expected, or isn't valid in this context

USAGE:
    greet [OPTIONS] [SUBCOMMAND]

For more information try --help

No, it doesn't. But it's great to see that the usage description is generated according to the fact we have added options and subcommands.

Ok, now let's try our program out:

$ cargo r -q
No idea what to do!
$ cargo r -q -- --name Matt
No idea what to do!
$ cargo r -q -- --name Matt hello
Hello Matt
$ cargo r -q -- --name Bob bye
Goodbye Bob
$ cargo r -q -- hello --name Matt
error: Found argument '--name' which wasn't expected, or isn't valid in this context

USAGE:
    greet hello

For more information try --help

Well, that's not good. It seems that with Clap, the options before a subcommand are considered different than the options passed after a subcommand. We can add arguments to the subcommands:

use clap::{App, Arg, SubCommand};

fn main() {
    let matches = App::new("The Amazing Greeter")
        .version("1.0")
        .author("Matt Davies")
        .about("The best greeter in town!")
        .arg(
            Arg::with_name("name")
                .short("n")
                .long("name")
                .value_name("NAME")
                .help("Provides a name to use in the greeting")
                .takes_value(true)
                .default_value("World"),
        )
        .subcommand(
            SubCommand::with_name("hello")
                .about("Let's meet!")
                .version("1.0")
                .author("Matt Davies (again!)")
                .arg(
                    Arg::with_name("name")
                        .short("n")
                        .long("name")
                        .value_name("NAME")
                        .help("Provides a name to use in the greeting")
                        .takes_value(true),
                ),
        )
        .subcommand(
            SubCommand::with_name("bye")
                .about("We part ways")
                .version("1.0")
                .author("Matt Davies (again!)")
                .arg(
                    Arg::with_name("name")
                        .short("n")
                        .long("name")
                        .value_name("NAME")
                        .help("Provides a name to use in the greeting")
                        .takes_value(true),
                ),
        )
        .get_matches();

    let name = matches.value_of("name").unwrap();

    match matches.subcommand_name() {
        Some("hello") => {
            let submatches = matches.subcommand_matches("hello").unwrap();
            let name = submatches.value_of("name").unwrap_or(name);
            println!("Hello {}", name);
        }
        Some("bye") => {
            let submatches = matches.subcommand_matches("bye").unwrap();
            let name = submatches.value_of("name").unwrap_or(name);
            println!("Hello {}", name);
        }
        _ => println!("No idea what to do!"),
    }
}

A few things to take note of. Firstly, we don't provide default values for the subcommand options as the global option can provide that.

Secondly, to access the local options of a subcommand we need to grab its matches with subcommand_matches() that may or may not exist. Since we've already matched the name, we can unwrap it here with safety. Then we fetch the value if it exists, or use the global name value. The global name should always have a value because it has a default one.

This works quite well now:

$ cargo r -q -- hello --name Matt
Hello Matt
$ cargo r -q -- bye --name Matt
Goodbye Matt
$ cargo r -q -- hello
Hello World
$ cargo r -q -- --name Matt hello
Hello Matt
$ cargo r -q -- --name Matt hello --name Bob
Hello Bob

OK, it's not perfect since it doesn't detect two --name options, but perhaps we can remove the global option and ensure that the two subcommand options have default values.

There is a lot more you can do with Clap and I encourage you to explore the documentation.

But there's a better, more concise way to interact with Clap.

`Structopt` crate

What problem with Clap is that it is very verbose to describe the options and to use them. A better way to access the command line data would be to place it all in a structure hierarchy. Something like this:

struct CommandLineData {
    subcommand: SubCommand,
}

enum SubCommand {
    Hello(HelloData),
    Bye(ByeData),
}

struct HelloData {
    name: String,
}

struct ByeData {
    name: String,
}

But we still have to extract the information from Clap and insert it into that data structure. And, you've guessed it, is exactly what Structopt strives to do.

Through the use of macros and a function call, Structopt can generate the calls to Clap to declare the configuration and to extract the data into your structures. It is an extremely useful crate.

First, replace the clap entry in Cargo.toml with structopt = "0.3". And replace main.rs with:

use structopt::StructOpt;

#[derive(Debug, StructOpt)]
#[structopt(
    name = "The Amazing Greeter",
    about = "The best greeter in town",
    author = "Matt Davies"
)]
struct CommandLineData {
    #[structopt(subcommand)]
    subcommand: Option<SubCommand>,
}

#[derive(Debug, StructOpt)]
enum SubCommand {
    /// Let's meet!
    Hello(HelloData),

    /// Let's part ways
    Bye(ByeData),
}

#[derive(Debug, StructOpt)]
#[structopt(author = "Matt Davies")]
struct HelloData {
    /// Provides a name to use in the greeting.
    #[structopt(short = "n", long = "name", default_value = "World")]
    name: String,
}

#[derive(Debug, StructOpt)]
#[structopt(author = "Matt Davies (again!)")]
struct ByeData {
    /// Provides a name to use in the greeting.
    #[structopt(short = "n", long = "name", default_value = "World")]
    name: String,
}

fn main() {
    let opt = CommandLineData::from_args();

    match opt.subcommand {
        Some(SubCommand::Hello(data)) => println!("Hello {}", data.name),
        Some(SubCommand::Bye(data)) => println!("Goodbye {}", data.name),
        None => println!("Not sure what to do!"),
    }
}

Every structure and enum used to house command-line data must start with #[derive(Debug, StructOpt)]. The Debug trait needs to be there to provide error messages when things go wrong. The StructOpt trait is there to do the magic. Later in the structures and enums, we have #[structopt()] attribute macros and documentation comments to help with the meta-data. For example, the document comments for the Hello variant in SubCommand is used to generate the string that's passed to about() in Clap and is used for documentation. Two birds with one stone (that expression seems very cruel to me!).

One difference I noticed was that version information throughout the declaration is lifted directly from Cargo.toml and so different subcommands cannot have different versions. For me, this makes much more sense!

Finally, a quick call to from_args() method on your top-level data structure will construct an instance. The code that uses the command line data ends up being very concise.

As with Clap, StructOpt has lots of features that I encourage you to discover in the documentation. For example, you can declare an option as being required if another option is used. Also, you can collect all non-flag and non-option arguments into an array by just declaring a Vec field in your structure. Flags are easily produced using a bool type field. You may have noticed that name didn't require explicit information that it takes an argument. StructOpt was able to figure that out all by itself because it was of type String.

But there's one more thing we can do...

`Paw` crate

Paw allows us to treat the command line data structure as an argument to main(). In C and C++, you access the command line arguments via arguments passed to the main function but Rust handles it differently. You have to call a function to fetch them using std::env::args(). But with Paw, you can have the same functionality.

In Cargo.toml, change the dependencies to:

[dependencies]
structopt = { version = "0.3", features = ["paw"] }
paw = "1.0"

and now main can be:

#[paw::main]
fn main(opt: CommandLineData) {
    match opt.subcommand {
        Some(SubCommand::Hello(data)) => println!("Hello {}", data.name),
        Some(SubCommand::Bye(data)) => println!("Goodbye {}", data.name),
        None => println!("Not sure what to do!"),
    }
}

The paw::main macro wraps our main and transforms it into one that can take a single argument containing our command line data. No more calls to from_args(). You're not saving much but I thought this crate was quite cute. Very rust-like.

Conclusion

I showed you a terrible naive way of writing a program that processed the command-line, and then I showed you how to improve the situation drastically using Clap. Following that, I showed you that you didn't require much code at all and StructOpt allowed you to pack all your command line data into your own data structures whose very structure described the command-line use. Finally, I showed you Paw that allowed the command-line structure to be passed directly to your main function.

Hopefully, I have shown you that Rust, along with a few crates, is the best language to use for processing command-line arguments.

So, journey on and write some amazing CLI programs!

Rust #6: Exploring crates

Matt Davies — Sat, 24 Jul 2021 15:33:44 +0000

I often install tools via cargo and use crates for my code that have many dependencies. If you're like me, you are wondering when downloading and compiling what all those crates do. I could just look at the top N popular crates on http://crates.io but I thought that was boring. Rather, I thought I'd clone the exa command-line tool from Github and see what crates it used. Better to look at a released tool that is out in the wild?

The 'exa' tool is a Rust-built drop-in replacement for the Unix command ls. It allows me to produce listings like:

A simple command shows us the crate dependency hierarchy for this tool:

$ cargo tree

The top-level list of dependencies are:

ansi_term
datetime
git2
glob
lazy_static
libc
locale
log
natord
num_cpus
number_prefix
scoped_threadpool
term_grid
term_size
unicode-width
users
zoneinfo_compiled

Some interesting crates that I recognise and are dependencies of the above also appeared:

bitflags
byteorder
matches
pad
tinyvec
url

So now I will boot up my favourite browser, go to http://docs.rs and figure out what all these crates do and see if any are useful. At least cursory knowledge of them will stop me from reinventing the wheel if I need their functionality.

`ansi_term`

This is a library that allows the generation of ANSI control codes that allow for colour and formatting (i.e. bold, italic, etc.) on your terminal. If you are not sure what ANSI control codes are, this link here at Wikipedia should explain it. It provides styles and colours via the builder pattern or a colour enum. For example:

use ansi_term::Colour::Yellow;

println!("My name is {}", Yellow.bold().paint("Matt"));

There's support for blink, bold, italic, underline, inverse (confusingly call reverse here), 256 colours and 24-bit colours. This is a very useful crate for wrapping strings with the ANSI control codes that you need.

But I have seen crates that do this with different syntax using extension traits on &str and String. The one that I use frequently is colored. For example, to recreate the last snippet using colored, it is:

use colored::Colorize;

println!("My name is {}", "Matt".yellow().bold());

I prefer the latter form, but it is totally subjective.

`datetime`

This is one crate I am familiar with. Unfortunately, I found the standard's library time and date support lacking for a tool I was writing. Some research unearthed this crate that provides a lot more functionality. Be careful though, there is another crate called date_time that does similar stuff. The datetime library here provides structures for representing a calendar date and time of day in both timezone and local form. It also provides formatting functions that convert a date or time into a string and back again, but unfortunately no documentation on how to use it. The format function takes a time value but also takes a locale of type Time but no information on how to generate that. I couldn't figure it myself.

My go-to date and time crate that I use is chrono. It is fully featured, efficient and better documented. You can do UTC, local and fixed offset times (times in a certain timezone). It can format to and parse from strings containing times and dates. Times also have nanosecond accuracy. Times and dates are complicated things and chrono is a useful weapon in your coding arsenal.

`bitflags`

This is a very useful crate for generating enums that are bitmasks. You often see this in low-level programming or when dealing the operating system interfaces. Unlike normal enums, bitflag enums can be combined using logical operators. This is trivial to do in C but is not supported in Rust. In C, you could do something like:

typedef enum Flags {
    A = 0x01,
    B = 0x02,
    C = 0x04,
    ABC = A | B | C,
};

int ab = A | B;

In Rust, enums are distinct and cannot be combined like that. With bitflags you can:

use bitflags::bitflags;

bitflags! {
    struct Flags: u8 {
        const A = 0b001;
        const B = 0b010;
        const C = 0b100;
        const ABC = Self::A.bits | Self::B.bits | Self::C.bits;
    }
}

let ab = Flags::A | Flags::C;

Not as eloquent as C, but at least the enumerations are scoped like C++'s enum class. However, Rust's bitflags crate does support set difference using the - operator. This would go very wrong in C and C++ as - would be treated as a normal integer subtract. For example:

let ac = Flags::ABC - Flags::B;
let we_dont_want_b = ac - Flags::B;

would do the right thing. The equivalent code in C would not.

`byteorder`

This simple crate allows reading and writing values in little-endian or big-endian byte order. The standard library does have some support with the to_le_bytes et al. family of functions on the integer primitive types so a lot of this crate is redundant now. Where this crate is useful is with implementing the Read and Write interfaces.

If you're wondering what endian means, it refers to how computers store numeric values that require more than a byte to store. For example, with u32 it takes 4 bytes of storage. There are 2 conventional ways of storing this value. You could put the high bytes first (big-endian) or the low bytes first (little-endian) into memory. So for example, the number 42 could be stored as the bytes 42, 0, 0, 0 or the bytes 0, 0, 0, 42. Most modern CPUs, by default, support the former, which is little-endian. However, data that goes over the network is usually big-endian. So these routines are critical for putting the data in the correct form. There is also a third convention called native-endian that is either little or big depending on the CPU's preferred form.

`git2`

This crate offers bindings over the C-based libgit2 library. exa uses this to implement its .gitignore support. This is a large crate and way beyond the scope of this article.

`glob`

One of the main obvious jobs of exa is to iterate over all the files in a directory. glob does this with the bonus that you can use the wildcards * and **. It provides a single function glob that takes a file pattern and gives back an iterator returning paths. For example:

// Read in all the markdown articles under all folders in my blogs folder.
use glob::glob;

for entry in glob("~/blogs/**/*.md").unwrap() {
    match entry {
        // path is a PathBuf
        Ok(path) => println!("{}", path),

        // e is a GlobError
        Err(e) => eprintln!("{}", e),
    }
}

Of course, being an iterator, you can run it through all the iteration operators, such as filter, map etc. But there is no need to sort since paths are yielded in alphabetical order.

`lazy_static`

Now this is a crate that is often used. Normally, static variables cannot have run-time calculated values. Try this:

fn main() {
    println!("Number = {}", MY_STATIC);
}

static MY_STATIC: u32 = foo();

fn foo() -> u32 {
    42
}

You will be greeted with the error:

error[E0015]: calls in statics are limited to constant functions, tuple structs and tuple variants
 --> src/main.rs:5:25
  |
5 | static MY_STATIC: u32 = foo();
  |                         ^^^^^

Using lazy_static it becomes:

use lazy_static::lazy_static;

fn main() {
    println!("Number = {}", *MY_STATIC);
}

lazy_static! {
    static ref MY_STATIC: u32 = foo();
}

fn foo() -> u32 {
    42
}

There are three main changes to the code. Firstly, there's the lazy_static! macro to wrap the static declarations. Secondly, there's an added ref keyword. The statics returned here are references to your type. Using them invokes the Deref trait. This means that thirdly, I had to dereference it so that the Display trait was detectable for u32. In Rust, Deref is not invoked when looking for traits so I had to do it manually.

`libc`

There is a vast sea of C code out there implementing many useful libraries. To speed up Rust's adoption, it was required not to rewrite many of these libraries in Rust. Fortunately, the designers of Rust realised that and made it easy to interoperate with C. libc provides more support to interoperate with C code. It adds type definitions (like c_int), constants and function headers for standard C functions (e.g. malloc).

`locale`

This crate is documented as mostly useless as it is being rewritten for its version 0.3. This provides information on how to format numbers and time.

use locale::*;

fn main() {
    let mut l = user_locale_factory();

    let numeric_locale = l.get_numeric().unwrap();
    println!(
        "Numbers: decimal sep: {} thousands sep: {}",
        numeric_locale.decimal_sep, numeric_locale.thousands_sep
    );

    let time = l.get_time().unwrap();
    println!("Time:");
    println!(
        "  January: Long: {}, Short: {}",
        time.long_month_name(0),
        time.short_month_name(0)
    );
    println!(
        "  Monday: Long: {}, Short: {}",
        time.long_day_name(1),
        time.short_day_name(1)
    );
}

This outputs:

Numbers: decimal sep: . thousands sep: ,
Time:
  January: Long: January, Short: Jan
  Monday: Long: Mon, Short: Mon

Mmmm... there seems to be a bug at the time of writing. Surely time.long_day_name(1) should return Monday and not Mon. Whether this is an operating system issue or a problem with locale, I am not sure.

`log`

This crate provides an interface for logging. The user is expected to provide the implementation of the logger. This can be done through other crates such as env_logger, simple_logger and few other crates. log is not used directly by exa itself, but rather some of its dependencies.

Essentially, it provides a few macros such as error and warn! to pass formatted messages to a logger. There are multiple levels of logging and they range from trace! to error! in order of rising priority:

trace!
debug!
info!
warn!
error!

I think this crate is missing a fatal! as most logging systems contain these levels.

Log messages can be filtered by log level, with the lowest level restricting more lower-level messages and the highest level showing all messages. This is set using the set_max_level function. By default, it is set to Off and no messages are sent to loggers. Levels can also be set at compile-time using various features. All of this is described in the documentation.

Loggers implement the Log trait and users install them by calling the set_logger function.

How should you use the logging levels? Below I provide some opinionated guidance:

Trace

Very fine-grained information is provided at this level. This is very verbose and high traffic. You could use this to annotate each step of an algorithm. Or for logging function parameters.

Debug

Used for everyday use and diagnosing issues. You should rarely submit code that outputs to debug level. At the very least it shouldn't output in release builds.

Info

The standard logging level for describing changes in application state. For example, logging that a user has been created. This should be purely informative and not contain important information.

Warn

This describes that something unexpected happened in the application. However, this does not mean that the application failed and as a result work can continue. Perhaps a warning could be a missing file that the application tried to load but does not necessarily require it for running (e.g. a configuration file).

Error

Something bad happened to stop the application from performing a task.

`matches`

Allows a check to see if an expression matches a Rust pattern via a macro:

// Macro version
let b = macros!(my_expr, Foo::A(_));

// is the same as:
let b = match my_expr {
    Foo::A(_) => true,
    _ => false,
}

// or even:
let b = if let Foo::A(_) = my_expr { true } else { false };

It also provides assert versions as well. It is just a small convenience crate.

`natord`

If you were to sort these strings using normal method: ["foo169", "foo42", "foo2"], you would get the sequence ["foo169", "foo2", "foo42"]. This might not be the order you would prefer. What you might want is sometimes referred to as normal ordering. You might prefer the order ["foo2", "foo42", "foo169"] where the numbers in the strings increase in value and not by ASCII ordering.

This functionality is what natord provides. Natural ordering can work well for filenames with numbering inside them and IP addresses to name just a couple. For example, natural ordering will handle strings where the numbers are in the middle (e.g. "myfile42.txt") and not just at the end.

I am sure I will make use of this crate at some point in the future.

`num_cpus`

Short and sweet this one. It provides a get() function to obtain the number of logical cores on the running system, and get_physical() function to obtain the number of physical cores. Very useful if you want to set up a worker thread pool.

`number_prefix`

This crate determines the prefix of numerical units. Given a number, it determines whether there should be a prefix such as "kilo", "mega", "K", "M" etc., or not. It can even handle prefixes that describe binary multipliers such as 1024, something that many programmers will appreciate, but a prefix like K will not be used. Rather Ki will be used.

And given a prefix, it can be converted into its full name. For example, the prefix for 1000 would be K, but call upper() on it and you will get KILO, call lower() for kilo and caps() for Kilo.

This is very useful for listing file lengths as exa is required to do.

`pad`

This crate is used for padding strings at run-time. The standard library format! function can do a lot of padding functionality but this crate can do more.

For example, it can add spaces at the front to right-align a string within a set field width:

let s = "to the right!".pad_to_width_with_alignment(20, Alignment::Right);

Neat huh?

You don't have to pad with spaces either. It can use any character you wish. It will also truncate strings that are too long for a particular width.

If you need to format the textual output on a terminal, you may need to look this crate up.

`scoped_threadpool`

This can produce a struct that manages a pool of threads. These threads can be used to run closures that can access variables in the original scope. This is useful because normally threads can only access values of 'static lifetime or are entirely owned inside the thread.

Let us look at the example in the documentation.

fn main() {
    // We create a pool of 4 threads to be utilised later.
    let mut pool = Pool::new(4);

    // Some data we can do work on and reference to.
    let mut vec = vec![0, 1, 2, 3, 4, 5, 6, 7];

    // Use the pool of threads and give them access to the scope of `main`.
    // `vec` is guaranteed to have a lifetime longer than the work we
    // will do on the threads.
    pool.scoped(|scope| {
        // Now we grab a reference to each element in the vector.
        // Remember `vec` is still around during `pool.scoped()`.
        for e in &mut vec {
            // Do some work on the threads - we move the reference
            // in as its mutably borrowed.  We still cannot mutably
            // borrow a reference in the for loop and the thread.
            scope.execute(move || {
                // Mutate the elements.  This is allowed as the lifetime
                // of the element is guaranteed to be longer than the
                // work we're doing on the thread.
                *e += 1;
            });
        }
    });
}

This crate allows us to create works on threads that we know will not outlive other variables in the same scope. pool.scoped must block until all work is done to allow this to happen.

This is very useful for quickly doing short-lived jobs in parallel.

`term_grid`

To produce standard ls output, exa must show filenames in a grid formation. Given a width, this crate provides a function fit_into_width that can help to produce a grid of strings. By working out the longest string in a collection, it can calculate how many of those strings can fit in horizontal space, like say, the line on a terminal.

`term_size`

Very simple but crucial if you want to provide textual output on a terminal in a highly formatted way. It uses ANSI control codes to communicate with your terminal to figure out the size of your terminal view.

let (width, height) = match term_size::dimensions() {
    Some((width, height)) => (width, height),
    None => (80, 25),
}

The terminal, possibly, may not report the dimensions (although most do) and so the result of dimensions() is an Option<(usize, usize)>.

`tinyvec`

This mainly provides 2 vector types: ArrayVec and TinyVec.

ArrayVec is a safe array-backed drop-in replacement for Vec. This means that a fixed array is allocated to act as storage for ArrayVec. In any other way, ArrayVec acts like a Vec but it cannot reallocate that storage. If the array becomes too big a panic will occur.

This is very useful to avoid reallocations all the time. ArrayVec even allows access to the unused space in the backing array that is currently being used for elements.

Because ArrayVec uses a fixed array, a heap allocation does not occur.

TinyVec is a drop-in replacement for Vec too. However, for small arrays, it starts life as an ArrayVec using a fixed array as the backing store. As soon as it becomes too big, it will automatically revert to use a normal Vec, hence using the heap to store the array.

This requires the alloc feature to be activated.

It is basically an enum that can be an ArrayVec or a Vec:

enum TinyVec<A: Array> {
    Inline(ArrayVec<A>),
    Heap(Vec<A::Item>),
}

You have got to love algebraic types! You have to provide the backing store type and a macro helps you do this:

let my_vec = tiny_vec!([u8; 16] => 1, 2, 3);

This example can only store the first 16 elements on the stack before it switches to a more regular Vec<u8>.

`unicode-width`

Unicode characters can be wide, and on the terminal, it's important to know how many characters a Unicode character or string might take up. There is a standard for working out that information and it is called the Unicode Standard Annex #11.

This is very useful when displaying filenames with international characters, which is exactly what exa needs to deal with. Fortunately, there is a crate that can provide that information.

The documentation for unicode-width does remind us that the character width may not match the rendered width. So be careful but I don't think there is much you can do in these situations.

`url`

A typical URL can code much information including:

the protocol scheme (e.g. http)
the host (e.g. docs.rs)
a port number (e.g. the number in http://docs.rs:8080)
a username and password
a path (e.g. foo/bar in the URL http://docs.rs/foo/bar)
a query (e.g. everything after the ? in myurl.com/foo?var=value)

A lot of these are optional and so a URL parsing library will need to handle this too.

This crate provides a Url::parse function that constructs an object describing the various parts of an URL. Various methods can then be called to provide string slices into the URL such as scheme() or host().

This crate also provides serde support too so if you know what that is, you will understand what this means. Maybe I will write about serde in a future article.

`users`

This is a very Unixy thing and so is useful on Mac OSX too. To handle various permissions in a Unix operating system, a process is assigned an effective user ID. This user ID and the various group IDs the user belongs to determines the permissions a process has within the operating system, including file and directory access.

The users crate provides this functionality by wrapping the standard C library functions that can obtain user and group information.

If you have trace level logging on, this crate will log all interactions with the system.

This crate also has a mocking feature to use pretend users and groups that you can set up for purposes of development. You do not necessarily want to access real IDs.

Even though this is a very Unixy thing, Windows does have similar permission features. So this crate should work on Windows too. Although I haven't tested this and the documentation does not say so. But I have seen exa work on Windows so I am assuming.

`zoneinfo_compiled`

This crate seems to get information directly from the operating system via zoneinfo files. This information allows you to obtain information on leap seconds and daylight savings time for your current time zone.

This information is maintained by a single person and distributed and stored on your harddisks if you use Unix-based systems. You can find this data in the /usr/share/zoneinfo folder. Each timezone has a binary file and this crate can parse this file and extract the information it holds.

The whole proper handling of time and time zones is beyond me at this moment in time. So I am not sure how exa would be using this crate. It's a very complex topic and something I would love to dig in deeper with and hopefully not go insane at the same time.

Another name for the database is the tz database and you can find more information about it, if you so desire, at Wikipedia.

Conclusion

I hope you enjoyed this little trip into a few crates used by a single project. I encourage all of you to try this simple exercise. I learnt a lot by researching for this article and I still feel I have not even scratched the surface.

Please write in the discussion below about interesting crates that you have found and used. I would love to hear about them.

Until next time!

Rust #5: Naming conventions

Matt Davies — Sat, 17 Jul 2021 12:58:37 +0000

This week, I wanted to clarify in my head what the naming conventions are in the standard library if any, and to see if they help with communicating concepts in the API.

A good API has well-established concepts and paradigms and a good language to communicate that. When you see a certain proposition in a name or verb, you understand what that API is trying to do. Good examples are MacOSX's Cocoa API. Bad examples are the Windows Win32 API. For me, the most important thing is consistency.

I feel that Rust's std library API is trying to achieve the same thing, and I want to explore what that language is.

I have already seen some of this with the prefixes and suffixes of some methods of Option and Result, such as or and or_else and more simply the use of is.

Let us go through the tools that the std library uses to convey information to us.

Casing

The Rust compiler is very opinionated about what casing and style you use to name things, even giving warnings when you break its rules. You can disable those warnings if you wish. Initially, I HATED the casing rules as I was used to my styles with my C++ background, and I have to admit, was a contention point for me in using Rust. But after a week of writing Rust code, I soon got used to it and appreciate there is consistency if not exactly what I'd prefer.

Here is a quick overview of the rules:

Convention	Types that use it
`snake_case`	Crates, modules, functions, methods, local variables and parameters, lifetimes.
`CamelCase`	Types (including traits and enums), type parameters in generics.
`SCREAMING_SNAKE_CASE`	Constant and static variables.

This means that when you have a type name and you want to refer to it in a function name, you have to convert. So YourType will become your_type. For example:

struct BankAccount {
    id: u64,
    name: String,
    amount: u128,
}

fn find_bank_account(id: u64) -> Rc<BankAccount> { ... }

Ok, this is general Rust stuff and not specific to the std library, but I thought it was worth going over.

Basic types

The standard library naming convention documentation lists various pieces of text that are in method names based on the types that they operate on:

Type name	Text in methods
&T	ref
&mut T	mut
&[T]	slice
&mut [T]	mut_slice
&[u8]	bytes
&str	str
*const T	ptr
*mut T	mut_ptr

Get and Set methods

Methods that fetch or set a value within their struct are usually called Getter and Setter methods.

For Setter methods, the prefix set_ is used with the field name. For Getter methods, no prefix is used and just the field's name is. For example:

struct Person {
    name: String,
    age: u16,
}

impl Person {
    fn new(name: String, age: u16) -> Self {
        Person { name, age }
    }

    // Getters

    fn name(&self) -> &str { &self.name }
    fn age(&self) -> u16 { self.age }

    // Setters

    fn set_name(&mut self, name: &str) { self.name = name.to_string(); }
    fn set_age(&mut self, age: u16) { self.age = age; }
}

Suffixes

Suffixes are pieces of text that appear at the end of a name. And the std library has a few. Below is a list I've discovered.

Suffix	Meaning	Example
err	Deals with the error part of a result	Result::map_err()
in	This function uses a given allocator	Vec::new_in
move	Owned value version for a method that normally returns a referenced value
mut	Mutable borrow version of method that normally returns owned or referenced value	&str::split_at_mut()
ref	Reference version for a method that normally returns an owned value	Chunks::from_ref()
unchecked	This function is unsafe - there may be dragons beyond!	&str::get_unchecked()

Due to the nature of Rust, many methods have variants. If a method foo returns an immutably borrowed value, then there are sometimes variants that return a mutably borrowed or an ownable copy. The suffixes mut, move and ref are used for this. For example:

fn foo() -> &T;             // original method
fn foo_mut() -> &mut T;     // mutable version
fn foo_move() -> T;         // owned version

fn bar() -> T;              // original version
fn bar_ref() -> &T;         // immutably borrowed version
fn bar_mut() -> &mut T;     // mutably borrowed version

However, there are exceptions. For conversion to an iterator returning owned values, into_iter() is used instead of iter_move(). Also, if the method contains a type name and it returns a mutable borrow, the mut appears before the type. For example:

fn as_bar(foo: &Foo) -> &Bar;
fn as_mut_bar(foo: &Foo) -> &mut Bar;

err is used with Results for methods that work with the error part instead of the OK part.

unchecked states that this method or function is unsafe. Make sure you add your protections around it. This suffix should be a BIG red flag to any user wanting to stay within Rust's safety. Usually, the documentation contains information on what you should do around this function to make sure it's still safe. For example, on &str::get_unchecked(i) the documentation asks that you:

The starting index must not exceed the ending index.
Indices must be within bounds of the original slice.
Indices must lie on UTF-8 sequence boundaries.

Now, it may be that given index i, your program has already ensured that these conditions are true; in which case you can go ahead and use get_unchecked instead of get because it will be more efficient. However, as a programmer, whenever you see the unchecked suffix, you should be reaching for the documentation to see what those conditions are.

in is not stable yet and Standard Library methods that use this suffix are only in the nightly compiler as of the time of writing. This indicates a variant that allows the user to pass in their allocator for allocation. For example, whereas Vec::new creates a vector that will use the default allocator when adding elements, Vec::new_in requires an allocator object that implements the Allocator trait that will be responsible for allocations.

Occasionally, there are times where you want to have multiple suffixes. If you do and one of them is mut, the convention is that mut should be last.

Prefixes

Prefix	Meaning	Example
as	Free conversion from original	&[u8]::as_bytes()
into	Conversion that consumes original value	String::into_bytes()
is	Query method that returns a bool	Vec::is_empty()
new	Indicates a constructor	Box::new_zeroed()
to	Expensive conversion from original	&str::to_owned()
try	This variant returns a Result instead panicking	Box::try_new()
with	The variant uses a function to provide a value instead of directly	RefCell::replace_with()

as, to and into prefixes indicate conversions. But there are different types of conversions. There are conversions that are free because they are basically no-ops. There are conversions that are expensive because a new type is constructed from the old. And there are conversions that consume the original value to produce the new one. Rust differentiates between these conversions by using prefixes.

as essentially exposes a view to an underlying representation that is present in the original value. For example, a slice from a vector. as methods do not affect the original value and usually produce a borrowed reference. This means that the conversion's lifetime must be shorter than the original value.

to is an expensive conversion that constructs a new value from the old one. For example, creating a new String from a string slice. These methods usually produce an ownable value.

into can be expensive and can be a no-op depending on what it does. However, it consumes the original value. An example of this is converting a String to a Vec<u8> using into_bytes(). The String contains a Vec<u8> under the bonnet, and so producing it is effectively a no-op. But because it's given away the underlying structure, the String cannot exist anymore.

Another example is into_iter(), which consumes the original container and produces an iterator that can give back its values. It can do this because the new iterator value owns the values and not the original container.

These prefixes are essential in understanding the semantics of the operations their methods provide.

new is often the complete name of a static method for constructing an instance of a struct. However, sometimes there are different ways to do construction and therefore multiple variants of new. These all have the new prefix.

try, often used with new for the try_new prefix indicates a version of a method that can fail gracefully. For example:

// This function will panic if the id does not exist.
fn get_config(id: u32) -> Config { ... }

// This will not panic and will return a Result instead.
fn try_get_config(id: u3) -> Result<Config, ConfigError> { ...  }

If the Result version is the only version, then there is no need for the prefix try. It's only used for variants.

with indicates a method that whereas the original function passed a value as a parameter, this variant will be passed a function that provides that value instead. To be fair, I have not seen this suffix used that often. I have even seen other suffixes used to mean the same thing. For example, should Option::ok_or_else really be Option::ok_or_with?

Conclusion

I found this research to be enlightening and helpful in understanding the Standard Library better.

Please comment below about any other prefixes or suffixes that I might have missed - I am sure that there are many.

Also, if you've been enjoying my series of random articles, please let me know what you want me to look into next.

Until next week!

Rust #4: Options and Results (Part 2)

Matt Davies — Sat, 10 Jul 2021 15:54:54 +0000

Last week I wrote about basic use of Option, Result and creating and using errors derived from std::error::Error. But both Option and Result have a host of methods that I wanted to explore and describe in today's post. Below is an overview of some of the Option methods I want to talk about:

Method	Use Description	Return Type
and	Testing two `Option`s are not `None`	Option<U>
and_then	Chaining `Option`s	Option<U>
expect	Panic if None	T
filter	Filter the `Option` with a predicate	Option<T>
flatten	Removes nested `Option`s	Option<T>
is_none	Test the `Option` type	bool
is_some	Test the `Option` type	bool
iter	Iterate over its single or no value	an iterator
iter_mut	Iterate over its single or no value	an iterator
map	Transform the value into another	Option<U>
map_or	Transform the value into another	U
map_or_else	Transform the value into another	U
ok_or	Transform the `Option` to a `Result`	Result
ok_or_else	Transform the `Option` to a `Result`	Result
or	Provide a new value if `None`	Option<T>
or_else	Provide a value if `None`	Option<T>
replace	Change the value to a `Some` while returning previous value	Option<T>
take	Change the value to `None` while returning original	Option<T>
transpose	Change `Option` of `Result` to `Result` of `Option`	Result, E>
unwrap	Extract value	T
unwrap_or	Extract value	T
unwrap_or_default	Extract value	T
unwrap_or_else	Extract value	T
xor	Return one of the contained values	Option<T>
zip	Merge `Options`	Option<(T, U)>

This is a non-exhaustive list. And below is an overview of the Result methods I want to talk about:

Method	Use Description	Return Type
and	Testing two `Results` are not errors	Result
and_then	Chaining `Results`	Result
err	Extract the error	Option<E>
expect	Panic if `Err`	T
expect_err	Panic if `Ok`	E
is_err	Test the `Result` type	bool
is_ok	Test the `Result` type	bool
iter	Iterate over its single or no value	an iterator
iter_mut	Iterate over its single or no vlaue	an iterator
map	Transform the value into another	Result
map_err	Transform the value into another	Result
map_or	Transform the value into another	U
map_or_else	Transform the value into another	U
ok	Converts `Result` into `Option`	Option<T>
or	Provide a new `Result` if `Err`	Result
or_else	Provide a new `Result` if `Err`	Result
transpose	Change `Result` of `Option` to `Option` of `Result`	Option<Result<T,E>>
unwrap	Extract value	T
unwrap_err	Extract error	E
unwrap_or	Extract value	T
unwrap_or_default	Extract value	T
unwrap_or_else	Extract value	T

You can see from the above tables thit's the absense of a result Option and Result have similar methods and act in similar ways. This is not surprising if you think about it because both can return results or a non-result. In Option's case, the non-result is an absence of a result (None) and in Result's case, the non-result is an error.

Both have logical operations that combine each other: and, or and in Option's case, xor.

Both can be treated like an iterator with one or no values.

Both can transform their values via the map family of methods and extract values via the unwrap methods.

Also, you may notice some common suffixes on the functions, particularly or versus or_else. The difference between these functions is how they provide a default value. The or signifies that if there is no result (i.e. None or Err), then here is one I provide for you! The else part signifies that I will provide the default value, but via a function. This allows a default result to be provided lazily. Passing a value to an or type method will mean it is evaluated regardless of the original value. This is because all parameters in Rust are evaluated before or is called. Because or_else uses a function to provide the default value, this means it is not evaluated unless the original value is an None or Err.

Good and Bad

Because I will be talking about Option and Result generically as they share so much in common, I will use different terminology for the types of values they can have.

For Some and Ok values, I will call them good values.

For None and Err values, I will call them bad values.

There is no semantic meaning to the terms other than to distinguish between them. I could use positive and negative, or result and non-result. But the words good and bad are easier to type!

Extracting Good Values and Result Errors

A common use pattern is to extract the good value regardless of whether it is good. This is done by the unwrap family of methods. How they differ is what happens when the value is not good. If it is a good value, they just convert the value into the contained value. That is, an Option<T> or Result<T,E> becomes a T.

If you want to panic on a bad value, just use unwrap. I wouldn't recommend using this as panicking is so user unfriendly, but it's good for prototype work and for during development.

If you want to provide a default value, then there are two ways to do this: eagerly; and lazily. The unwrap_or and unwrap_or_else do this and I've already explained what or and or_else means above.

If you want to provide the type's default value via the Default trait, use unwrap_or_default.

Finally, Result has a couple more unwrapping methods for errors: err and unwrap_err. err will return an Option<E> value, which is None if no error occurred. unwrap_err extracts the error value.

let maybe_name: Option<String> = get_dog_name();

let name1: String = maybe_name.unwrap();
let name2: String = maybe_name.unwrap_or(String::from("Fido"));
let name3: String = maybe_name.unwrap_or_else(|| String::from("Fido"));
let name4: String = maybe_name.unwrap_or_default();

let maybe_file: Result<std::fs::File, std::io::Error> = File::open("foo.txt");
let file_error: std::io::Error = maybe_file.unwrap_err();
let maybe_file_error: Option<std::io::Error> = maybe_file.err();

Iteration and Transforming Values

Both Option and Result can be treated as containers that have one or zero values in it. By iterating over a single Ok or Some value, this opens up Options and Results to all the iterator functionality.

So iter provides the &T and iter_mut provides the &mut T on the "collection".

Some of the iterator methods have been brought into Option and Result, removing the requirement to convert to an iterator first. This is the map family of methods and for Option there is filter.

map calls a function that converts the good value of one type to a good value of another. The function moves the wrapped value into a function that returns a new one, which doesn't even have to share the same type. So it converts an Option<T> to a Option<U> or a Result<T,E> to a Result<U,E>. For bad values, it remains the bad value with no transformation.

map_or and map_or_else extends map's functionality by provided a default value in the case of a bad value. This follows the same or and or_else functionality as described above. However, the function you provide for the default in the Result's case is provided with the error value.

Result has one more version for transforming errors. This is map_err and is often used, for example, to adapt standard errors to your own.

struct Dog {
    name: String,
    breed: Breed,
}

let maybe_name: Option<String> = get_dog_name();

let dog1: Option<Dog> = maybe_name.map(|dog_name| Dog {
    name: dog_name, 
    breed: Breed::Labrador 
});
let dog2 = maybe_name.map_or(
    Dog {
        name: String::from("Fido"),
        breed: Breed::Labrador,
    },
    |dog_name| Dog {
        name: dog_name,
        breed: Breed::Labrador,
    }
);
let dog3 = maybe_name.map_or_else(
    || Dog {    // This lambda is passed an error argument in Result case.
        name: String::from("Fido"),
        breed: Breed::Labrador,
    },
    |dog_name| Dog {
        name: dog_name,
        breed: Breed::Labrador,
    },
);

let maybe_file: Result<File, std::io::Error> = std::fs::File::open("foo.txt");
let new_file: Result<File, MyError> = maybe_file.map_err(|_error| Err(MyError::BadStuff));_

Querying Value Types

Option provides is_some and is_none to quickly determine if it contains a good or bad value.

Result also provides is_ok and is_err for the same reasons.

There really isn't much to say about this. I would add that these are not used that often because the tests are implicit with the if let and match syntaxes.

let maybe_name = get_dog_name();
match maybe_name {
    Some(name) => Dog { name, breed: Breed::Labrador },
    None => Dog::default(),
}

// instead of:
let dog = if maybe_name.is_some() {
    // You still need to deconstruct the value here so the 
    // is_some check is redundant.
    if let Some(name) = maybe_name {
        Dog {
            name,
            breed: Breed::GermanShepherd,
        }
    } else {
        Dog::default()
    }
};

They are really only useful if you need to convert an option or result to a boolean.

Logical combinations

Both Option and Result provide and and or methods that combine two values according to their logical rules.

With and, both values need to be good, otherwise the result is bad. Also, when all values are good, the result is the final good value. This makes and a really good way of chaining operations if you don't care about the results of anything but the last operation. For example:

// All these operations must be good
fn get_name(person: &Person) -> Option<String> { ... }
fn clone_person(person: &Person, name: String) -> Option<Person> { ... }

// We only create a new clone if the first clone has a name.  For some
// reason unnamed clones are not allowed to be cloned again!
let new_person = get_name(old_person).and(clone_person(old_person, "Brad"));

// new_person will either be None or Some(Person)

But even more useful is the method and_then. This allows you to chain operations together but passing the result of one to the function of the next.

For example, perhaps I want to try to open a file, and read the first line. Both of these operations can fail with a bad Result. and_then is perfect for this, because I only care about the first line and none of the intermediate data such as the open file:

use std::result::Result;
use std::fs::File;

fn read_first_line(file: &mut File) -> Result<String> { ... }

let first_line: Result<String> = File::open("foo.txt").and_then(|file| {
    read_first_line(&mut file)
});

The or method will check the first result and will return that result only if it is good. If it is bad, it will return the second result. This is useful for finding alternative operations. If one fails, then let's try the other.

struct Disease { ...  }

fn get_doctors_diagnosis() -> Option<Disease> { ... }
fn get_vets_diagnosis() -> Option<Disease> { ... }

// We're desperate!  Let's ask a vet if the doctor doesn't know.
let disease: Option<Disease> = get_doctors_diagnosis().or(get_vets_diagnosis());

Notice how or differs to and in that all operations must wrap the same type. In the and example, the first operation wrapped a File, then resulte in a wrapped String. This is possible due to the nature of the logical AND. As soon as one operation is bad, all results are bad. This is not so for logical OR. Any operation could be bad and we will still get a good result if at least one of them is good. This means that all wrapped values must be the same. In the example, all wrapped values were of type Disease.

Option provides one more logical combination and that is the logical XOR operation. The result is good if and only if only one of the operations is good. You can either take this result or the other, but not a combination of both.

`and_then` versus `map`

It may not be obvious, but and_then and map do very similar things. They can transform a wrapped value into another. In the example above, and_then changed an Option<File> into an Option<String>. With the map example, we changed an Option<String> into an Option<Dog>. But they do differ and I want to talk about how they differ.

Both methods take a function that receives the wrapped value if good but the difference is in what those functions return.

In the case of and_then it returns a value of type Option<U>, whereas map returns a value of type U. This means that and_then can change a value from a good one to a bad one but map cannot; once good always good. map is purely for the transformation of a wrapped value to another value and even type but not goodness. That is, that transformation step cannot fail by becoming Err in the case of Result, or None in the case of Option. With and_then the transformation can fail. The function passed can return None or an Err.

It took a while for it to click for me why you use one over the other. I hopes this helps to clarify the reasons.

Converting between Option and Results

Because these types are used in very similar circumstances, it is often useful to convert between them. Let's talk about how we do this.

Some(T) -> Ok(T), None -> Err(E)

We have an option and we want to convert to a result. You could use a match:

match opt {
    Some(t) => Ok(t),
    None => Err(MyError::new()),
}

That's a little verbose, but you can use ok_or and ok_or_else to provide the error if the option is None:

let res = opt.ok_or(MyError::new());
let res = opt.ok_or_else(|| MyError::new());

Ok(T) -> Some(T), Err(E) -> None

Now we have a result and we want to convert to an option. Again with match:

match res {
    Ok(t) => Some(t),
    Err(_) => None,
}

Result uses a method ok to do the conversion:

let opt = res.ok();

Transposition and Flattening

Both Result and Option are container types that wrap a value of type T. But that type T can just as well be a Result and an Option too.

Transposition is the operation to swap a Result and Option in a nested type. So, for example, a Result<Option<T>> becomes a Option<Result<T>> or vice versa. Both Result and Option offer a transpose method to do that swapping.

Also, it is useful to convert a Result<Result<T,E>,E> into a Result<T,E>, or convert an Option<Option<T>> into an Option<T>. This operation is called flattening and both methods offer a flatten method to do so. But, as of writing, Result::flatten is only available in nightly builds and so therefore, it has not been stablised yet. This is why this article only shows flatten in the Option table at the beginning.

Miscellaneous Operations

We've covered the lion's share of the methods that are used with Option.

Firstly, replace and take are used to move values in and out of the Option. replace transfers ownership of a value into the Option and returns the old value so something else can own it. This is useful to make sure the option is not in a valid state. You cannot just move its value out without replacing it with a new value. The borrow checker will not let you do that. Quite often this is used for options within structures.

It is very common that when you do a replace, you set the option to None as you extract the value you want. That is, you want to do something like this:

let mut x = Some(42);

// Extract the 42 to another owner, but now make x own `None`.
let extracted_value = x.replace(None)

But, unfortunately, replace takes a value of type T, meaning you can only replace it with a Some variant. Option provides another method take to do this:

let mut x = Some(42);
let extracted_value = x.take();  // x will be None after this.

Summary

I hope I have provided a reasonable guide to using Option and Result effectively over the past two articles. It's a lot to take in and I will certainly be referring to my own articles to help me remember how I should use them.

We looked at what they are and how to create and use errors. We looked at methods to transform values, extract values, logically combine, manipulate structure and convert between them. Rust standard library provides a lot of functionality. There are more methods that I didn't go into, mainly because they were with advanced or unstable.

Next week I will write about naming conventions in the Rust Standard Library. As you have seen here, there are common suffixes that describe what the function may do. In the Standard Library there are prefixes too that help convey concepts to functions that share them. I'd like to understand them better and so I can understand what the function might be doing at a glance and perhaps reused them for my own code so that others can understand it too.

Rust #3: Options, Results and Errors (Part 1)

Matt Davies — Sat, 03 Jul 2021 18:16:34 +0000

Options and Results

The Option and Result types in Rust will be two of the most used types you will have at your disposal when writing your programs. Their concepts are simple but their use can be confusing at times for beginners. It was for me. This blog entry is an attempt to help explain how to use them effectively.

A Result can also wrap a Rust error and this blog article will cover how to create those easily too.

Let's look at the basics.

Option

The Option type allows you to have a variable that may or may not contain a value. This is useful for passing optional parameters or as a return value from a function that may or may not succeed.

Its definition is

enum Option<T> {
    Some(T),
    None,
}

So it can either contain a single value via the Some variant or no value at all via the None variant.

Enums in Rust are implemented with a discriminant value that tells Rust what type of variant is stored and a union of all the data in the variants. So, if you were to implement the same thing in C, it would look something like:

struct Option
{
    int type;
    union 
    {
        struct Some 
        {
            T t;
        };
        struct None {};
    };
};

So, the size of the enum is usually the size of the largest variant plus the size of the type value.

But Rust has a neat optimisation. If one variant has no data and the other has a single value that is a non-null pointer (such as references, boxes, function pointers), Rust will optimise the enum type so that its size is the same as the type T. It accomplishes this by representing the no-value variant (e.g. None) as a null pointer. This means something like Option<&T> is the same size as &T. Effectively, this is like normal C pointers with the extra type safety.

For more information about this check out the documentation of Option here under the section titled 'Representation'.

Below is an example of how we can use Option for a generic function that returns the first item:

fn first_item<T>(v: &Vec<T>) -> Option<T>
where T: Clone {
    if v.len() > 0 {
        Some(v[0].clone())
    } else {
        None
    }
}

The function first_item can only return a value if the vector being passed is not empty. This is a good candidate for Option. If the vector is empty, we return None, otherwise we return a copy of the value via Some.

The None variant forces the programmer to consider the case where the information required is not forthcoming.

Result

Result is similar to Option in that it can either return a value or it doesn't and is usually used as a return value from a function. But instead of returning a None value, it returns an error value that hopefully encapsulates the information of why it went wrong.

Its form is:

enum Result<T, E> {
    Ok(T),
    Err(E),
}

If everything goes well, a function can return an Ok variant along with the final result of the function. However, if something fails within the function it can return an Err variant along with the error value.

Let's look at an example:

use std::fs::File;
use std::io::{BufRead, BufReader, Error};

fn first_line(path: &str) -> Result<String, Error> {
    let f = File::open(path);

    match f {
        Ok(f) => {
            let mut buf = BufReader::new(f);
            let mut line = String::new();
            match buf.read_line(&mut line) {
                Ok(_) => Ok(line),
                Err(e) => Err(e),
            }
        }
        Err(e) => Err(e),
    }
}

std::fs::File::open will return a Result<std::fs::File, std::io::Error>. That is, it will either return a file handle if everything goes OK, or it will return an I/O error if it doesn't. We can match on this. If it's an error, we just return it immediately. Otherwise, we try to read the first line of that file via the std::io::BufReader type.

The read_line method returns a Result<String, std::io:Error> and once again we match on this. If it was an error, we return it immediately. Notice that the error type for both the open and read_line methods is std::io::Error. If they were different, this function wouldn't compile. We will deal with differing error types later.

However, if we were successful, we return the first line as a string via the Ok variant.

The ? operator

Rust introduced an operator ? that made handling errors less verbose. Basically, it turns code like this:

let x = function_that_may_fail();
let value = match x {
    Ok(v) => value,
    Err(e) => return Err(e);
}

into:

let value = function_that_may_fail()?;

The ? operator changes the Result<T,E> value into one of type T. However, if the result was an error, the current function exits immediately with the same 'Err' variant. It unwraps the result if everything went OK, or it causes the function to return with an error if not.

With this in mind, we can simplify the first_line demo function above:

use std::fs::File;
use std::io::{BufRead, BufReader, Error};

fn first_line(path: &str) -> Result<String, Error> {
    let f = File::open(path)?;
    let mut buf = BufReader::new(f);
    let mut line = String::new();
    buf.read_line(&mut line)?;
    Ok(line)
}

I think we can all agree this is a lot easier to read.

Errors

The error type in a Result can be any type, like, for example, a String. However, it is recommended to use a type that implements the trait std::error::Error. By using this standard trait, users can handle your errors better and even aggregate them.

Traits are interfaces that structures can implement as methods to extend them. I might write a blog article about traits in the future, but if you are not sure what they are, please read this article.

Here is the Error trait in all its glory:

trait Error: Debug + Display {
    fn source(&self) -> Option<&(dyn Error + 'static)> { None };
    fn backtrace(&self) -> Option<&Backtrace> { None };
}

The backtrace method is only defined in the nightly version of the compiler, at time of writing, and so only source is defined for the stable version. source can be implemented to return an earlier error that this current error would be chained to. But if there is no previous error, None is returned. Returning None is the default implementation of this method.

A type that implements Error must also implement Debug and Display traits.

Errors can be enums too. Below is an example of possible errors that can occur when reading from a file-based database:

use std::fmt::{Result, Formatter};
use std::fs::File;

#[derive(Debug)]
enum MyError {
    DatabaseNotFound(String),
    CannotOpenDatabase(String),
    CannotReadDatabase(String, File),
}

impl std::error::Error for MyError{}

impl std::fmt::Display for MyError {
    fn fmt(&self, f: &mut Formatter<'_>) -> Result {
        match self {
            Self::DatabaseNotFound(ref str) => write!(f, "File `{}` not found", str),
            Self::CannotOpenDatabase(ref String) => write!(f, "Cannot open database: {}", str),
            Self::CannotReadDatabase(ref String, _) => write!(f, "Cannot read database: {}", str),
        }
    }
}

First we define the enum with the possible error states and their associative data (e.g. filename of file that was not found). Notice the derive macro that implements the Debug trait for us. Unfortunately, we cannot do that for Display traits.

Secondly, we implement the Error trait for compatibility with other error systems. Since we're not chaining errors, the default implementation will do.

Finally, we implement the Display trait, which is a requirement of the Errortrait.

This is a lot to write for an error type, but fortunately there are some popular crates that allow us to write and use errors more easily.

Error Crates

As just shown, implementing an error type to be passed with a Result's Err variant can be tedious to write. Some consider the Error trait lacking in functionality too. Various crates have been written to combat the boilerplate and to increase the usefulness of the types of error values you can generate.

I will explore a few of them in this article that I have found to be the most popular.

Let us imagine we want to implement this function:

fn first_line(path: &str) -> Result<String, FirstLineError> { ... }

We will investigate how we implement FirstLineError in each of these crates. The basic foundation of the error will be this enum:

enum FirstLineError {
    CannotOpenFile { name: String },
    NoLines,
}

`failure` crate

failure provides 2 major concepts: the Fail trait and an Error type.

The Fail trait is a new custom error type specifically to hold better error information. This trait is used by libraries to define new error types.

The Error trait is a wrapper around the Fail types that can be used to compose higher-level errors. For example, a file open error can be linked to a database open error. The user would deal with the database open error, and could dig down further and obtain the original file error if they wanted.

Generally, crate writers would use Fail and crate users would interact with the Error types.

Failure also supports backtraces if the crate feature backtrace is enabled and the RUST_BACKTRACE environment variable is set to 1.

This is how we would create the FirstLineError error type using this crate:

use std::fs::File;
use std::io::{BufRead, BufReader};

use failure::Fail;

#[derive(Fail, Debug)]
enum FirstLineError {
    #[fail(display = "Cannot open file `{}`", name)]
    CannotOpenFile { name: String },
    #[fail(display = "No lines found")]
    NoLines,
}

fn first_line(path: &str) -> Result<String, FirstLineError> {
    let f = File::open(path).map_err(|_| FirstLineError::CannotOpenFile {
        name: String::from(path),
    })?;
    let mut buf = BufReader::new(f);
    let mut line = String::new();
    buf.read_line(&mut line)
        .map_err(|_| FirstLineError::NoLines)?;
    Ok(line)
}

The derive macro implements the Fail and Display traits automatically for us. It uses the fail attributes to help it construct those traits.

But we do have another problem. The File::open and BufRead::read_line methods return a result based on the std::io::Error type and not the FirstLineError type that we require. We use the Result's map_err method to convert one error type to another.

I will cover map_err and other methods for Option and Result in my next blog article, but for now I will describe this one. If the result is an error, map_err will call the closure given with the error value allowing us an opportunity to replace it with a different error value.

So, recall that File::open returns a Result<(), std::io::Error value. By calling map_err we now return a Result<(), FirstLineError> value. This is because the closure given returns a FirstLineError value and through type inference, we get the new result type. If the result is an error, that closure will provide the value to associate with the Err variant.

But the value returned from File::open is still a Result type so we use the ? operator to exit immediately if an error occurs.

Now we can do things like:

match first_line("foo.txt") {
    Ok(line) => println!("First line: {}", line),
    Err(e) => println!("Error occurred: {}", e),
}

Failure can even allow you to create errors on the fly that are compatible with failure::Error. For example,

use failure::{ensure, Error};

fn check_even(num: i32) -> Result<(), Error> {
    ensure!(num % 2 == 0, "Number is not even");
    Ok(())
}

fn main() {
    match check_even(41) {
        Ok(()) => println!("It's even!"),
        Err(e) => println!("{}", e),
    }
}

This program will output Number is not even as expected via the Display trait of the error.

There are other ways to create errors on the fly too with failure. format_err! will create a string based error:

let err = format_err!("File not found: {}", file_name);

And finally, there's a macro that combines format_err! with a return:

bail!("File not found: {}", file_name);

`snafu` crate

This is similar to failure but solves the issue where the actual error that occurred is not the error we want to report.

If you recall above, we use map_err to convert the std::io::Error into one of our FirstLineError variants. snafu makes this easier by providing a context method that allows the programmer to pass in the actual error they wish to report.

Let's redefine our error type and function using snafu:

use std::fs::File;
use std::io::{BufRead, BufReader};

use snafu::{Snafu, ResultExt};

#[derive(Snafu, Debug)]
enum FirstLineError {
    #[snafu(display("Cannot open file {} because: {}", name, source))]
    CannotOpenFile {
        name: String,
        source: std::io::Error,
    },
    #[snafu(display("No lines found because: {}", source))]
    NoLines { source: std::io::Error },
}

fn first_line(path: &str) -> Result<String, FirstLineError> {
    let f = File::open(path).context(CannotOpenFile {
        name: String::from(path),
    })?;
    let mut buf = BufReader::new(f);
    let mut line = String::new();
    buf.read_line(&mut line)
        .context(NoLines)?;
    Ok(line)
}

To use context(), there needs to be a source field in the variant. Notice that the enum type FirstLineError is not included. We wrote CannotOpenFile, not FirstLineError::CannotOpenFile. And the source field is automatically set! There's some black magic going on there!

If you don't want to use the name source for your underlying cause, you can rename it by marking the field you do want to be the source with #[snafu(source)]. Also, if there is a field called source that you don't want to be treated as snafu's source field, mark it with #[snafu(source(false))].

Similarly, snafu supports the backtrace field too to store a backtrace at point of error. #[snafu(backtrace)] et al. controls those fields like the source.

On top of this, you have the ensure! macro that functions like failure's.

`anyhow` crate

This crate provides dynamic error support via its anyhow::Result<T>. This type can receive any error. It can create an ad-hoc error from a string using anyhow!:

let err = anyhow!("File not found: {}", file_name);

It also defines bail! and ensure! like other crates. anyhow results can extend the errors using a context() method:

let err = anyhow!("File not found: {}", file_name)
    .context("Tried to load the configuration file");

Here's the first_line method implemented using anyhow:

use anyhow::Result;

#[derive(Debug)]
enum FirstLineError {
    CannotOpenFile {
        name: String,
        source: std::io::Error,
    },
    NoLines {
        source: std::io::Error,
    },
}

impl std::error::Error for FirstLineError {}

impl std::fmt::Display for FirstLineError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            FirstLineError::CannotOpenFile { name, source } => {
                write!(f, "Cannot open file `{}` because: {}", name, source)
            }
            FirstLineError::NoLines { source } => {
                write!(f, "Cannot find line in file because: {}", source)
            }
        }
    }
}

fn first_line(path: &str) -> Result<String> {
    let f = File::open(path).map_err(|e| FirstLineError::CannotOpenFile {
        name: String::from(path),
        source: e,
    })?;
    let mut buf = BufReader::new(f);
    let mut line = String::new();
    buf.read_line(&mut line)
        .map_err(|e| FirstLineError::NoLines { source: e })?;
    Ok(line)
}

anyhow doesn't define the Display trait for us so we have to do that ourselves. Also map_err has to come back if we want to convert error values from one domain to another. But, this time we use Result<String> and we don't need to define which error is returned.

`thiserror` crate

This crate makes it easier to define the error type, and can be used in conjunction with anyhow. It uses #[derive(thiserror::Error)] to generate all the Display and std::error::Error boilerplate like other crates do.

But thiserror makes it easier to chain lower-level errors using the #[from] attribute. For example:

#[derive(Error, Debug)]
enum MyError {
    #[error("Everything blew up!")]
    BlewUp,

    #[error(transparent)]
    IoError(#[from] std::io::Error)
}

This will allow auto-casting from std::io::Error to MyError::IoError.

Let's look at our demo with anyhow for results, and thiserror for errors:

use std::fs::File;
use std::io::{BufRead, BufReader};

use anyhow::Result;
use thiserror::Error;

#[derive(Debug, Error)]
enum FirstLineError {
    #[error("Cannot open file `{name}` because: {source}")]
    CannotOpenFile {
        name: String,
        source: std::io::Error,
    },
    #[error("Cannot find line in file because: {source}")]
    NoLines {
        source: std::io::Error,
    },
}

fn first_line(path: &str) -> Result<String> {
    let f = File::open(path).map_err(|e| FirstLineError::CannotOpenFile {
        name: String::from(path),
        source: e,
    })?;
    let mut buf = BufReader::new(f);
    let mut line = String::new();
    buf.read_line(&mut line)
        .map_err(|e| FirstLineError::NoLines { source: e })?;
    Ok(line)
}

Notice the neat embedded field names in the strings on the #[error(...)] lines.

Main function

A quick note on the main function. Rust Edition 2018 as added a feature that allows main to return a Result. If main returns an Err variant, it will return an error code other than 0 to the operating system (signifying a fail condition), and output the error using the Debug trait.

If you wanted the Display trait, then you have to put your main function in another, then have your new main call it and println! the result:

fn main() -> i32 {
    if let Err(e) = run() {
        println!("{}", e);
        return 1;
    }

    return 0;
}

fn run() -> Result<(), Error> { ... }

If the Debug trait is good enough for printing out your error, you can use:

fn main() -> Result<(), Error> { ...  }

How does the program know what error code to return? It uses the new Termination trait:

trait Termination {
    fn report(self) -> i32;
}

The compiler will call report() on the type you return from main.

And there's much more...

But not this week...

I wanted to talk about the methods on Option and Result, like map_err but this article is already too long. I will cover them next time.

Rust #2: Lifetimes, Owners and Borrowers, OH MY!

Matt Davies — Sat, 26 Jun 2021 13:22:22 +0000

One of the abilities that Rust promises is memory safety, and I'm here to say that it delivers. So much so, that I will not use a programming language for low-level systems programming that doesn't have the same promises now.

Microsoft discovered that 70% of their bugs were memory safety issues (reference), which means such bugs would be impossible to create if their software were written using Rust.

So what are the memory safety issues, and how does Rust make them impossible to produce?

Memory safety

Memory safety, in a nutshell, is making it impossible to access (reading or writing) memory that you shouldn't. For example:

Writing to memory that is read-only. Examples of read-only memory are your code and string literals.
Reading from or writing to memory you haven't allocated specifically from the your operating system.
Accessing the area of your stack that isn't used by local variables or parameters.
Dereferencing a null pointer.
Writing to memory at the same time as another thread of your program is accessing it.

So how do such bad memory operations occur? Other languages (such as C++) make it easy for you to execute such bad operations as they do not protect you from them. This is why much software crashes all the time. Let's go through them.

Use after free

This means that after you free your memory and give it back to the operating system for reuse, you go ahead and continue using it. This can be caused in most languages by pointer aliasing. This means that your code has more than one pointer pointing to some memory, perhaps in different parts of your code. When one pointer is used to free some memory, the other pointers now "dangle". This means, they point to nothing. And so the main reason use-after-free occurs is when these other pointers continue to be used. That can lead to a crash (or segmentation fault). Look at this C++ code to demonstrate how this can happen:

std::vector<int> v;
v.push(1);
const int& ref_v = v[0];  // a
v.push(2); // b
std::cout << "v[0] = " << ref_v; // c

In the line a, we take a reference, which is a pointer, to the first element of the C++ vector. After we push the next value in line b, it is possible that the memory used by the vector is moved because it needs to be enlarged. This will invalidate the reference stored in ref_v. It is, in effect, pointing to garbage. Then we use it in line c - a use-after-free bug.

Double free

Another problem, which is really a use-after-free bug, is when memory is given back to the operating system more than once, either through the same pointer, or via aliasing pointers. This can lead to a crash.

Using uninitialised memory

This is when memory that is either allocated on the heap or the stack is not initialised to good data, and then later the program uses it assuming it is. This means your data will have random values in it, and if your data is a pointer, it means that your code will be more likely to access invalid memory addresses. That can lead to a crash.

Buffer underflow and overflow

This happens if while reading and writing to a block of memory, you access past the end or beginning of that block. For example, you allocate an array of 10 items, but ask for the 11th. This can, if you're lucky, cause a segmentation fault and crash your program. If you're unlucky, it can corrupt the data around that memory area and you won't see the effects of that issue until much later, making it hard to pinpoint where the problem occurred. Buffer overflows and underflows are the source of many security issues too.

Null dereferencing

Tony Hoare once called this his "billion dollar mistake". Setting a pointer to address 0, otherwise called null, usually means, in most programming languages, "this pointer points to nothing". Unfortunately, in those languages, there is nothing stopping you from using the pointer still. Trying to access address 0 on most operating systems will result in an immediate crash. This is usually the easiest bug to track down and fix, but still, you don't want them occurring after you release your software.

Data races

A data race occurs when one thread is writing to memory and another thread is reading or writing to it at the same time. This can lead to any thread reading or writing the wrong information.

Imagine this function being run at exactly the same time on two threads:

int gCounter = 0;

void increment_counter()
{
    int counter = gCounter; // a
    counter = counter + 1; // b
    gCounter = counter; // c
}

You'd expect the answer to be 2 after both functions have completed. However, if thread 1 executes lines a and b after thread 2 has executed line a and before it has executed line c, both threads will write 1 to gCounter. This is called a data race. Although this is often not a crash bug, it can be amazingly hard to track down. This is because unless those or similar conditions happen, you will not have a data race. So, in more complex situations, your application may run correctly 99% of the time, resulting in you desperately trying to recreate that 1% occurrence to track down the bug.

In software development, there is nothing worse than trying to fix a bug you cannot recreate.

Rust to the Rescue

All those bugs I've described above are impossible to occur if you write your program in Rust and avoid its unsafe keyword (used to lower Rust's shields when you need to). They are impossible to occur because any such attempt to produce those bugs results in a compilation error. This means that you can't even start a program with those bugs. This is a huge win for programmers.

So how does it do this? It is achieved using its memory model of ownership and borrowing. The 'borrow checker' is the analytical part of the compiler that will track and detect these memory issues. And when you first use Rust, you will butt heads with the borrow checker and hate it. But eventually, as you realise the borrow checker is always right, you will learn to love it and it will be your best friend in writing software.

Ownership

Every data used in Rust is owned by a single variable and ONLY one variable. When this variable goes out of scope, the data is deleted. Rust, therefore, always knows when to deallocate memory: as soon as the owner goes out of scope. This is why you can allocate memory and not care about deallocating it. You cannot do this in other languages because there might be two references to the data. Deallocating when one reference goes out of scope will cause a dangling pointer so those languages do not automatically deallocate.

When you assign a variable to another, or pass a variable to a function, you transfer ownership because you cannot break the one-owner rule. Examine this code:

let a = vec![1, 2, 3];
let b = a;  // Ownership of vector is transferred to b.
let c = a;  // Compiler error!  a does not own any data.

In line 1, the a variable owns the vector [1, 2, 3]. In line 2, by assigning a to b, you transfer ownership. The variable b now owns the vector, and as a result a owns nothing. It is a compilation error to use a afterwards. So as a result, in line 3 you would get a compilation error.

The same thing occurs when passing variables to functions:

fn print_vector(v: Vec<i32>) {
    println!("{}", v);
}
...
let a = vec![1, 2, 3]
print_vector(a);
println!("{}", a);   // Compilation error!!!

By using a as a parameter to print_vector, you transfer ownership to print_vector's local variable v. When v goes out of scope as the function ends, the compiler knows to destroy the vector. Using a afterwards is a compilation error.

If we just had these ownership rules, programming in Rust would be very hard to do, because as soon as you call a function, you've lost the use of your variables. This is where borrowing comes in.

Borrowing

When you own a book and you want to let someone else read it, but still want to own the book afterwards, you lend it. That is, the other person receiving your book 'borrows' it with the understanding that they will give it back. This is essentially Rust's borrowing model.

But there's slightly more to it. When you lend out some data, you can choose to do it mutably or immutably. That is, the borrower may have permission to change the data, or not. This is important. Rust allows infinite immutable borrows as long as there is not a single mutable borrow. Likewise, Rust allows only a single mutable borrow, if and only if, there are no immutable borrows. This simple rule makes data races impossible. Put a different way, if your function has a mutable borrow, you know for a fact that it is the ONLY reference to the data and you're free to change it.

A borrow in Rust is symbolised by a & symbol before the type. You are essentially creating a pointer. Looking at an example:

let a = vec![1, 2, 3];
let b = &a;
let c = &a;

This time line 3 is allowed and this code will compile. Rust allows as many immutable references (they are immutable by default) as you would like. Also bear in mind that the owner can still mutate the vector as in this example:

let mut a = vec![1, 2, 3];
let b = &a;
let c = &a;
a.push(4);

But didn't we break the rule that we can't mutate data if we have immutable borrows? No, because Rust is smart enough to see, in this example, that b and c are not used any more and a is free to mutate its data.

If we had this, however:

let mut a = vec![1, 2, 3];
let b = &a;
let c = &a;
a.push(4);
println!("{:?}", b);

then we would have a compilation error. By using b in the println! statement, we are using an immutable borrow at the same time as we are mutating the data. Remember, as in the C++ example above, pushing more data can move the memory used by the vector and invalidate b and c causing a use-after-free bug.

To mutably borrow we use the mut keyword after the &. Let us look at the increment_counter function above rewritten in Rust:

fn increment_counter(counter: &mut i32) {
    *counter = *counter + 1;
}

Note, you can't have global mutable variables in Rust unless you use the unsafe keyword to change them, so we pass it as a parameter. We're only talking about safe Rust in this article for now.

Also note, for mutable borrows you must use the * operator to dereference them.

Slices

Slices are views on an area of memory. They are sometimes called fat pointers. They consist of a pointer and a length. Unlike C++, where a pointer can pointer to a single or multiple values, in Rust a better distinction is made. A borrow, mutable or otherwise, always references a single data value. To reference multiple data values, slices are used.

Slices cannot own data because, by definition, they are views on already existing data. So they are borrows. You can always identify a slice by the type &[T] where T is any data type. Without the ampersand, the type [T] is just a fixed array and needs to be owned.

Indexing a slice at runtime incurs a bounds check. Trying to access a slice using an index that is equal or larger than its size will cause a panic and your program will exit if the panic is not trapped. This stops the buffer overflow bug. Also, buffer underflows cannot occur because slice indices are always unsigned. That is, you cannot have an index less than 0.

As in other borrows, mutable slices are written as &mut [T].

Also, in Rust there is a type str, which is an alias for [char], so whenever you see &str, think of it as a slice of a character array. String literals will have the type &str as they are sliced views on static data in your executable. So in this code:

let a = "hello";

a is of type &str.

Lifetimes

If you store a borrowed reference to some data and its owner goes out of scope, will you not have a dangling pointer? In short, no, and that is where lifetimes come in.

Every variable in Rust has a lifetime and Rust ensures that ALL borrowed references have the same lifetime or shorter than the owner that lent the reference. So in the example above, Rust would not allow you to assign a borrowed reference to another variable unless Rust could prove that the variable had a equal or shorter lifetime than the owner.

This means, that when the owner goes out of scope, Rust knows for a fact there are no other references and it is safe to destroy the data.

Lifetime syntax is written as 'name where name can be anything. Usually, these lifetime annotations can be implied. For example, when you write:

fn substr(s: &str, index: usize, len: usize) -> &str;

the compiler is really seeing:

fn substr<'a>(s: &'a str, index: usize, len: usize) -> &'a str;

I have chosen a as the name of the lifetime. And as you can see, the lifetime of the result must match the lifetime of the input parameter s. This makes sense as the function will return a reference to the data passed in as s. Notice, that the lifetime annotation is declared within <...> next to the function's name.

The implied lifetime annotations are called lifetime elision. Rust has various rules for eliding lifetimes. If your function is more complicated, the compiler might complain and you will have to add the annotations yourself. If you're interested you can find more information here.

There is another lifetime that data can have and that is 'static. This means that the data lasts as long as the program itself. Functions have lifetimes of 'static for example. Also literal strings are immutable character slices (i.e. &str) with a lifetime of 'static.

Another place where lifetime annotation occurs is if you have references (immutably borrowed or otherwise) inside structures. For example:

struct Person {
    name: &str,
    age: u32,
}

will give a compilation error:

error[E0106]: missing lifetime specifier
 --> test.rs:2:11
  |
2 |     name: &str,
  |           ^ expected named lifetime parameter
  |
help: consider introducing a named lifetime parameter
  |
1 | struct Person<'a> {
2 |     name: &'a str,
  |

and the Rust compiler will give you a nice hint on how to fix it. This ensures that an instance of the Person structure cannot outlive the data that name references. If it did, we'd have a dangling pointer.

Summary of Syntax

Below, for convenience, I summarise the syntax used for the different types of data:

Type	Syntax
Owner	T
Immutably borrowed	&T
Mutably borrowed	&mut T
Immutable borrowed slice	&[T]
Mutably borrowed slice	&mut [T]

Bugs begone!

I'd like to go over the different bug classes again and discuss how Rust makes them impossible.

Use After Free

Ownership and lifetime rules ensure that when the owner finally needs to be destroyed, there are no other references to it. This is because references caused by borrowing cannot outlive the owners.

Double Delete

Deletion can only occur when the owner goes out of scope. Because Rust only allows a single owner of data, there cannot be another owner going out of scope later. You cannot delete data via a borrowed reference either.

Uninitialised Data

I didn't talk about this before, but Rust insists on all new variables to be initialised. You cannot do this, for example:

let mut a: i32;
println!("{}", a);
a = 42;

a is undefined when println! is executed and so the compiler stops this from happening. However, it is OK to initialise a on a different statement as long as you don't use a before you do so:

// This is OK
let a: i32;
a = 42;

Note that a doesn't even have to be defined mut because Rust sees that the a = 42 statement is in fact an initialisation and not a mutation.

Buffer Overflow and Underflow

Buffer overflows are impossible because Rust, at compile time and runtime, bounds-checks array accesses. Any violation will result in a panic.

Buffer underflows are impossible because you cannot use signed indices.

Null Dereferencing

Safe Rust doesn't allow you to dereference pointers. Also, borrowed references cannot be null either. Therefore, it is impossible to dereference a null pointer. If you do wish to have a type that could point to some data or not, Rust provides the Option enum, which I will be visiting in my next blog.

Data Races

Rust does not allow more than one mutable reference (via borrowing or ownership), and it doesn't allow any mutable references if a single immutable reference exists. This means that given a mutable reference, Rust guarantees that this is the only way you can mutate it at that time. Therefore, data races cannot occur.

A Word about the Copy Trait

When I said that passing a variable to a function or assigning it to another transfers ownership, I was telling you half the story. For some built-in types, by default, are marked by the Copy trait. I will talk about traits another time, but essentially it means that the type is marked with a property. That property allows the compiler to copy the data byte-for-byte into the new variable or parameter. This means that ownership is not transferred but a new owner is created with a copy of the data.

This is why in the examples above I used vectors to demonstrate transfer of ownership as Vec does not implement the Copy trait. Types that do implement it are so-called POD types (Pieces Of Data) such as i32, u32, f32, char etc. Because of this, the following code is correct:

let a = 42; // a is an i32, which implements the Copy trait
let b = a;
let c = a;

Both b and c become owners of the copy of a's data. So, both b and c will hold the value 42.

You can also implement the Copy trait easily for a structure as long as all its fields also implement the Copy trait:

#[derive(Copy, Clone)]
struct Person {
    name: String,
    age: u32,
}

Also note that the Clone trait must also be implemented if Copy is. The Clone trait provides the clone() method on the type and is required to do the actual copy. After adding the #[derive(Copy, Clone)] statement, this code is possible:

let p = Person { name: String::from("Matt"), age: 48 };
let a = p;
let b = p;

Both b and c will hold different instances of Person but contain the same values.

Conclusion

I hope this helps with groking what borrowing an ownership means and how it a) avoids memory bugs; and b) allows the compiler to deallocate data at the right time so Rust does not need a garbage collector.

Please let me know in the discussion if there are details I have wrong or left out. I am always interested in improving this text.

Next blog will talk about Option and Result enum types.

Rust #1: Creating your development environment

Matt Davies — Sat, 19 Jun 2021 13:32:32 +0000

Overview

This blog is my attempt at documenting what I've learnt on my journey using Rust. I have worked as a video game programmer for over 25 years and as a result have used C++ as my main language of choice, even for personal projects. Today, I will never touch C++ willingly again since discovering and learning about Rust. I will restrict C++'s use to my day job, and that's all. In my opinion, Rust has made C++ redundant for writing new software.

I have managed to hone my development environment for Rust programming over the last few months and I want to share my approach. This is what my first article is about - setting up a development environment for effective Rust programming.

Prerequisites

I will assume the reader knows their way around a terminal (or shell). That is, they know how to make folders, set the current folder, and launch commands all via their terminal.

I will also assume that the reader has prior programming experience, perhaps in C, C++ or similar language. These guides will not be teaching you how to program, but rather how to apply your current programming skills to a new language.

You will also need an internet connection. The Rust build system will constantly download packages (called crates) from the world wide web.

Installing Rust

The best way to install Rust is using the tool Rustup. Windows' users will require downloading an executable, but for operating systems with more sane command shells, a single cut & paste will suffice.

Hop over to http://rustup.rs and follow the instructions. After running the executable rustup-init.exe on Windows, or executing the shell command on other operating systems, you will see a screen-full of text and 3 options:

Go ahead and choose option 1 and you will have Rust and its associated tools installed. To make sure, open up your favourite terminal and type:

$ rustc --version

if everything went well (and I hope it does because this guide will not help you otherwise), you should see something like:

rustc 1.53.0 (53cb7b09b 2021-06-17)

The installation procedure involves rustup downloading and installing "components". The components you might have seen being installed would be:

cargo - the ultimate, bad-ass package/build/do-everything tool that will be the centre of your Rust-programming experience.
clippy - this is a linting tool that can scan your code and make suggestions on how to improve it, or catch problems. This is an essential part of your workflow.
rust-docs - a cargo plug-in that provides auto-generation of documentation from your source code. If you're familiar with Doxygen, it's similar to that.
rustc - the compiler, nuff said!
rustfmt - Rust's very opinionated code formatter. It will auto-format your code according to a set of standards so that everyone's Rust code looks the same. This can be annoying at first but you reap the benefits if you stick with it.

Your first Rust program

Let's try the obligatory Hello World program. And because of Cargo, you can do this without typing a line of code! Most guides, will demonstrate how to use rustc but you do not need to touch this directly. You will use Cargo to check, build, test, benchmark and run your code as well as many other things. So jump into your favourite terminal, cd to your folder you've set aside for Rust development (I use Windows and so my folder of choice will be r:\) and type:

$ cargo new hello
$ cd hello

You will get a message back telling you that you've created a binary package called 'hello'.

Let's look inside your folder that you've just created. First, you will see a Cargo.toml file. This gives information to Cargo on how to build your project and other meta-data (like authorship, version etc). We'll cover this later.

Secondly, there is a hidden file and directory related to git, which is the default source control system used by most Rust programmers. You can switch it to Mercurial or something else. To read about that, rush over to https://doc.rust-lang.org/cargo/commands/cargo-new.html and checkout the --vcs command line option. I will be ignoring this file and folder in this guide.

Finally, there is a src folder, and that is where all your rust source code lies. If you look inside, you will see a lonely file called main.rs. Let's view it. You can use type (Windows) or cat (other operating systems) to show the contents of src/main.rs:

fn main() {
    println!("Hello, world!");
}

The hello world program has already been written, courtesy of Cargo. Let's examine the code:

fn marks the start of a function and in this case defines the function called main. All Rust applications start at this function, like C/C++ does. If you build with Cargo, which we will, there also has to be a main.rs file in the root of the src folder. The parentheses following it usually contain parameters (we have none right now), and the code belonging to that function is in between the braces. If you are used to C, C++, C#, JavaScript etc, all this will be familiar to you.
println! will print text to standard output. The exclamation mark in the name indicates that println! is not a function but a macro. As in Lisp macros, not the useless C style macros. Rust can generate code during compilation. In this case, println! will expand to many statements, depending on its arguments, that will be inserted at the point of the call.

OK, let's run it. Type:

$ cargo run
   Compiling hello v0.1.0 (R:\hello)
    Finished dev [unoptimized + debuginfo] target(s) in 1.11s
     Running `target\debug\hello.exe`
Hello, world!

This will compile the code and run it. The astute of you will have noticed a new folder appearing called target. Dig deeper and inside that you will see a debug folder. Finally, inside that you will see the hello.exe you just built and that Cargo just ran.

In fact, the compiler output before the 'Hello, world!' message describes the type of application you just built (unoptimised and including debug information) and where to find it.

We can build and run an optimised one too:

$ cargo run --release
   Compiling hello v0.1.0 (R:\hello)
    Finished release [optimized] target(s) in 0.66s
     Running `target\release\hello.exe`
Hello, world!

Here are some other Cargo commands worth knowing to begin with:

cargo build - this will just build the final executable. cargo run will call this implicitly if it needs to.
cargo clean - this will delete the target folder and all the generated files within.
cargo check - useful while writing code. This will compile your code but without generating the code. This is a lot faster than cargo build and is useful if you just want to check for syntax errors. Because of the excellent IDE integration that I will demonstrate later, this is rarely needed.

One more point, after building your executables, a Cargo.lock file appeared too. This file is generated to lock down versions of other packages you might have used to build your program. This allows the Rust compiler to recreate the exact executable after a clean. It ensures that the same versions of packages are downloaded, built and linked with your application as before. For now, we don't use packages so this file contains very little information. If you use source control (like Git), you should include this file during submission, along with Cargo.toml and your source code.

The Editor

The second step in starting to learn Rust is setting up your programming environment just right so as to minimise friction between you and your work. The main part of that decision is which editor to use. I've tried writing Rust code in Emacs, Vim, Sublime but the text editor I eventually settled on is Visual Studio Code. For me, it's the best product Microsoft have ever created.

Why Visual Studio Code? In my opinion, it has the most mature tooling for writing Rust code, and for interacting with Rust's ecosystem and build tools (i.e. Cargo).

So go over to https://code.visualstudio.com/ and download and install it.

Restart your terminal, and enter:

$ code .

And lo and behold:

The terminal is nice and all but there's all that typing. Don't we do enough while coding? So let's see if we can build from Visual Studio Code (hereafter referred to as VS Code). Currently, it can't because it needs extensions.

VS Code Extensions

To turn VS Code into a powerhouse editor of Rust, we need to install some extensions. By clicking on the 5th icon on the left (it looks like 4 squares, with one of them offset) you open up the extensions interface. Use the search box to find the necessary extensions listed below, then click on the blue 'install' buttons beside the extension's entry:

rust-analyzer - provides the 'intellisense' for Rust and many other Rust-related utilities.
C/C++ - what??? You need a C/C++ extension? Yes, it contains the debugger and it's useful for inserting breakpoints and stepping through your code. You can also use CodeLLDB but for me, the C/C++ one feels quicker.

Also, there are some useful extensions to install, that while unnecessary, are very useful all the same:

crates - checks the versions of any crates you reference (more on that later).
Markdown All in One - you will be writing several markdown files (files with .md extensions). This extension makes it easier to do so and provides a useful preview mode.
Even Better TOML - some editor support for editing TOML files such as Cargo.TOML.
Rewrap - not related to Rust but provides the ability to reformat text (in markdown files and Rust comments) so that they fit on the lines better. By pressing Alt+Q, you can change a couple of long comments into several shorter ones that fit on the screen. As you will eventually see, Rust comments will take a big part in document generation.

If you clear the search box, you will see the extensions you've installed. Some buttons beside them may be labelled 'Reload Required', and if you do, press it to restart VS Code in a matter of seconds.

After installing rust-analyzer, you will see a dialog box pop up in the bottom right corner of the window. If you miss it, click on the notifications icon in the bottom-right of the window. It should look like this:

Click the 'Download now' button. If you don't see it, restart VS Code and it should show again. This will download the Rust Language Server (RLS), which is the external program that can analyse your source code and communicate back to VS Code, via the extension, to display useful information about your program, including auto-complete.

When downloaded, 'rust-analyzer' will get to work looking at your code. Open up src/main.rs using VS Code and you will see two new links above the main function:

Click on Run and VS Code will build and run your program. You can see the results in the terminal window that automatically opens up.

Alternatively, you can click on Debug to do the same thing. But right now VS Code isn't properly set up for debugging.

Debugging in VS Code

If you haven't used VS Code before, it probably isn't set up for debugging facilities like breakpoints. So press Ctrl+Comma to open up VS Code's settings window. Type breakpoints in the search bar at the top of this window, and make sure Debug: Allow Breakpoints Everywhere is selected:

While you're in the settings window, retype the search parameter to be formatter. Ensure Editor: Default Formatter is set to rust-analyzer and Editor: Format On Save is checked. This will allow the rustfmt component mentioned earlier to do its magic every time you save a file. It's a really nice time-saver. You can type code as messily as you want in the sweet knowledge that when you save, everything will move to its correct place, and missing trailing commas and spacing will be inserted.

Now you've done this, you can hit F9 while editing to add a breakpoint to the very line you're on. Try doing this now on the println! statement in src/main.rs:

Click on 'Debug' and see the code stop on the line before it executes it. Tap F10 to step over the next line and you will see 'Hello, world!' appear as it executes it. Press F5 to continue running, or Shift+F5 to stop debugging altogether. Don't forget to tap F9 again on the line with the breakpoint to remove it!

Launchers and Tasks in VS Code

These links in your code are very convenient, but it requires that you have src/main.rs open at the main function to see them. Wouldn't it be better if we could press a key and have it run or build?

First step is coercing VS Code to build your source code at a press of a key. Press F1 to bring up the command palette, which is an interface in VS Code that contains all the possible commands you can give to it. Now type build and the command menu will whittle down to a few entries, one of which is Tasks: Configure Default Build Task.

Select it. A new menu will appear with a list of possible Cargo commands. We want to choose rust: cargo build. This will create a new file in your project at .vscode/tasks.json. This file defines a task that VS Code can run. It calls cargo build and that's all. But let's improve on it.

Firstly, change the label entry to something shorter. It can be anything you want, but I usually change it to build. It will be referenced by another file later.

Secondly, we would like the terminal window in VS Code to clear every time we start a build so we can clear distinguish current messages from a previous build's messages. Add a comma at the end of the label line and then start typing presentation. After a few characters you should see an intellisense window open up suggesting presentation. Hit ENTER and VS Code will auto-complete the entry. Very handy! Now change the clear entry so it says true instead of false. The final version should look like this:

{
  "version": "2.0.0",
  "tasks": [
    {
      "type": "cargo",
      "command": "build",
      "problemMatcher": [
        "$rustc"
      ],
      "group": {
        "kind": "build",
        "isDefault": true
      },
      "label": "build",
      "presentation": {
        "echo": true,
        "reveal": "always",
        "focus": false,
        "panel": "shared",
        "showReuseMessage": true,
        "clear": true
      }
    }
  ]
}

Now, select File > Preferences > Keyboard Shortcuts from the menu and type build in the search bar of the new keyboard shortcuts window.

Double-click under the Keybinding column on the row that says Tasks: Run Build Task. This will allow you to bind a key to that command. Type F7 followed by ENTER and you're done. VS Code will warn you that other bindings use the same key, but ignore those as they will not conflict. I chose F7 because that's the usual key on the Visual Studio IDE for building and I am used to it. Of course, you can choose any other key combination that suits you.

Let's test it! Hit F7 now (or whichever key combination you chose) and the terminal window in VS Code will clear and start your build!

Now have it building at a press of a key, we want it to run (after building) at a press of another key. VS Code already has F5 set up for that task, but currently doesn't know how.

Click on the 4th icon on the left side of VS Code, the one that looks like a triangle with a bug on the corner. When you do this, you will see a new interface with a big Run and Debug button (but don't press it!).

Under that button, you should see a link that says create a launch.json file. To run your program at a press of a button, we need to create a launcher. Launchers are configured via this launch.json file, so go ahead and click on that link. You will see a dialog asking you to select and environment.

Choose C++ (Windows) if you're on Windows and you used the C/C++ extension, or choose C++ (GDB/LLDB) otherwise. This will create a new file called .vscode/launch.json. As before, let's make a few changes:

Change the program value so it reads ${workspaceFolder}/target/debug/hello.exe. This needs to match the final executable you're running.
Change the console value so it reads integratedTerminal and add a comma at the end.
After the console entry, and the reason you added a comma, add the entry "preLaunchTask": "build". This is the line that references the label text you entered in tasks.json. They must match. This line tells the launcher to ensure that the build task is run first. We want to run the latest version of our program, don't we?

The final result will be:

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "(Windows) Launch",
      "type": "cppvsdbg",
      "request": "launch",
      "program": "${workspaceFolder}/target/debug/hello.exe",
      "args": [],
      "stopAtEntry": false,
      "cwd": "${fileDirname}",
      "environment": [],
      "preLaunchTask": "build"
    }
  ]
}

Press F5 and the launcher should run. You should see in your terminal, build messages and the output of the program.

Occasionally, I have seen VS Code struggle with this. If it doesn't run, check your tasks.json and launch.json files, and run cargo clean from the terminal within your project folder.

Compiler Errors

This blog is getting a bit long now, so one last thing to talk about. What happens if you have errors in your code. The good news is that the rust-analyzer extension and VS Code will conspire to help you out. Let's add an error in our code. After the println! statement, add a new one:

fn main() {
    println!("Hello, world!");
    foo();
}

Then hit save (Ctrl+S). rust-analyzer will generally only see your changes after saving. Now the error is in the program, without building multiple things will happen:

A red line will be visible under foo. Move your mouse pointer over it and you will see a popup describing the error.
Looking at the tab above your editing window, you will see that the filename has turned red with a number after it. (There is also a letter U but that's related to source control so ignore it). That number is how many errors there are. The tab text can turn yellow too, which means there are warnings but no errors. This time the number counts how many warnings there are. At a glance you can see how many errors in your program.
On the scrollbar in your window, you will see a little red marker. This shows roughly where your error is.
If your terminal window is still open in VS Code, you will see a number next to the Problems tab. Inside this tab, you will see a list of errors. Click on them to jump to the line of code with the error.

Wow, that's a lot of clues something has gone wrong with your program. Now hit your build key to run the build task (remember mine was F7). In the Terminal tab below you will see the output of the compiler. It usually has a better error message than seen elsewhere:

Rust provides the best error messages EVER! You can even go to your terminal and type rustc --explain E0765, as described, to explain the error above in more detail. Truly amazing!

Summary

That was a lot to go over. But we installed Rust and it's tools, installed VS Code and extended it to support Rust, and then showed how to use VS Code to build and run your program.

Part #2 will talk about the borrow checker, ownership, lifetimes and all those nice things. See you then!

DEV Community: Matt Davies

Rust #8: Strings

ASCII

Unicode

UTF-8

Ownership and slices

String

OsString

PathBuf

CString

Conversions

From owned types to slices

From slices to owned types

Conversions between string types

Holy Cow!

Summary

Rust #7: Command-Line interfaces

clap crate

Structopt crate

Paw crate

Conclusion

Rust #6: Exploring crates

ansi_term

datetime

bitflags

byteorder

git2

glob

lazy_static

libc

locale

log

Trace

Debug

Info

Warn

Error

matches

natord

num_cpus

number_prefix

pad

scoped_threadpool

term_grid

term_size

tinyvec

unicode-width

url

users

zoneinfo_compiled

Conclusion

Rust #5: Naming conventions

Casing

Basic types

Get and Set methods

Suffixes

Prefixes

Conclusion

Rust #4: Options and Results (Part 2)

Good and Bad

Extracting Good Values and Result Errors

Iteration and Transforming Values

Querying Value Types

Logical combinations

and_then versus map

Converting between Option and Results

Some(T) -> Ok(T), None -> Err(E)

Ok(T) -> Some(T), Err(E) -> None

Transposition and Flattening

Miscellaneous Operations

Summary

Rust #3: Options, Results and Errors (Part 1)

Options and Results

Option

Result

The ? operator

Errors

Error Crates

failure crate

snafu crate

`String`

`OsString`

`PathBuf`

`CString`

`clap` crate

`Structopt` crate

`Paw` crate

`ansi_term`

`datetime`

`bitflags`

`byteorder`

`git2`

`glob`

`lazy_static`

`libc`

`locale`

`log`

`matches`

`natord`

`num_cpus`

`number_prefix`

`pad`

`scoped_threadpool`

`term_grid`

`term_size`

`tinyvec`

`unicode-width`

`url`

`users`

`zoneinfo_compiled`

`and_then` versus `map`

`failure` crate

`snafu` crate

`anyhow` crate

`thiserror` crate