DEV Community: Steven Trotter

AWS CDK - A Game Changer in IaC

Steven Trotter — Fri, 18 Dec 2020 11:06:15 +0000

Introduction

Towards the end of 2019, I said I would look into the Infrastructure as Code setup as my current workplace and try and improve it. Specifically I've promised that I'll get our DevOps capability to a point where I can build our system in full and tear it down once I'm finished, that it'll be easy for developers to use and change, that it'll be reliable and version controlled... basically a whole lot of standard promises that you seem to make as DevOps/Site Reliability Engineers. I looked into a variety of technologies to achieve this and I've more or less settled now on AWS CDK (at least for the AWS stuff). Having been using it for a bit I decided it's well worthy of a quick article, I haven't been able to find an example of an article from someone whose really tried to use it in anger so hopefully this is of interest to some.

In a nutshell: CDK is great. I really think it's a gamechanger in the DevOps space. In this article I'll explain why I think this, what the motivation was for using it and what sets CDK out from the other options. It's quickly become my favourite IaC tool on the market having previously always used things like Terraform, bespoke bash scripts, Azure ARM templates, that kind of thing.

I'm intrigued to see as well where CDK for Terraform goes, however this article does not touch on this at all and is purely about AWS CDK which is almost certainly the more mature of the two at time of writing.

What's not so great about CDK

So let's get the downsides out of the way first. CDK is quirky, sometimes behaves unexpectedly, has some work to do on documentation yet (though it's pretty good), feels a bit immature still, even buggy in places and I wouldn't blame folks for waiting around a bit for it to "stablise" and "become a bit more production ready". As with any new tool, eventually common problems will either dissipate out as they find better solutions, or StackOverflow will become flooded with helpful tips/tricks, or more people will write helpful articles about it.

The biggest drawback really for CDK is that it's AWS specific and uses CloudFormation under the hood. I mean that's design really, but it is kind of the most disappointing thing about it. I think you can write modules in it but there's little to no documentation at present and even having read through some of the code I wouldn't know where to begin (I can't find many examples of people who have done this yet). Compared to something like Terraform that has a multitude of providers, CDK doesn't (and I'm guessing won't any time soon) come anywhere near that level of support. Even for AWS there are things I've found missing in places that Terraform can fairly well do like posting swagger to an AWS API Gateway. I have managed to find workarounds (I'll be posting one about using Terraform with CDK and custom resources in Lambdas soon hopefully) but they're not ideal.

CDK is clearly bound by CF (CloudFormation) under the hood which at times is just a bit ropey. It also means you have to get to grips with both CDK and CF. Sometimes CDK can improve on this which is great but they're still using CF under the hood and you can tell. Examples I've come across are:

I wanted to create two indexes on a DynamoDB table: you can't do this in parallel. If you try to create two indexes in CF it will try to run them in parallel and fail to do so. There is a workaround I found here where you create them in a Lambda synchronously (one after the other) and then create a CDK CustomResource that polls that Lambda. Sadly it would be much simpler if CF could just know it needs to do this synchronously (in particular sequentially). In my opinion this is a CF weakness that CDK could get over this way potentially, but it seems they shouldn't really need to.
CloudFormation is not great at deleting stuff and therefore neither is CDK. Sometimes this is great when you would otherwise accidentally delete stuff but sometimes this can be very frustrating and time consuming. When I'm developing I want to be able to destroy a stack and then recreate it; however with CF I need to go through and destroy the DynamoDB tables, S3 buckets, Route 53 hosted zones, etc etc, otherwise it will fail when I rerun it as these are already there. This is almost certainly a CF problem but it still effects CDK.

Also recently we moved to using nested stacks and now we can no longer get diffs: they just tell you your asset hash changed but not what changed within them. Worse still it auto deploys everything without prompting me when I run a deploy, I have no idea what it will deploy without trying it any more. This one was a real doozy as we ended up rolling back some EC2's that were being used by the dev team by just putting a syntax error in something, they were not happy, had it been prod this would have been a disaster (though yes you're right, we shouldn't be trying stuff out on stuff that's being used, you're quite right, that would be our bad and we're not any more).

On a similar note: I've seen it so many times where if I put a syntax error in my code it will just deploy whatever it has found so far but also it will destroy everything it hasn't found (if I auto-approve or use nested stacks anyway). Surely the default would be if there's a syntax error just do nothing. Maybe this one is JS specific as compilers would catch it in Java say but surely if I put a syntax error in I should fix that before CDK does anything. However if I put a syntax error in right at the top it will literally teardown everything.

Why do I love CDK?

So with all these drawbacks, and some of them are big ones, why do I think it's the best thing since sliced bread? Well, it's got a great helpful community who are happy to answer questions, it's frequently coming out with updates that make it tangibly and noticeably better, it's fast and it's fun to work with.

Fundamentally though, it all comes down to one specific reason, code is better suited to this task than configuration or DSL's. It really is true Infrastructure as Code, this is what makes it such a game changer for me. It's not complex configs that are unreadable (I've seen beforehand JSON/YAML files of the order of thousands of lines that just seems like a nightmare to maintain). It's possible to put logic into it without having to learn a whole extra DSL (yes Terraform, I'm looking at you). I get to choose from TypeScript, Python, Java or any .NET language (more are likely to be added in time I understand). JS/TS is the one you'll get most examples in and help from the community with, plus some tools exist there where they don't in the others (like a testing assertion library which is really cool). However aside from this, from what I can tell they're all about as good as each other and I can define my whole AWS infrastructure in an AWS account using that language of choice (NB: I've only tried Python though, and did change to JS in the end because of the previous listed reasons).

Now I can hear a lot of DevOps people shouting, but Infrastructure as Code should be configs, if it's not fairly straight forward configs it's just too complex and that should be a huge code smell. I hear you, but I disagree for the following reasons:

Firstly code can be much more readable than configs. Configs I have to use JSON or YAML or whatever and have little control over the structure, I'm constantly having to refer to documentation to work out what something does, in JSON I can't even put comments in explaining why I'm doing something (yes I shouldn't have to in an ideal world, we all know it isn't). With code, I can write it in such a way that it's as clear as I want. Badly written code won't have this of course but that's on the head of the guy who wrote it. I'd much rather see something like the following:

        service_abc_lambda = aws_lambda.Function(
            self,
            "LambdaServiceABC",
            runtime=aws_lambda.Runtime.PYTHON_3_6,
            handler=lambda_function.handler,
            code=aws_lambda.S3Code.from_bucket(code_bucket_name, lambda_code),
            environment=environment,
            memory_size=128,
            role=lambda_role,
            timeout=aws_core.Duration.seconds(lambda_config.get("timeout", 30)),
        )

        bucket_name = "service-abc-awesome-bucket"

        bucket = aws_s3.Bucket(
            self,
            "BucketForServiceABC",
            bucket_name=bucket_name,
            encryption=aws_s3.BucketEncryption.S3_MANAGED,
            block_public_access=aws_s3.BlockPublicAccess.BLOCK_ALL,
        )

        principals = [aws_iam.ArnPrincipal(service_abc_lambda.role.role_arn)]

        bucket.add_to_resource_policy(
            aws_iam.PolicyStatement(
                resources=[bucket.bucket_arn, bucket.bucket_arn + "/*"],
                actions=[
                    "s3:GetObject*",
                    "s3:ListBucket",
                    "s3:PutObject*",
                    "s3:DeleteObject",
                ],
                principals=principals,
            )
        )

        service_abc_lambda.add_environment(
            "MY_BUCKET", bucket_name
        )

I've changed the names to protect the innocent and simplified it, but fundamentally this is pretty close to code I've written and something like this may be used in our end product. To me this is highly readable, I'm creating a Lambda for something called Service ABC, it'll be a Python 3.6 function, 128 meg, I've cheated a bit as the environment and lambda_role were defined above that I've cut out but you can see clearly it's creating a Lambda function. Then I create a bucket and attach permissions to that bucket that allow the Lambda to perform various read/write operations on that bucket and finally I add an environment variable to the Lambda so the Lambda knows the bucket name to interact with.

I could do most of this in Terraform too and it might be readable there as well with modules and the like, but once I get beyond this into ifs and loops, I can just write Python or JS in a much nicer, more readable and more reusable way.

It saves me (and the rest of my team) having to learn a new DSL and I can leverage existing skills (hopefully, or at least encourage people to learn really useful new skills, that's still great).
When I do want to do something complicated, I can do it. I might end up with just enough rope to hang myself but then that's on me if I write something horrific and unmaintainable, just as the case with application development. It should not be on the tool to decide how I write my IaC, it should be on me and my team to make sure that what we have is good, correct, reliable, etc.
If at some point we decide to change what we're using, assuming whatever we're changing to can be written in JS then I don't have to restart everything. Yeah I mean I'll have to change a lot potentially but I won't have to start again. Or I may decide to start again and base my new tool off the old one, in the places where it was good enough copy and paste can save me time (alright this isn't always a good thing).

Now I know that this isn't the case for very much at the moment, the only other thing I've seen that can do this at present is Pulumi which I discounted for my use case because of the price. I predict however a lot of things will go this way in time for the reasons I've listed here. Alright it's more complicated for the writers of the tool to allow people to write Python or TypeScript or Java or whatever but AWS have proved it's possible here.

If (as was my case) I'm taking over an existing codebase, assuming that codebase is one of the language options then I can leverage that existing codebase. If I decide to go down a Terraform route I've got to start all over again and for the existing complexities, I need to rethink all of them (that might not be a bad thing but if it's complex enough it'll be a huge pain and time sink).
If I really want configs I can write my code to read directly from configs and then just update configs in the future, I could even have configs that live on another server or code repository somewhere if I really wanted. It really does give me the best of both worlds here and I've certainly done this in my use of it. I'm not against configs, far from it, I just think code is always going to be better when it comes to something complex.

Experience with CDK

I won't go into too much details about how the project I've been running has gone because that's not the point of the article. However I will say that after a year we're still using it, more people have gotten involved in the project and they like it too and we're still actively developing solutions with this. It's quite a bold move (given the disadvantages section, some of them are big) to use CDK instead of Terraform, but I think it's one that will pay off in time still.

Would I use CDK again in the future? Well depends on the requirements of the project. If they're even remotely thinking about multi-cloud then no sadly. If they're fixed along the AWS route then I'd certainly think about it. For projects that are AWS based I think it will eventually be the goto tool of choice, AWS put so much effort into this and I really believe the disadvantages section will shrink over time. Don't be surprised if in a years time this article is just not relevant any longer (or if I've time and inclination gets a big overhaul).

Conclusion

Fundamentally, when it comes down to it I'm from a development background and I like writing code, I admit this. I know that there are other folks who come from an ops background who are more used to thinking of infrastructure in terms of UNIX configs and the like, I get that. My personal opinion is that people trying to find their way in DevOps who are weak in either development or operations are going to get left behind by those who can do both. Part of the DevOps movement I would hope is that we're moving to a world where people can do both and this should be encouraged. I had to learn my way around infrastructure in my transition to a Platform engineer/DevOps engineer/SRE (or whatever you want to call it, I do not wish to get into that debate), I'd strongly encourage those with an ops background to learn Python or something like that, you'll thank yourself for it later I promise (maybe I should write a Learning Python for those with an Operations Background to help, hmm).

I really truly hope other tools go this way, particularly open source tools (if I've missed any please ping them in the comments, I'd love to hear about them). It's a bigger challenge for those writing the tools but I think the benefit is worth it. AWS really seem to have shown us a potentially better way of life here, fingers crossed anyway.

The Not So Hard Bits In Practice

Steven Trotter — Mon, 09 Dec 2019 12:30:12 +0000

Writing Basic Programs

Introduction

Since the previous entry a lot has happened both for me and for Rust itself so I'll give a brief overview. It took me a while to write Article 2 in this series and I'm sorry about that, my initial thought was one article every month and maybe 3 months at worst case. I'm nowhere near that; I'll try and do better in future.

In Rust a lot has changed too. The big one for me is, we've got async in Rust now. If you don't know what this means that's fine, but if you're familiar with green threading and things like Go and Node you'll know this is potentially a big deal. We're early days, I'm currently playing around a lot with Tokio 0.2 and it's a big step forward in my view. It's early days, and a lot of other crates still need updating to Tokio 0.2 and we are still on the road to Tokio 1.0. But it really feels like the Rust guys took a while to make sure it's correct and it's an exciting time to be involved in Rust.

So basically my fourth reason for not using Rust in article 1 is now disappearing, not so sure the others are but I'll take that. I truly think Rust is going have a great year in 2020 and I can't wait to see where we are in a years time.

Anyway, to this article. In the previous entry in this series I talked a lot of theory. I mean if you're still here congratulations on getting through all that. I felt satisfied with my effort given that there is a lot of theory that does need to be covered and I think the hardest bits are done. However I didn't want it to become the case that you'd have all the theory and none of the practice. So in this second article in the series I'll basically be showing how to use Rust practically and how to use what we learned in the previous article. I'll introduce a lot of stuff along the way as well so that you can start using Rust in practice. After this hopefully you'll understand how to write a fairly basic Rust program using the cargo build tool and with help from the documentation you'll be able to do quite a lot. We'll cover strings, results, matching, enums, options, vectors and hashmaps, all of which you'll need to be comfortable with to be able to write good programs in Rust.

Together we're going to build a very basic calculator command line tool in this article. It'll be far from complete but it'll teach you a lot about Rust. I want to build a program that will take input on a command line through stdin and on pressing enter will perform the calculation and spit out the result. I'll go through a few iterations to get to where I want in order to help introduce as much as I can along the way. I'll make mistakes on purpose (honest) and show you what to do to correct them.

I'll assume you've got a command line system (I'm using a terminal on a Mac, Linux would work just as well, I'm sure Windows has a similar thing) with the rustc and cargo programs fully installed. See here for how to install Rust, rustup is the right way to do this.

I warn that this is a long article but overall it's almost certainly easier than the previous article. I'd recommend taking the time to try thing out yourself and I'd recommend reading the book alongside this, particularly if something is confusing.

Calculator - Iteration 1

So let's get started. Firstly I want a program that will do the following:

Take an input string that consists of two integers separated by an operator each of which is separated by spaces
When I press enter it will interpret this string and perform the sum specified and output that to the screen
Any division will be rounded down to the nearest integer to print out an integer at the end
All numbers will be i64's (signed 64 bit integers)

We may add more requirements as we go but let's keep things simple to begin with.

Creating a basic program with `cargo`

The first thing we're going to do is create a new program called calculator. To begin with run the following command in a terminal to create a calculator directory with the necessary bare bones:

> cargo new calculator
     Created binary (application) `calculator` package

We can then change to this directory and build this as follows:

> cd calculator
> cargo build
   Compiling calculator v0.1.0 (<snip>/rust/calculator)
    Finished dev [unoptimized + debuginfo] target(s) in 1.93s
>

We can then do cargo run to run the program and this will just print "Hello, World!" and exit as it's a skeleton program. We always use cargo run when we want to run the program going forward and it will recompile as necessary.

Let's take a look under the hood of what we have. In this directory we have the following:

Cargo.lock
Cargo.toml
src/
  main.rs
target/
  ...

So we have two files, Cargo.lock and Cargo.toml. I won't discuss much the Cargo.lock file for now, the Cargo.toml file is the starting point. In a Rust program we have a variety of third party libraries we can use (as any decent language really) called Crates. If you look on crates.io you'll be able to search for these Crates there. For example if we search for the tokio crate we get directed here and we can see links for the documentation and code repository for Tokio. We'll be adding a crate to our program shortly.

Additionally we have a src/ directory with one file in it, main.rs and we have a target/ directory. The latter for the most part we can ignore as that's where programs live, we can run them with cargo run or we can do ./target/debug/calculator. In src/ is where all our code will live and we'll just work with main.rs in this article. I recommend reading the book for how to split your work to be more modular but there's nothing excessively hard there so I won't cover it here.

Adding Crates

It's generally not advised to reinvent the wheel in programming unless you're doing something purely as an exercise (which has merits) or unless you have a really compelling reason to; that is don't rewrite something that has already been written. In this case we want something to read lines that input to a program. Surely we can do this with either a) an existing crate or b) something in the standard library. Well we certainly could do the latter and just use std::io::stdin, see here for details on how to do that. With this though we just get something very basic, let's have something cooler.

A quick search for readline in crates.io shows me this page that looks interesting. According to that page Rustyline is a Readline implementation in Rust, sounds about what we want. When writing this that package was on version 5.0.4, it's possible (even likely) that by the time you're reading this that's progressed on. The following should still work anyway as we pin it to 5.0.4, but as an exercise feel free to take the latest version, it'll probably just work and you've got the latest and greatest but if it doesn't then that's a good exercise to get it to work in itself (though in that case don't spend ages on something you can't get working of course).

I also recommend reading up about Cargo in general and specifically about specifying packages. We can always take the latest by specifying * instead of 5.0.4 for example and a variety of other things. We might want to pin to a major version say. For now we'll just pin to 5.0.4 to make life easier.

So how do we use it. Well it's quite easy actually, let's add the following to the end of cargo.toml:

[dependencies]
rustyline = "5.0.4"

[dev-dependencies]
spectral = "0.6.0"

The [dependencies] line is already present of course so just append to this. I've sneaked in a dev dependency called spectral that will allow me better assertion writing below when we start writing tests. Now let's run cargo build again. I get the following (yours may vary):

    Updating crates.io index
  Downloaded unicode-width v0.1.7
   Compiling libc v0.2.66
   Compiling cfg-if v0.1.10
<snip>
   Compiling rustyline v5.0.4
   Compiling calculator v0.1.0 (/Users/strotter/work/programming/rust/calculator)

Now if I compile again it's much quicker as it has already compiled all these crates for us but it won't recompile them now. I'm mostly just going to copy and paste the Rustyline example into my src/main.rs and run it (ignoring the history stuff, I don't want that). So I end up with a src/main.rs as follows:

use rustyline::error::ReadlineError;
use rustyline::Editor;

fn main() {
    let mut rl = Editor::<()>::new();
    loop {
        let readline = rl.readline(">> ");
        match readline {
            Ok(line) => {
                println!("Line: {}", line);
            },
            Err(ReadlineError::Interrupted) => {
                println!("CTRL-C");
                break
            },
            Err(ReadlineError::Eof) => {
                println!("CTRL-D");
                break
            },
            Err(err) => {
                println!("Error: {:?}", err);
                break
            }
        }
    }
}

We'll cover the match command later more thoroughly but basically on successfully taking input we run the first println! macro, if we send Ctrl-C or Ctrl-D we end up in the next two blocks and a generic error we end up in the last one.

Now if I run it I get the following:

> cargo run
   Compiling calculator v0.1.0 (/Users/strotter/work/programming/rust/calculator)
    Finished dev [unoptimized + debuginfo] target(s) in 1.52s
     Running `target/debug/calculator`
>> testing
Line: testing
>> 1 + 2 = 3
Line: 1 + 2 = 3
>>
CTRL-D
>

How great is that, I can use left and right arrow keys, I can press ctrl-c or ctrl-d to quit and it prints to say that's what I've done. I've basically done hardly anything and got a program that accepts input and does it well.

But of course this does nothing yet really. I need to make this do something. So we've learned a little about Cargo and getting going, Cargo is awesome and you definitely should use it in Rust development. I'm not going to search for any more crates here however as the point going forward is to teach you Rust properly, but I'd recommend doing so in your day to day usage, make your life as easy as possible.

Interpreting the line

Now we want to write another function basically that will perform a calculation given an input string. I personally love writing unit tests for this kind of thing, I hope you do too. I'll demonstrate how to do this kind of thing with unit tests in Rust now.

So I'm believe I'm going to have to write a function that will take as input a string, and will return an integer (you could try floating point numbers as an exercise, we're just using integers for simplicity as the point is teaching, not to have a fully fledged calculator). Let's add this function to the top of main.rs:

fn calculate(calculation_string: &str) -> i64 {
    0
}

We pause now for a quick discussion about strings as we'll need to know some basics before proceeding.

Aside: Strings in Rust

So let's briefly talk about strings in Rust first. The sad thing is it's a bit complicated (though I think less than C++ say but more so than Python or JavaScript certainly). There are essentially two string types we care about in Rust, String and &str. You'll notice the latter is actually a reference and you'd be right, but you basically never use the str type directly, you can't even do that for reasons I'll not go into here. Roughly I'll use String if I want to be able to build and change the string around and &str if I just want to pass a static string around, though you'll learn and understand more about this as you go through this article (I hope).

So let's demo creating an &str now. I can create one as follows:

fn main() {
    let example_string = "STRING";
    println!("{}", example_string);
}

Easy enough it seems, but how do I know this is an &str, doesn't look much like a reference? Well, I'll show you a neat trick now that I learned from The Rust Crash Course and I use all the time now for how to debug this kind of thing. Basically I change the above program as follows:

fn main() {
    let example_string = "STRING";
    let a: bool = example_string;
    println!("{}", example_string);
}

So the third line here is now saying I'm creating a variable a of type bool from whatever example_string is. Well Rust won't be able to just assign pretty much anything to a bool without being explicitly told so, so this will throw an error when compiled (unless it is a bool, but then you'd know that I guess). So let's try and compile this now, I get:

error[E0308]: mismatched types
 --> src/main.rs:3:19
  |
3 |     let a: bool = example_string;
  |            ----   ^^^^^^^^^^^^^^ expected `bool`, found `&str`
  |            |
  |            expected due to this

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: could not compile `calculator`.

Oh it told me at the end of the 5th line, I have an &str as promised.

Let's remove this and try and do something with this (spoiler alert, I can't do much in terms of writing but reading is mostly OK). I can try the following:

fn main() {
    let example_string = "STRING";
    if example_string == "STRING" {
        println!("{}", example_string);
    }
}

This works fine, I've checked if our example_string has value "STRING" and as it does the if condition is evaluated to true. Now let's try and append to it though:

fn main() {
    let example_string = "STRING";
    let example_string = example_string + "TEST";
    if example_string == "STRINGTEST" {
        println!("{}", example_string);
    }
}

Well that didn't work, when I ran it I got:

   Compiling calculator v0.1.0 (/Users/strotter/work/programming/rust/calculator)
error[E0369]: binary operation `+` cannot be applied to type `&str`
 --> src/main.rs:3:41
  |
3 |     let example_string = example_string + "TEST";
  |                          -------------- ^ ------ &str
  |                          |              |
  |                          |              `+` cannot be used to concatenate two `&str` strings
  |                          &str
  |
help: `to_owned()` can be used to create an owned `String` from a string reference. String concatenation appends the string on the right to the string on the left and may require reallocation. This requires ownership of the string on the left
  |
3 |     let example_string = example_string.to_owned() + "TEST";
  |                          ^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

Yikes, what is that? Actually it's a pretty good error, it tells me exactly how to fix it which is basically turn my &str here into a String and then I can do this. I can even then add an &str to a String and it's all fine, if I do what it says it works.

On the other hand maybe I have a String and I'm trying to pass to a function that takes an &str, what then? Example:

fn print_data(to_print: &str) {
    println!("Output: {}", to_print);
}

fn main() {
    let example_string = "STRING".to_owned();
    print_data(example_string);
}

Doesn't like that either:

error[E0308]: mismatched types
 --> src/main.rs:7:16
  |
7 |     print_data(example_string);
  |                ^^^^^^^^^^^^^^
  |                |
  |                expected `&str`, found struct `std::string::String`
  |                help: consider borrowing here: `&example_string`

error: aborting due to previous error

Clearly I need to convert from a String here to an &str. How do I do that? Easy here actually, I can just prepend an & and I'm good:

fn print_data(to_print: &str) {
    println!("Output: {}", to_print);
}

fn main() {
    let example_string = "STRING".to_owned();
    print_data(&example_string[..]);
}

See, it's all easy enough right. Well actually this isn't the full story and it does get a bit more complicated. You've two choices now, you can either get the full picture from the article about slices in the Rust book, or you can basically just stick to the following algorithm till you need to know more and it will pretty much serve you well:

If something wants a String and it's complaining it is a &str add a .to_owned() to it, e.g. "123".to_owned() converts a &str 123 to a String 123. Alternatively you can use the format! macro (don't worry about what a macro is for now, just think of it as a command) if you want to do something more complicated, e.g. I could do format!("Converting to String: {}", s) which converts an &str variable s to a String with "Converting to String: " prepended to it.
If something wants a &str and it's complaining it is a String then prepend an & to it and append [..] to it, e.g. the following converts a String s with value "123" to a &str with value "123":

let s = "123".to_owned();
print_data(&s[..]);

So why would I ever use &str is probably the next question. Well one answer is sometimes the standard library and Crates you use will insist you do. Other than that you can really just pass around String everywhere in your app if you want but it won't be the most efficient. String's can allocate more memory than needed and when passing by value you might need to think about cloning and thus using even more memory. If you want the gory details I'd say read the top response to this stackoverflow question but essentially the answer is, use &str wherever possible and where it isn't use String. We will get a better feeling for the answer to this question after reading this article.

Back to the Calculator

Anyway, back to our calculator. We had the following function:

fn calculate(calculation_string: &str) -> i64 {
    0
}

Obviously this doesn't do much for now and I'm not even going to plug it into my main function until my tests are all working. So TDD says I should write a test that fails and make it pass, let's do that now. Advantages of this are essentially I can run this really quickly and tests give me really fast feedback, they also serve to document to future developers exactly what my thought process was when writing this kind of thing. I don't want to get too side tracked on TDD so let's just jump in now and write the following test, add this to the bottom of main.rs:

#[cfg(test)]
mod tests {
    use spectral::prelude::*;

    #[test]
    fn test_addition() {
        let res = super::calculate("1 + 2");
        assert_that(&res).is_equal_to(&3);
    }
}

There's a lot going on here but basically it's all to allow cargo and the compiler to know to run this as a test. So anything marked with #[test] is ran as a test when we do cargo test. The use statement brings the spectral crate in so we have better assertions and error reporting. OK, let's run cargo test now:

> cargo test
  Downloaded spectral v0.6.0
<snip>
   Compiling spectral v0.6.0
   Compiling calculator v0.1.0 (/Users/strotter/work/programming/rust/calculator)
warning: unused variable: `calculation_string`
 --> src/main.rs:4:14
  |
4 | fn calculate(calculation_string: &str) -> i64 {
  |              ^^^^^^^^^^^^^^^^^^ help: consider prefixing with an underscore: `_calculation_string`
  |
  = note: `#[warn(unused_variables)]` on by default

    Finished test [unoptimized + debuginfo] target(s) in 10.72s
     Running target/debug/deps/calculator-3aae0d9719182d41

running 1 test
test tests::test_addition ... FAILED

failures:

---- tests::test_addition stdout ----
thread 'tests::test_addition' panicked at '
        expected: <3>
         but was: <0>
', /Users/strotter/.cargo/registry/src/github.com-1ecc6299db9ec823/spectral-0.6.0/src/lib.rs:343:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.


failures:
    tests::test_addition

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

Oh no, big lots of badness and errors. The first warning I can ignore as that will be fixed soon. As for the test error, well wouldn't I expect this though? It returns 0 which is not equal to 3. Well OK, let's fix it so it does. Obviously I could just change it to return 3, but who am I kidding? Myself? Let's try and parse the string properly now and make sense of it, another test coming on.

My first thought is I want another function tokenize that takes this &str and returns a Vector of strings. Wait, a what? What's a Vector? Time for another aside.

Aside: Vectors

Well you may have seen lists or arrays in other languages, it's kinda similar. A vector in Rust is a collection of elements that all have the same type and it can be added to along with various other operations. It's part of the standard library. We can create a vector in a few ways, but the easiest I find is the following:

let vector: Vec<i64> = vec!(1, 2, 3);

This creates a vector with 3 elements that are all i64's. I could of course do the same thing with 10 or 100 or no elements at all. I can also add to this by making it mutable and then pushing elements onto it as follows:

let mut vector: Vec<i64> = vec!(1, 2, 3);
vector.push(4);

It should be noted that a vector owns all its elements and once a vector gets dropped then all of its elements also get dropped. However if it owns references the references are dropped but not the things they're referencing.

I have the following example of a program (the {:?} in the println! means print with debugging info):

fn main() {
    let mut vector: Vec<i64> = vec!(1, 2, 3);
    vector.push(4);
    println!("{:?}", vector[0])
}

which I can run to get:

I can also loop over vectors with for loops as follows:

fn main() {
    let mut vector: Vec<i64> = vec!(1, 2, 3);
    vector.push(4);
    for i in &vector {
        println!("{:?}", &vector.get(*i as usize))
    }
}

There's not loads that truly tricky about vectors themselves, and you're welcome to read the Vector page of the book. It is, however, the case that you have to be careful with ownership (an anything in Rust, remember lesson 1?). If you have a vector that owns its entries directly (rather than references to) then when accessing them you need to either transfer ownership or take references. In transferring ownership have a few options, popping entries off or take entries out of vectors and leave something else in its place. For example we can do the following:

let mut vec: Vec<String> = vec!["TEST1".to_owned(), "TEST2".to_owned(), "TEST3".to_owned()];
let item = std::mem::replace(&mut vec[0], "TEST1_REPLACEMENT".to_owned());
println!("Vector: {:?}", vec);
println!("Item: {}", item);

We've used a standard library function std::mem::replace that will literally swap and replace safely an existing entry out. This can be extremely useful when trying to take ownership of something. It's also worth considering using an Option which we discuss below.

Back to the Calculator

Returning to the problem at hand, should my vector have String's or &str's as its type? Well we're supposed to have &str to begin with right so let's try that first, we're not going to have to do loads of changing the string it seems, just interpreting, so this seems like a good first shout.

So we have another function:

fn tokenize(string_to_parse: &str) -> Vec<&str> {
    let vec = vec!();
    vec
}

Note here that because the last line in this function doesn't have a semi-colon that means it's a return. I could just write return vec; instead but this is a more Rusty way of doing things so I do as my Rust overlords tell me.

This compiles and we're all good. Let's write another test now, put this inside the tests module again.

    #[test]
    fn test_parsing_string() {
        let res = super::tokenize("1 + 2");
        let expected = vec!("1", "+", "2");
        assert_that(&res).is_equal_to(expected);
    }

Now two tests failing, way to go me. Let's fix at least this one.

In the tokenize I want to split the string handed to me by whitespaces. If I look at the Rust standard library documentation forStrings I see there is a method split_ascii_whitespace that gives me an iterator. Iterators I haven't covered and won't be doing an aside for here but essentially I can iterate over iterators (duh). In particular I can do the following:

for token in string_to_parse.split_ascii_whitespace() {
    println!("Token: {}", token);
}

and then run with cargo test -- --nocapture to see the logged lines and I get:

Token: 1
Token: +
Token: 2

Cool, that's pretty much what I need. So I can now finish this function off:

fn tokenize(string_to_parse: &str) -> Vec<&str> {
    let mut vec = vec![];

    for token in string_to_parse.split_ascii_whitespace() {
        vec.push(token);
    }

    vec
}

I had to change the vector to be mutable to get it compiling but it compiles now. Now we run the tests and one test passes now, woohoo:

test tests::test_parsing_string ... ok
test tests::test_addition ... FAILED

failures:

failures:
    tests::test_addition

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filter

In fact I could have also written the function in a more compact and functional style as follows, but this involves knowledge of closures and iterators so I've not done this for now, but if you know these things I could also have done:

fn tokenize(string_to_parse: &str) -> Vec<&str> {
    string_to_parse.split_ascii_whitespace().collect()
}

Performing Calculations

Back to the tests anyway. Obviously the other one doesn't pass, how would it, it's not implemented yet. I can start implementing it now though with the tokenize function.

So we can start implementing by using the calculate function, we can do the following to begin with:

fn calculate(calculation_string: &str) -> i64 {
    let vec = tokenize(calculation_string);
    0
}

Now we want to turn the first element into an integer. I recommend at this point trying to parse the standard library documentation first and attacking these problems, start by looking at the String and str types to see if we have anything.

So did you find the parse function/turbofish operator? Well (this)[https://doc.rust-lang.org/std/string/struct.String.html#method.parse] is what I'd use for this (there may be other ways). So I can solve the problem using this, it works on both a &str and a String so let's try it on a reference:

fn calculate(calculation: &str) -> i64 {
    let vec = tokenize(calculation);
    let num1 = vec[0].parse::<i64>();
    println!("{}", num1);
    0
}

Note here that I can just take vec[0] and parse that because we have a vector of references and we don't need to worry about ownership. Had we have chosen to have a vector of String's we'd have had to do (&vec[0]).parse::<i64>() to prevent trying to take ownership. Alternatives of swapping out with std::mem::replace, see here for more details.

Here's an interesting error we get now though:

error[E0277]: `std::result::Result<i64, std::num::ParseIntError>` doesn't implement `std::fmt::Display`
 --> src/main.rs:7:20
  |
7 |     println!("{}", num1);
  |                    ^^^^ `std::result::Result<i64, std::num::ParseIntError>` cannot be formatted with the default formatter
  |
  = help: the trait `std::fmt::Display` is not implemented for `std::result::Result<i64, std::num::ParseIntError>`
  = note: in format strings you may be able to use `{:?}` (or {:#?} for pretty-print) instead
  = note: required because of the requirements on the impl of `std::fmt::Display` for `&std::result::Result<i64, std::num::ParseIntError>`
  = note: required by `std::fmt::Display::fmt`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0277`.

It's telling us num1 can't be displayed because it doesn't implement the Display trait. I was expecting an i64 but it doesn't look like it is an i64. What is it? It tells me, it's a std::result::Result<i64, std::num::ParseIntError>, whatever that is. It recommends I fix it by putting a {:?} inside the String, which would make the program compile but wouldn't give me what I want, I want num1 to be an i64 but the parse function/turbofish operator can't guarantee I didn't write "bananas" here, so it gives me a Result back instead. Let's do a quick aside on this one.

Aside: Results, Enums and Matching

So what is a Result? A Result is basically an enum that can be set to either something correct or an error. Great, what's an enum? Enums in Rust are probably a bit different to enums you're used to in other languages and it's well worth reading up about them here, but basically an enum is a type that has several options and an enum value will assume one (and only one) of those values. What really makes enums pop for me at least is a combination of the match keyword and that I can attach extra information to one entry. We'll discuss in the context of a Result as an example now or you can also read more about this in the book here.

In a Result we have two types, an error type in case something went wrong and an OK type for when we succeeded and I'll only get one or the other. A Result is defined in the standard library as follows:

pub enum Result<T, E> {
    Ok(T),
    Err(E),
}

I took out comments and things but this is it with T and E here being placeholders for a particular type. E will generally be for handling errors but T will be whatever you want essentially. In our case T is clearly i32 and E is whatever is returned by parse (doesn't matter for now). So if everything was alright with number 32 say I'd return Ok(32) which would be a Result<i32>. Or if something went wrong I might return Err(err) where I've built an appropriate variable err of type E.

But how can I get useful information out in our case. Well as I say match, let's try and pull the error out (as we'll clearly have an error here, I mean bananas integer, what was I thinking). We try the following:

let not_number = "bananas";
let number = not_number.parse::<i32>();
match number {
    Err(error) => {
        println!("Error: {}", error);
    }
};

This is saying that if number which is either Ok with type i32 or an Err with some error associated. This is saying if it's an Err then match that, set the variable error to whatever the error was and we can reference that within that block of code to handle this.

But if we try this we get:

error[E0004]: non-exhaustive patterns: `Ok(_)` not covered
 --> src/main.rs:4:11
  |
4 |     match number {
  |           ^^^^^^ pattern `Ok(_)` not covered
  |
  = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms

Well that's pretty helpful so let's add a leg for Ok too:

let not_number = "bananas";
let number = not_number.parse::<i32>();
match number {
    Ok(num) => {
        println!("Num: {}", num);
    },
    Err(error) => {
        println!("Error: {}", error);
    },
};

Now we can run it and we get as expected:

     Running `target/debug/calculator`
Error: invalid digit found in string

Let's pause now and look at this again just to make sure it's clear. We've ran a command match on the number variable which we know has two different value possibilities, its either worked and I get Ok(num) where num is an i32 or its not worked and I get Err(err) where err is some kind of error. The "arms" of the match command in lines 4 and 9 above match on those patterns and then run the code in between the braces dependent on which one matched. I can also use the variables num or err (called whatever I want) and they'll have the values they're supposed to have.

If I wanted to find out the error type I could play my trick above of doing this:

let not_number = "bananas";
let number = not_number.parse::<i32>();
match number {
    Ok(num) => {
        println!("Num: {}", num);
    },
    Err(error) => {
        let a: bool = error;
        println!("Error: {}", error);
    },
};

to find that our E has type std::num::ParseIntError:

error[E0308]: mismatched types
 --> src/main.rs:9:27
  |
9 |             let a: bool = error;
  |                    ----   ^^^^^ expected `bool`, found struct `std::num::ParseIntError`
  |                    |
  |                    expected due to this

error: aborting due to previous error

This really is just the tip of the iceberg where match is concerned, it's extremely powerful. It may just look a bit like a switch but I don't know of many languages that have this ability to specify values within the enums like this (I gather Haskell has this kind of thing, I don't know it personally but I understand a lot of the inspiration for Rust comes from functional languages like this, I'm not trying to report the history but I'll happily report that I really love this part of Rust).

Also here we've handled the error but we carry on afterwards. Sometimes this isn't what we want, sometimes we want to make sure that if we do get somewhere we just bail out straight away. Maybe it's dangerous to carry on in which case we can use things like the panic! macro:

let not_number = "bananas";
let number = not_number.parse::<i32>();
match number {
    Ok(num) => {
        println!("Num: {}", num);
    },
    Err(error) => {
        panic!("Bailing: {}", error);
    },
};

and we get:

thread 'main' panicked at 'Bailing: invalid digit found in string', src/main.rs:9:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

If I find I'm in this situation and I run it with the RUST_BACKTRACE variable exported it will, as it promises, give me a stack trace which can be really useful.

> RUST_BACKTRACE=1 cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.45s
     Running `target/debug/calculator`
thread 'main' panicked at 'Bailing: invalid digit found in string', src/main.rs:9:13
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/strotter/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /Users/strotter/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
<snip>
  21: std::rt::lang_start
             at /Users/strotter/code/third_party/rust/rust/src/libstd/rt.rs:67
  22: calculator::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

On mine number 13 shows me where I did this, yours may vary with Rust versions.

So let's fix it so I don't get an error:

let number_to_parse = "12";
let number = number_to_parse.parse::<i32>();
match number {
    Ok(num) => {
        println!("Num: {}", num);
    },
    Err(error) => {
        println!("Error: {}", error);
    },
};

and we get predictably:

Num: 12

We don't always need to go to the full effort of using match when we just expect it to work though. The match command is great when you want a lot of power but there are a few other ways of just going "this should work but if not panic" as above. The big one I use all the time is expect, it's a bit of a strange name for it at first but it's basically like saying "I expect this to work, if not throw the error I give you". You can also use unwrap and the ? operator but I tend to prefer expect because if it happens I then know exactly where and why. By using expect I always expect it will work and if it doesn't that implies there's a bug. If it should be handled then I should think about using match.

Example:

let number_to_parse = "bananas";
let number = number_to_parse.parse::<i32>().expect("Couldn't find integer");

and now I get:

thread 'main' panicked at 'Couldn't find integer: ParseIntError { kind: InvalidDigit }', src/libcore/result.rs:1189:5

Also I can use match on things other than enums, I can use it on strings, ints, whatever really. But it's going to be difficult to exhaustively list out all strings or ints or whatever. For this reason we have the _ matcher, this will match on anything else and should always come last otherwise anything after will be ignored. Example:

let string = "TESTING";
match string {
    "TESTING" => {
        println!("As expected");
    },
    _ => unreachable!()
}

Note unreachable! is a useful macro too for when you're really confident you can't get somewhere and if you do it's a bug. It'll just hard panic if you get to one of these. Here we never will clearly.

So we need the _ otherwise we'll get an error as match insists that you are exhaustive (cover everything):

error[E0004]: non-exhaustive patterns: `&_` not covered
 --> src/main.rs:3:11
  |
3 |     match string {
  |           ^^^^^^ pattern `&_` not covered
  |
  = help: ensure that all possible cases are being handled, possibly by adding wildcards or more match arms

error: aborting due to previous error

The match command being exhaustive is a great reminder to handle your errors and the like.

One other thing with the match command though that we need to make sure that each arm evaluates to the same type, or will panic! or whatever. For example the following will fail to compile:

let num = 32;
match num {
    1 => "TEST",
    32 => 33,
    _ => unreachable!(),
};

to give us:

error[E0308]: match arms have incompatible types
 --> src/main.rs:5:15
  |
3 | /     match num {
4 | |         1 => "TEST",
  | |              ------ this is found to be of type `&str`
5 | |         32 => 33,
  | |               ^^ expected `&str`, found integer
6 | |         _ => unreachable!(),
7 | |     }
  | |_____- `match` arms have incompatible types

But if we take out the 1 arm it's fine, note that we don't return an integer from the other pattern but that's fine because it's unreachable! and this means we just hard panic anyway. We need to return the same type from each arm because it's possible to set a variable to this value, for example, this will print ret: 33 out:

let num = 32;
let ret = match num {
    32 => 33,
    _ => unreachable!(),
};
println!("ret: {}", ret);

Back to the Calculator (again).

Ok, so we'll use expect against recommendation in the aside now as it's easier, to do this properly we should use match as we don't know for sure the string is as i64. Let's now try this and run the tests again:

fn calculate(calculation: &str) -> i64 {
    let vec = tokenize(calculation);
    let num1 = vec[0].parse::<i64>().expect("num1 should be a number");
    num1
}

Getting closer, it's complaining it has a 1 now instead of a 0. We'll do the same for num2 and in fact I could get this test passing by doing the following:

fn calculate(calculation: &str) -> i64 {
    let vec = tokenize(calculation);
    let num1 = vec[0].parse::<i64>().expect("num1 should be a number");
    let num2 = vec[2].parse::<i64>().expect("num2 should be a number");
    num1 + num2
}

But this is wrong as what if I'd put a - instead of a +, it'll add it anyway. I could add a subtraction test next and fix this, this kind of thing will be left as an exercise for the reader however.

To do this properly we use the match operator from the aside. So for our first example here we can give the following a go to match on the operator (remembering that _ matches anything):

fn calculate(calculation: &str) -> i64 {
    let vec = tokenize(calculation);
    let num1 = vec[0].parse::<i64>().expect("num1 should be a number");
    let num2 = vec[2].parse::<i64>().expect("num2 should be a number");
    let op = vec[1];
    match op {
        _ => {
            panic!("Bad operator");
        }
    }
}

This is not finished of course because it matches everything and panics. Let's try and finish this off properly now:

fn calculate(calculation: &str) -> i64 {
    let vec = tokenize(calculation);
    let num1 = vec[0].parse::<i64>().expect("num1 should be a number");
    let num2 = vec[2].parse::<i64>().expect("num2 should be a number");
    let op = vec[1];
    match op {
        "+" => num1 + num2,
        _ => {
            panic!("Bad operator");
        }
    }
}

The cargo test passes, woohoo. Now I glossed over a minor detail here, did you notice? You may have done (and if so well done) that there's no return here. Well we didn't put a semi-colon on the match statement so whatever that returned is what we returned, in this case it's what we want for addition.

Now we need to finish this and implement the same for subtraction, multiplication and division. I'll leave this as an exercise for the reader.

Finishing off

OK so we now (if you've done the exercises) covered off writing the tests, the function can interpret adding, subtracing, etc within a string (if you write it correctly). But I don't use the function yet. In fact I get warnings saying "Are you crazy? You meant to not use this stuff you've worked so hard on?":

warning: function is never used: `calculate`
warning: function is never used: `tokenize`
 --> src/main.rs:4:4
  |
4 | fn tokenize(string_to_parse: &str) -> Vec<&str> {
  |    ^^^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default

warning: function is never used: `calculate`
  --> src/main.rs:14:4
   |
14 | fn calculate(calculation: &str) -> i64 {
   |    ^^^^^^^^^

Well gosh, Rust is right it seems. Let's use it, it's no good yet.

Joking aside though when I wrote this, I knew before plugging it into the main function it was very likely to work because I'd tested it. Let's see this now, change main to:

fn main() {
    let mut rl = Editor::<()>::new();
    loop {
        let readline = rl.readline(">> ");
        match readline {
            Ok(line) => {
                println!("Result: {}", calculate(&line));
            }
            Err(ReadlineError::Interrupted) => {
                println!("CTRL-C");
                break;
            }
            Err(ReadlineError::Eof) => {
                println!("CTRL-D");
                break;
            }
            Err(err) => {
                println!("Error: {:?}", err);
                break;
            }
        }
    }
}

Now if I run it I get what I want:

>> 1 + 32
Result: 33
>> 2 + 33
Result: 35
>> 5 * 21
Result: 105
>> 2 - 12
Result: -10
>> 10 * 214
Result: 2140
>> 214 / 24
Result: 8
>>
CTRL-C

For completeness, here's the code I ended up with, yours may vary.

use rustyline::error::ReadlineError;
use rustyline::Editor;

fn tokenize(string_to_parse: &str) -> Vec<&str> {
    let mut vec = vec![];

    for token in string_to_parse.split_ascii_whitespace() {
        vec.push(token);
    }

    vec
}

fn calculate(calculation: &str) -> i64 {
    let vec = tokenize(calculation);
    let num1 = vec[0].parse::<i64>().expect("num1 should be a number");
    let num2 = vec[2].parse::<i64>().expect("num2 should be a number");
    let op = vec[1];
    match op {
        "+" => num1 + num2,
        "-" => num1 - num2,
        "*" => num1 * num2,
        "/" => num1 / num2,
        _ => {
            panic!("Bad operator");
        }
    }
}

fn main() {
    let mut rl = Editor::<()>::new();
    loop {
        let readline = rl.readline(">> ");
        match readline {
            Ok(line) => {
                println!("Result: {}", calculate(&line));
            }
            Err(ReadlineError::Interrupted) => {
                println!("CTRL-C");
                break;
            }
            Err(ReadlineError::Eof) => {
                println!("CTRL-D");
                break;
            }
            Err(err) => {
                println!("Error: {:?}", err);
                break;
            }
        }
    }
}

#[cfg(test)]
mod tests {
    use spectral::prelude::*;

    #[test]
    fn test_addition() {
        let res = super::calculate("1 + 2");
        assert_that(&res).is_equal_to(&3);
    }

    #[test]
    fn test_subtraction() {
        let res = super::calculate("2 - 1");
        assert_that(&res).is_equal_to(&1);
    }

    #[test]
    fn test_multiplication() {
        let res = super::calculate("2 * 3");
        assert_that(&res).is_equal_to(&6);
    }

    #[test]
    fn test_division() {
        let res = super::calculate("47 / 3");
        assert_that(&res).is_equal_to(&15);
    }

    #[test]
    fn test_parsing_string() {
        let res = super::tokenize("1 + 2");
        let expected = vec!["1", "+", "2"];
        assert_that(&res).is_equal_to(expected);
    }
}

Cool huh? Well, not really I guess, we're just on iteration 1 though, we can do better. For example what happens if I write a letter instead of a number, or what if I write a bad operator, or what if I write nothing or multiple sums:

>> 1 + 2 + 3
Result: 3

>> abc * 1
thread 'main' panicked at 'num1 should be a number: ParseIntError { kind: InvalidDigit }', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

>>
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libcore/slice/mod.rs:2690:10
note: run with `RUST_BACKTRACE=1` environment variable to display a back

We won't handle the errors here because you have the skillset mostly to do this yourself now, just have it print more useful errors is about all you can do here anyway. But let's have a look at changing it so that we can have a variable abc say that we can set like abc = 42 and then use in calculations.

Calculator - Iteration 2

OK, so this time we'll change it to accept variables. Let's change it so that we have the following requirements:

We can set variables with the = operator and it will set a variable in the left as long as it starts with an alphabet character.
We can use variables within sums whenever they are set. We leave error handling as an exercise again, the point is to teach you things after all and there's no better way of learning than doing.

Variable Substitution

So we take care of number 2 first. You might think number 1 is better to tackle first but we do number 2 first here because otherwise we can't easily test number 1 without number 2. However for number 2 I need to know how I'll be specifying variables. It seems that we need an extra parameter to send to that contains variable names and their values. Some kind of HashMap (hint hint), try and see if you can find one in the standard library.

Well we have one and we can add the following line to the beginning of the program:

use std::collections::HashMap;

and we add a variable to the calculation:

fn calculate(calculation: &str, variables: HashMap<&str, i64>) -> i64 {
    <snip>
}

Now we need to alter main:

fn main() {
    let mut rl = Editor::<()>::new();
    let variables: HashMap<&str, i64> = HashMap::new();
    loop {
        let readline = rl.readline(">> ");
        match readline {
            Ok(line) => {
                println!("Result: {}", calculate(&line, variables));
            }
            <snip>
        }
    }
}

but we get an error now:

error[E0382]: use of moved value: `variables`
  --> src/main.rs:39:57
   |
34 |     let variables: HashMap<&str, i64> = HashMap::new();
   |         --------- move occurs because `variables` has type `std::collections::HashMap<&str, i64>`, which does not implement the `Copy` trait
...
39 |                 println!("Result: {}", calculate(&line, variables));
   |                                                         ^^^^^^^^^ value moved here, in previous iteration of loop

Well we don't want to copy it as it's hinting because we want one copy of global variables essentially. So let's change it to a reference that the calculate function takes:

fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> i64 {
    <snip>
}

and there's a few additional changes like taking a reference in main and adding this parameter to the tests, we'll leave this for the reader (sorry but it is for your own good).

Now let's add a new test for being able to use variables:

#[test]
fn test_variables_in_sums() {
    let mut variables: HashMap<&str, i64> = HashMap::new();
    variables.insert("abc", 123);
    let res = super::calculate("abc + 5", &variables);
    assert_that(&res).is_equal_to(&128);
}

Now when I try and implement this I find that I want to make this possible for both num1 and num2. I don't particularly want to repeat the logic for both so I'll write a new function that checks if the first character is a letter and if so tries to substitute the value in. I'll write the following signature:

fn get_value(token: &str, variables: &HashMap<&str, i64>) -> i64 {
    0
}

and write the following tests:

#[test]
fn test_substitute_variable() {
    let mut variables: HashMap<&str, i64> = HashMap::new();
    variables.insert("abc", 123);
    let res = super::get_value("abc", &variables);
    assert_that(&res).is_equal_to(&123);
}

#[test]
fn test_return_parsed_number() {
    let variables: HashMap<&str, i64> = HashMap::new();
    let res = super::get_value("123", &variables);
    assert_that(&res).is_equal_to(&123);
}

and now I can implement get_value by:

fn get_value(token: &str, variables: &HashMap<&str, i64>) -> i64 {
    match token.chars().next().unwrap().is_ascii_alphabetic() {
        true => *variables.get(token).unwrap(),
        false => token.parse::<i64>().expect("token should be a number"),
    }
}

and the tests just written now pass. I can easily fix the remaining test as follows:

fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> i64 {
    let vec = tokenize(calculation);
    let num1 = get_value(vec[0], variables);
    let num2 = get_value(vec[2], variables);
    <snip>
}

and everything is all good again.

Variable Setting

Now we want to be able to set variables by sending abc = 123 say. Let's add a test for this now:

#[test]
fn test_setting_variables() {
    let variables: HashMap<&str, i64> = HashMap::new();
    let res = super::calculate("abc = 5", &variables);
    assert_that(&res).is_equal_to(&5);
    assert_that(variables.get("abc").unwrap()).is_equal_to(5);
}

Because we currently have to return an i64 from calculate we have to either return an i64 for this case or change the return type and deal with it elsewhere. Ordinarily I'd probably choose to return what the variable gets set to here but as we're learning I'll give an example of an Option.

Aside: Options

As one final aside we have another similar entry to Result called Option that is equally as useful. This is an enum that takes two options, either something or nothing. So an Option<T> is either Some(T) or None. This is how we handle nulls and it forces us to consider both when something is null and exists when using matching.

So for example we can have:

let num: Option<i32> = Some(32);
println!("{:?}", num);

We can also use match and expect and that kind of thing on Options too in a similar way as we can with Result. E.g. here I can do num.unwrap() if I know that num is Some(32) and I'll get 32 out.

We don't use it here but it's so useful I point it out now anyway. We can take ownership of an entry in an Option with the take method which can be very useful. For example:

let mut option: Option<String> = Some("TEST".to_owned());
let item = option.take();
println!("Option: {:?}", option);
println!("Item: {:?}", item);

and we get

Option: None
Item: Some("TEST")

This take method here and the std::mem::replace function are examples of things I wish I'd known about sooner in my journey learning Rust. Both are extremely useful and can get you out of some tight spots in places you sometimes otherwise wouldn't know how to, keep them in mind. So often I had an Option that was Some(s), I could get references to s easy enough but taking ownership evaded me when I wanted to.

Let's finish off our calculator now anyway.

Back to the Calculator

Let's begin by changing the return type of calculate to an Option where we agree that if we return Some(num) then num is the evaluated result and if we return None then no result was evaluated. We have the following now:

fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> Option<i64> {
    <snip>
}

To fix the main function that is causing this we change the part calling calculate as follows:

Ok(line) => match calculate(&line, &variables) {
    Some(num) => println!("Result: {}", num),
    None => {}
},

and we can fix the first test as follows (with similar fixes for the others):

#[test]
fn test_addition() {
    let variables: HashMap<&str, i64> = HashMap::new();
    let res = super::calculate("1 + 2", &variables);
    assert_that(&res).is_equal_to(Some(3));
}

and we fix test_setting_variables test such that it checks we get None back as follows:

#[test]
fn test_setting_variables() {
    let variables: HashMap<&str, i64> = HashMap::new();
    let res = super::calculate("abc = 5", &variables);
    assert_that(&res).is_equal_to(None);
    assert_that(variables.get("abc").unwrap()).is_equal_to(5);
}

and finally we fix the calculate function as follows:

fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> Option<i64> {
    let vec = tokenize(calculation);
    let num1 = get_value(vec[0], variables);
    let num2 = get_value(vec[2], variables);
    let op = vec[1];
    match op {
        "+" => Some(num1 + num2),
        "-" => Some(num1 - num2),
        "*" => Some(num1 * num2),
        "/" => Some(num1 / num2),
        _ => {
            panic!("Bad operator");
        }
    }
}

Now we need to refactor calculate a little bit further. We need to perform a calculation in the case we have one of the 4 operators +, -, * or / but in the case of an equals sign we want to do something else. We'll try and refactor this a bit as follows:

fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> Option<i64> {
    let vec = tokenize(calculation);
    let num2 = get_value(vec[2], variables);
    let op = vec[1];
    match op {
        "+" => Some(get_value(vec[0], variables) + num2),
        "-" => Some(get_value(vec[0], variables) - num2),
        "*" => Some(get_value(vec[0], variables) * num2),
        "/" => Some(get_value(vec[0], variables) / num2),
        _ => {
            panic!("Bad operator");
        }
    }
}

I've also refactored out the check on whether something is a variable as this is useful here too:

fn is_variable(token: &str) -> bool {
    token.chars().next().unwrap().is_ascii_alphabetic()
}

fn get_value(token: &str, variables: &HashMap<&str, i64>) -> i64 {
    match is_variable(token) {
        true => *variables.get(token).unwrap(),
        false => token.parse::<i64>().expect("token should be a number"),
    }
}

We can have faith that this is a good refactor as the same 8 tests are still passing. We now just need to implement the one remaining test.

fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> Option<i64> {
    let vec = tokenize(calculation);
    let num2 = get_value(vec[2], variables);
    let op = vec[1];
    match op {
        "+" => Some(get_value(vec[0], variables) + num2),
        "-" => Some(get_value(vec[0], variables) - num2),
        "*" => Some(get_value(vec[0], variables) * num2),
        "/" => Some(get_value(vec[0], variables) / num2),
        "=" => match is_variable(vec[0]) {
            true => {
                variables.insert(vec[0], num2);
                None
            }
            false => {
                panic!("Bad variable name");
            }
        },
        _ => {
            panic!("Bad operator");
        }
    }
}

Now let's try and run this final test and we should be good.

error[E0623]: lifetime mismatch
  --> src/main.rs:38:34
   |
27 | fn calculate(calculation: &str, variables: &HashMap<&str, i64>) -> Option<i64> {
   |                           ----                      ----
   |                           |
   |                           these two types are declared with different lifetimes...
...
38 |                 variables.insert(vec[0], num2);
   |                                  ^^^^^^^ ...but data from `calculation` flows into `variables` here

Oh dear. This error looks bad. We've not covered lifetimes in this article and I don't propose to yet, at least not fully, that's a later article. But basically what's happening here is that we've two references, one under calculation and one under the key for the variables HashMap. Now that we're using the first in the same place as the second, Rust needs to know that what is under the calculation reference will live at least as long as the variables HashMap will, we wouldn't want to free the memory for calculation while it's still being used in variables, memory corruption issues be here.

So suffice is it to say that these "lifetime problems" can occur when working with references like here. In fact even if we understood lifetimes it's not straight forward to fix this problem and we may want to consider just changing tact completely, which is what we're going to do now. We're going to use String instead as the key type in variables and that gets rid of the lifetimes problem. At this point we no longer worry about the underlying key living long enough as the HashMap owns it. In fact it's generally not a bad idea to have the HashMap own the keys like this for this very reason.

So now let's change variables to be of type HashMap<String, i64>. There's a few changes to make, including in the tests and we leave this to the reader, basically everywhere you see HashMap<&str, i64> change it to HashMap<String, i64>. Once you've done this you'll need to change the places that you're adding stuff to variables to add a String instead, use the .to_owned() method for this, e.g.:

        "=" => match is_variable(vec[0]) {
            true => {
                variables.insert(vec[0].to_owned(), num2);
                None
            }
            false => {
                panic!("Bad variable name");
            }
        },

and similarly in the tests.

Now we get an error:

error[E0596]: cannot borrow `*variables` as mutable, as it is behind a `&` reference
  --> src/main.rs:38:17
   |
27 | fn calculate(calculation: &str, variables: &HashMap<String, i64>) -> Option<i64> {
   |                                            --------------------- help: consider changing this to be a mutable reference: `&mut std::collections::HashMap<std::string::String, i64>`
...
38 |                 variables.insert(vec[0].to_owned(), num2);
   |                 ^^^^^^^^^ `variables` is a `&` reference, so the data it refers to cannot be borrowed as mutable

Does make sense actually as we're changing variables, however we've just got a reference, not a mutable reference. Thankfully in our case we can just change this to a mutable reference and after a bit of thought this does make sense to do here. Let's do that, there's a fair amount of error fixing because we need to change all our tests to have variables be mutable and pass a mutable reference to calculate to allow it to change things. We end up with the following code finally.

use std::collections::HashMap;

use rustyline::error::ReadlineError;
use rustyline::Editor;

fn tokenize(string_to_parse: &str) -> Vec<&str> {
    let mut vec = vec![];

    for token in string_to_parse.split_ascii_whitespace() {
        vec.push(token);
    }

    vec
}

fn is_variable(token: &str) -> bool {
    token.chars().next().unwrap().is_ascii_alphabetic()
}

fn get_value(token: &str, variables: &HashMap<String, i64>) -> i64 {
    match is_variable(token) {
        true => *variables.get(token).unwrap(),
        false => token.parse::<i64>().expect("token should be a number"),
    }
}

fn calculate(calculation: &str, variables: &mut HashMap<String, i64>) -> Option<i64> {
    let vec = tokenize(calculation);
    let num2 = get_value(vec[2], variables);
    let op = vec[1];
    match op {
        "+" => Some(get_value(vec[0], variables) + num2),
        "-" => Some(get_value(vec[0], variables) - num2),
        "*" => Some(get_value(vec[0], variables) * num2),
        "/" => Some(get_value(vec[0], variables) / num2),
        "=" => match is_variable(vec[0]) {
            true => {
                variables.insert(vec[0].to_owned(), num2);
                None
            }
            false => {
                panic!("Bad variable name");
            }
        },
        _ => {
            panic!("Bad operator");
        }
    }
}

fn main() {
    let mut rl = Editor::<()>::new();
    let mut variables: HashMap<String, i64> = HashMap::new();
    loop {
        let readline = rl.readline(">> ");
        match readline {
            Ok(line) => match calculate(&line, &mut variables) {
                Some(num) => println!("Result: {}", num),
                None => {}
            },
            Err(ReadlineError::Interrupted) => {
                println!("CTRL-C");
                break;
            }
            Err(ReadlineError::Eof) => {
                println!("CTRL-D");
                break;
            }
            Err(err) => {
                println!("Error: {:?}", err);
                break;
            }
        }
    }
}

#[cfg(test)]
mod tests {
    use std::collections::HashMap;

    use spectral::prelude::*;

    #[test]
    fn test_addition() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        let res = super::calculate("1 + 2", &mut variables);
        assert_that(&res).is_equal_to(Some(3));
    }

    #[test]
    fn test_subtraction() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        let res = super::calculate("2 - 1", &mut variables);
        assert_that(&res).is_equal_to(Some(1));
    }

    #[test]
    fn test_multiplication() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        let res = super::calculate("2 * 3", &mut variables);
        assert_that(&res).is_equal_to(Some(6));
    }

    #[test]
    fn test_division() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        let res = super::calculate("47 / 3", &mut variables);
        assert_that(&res).is_equal_to(Some(15));
    }

    #[test]
    fn test_variables_in_sums() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        variables.insert("abc".to_owned(), 123);
        let res = super::calculate("abc + 5", &mut variables);
        assert_that(&res).is_equal_to(Some(128));
    }

    #[test]
    fn test_parsing_string() {
        let res = super::tokenize("1 + 2");
        let expected = vec!["1", "+", "2"];
        assert_that(&res).is_equal_to(expected);
    }

    #[test]
    fn test_substitute_variable() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        variables.insert("abc".to_owned(), 123);
        let res = super::get_value("abc", &mut variables);
        assert_that(&res).is_equal_to(&123);
    }

    #[test]
    fn test_return_parsed_number() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        let res = super::get_value("123", &mut variables);
        assert_that(&res).is_equal_to(&123);
    }

    #[test]
    fn test_setting_variables() {
        let mut variables: HashMap<String, i64> = HashMap::new();
        let res = super::calculate("abc = 5", &mut variables);
        assert_that(&res).is_equal_to(None);
        assert_that(variables.get("abc").unwrap()).is_equal_to(5);
    }
}

This then passes all tests and I can use it:

> cargo run
   Compiling calculator v0.1.0 (/Users/strotter/work/programming/rust/calculator)
    Finished dev [unoptimized + debuginfo] target(s) in 1.54s
     Running `target/debug/calculator`
>> abc = 1
>> abc + 123
Result: 124

Summary

This is a pretty good outcome and I'm happy with it, it's not perfect but it's a fairly efficient and decent design. I haven't error handled but I believe you now have the skills to do so. What would be good next is to be able to support multiple sums and brackets but that may need to wait for a later date.

We've still to cover a few bits and bobs, the difficult ones in the next article are likely to be lifetimes and closures, both of which you need to understand if you're truly going to be a good Rust programmer. Thanks for reading this far and I'll see you in the next part.

The Borrow Checker and Memory Management

Steven Trotter — Sun, 28 Jul 2019 12:40:15 +0000

Introduction

I've been playing around with Rust for about 12 months now on and off and one of the things I've learned is I love Rust. Whether I can justify Rust as something I would happily use for production stuff in my day to day job is another story (I think it's real close though). But I'd love to one day, I mean really love to.

So why is Rust great and worth your time. Well, recently MicroSoft said they're interested in it (see here), it's used behind the scenes in places in both AWS and Azure (see this link for AWS and this link for Azure), it can do green threading (https://tokio.rs/) and the performance is incredible (comparable to C/C++, from various internet sources and experience). Essentially it's worthy of several articles in itself, google "why do developers love Rust".

So why do I say I can't justify moving away from higher level languages like Java, JavaScript or Python. A few reasons:

No pulling punches, it's hard to get started. One problem is can we actually find developers to develop and maintain existing Rust without paying a fortune because they're scarce? I'm hoping in time there will be but for now it's probably easier to find Java or JavaScript devs.
Unit testing isn't great in my view. They probably haven't concentrated on this much as the philosophy seems to be "if it compiles, it probably works". Honestly, even if that's true (which it isn't, even for Rust) I still want to unit test properly. It's not dreadful, it can do it, in fact it's got pretty decent foundations. But when I compare it to working with JavaScript or Python... it's missing decent mocking libraries badly. As an addendum I found this since writing this and maybe that fits my requirements, I haven't fully checked yet.
For cloud technologies we're in infancy here really. I haven't played with this much but a quick google suggests this to be the case. Whilst trying to get Java/Node/Python stuff running on AWS or Azure there's tonnes of stuff around (often created by the vendors), but Rust, it's just not on peoples radar enough yet. Again I hope and think it will be more so in the next few years. I notice that we can write serverless stuff in Rust now though which is great so this stuff is certainly going somewhere.
Green threading needs work still. Comparing this with Go here which, whilst I strongly prefer Rust overall, gives me super nice green threading. In Rust it's just harder to achieve the same thing. I kinda prefer that Rust doesn't have this kind of thing coded into its runtime and Rust can do this with the Tokio crate which is great, but it's hard. I'd still recommend using Tokio and there is some serious stuff written using it, but for the faint hearted it is not. I believe this will improve dramatically in time, not even necessarily that long from now and it's not really a fair complaint as the core team are just getting futures and async/await sorted out properly now, still I want it. Normal multi-threading however is totally awesome, largely as a result of the stuff in this article.

So I was thinking to myself what can I do to help with the above problems. I mean I wanna use this thing, it's great. For any low level systems programming stuff I'd use it in a heartbeat but I wanna use it day to day in building microservices in the cloud. I decided maybe I can help with by writing a series of articles as tutorials on the challenging parts of learning Rust to give people a starting point. Now there are loads of articles about this kind of thing and I'll reference this truly excellent series here but this references Haskell similarities too much for my liking. I don't promise mine is really any better but I'm trying to aim it at a slightly lower level, ideally I'd like to aim this at those without programming experience but reasonably it's unlikely without a bit of experience, however I'd like to think those with some basic programming knowledge can gain something from these articles. Knowing roughly what a function, module, string, integer and memory is in some languages (not necessarily Rust) will be assumed, but just a basic grasp should suffice for this article to make sense.

I feel that the main initial stumbling block with Rust is the borrow checker and memory management so that will be the subject of this first article. This is super unfortunate in a lot of ways as the borrow checker is the best thing about Rust. My advice to those new to Rust is don't be afraid of it: it loves you, it wants you to write great bug free code, it wants to help you. Whilst Rust has a difficult and steep learning curve it gives super helpful compile time errors (as I'll try and persuade in this article). I've spoken to people who've given up on Rust because of the borrow checker, if that's true then this article is for you. Please give it another go after reading this and if you still don't understand something please ask me afterwards.

I have also noticed that reading up about Rust it's way better than it used to be. It does have this reputation of being super hard to use still but the Rust community is great, they listen to problems and they try and make it better and they have (see this article on non-lexical lifetimes for example). Whilst green threading isn't quite as good as I'd like yet, I know they're working super hard on it to improve things and before long I'm pretty sure it will be.

This will be a series of articles about the difficult parts of Rust but it is not meant to be exhaustive, I'm not trying to replace the Rust book and I never will, the Rust book is great but it's a better reference than it is a tutorial. I'd strongly recommend you to try and get to grips with the Rust book too, you'll need it. If this article is confusing you could read the first 3 chapters of the Rust book (the easy chapters :) ).

Whilst Rust is hard, it is workable I promise and I will try my best to hold your hand (metaphorically of course) and guide you through the initial stages. There is nothing in this article that is rocket science or even that difficult, but there is a lot to learn in Rust and even for experienced programmers you may need to learn stuff from the ground up. These articles should prepare you well here and should have everything you need (once they're all written anyway, but even just this one will be enough for experienced programmers).

At the end of the day though, if you're after a nice easy programming language, Rust just isn't it. Rust I believe is a strong competitor for the lower level C/C++'s of the world and for complex programs in higher level languages like Python or Java. If you want a quick simple program and/or rapid prototyping Rust isn't what you're after.

It's recommended to try and run the programs below and experiment with them yourself. If you don't want to install Rust try the Rust playground. There will also be compiler warnings like warning: unused variable: i, this I would not allow in production stuff I'm writing, in tutorials I don't want to get bogged down with this but basically if I used it this would go away. It's telling me if I take away this variable the program is unchanged, it's trying to help me again. You should always fix compiler warnings, always, always, always.

Finally, I make no pretence of being a Rust expert but I've probably solved enough basics problems in it to be a bit above beginner now. I'd love to get some feedback from any Rustaceans out there as to what they think, please ping me.

The Borrow Checker

The main content of this article is all about the borrow checker. We'll discuss further things in future articles but for now we need to get a handle on this fully first. Once you understand the borrow checker you fairly well understand Rust and many other things will fall into place after this that maybe didn't before. We'll introduce some bits and pieces of syntax too but mostly this is a theoretical article about the hardest obstacle to learning Rust (in my view).

Ownership

OK, let's get started. First topic is ownership and always will be. It's important, learn and digest how this works. Once you do you're well on your way to doing cool stuff with Rust, I promise.

Memory management is the theme here basically. If you've programmed in C or C++ before you likely know about the pains memory management can cause. If you've programmed in higher level languages you may know less as it deals with it for you, but it does so with the expense of a garbage collector.

You may never have even cared about the garbage collector, it will be slowing things down for you and it only solves the freeing memory problem, Rust does more but I'll get to that later. But really garbage collectors are great, I much prefer that to C/C++ where memory management just gets in the way of what you're trying to do. Rust offers something in between essentially and in a lot of ways becomes superior as a result.

If you've not programmed in C/C++ before, well if you want to reserve memory in a dynamic way (now I need a 10 element list but soon I'll need 12, that kind of thing) you need to reserve memory and free it when done. For example, to reserve memory for an integer on the heap (the dynamic memory store basically, you don't really need to worry about this for now) I need to do tell C++ to create it as follows (don't worry about C++ syntax, it's not needed for this article):

int* i = new int;
*i = 1;

and then when I'm done I need to do:

delete i;

If I forgot the latter or do it twice then badness ensues, particularly in the latter case. It's as simple as that, the standards say if I free twice basically anything can happen. I tried freeing twice on Linux and nothing really happened, on a Mac I got the following (your results may vary):

➜  ./a.out
a.out(67865,0x1047a05c0) malloc: *** error for object 0x7fa4e8c02ac0: pointer being freed was not allocated
a.out(67865,0x1047a05c0) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    67865 abort      ./a.out

In C++ you need to think about the different types of memory here (stack and heap to those familiar), in Rust less so. The stack and heap still exist but honestly to begin with you don't need to worry so much. If you get deeper into Rust maybe but I've honestly not had to care yet. Nor is there GC (garbage collection) but it's as safe as when there is GC, more safe. I can't get the above runtime error from Rust, I've tried (even in unsafe Rust, which we're not touching here, I couldn't get it. Stay away from unsafe Rust to begin with, there be dragons). I get compiler errors but no runtime errors and of course compiler errors are better because I get them sooner and I know I must have broken something.

The reason has to do with ownership, in Rust everything has to be "owned" by something. Once it's no longer owned by something the memory will be cleared for us at that point. It's as if the compiler went and added the delete statement in C++ for me, (how cool is that, I don't need to remember it). But what does it mean to be "owned" by something.

Let take a very simple example here:

fn main() {
    let i: u32 = 32;
}

That's an ultimately pointless Rust program. This declares a variable i to be of type u32 (unsigned 32 bit integer, Rust's type system is way more sane than C/C++) and sets it to have value 32. Rust's compiler notices that when that function returns, i is no longer needed so it does any cleanup necessary (technically this would work in C/C++ too because it's on the stack, but this would also work if it were on the heap in Rust). Essentially you don't need to worry about memory management, Rust has taken care of it for you.

Similarly, we can consider the following program

fn main() {                        // Scope of `main` starts here
    let i: u32 = 32;
    {                              // A new inner scope starts here
        let j: u32 = i + 5;
        println!("j is: {}", j);
    }                              // The inner scope ends here and `j` is dropped
}                                  // The `main` scope ends here and `i` is dropped

Running this program prints the line j is: 37. We won't explain the println statement thoroughly but essentially it prints the string given and every time we have a {} inside that string we can substitute a variable value. Here the {} is replaced with the value of j.

Here we have created another scope in the middle with the braces. If you're not familiar with a scope like this from other C-family languages, basically a scope is created by these curly braces, anything between a left curly brace { and a right curly brace } is a scope. In this case I have a scope for the main function created between lines 1 and 7 an inner scope created after defining i in line 3 that ends on line 6. Everything in this inner scope has access to the variables in the scope above it but the main scope itself doesn't have access to stuff in the middle. So for example the inner scope has i and j but the main scope only has i. Having inner scopes like this is not uncommon in Rust (a bit like JavaScript in this sense). We can limit the time we need variables for with scopes like this. You can also shadow variables by calling them the same name in inner scopes, see here for example.

Now returning to the code example, we have i as before that goes out of scope once main returns. However, now we have another scope by the inner pair of braces. In this scope we create a new u32 variable j, we also have access to i so we can set j equal to i + 5, or indeed 37. Now once we get to the end of the inner scope, i.e. once the program goes past the right curly brace on line 6, then we can no longer access j. For example consider the following program:

fn main() {
    let i: u32 = 32;
    {
        let j: u32 = i + 5;
    }
    println!("j is: {}", j);
}

When we compile this we get:

➜ rustc simple_rust_program.rs
error[E0425]: cannot find value `j` in this scope
 --> simple_rust_program.rs:6:26
  |
6 |     println!("j is: {}", j);
  |                          ^ help: a local variable with a similar name exists: `i`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.

Let's take a moment now and look at how brilliant this error output is here. This tells me exactly what I need to know, j doesn't exist at line 6. It even tells me a command to explain the error more fully and it thinks maybe I meant i. I know exactly what it's complaining about and it really tries to help me fix it. I really love the borrow checker for this, it's really trying to help (I recommend trying to remember this).

So we've covered what ownership means, it's pretty straight forward right, what's the big deal? Well let's move onto the next topic where we pass stuff around.

Transferring ownership

Sometimes of course we want to pass variables around, in functions calls for example we may want to pass in a variable as a parameter. Now we have 3 ways (these 3 ways come up a lot in Rust) to do this, transfer ownership, pass by reference or pass by mutable reference. We cover transferring ownership now (The other two will be covered under the Borrowing section below).

Transferring ownership is really easy syntactically and just what you'd expect. The following program shows something being passed by ownership:

fn take_ownership(a: String) {
    println!("I have a: {}", a);                 // and a is now owned by take_ownership
}

fn main() {
    let a: String = "Test String".to_string();   // a is owned by main
    take_ownership(a);                           // Now we pass a to take_ownership
    //println!("I try to print a: {}", a);
    // If I uncomment this line above then I get an error because this scope no longer owns a
}

To explain briefly, the .to_string() creates a String that we can alter out of a String literal. See Section 8.2 of the book for more details.

So if you pass ownership like this, it's gone from the original function, only one thing can own a variable at any one time. Unless you pass it back of course. I can alter the above as follows:

fn take_ownership(a: String) -> String {
    println!("I have a: {}", a);
    return a;
}

fn main() {
    let a: String = "Test String".to_string();
    let a = take_ownership(a);
    println!("a: {}", a);            // We've passed a back so we're OK now
}

As an aside, the above program doesn't return in a very Rust like way. Equivalently this function can be written:

fn take_ownership(a: String) -> String {
    println!("I have a: {}", a);
    a
}

This is exactly the same thing. Because there's no semi-colon on the last line Rust knows this is a return. There's an explanation involving statements and expressions but I don't want to go into this here, just be aware if you see this, this is just a return.

So simple enough right. Well no, not quite. There's a complication that we'll talk about shortly but first we cover mutability now.

Mutability

Next on the agenda is to cover mutability, or you may prefer to think of non-constant variables or variables you can alter (or just variables maybe, surely I can alter a variable or it doesn't vary, just go with it). Essentially, by default, everything in Rust is immutable, that is it's constant and can't have its value altered. If I have the following program:

fn main() {
    let i: u32 = 1;
    i = 2;
    println!("i is: {}", i);
}

If I try I get something like the following:

error[E0384]: cannot assign twice to immutable variable `i`
 --> src/main.rs:3:5
  |
2 |     let i: u32 = 1;
  |         -
  |         |
  |         first assignment to `i`
  |         help: make this binding mutable: `mut i`
3 |     i = 2;
  |     ^^^^^ cannot assign twice to immutable variable

error: aborting due to previous error

For more information about this error, try `rustc --explain E0384`.

Pretty clear where the problem is, right? But how do I do that? Quite easy actually, we can create a mutable variable (one that can be changed) by simply adding the keyword mut as per the code below. In fact it tells me in the error exactly how to do this.

fn main() {
    let mut i: u32 = 1;        // Added mut, now I can change it
    i = 2;
    println!("i is: {}", i);
}

What's interesting about this is that I can pass ownership of immutable variable around to be a mutable variable and that's absolutely fine. For example:

fn take_ownership_and_pass_back(mut a: String) -> String {
    a = a + "2";
    a
}

fn main() {
    let a: String = "TEST".to_string();
    let a = take_ownership_and_pass_back(a);
    println!("a: {}", a);
}

So I change a from immutable to mutable, and that's fine. If I run this I get a line a: TEST2. But a was immutable? Well Rust makes no guarantees you won't purposefully change it to mutable but you've got to try to change it. It's not trying to tell you what to do it's trying to get you to stop changing things you never meant to. Generally, immutable variables are a great thing for this reason and that's why they're the default. The point really is that I can limit the amount of time something needs to be mutable for and thus limit the places where it's possible to alter values (and create bugs according to functional programming, we wouldn't want that).

Another interesting code snippet is the following:

fn main() {
    let mut s = "TEST1".to_string();
    let t = s;
    // Can't do the following as s is now moved
    //println!("s={}", s);
    s = "TEST2".to_string();
    println!("s={}, t={}", s, t);
}

Here we create a mutable string s with value "TEST1". Next we let t take ownership of this string, at this point I can no longer read from s as it's moved its value elsewhere and is empty. However I can reassign to s as it's mutable and then read from it as it's now initialised, that's absolutely fine as shown by running the above. I get the output s=TEST2, t=TEST1.

There's another common gotcha here that we're not allowed to do, let's take a look at the following.

fn main() {
    let x = "TEST".to_string();
    for i in 0..10 {
        let y = x + &i.to_string();
        println!("{}", y);
    }
}

Here we try to loop 10 times and try to build a string from concatenating "TEST" and i the integer. Unfortunately we have a problem, can you see why without compiling?

If we try to compile this we get the following error:

error[E0382]: use of moved value: `x`
 --> src/main.rs:4:17
  |
2 |     let x = "TEST".to_string();
  |         - move occurs because `x` has type `std::string::String`, which does not implement the `Copy` trait
3 |     for i in 0..10 {
4 |         let y = x + &i.to_string();
  |                 ^ value moved here, in previous iteration of loop

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.

So because x was moved on the first loop and then goes out of scope at the end of this, x is dropped after one loop and thus we have no access to x on the second loop. The compiler recognises this and throws the above error. There are a variety of ways around this, including the following initially surprising (though after thought reasonable) method:

fn main() {
    let mut x = "TEST".to_string();
    for i in 0..10 {
        let y = x + &i.to_string();
        println!("{}", y);
        x = "TEST".to_string();
    }
}

We now try and look at copying and cloning as a way around this kind of thing. Using references from later will also work here.

Copying and Cloning

So I said earlier that you can transfer ownership of a variable and I showed you how to do it. I also told you there's a minor complication, well here it is. Let's have a look at the following example:

fn main() {
    let i: u32 = 123;

    {
        let mut j = i;     // <-- If i didn't satisfy copy this would be a move

        j = 124;

        println!("Not finished and j is: {}", j);
    }

    println!("Finished and i is: {}", i);
    // ^^ And if i didn't satisfy copy and hence moved above I wouldn't be able to read i here
}

By the logic I mentioned previously, here i should be moved into j in the inner block when j is created, but then I can no longer access i in the final println as it's gone. Yet I run it, it compiles successfully and I get:

Not finished and j is: 124
Finished and i is: 123

So I can see I've copied the value of i into j, not moved anything as I thought, and I can happily carry on using an unchanged i. Rust has helped me because it knows u32's satisfy the Copy trait. See here for more details about the Copy trait but I'll explain what this means briefly. If you've come across interfaces from other languages, traits are kinda like interfaces, read "interface" for now instead of trait and it's close enough. Essentially a trait is something that can be satisfied and then things can act accordingly based on this. In this case anything that satisfies the Copy trait, the compiler knows its value gets copied rather than moved.

On the other hand if I change i to be a String s as in the following code:

fn main() {
    let s: String = "123".to_string();

    {
        let mut t = s;

        t = "124".to_string();

        println!("Not finished and t is: {}", t);
    }

    println!("Finished and s is: {}", s);
}

Then I get an error as follows:

error[E0382]: borrow of moved value: `s`
  --> src/main.rs:12:39
   |
2  |     let s: String = "123".to_string();
   |         - move occurs because `s` has type `std::string::String`, which does not implement the `Copy` trait
...
5  |         let mut t = s;
   |                     - value moved here
...
12 |     println!("Finished and s is: {}", s);
   |                                       ^ value borrowed here after move

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.

Again, look at the amazing output (gcc or clang eat your heart out). This tells me exactly what I need to know, s was created here on line 2, moved here on line 5, but now I've tried to use it on line 12 and I'm not allowed.

So the error is great but what do I do here now? Well there's a few things you can do. If you're happy with the copy you can simply add a .clone() to the line changing the value for t, as in:

        t = "124".to_string().clone();

Now Rust copies it because I've told it to. Whatever's being cloned needs to satisfy the Clone trait, most things tend to but not always. Cloning means the whole memory of the value of that variable is copied into memory elsewhere and if something is likely to be particularly big, you may not want it to satisfy this trait so you can't accidentally do this.

But basically what this really means is that whenever we pass a u32 like this it copies the data rather than moving it. So by copying it I get a whole new variable and it's totally unrelated to the original one. Rust is not alone here, I've seen a few languages that do this, Python for example. It's usually small fixed size things this is true of by default (booleans and numbers mostly). If you ever find something like this worked when it looked like it shouldn't then probably just Rust copied it and is trying to help you. Of course if this happens and you change the second variable the first is unchanged, one to be careful of.

So now I've covered most of what you need to know to get going with ownership. But we won't get very far unless it's a pretty simple program (and I told you not to bother using Rust in that case). Sometimes we just want to borrow something which we'll take a look at now.

Borrowing

Now we know how to move and copy stuff, but what if I don't want to move or copy it? I want access to it I guess to see what's in there but it's not mine, or even I want to change some stuff but it's not mine. Well you've a few choices as follows:

If I want to only be able to read from it I can create a "reference" to it, I can have as many of these as I like (as long as no "mutable references" from the next exist).
If I want to be able to update it too I can create a "mutable reference" to it, but I can only have one of these, period. If I have a "mutable reference" the compiler will stop me from creating any other "references" mutable or not ("references" are immutable by default, as indeed everything is in Rust).
I can create "smart pointers" to it. I'll discuss this in a future article, this can be a bit of a last ditch effort but sometimes is needed.

So what are references and mutable references? Well instead of passing ownership I pass a reference (kinda like a pointer in C if you're familiar with this). A reference holds a memory address essentially, that points to where the original is. When you take a reference a of a variable b you leave the ownership of a alone but you say to b "you can point over here and take a look at a, or even you can point at a and alter it if it's a mutable reference I've taken, but a still belongs to me". We'll look at an example now.

For the code example above I really wanted to be able to change it in another scope without taking ownership, maybe in another function or something. Well I could do this and this works:

fn main() {
    let mut s: String = "123".to_string();

    {
        let t = &mut s;

        *t = "124".to_string();

        println!("Not finished and t is: {}", t);
    }

    println!("Finished and s is: {}", s);
}

If I run this I get what I want:

Not finished and t is: 124
Finished and s is: 124

So what have I done here? First, I've had to change the original string to be mutable because I'm changing it, I didn't need to before because I never wanted to change the original. I also created a mutable reference t by doing let t = &mut s;. This & is the syntax for me taking a reference and mut implies I want a mutable reference, something I can change. If I'd just written let t = &s; then t would just be a reference and I could only read from it. When I want to perform a read or write (if it's mutable) to the original I need to dereference it and I do so by writing *t, the * here means the value this address is pointing to. If I had written t = "124".to_string() as you might think I'd get the following error:

 --> src/main.rs:7:13
  |
7 |         t = "124".to_string();
  |             ^^^^^^^^^^^^^^^^^ expected mutable reference, found struct `std::string::String`
  |
  = note: expected type `&mut std::string::String`
             found type `std::string::String`
help: consider dereferencing here to assign to the mutable borrowed piece of memory
  |
7 |         *t = "124".to_string();
  |         ^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.

I'll stop gushing about Rust's error reporting after this one but look, told me exactly how to fix it.

So we understand now from above about dereferencing and anyone whose looked at Rust before is probably thinking "I never do that in most cases". Quite right, I lied to you a moment ago, I'm sorry. Well the truth is, thankfully, Rust is smart. In non-ambiguous cases the Rust compiler knows places where I clearly meant to be dereferencing this and adds the dereference operator automatically. So I put the * to dereference the t in the line *t = "124".to_string(); because, as we see above, Rust doesn't know I don't want to assign directly to t. But in the next line where I print the variable I can see that I don't need to dereference, there's no harm in doing so, but if I don't the compiler knows "well he obviously meant to". In fact most Rust devs from what I can tell wouldn't bother putting the * on this line either.

Now I said earlier I can only take one mutable reference and if I have one I can't take any other references. Let's try and create another reference after creating the immutable reference and see what happens. Well we'd get an error similar to the following:

error[E0502]: cannot borrow `s` as immutable because it is also borrowed as mutable
 --> src/main.rs:6:17
  |
5 |         let t = &mut s;
  |                 ------ mutable borrow occurs here
6 |         let u = &s;
  |                 ^^ immutable borrow occurs here
7 |
8 |         *t = "124".to_string();
  |         -- mutable borrow later used here

error: aborting due to previous error

For more information about this error, try `rustc --explain E0502`.

This is because as I hinted earlier, if I try and create any other reference when a mutable reference exists then Rust will block me. The reasons are to do with enforcing safe multi-threading programming, race conditions are no fun and they are largely eliminited by imposing these rules. So if one thread had a mutable reference that could be altering stuff away, I might get data race conditions if I try and read it in another thread. So Rust absolutely guarantees safety here, it makes sure you can't mess stuff up in multithreading environments at the cost of having to understand this kind of article. Once you've worked out the stuff in this article the rest of the theory is a lot easier.

Because of the rules around references I've just explained, limiting the scope by creating an inner scope is a common thing to see in Rust code. Once the scope changes things get cleared up and it's a great way out of some problems you'll encounter. It's a common pattern to create a really small scope for changing stuff.

Similarly if I try and create a mutable reference after a reference exists (I have to use it though otherwise Rust is smart enough to know there's no problem):

error[E0502]: cannot borrow `s` as mutable because it is also borrowed as immutable
  --> src/main.rs:6:17
   |
5  |         let u = &s;
   |                 -- immutable borrow occurs here
6  |         let t = &mut s;
   |                 ^^^^^^ mutable borrow occurs here
...
11 |         println!("Not finished and u is: {}", u);
   |                                               - immutable borrow later used here

error: aborting due to previous error

For more information about this error, try `rustc --explain E0502`.

I often wondered in my early dealings with Rust, how is this stuff safe if I have a reference to a mutable variable and the mutable variable starts getting changed. I should have just tested this back then but instead I just got confused. What I mean is, consider the following program:

fn main() {
    let mut x = "123".to_string();
    let y = &x;
    x += "456";
    println!("{}", y);
}

I can't take a mutable reference to x after the reference because that's dangerous. But if x can still change the value then how is this any different. Well this fails compilation too because Rust blocks even the owner making changes while these references exist. I'm not sure this is very well documented, I certainly wasn't aware of this for a long time but this is how it ensures safe multi-threading. If I compile the above program I get:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
 --> src/main.rs:4:5
  |
3 |     let y = &x;
  |             -- immutable borrow occurs here
4 |     x += "456";
  |     ^^^^^^^^^^ mutable borrow occurs here
5 |     println!("{}", y);
  |                    - immutable borrow later used here

error: aborting due to previous error

For more information about this error, try `rustc --explain E0502`.

Similarly the following program fails:

fn main() {
    let mut x = "123".to_string();
    let y = &mut x;
    x += "456";
    println!("{}", y);
}

Once I've created the mutable reference I can only use that to change things while it exists, even the owner can't change it any more. Compiler error this time:

error[E0499]: cannot borrow `x` as mutable more than once at a time
 --> src/main.rs:4:5
  |
3 |     let y = &mut x;
  |             ------ first mutable borrow occurs here
4 |     x += "456";
  |     ^ second mutable borrow occurs here
5 |     println!("{}", y);
  |                    - first borrow later used here

error: aborting due to previous error

For more information about this error, try `rustc --explain E0499`.

So in a nutshell Rust will ensure safety as we have:

Rust will make sure that if more than one thing can read a variable then nothing can change that variable at all.
Rust will make sure that if something is able to change a variable (mutable variable or mutable reference) then only that one thing is able to read and write from that variable.
The counterexample to either of the above is if Rust has been smart enough to know it's safe anyway which it occasionally is. For example when I tried to take a reference and a mutable reference in the same scope earlier it wasn't a problem until I used both of them.
These checks are all enforced at the compiler level, nothing in this article is done at runtime so you know sooner if you've done something potentially dangerous.

That just about covers the basics of the borrow checker. Easy right? Well there's more, of course, and I'll cover that off in future entries in this series. Hopefully you can now tackle up to Chapter 10 of the book will make sense now and you can hopefully read all up until Chapter 10 without finding anything too challenging. In the next few articles in this series I'll try to cover structs, options, threads, smart pointers, lifetimes and closures and how to work with the borrow checker with these things. Stay tuned.

DEV Community: Steven Trotter

AWS CDK - A Game Changer in IaC

Introduction

What's not so great about CDK

Why do I love CDK?

Experience with CDK

Conclusion

The Not So Hard Bits In Practice

Writing Basic Programs

Introduction

Calculator - Iteration 1

Creating a basic program with cargo

Adding Crates

Interpreting the line

Aside: Strings in Rust

Back to the Calculator

Aside: Vectors

Back to the Calculator

Performing Calculations

Aside: Results, Enums and Matching

Back to the Calculator (again).

Finishing off

Calculator - Iteration 2

Variable Substitution

Variable Setting

Aside: Options

Back to the Calculator

Summary

The Borrow Checker and Memory Management

Introduction

The Borrow Checker

Ownership

Transferring ownership

Mutability

Copying and Cloning

Borrowing

Creating a basic program with `cargo`