DEV Community: Roger Torres (he/him/ele)

[Rust] Tokio stack overview: Runtime

Roger Torres (he/him/ele) — Sun, 12 Sep 2021 13:39:39 +0000

Quoting its first announcement, "Tokio is a platform for writing fast networking code in Rust [and] is primarily intended as a foundation for other libraries".

The central piece of this platform, the runtime, is also named tokio, and that is what this post is about; for understanding tokio runtime is vital for understanding the platform as a whole and — given the current state of things — how to write asynchronous code in Rust.

What is an asynchronous runtime?

Core Rust provides the types to build asynchronous applications. However, when building, say, an asynchronous network application, we found ourselves in the need of a lot of boilerplate code. We can write it ourselves, or we can use a library that gives it to us ready-made (and probably better-made). And that is what an asynchronous runtime such as Tokio does, it provides the building blocks to construe such an application.

Futures

Let us start by taking a look at what core Rust brings to the table, so we can better understand what we would lack if we were to build an asynchronous network application all by ourselves.

P.S. I already wrote an introduction to async Rust, so this will be a dried out explanation.

Asynchronous Rust allows us to create concurrent applications. It does so via the syntax async/.await. Basically, blocks and functions declared with async desugar into a block or function that returns an implementation of a trait called Future. Future is a state machine, so it can keep up with the progress made in a certain operation, which means it can stop processing at some point and, when executing again, continue from where it stopped. On a higher level, we might say that a future is the representation of a value that may or may not be ready, a duality that is put forward using an enum called Poll that has two variants: Pending and Ready<T>. The Future trait also has a function, called poll(), that will try to make as much progress as possible within the future (thus driving the state machine forward). This function, poll(), is first executed when we .await the future. To .await the future is to deliver it to a scheduler (formerly known as executor) that will poll() it. If it is processed through completion, Ready<T> is returned, otherwise Pending is returned and the scheduler keeps the future aside, waiting for a request to poll() it again. This request comes from the driver (formerly known as reactor), which is an I/O event loop.

Rust does not provide these last two. That is why we need a crate to help us with that. Furthermore, we also need some time-related utilities to handle all this scheduling stuff.

Needless to say, this is precisely what the Tokio runtime provides. Quoting its documentation:

Unlike other Rust programs, asynchronous applications require runtime support. In particular, the following runtime services are necessary:

An I/O event loop, called the driver, which drives I/O resources and dispatches I/O events to tasks that depend on them.

A scheduler to execute tasks that use these I/O resources.

A timer for scheduling work to run after a set period of time.

Scheduler

When you code an async function for the first time, you realize that the place from which you are calling this function also has to be async. And if you go all the way up and try to make your main() function async, Rust will tell you that "main function is not allowed to be async".

Asking Rust to explain this error gives us a hint:

$ rustc --explain E0752

`fn main()` or the specified start function is not allowed to be `async`. Not having a correct async runtime library setup may cause this error.

A quick search on the web is enough to provide the solution: we got to import tokio and use this attribute macro:

#[tokio::main]
async fn main(){ 
    // ... 
}

However, even thought it certainly works, a question remains…

Why?

Because at some point the futures have to be dealt with, and there is nothing above the main() function in a Rust program, so whoever is handling them, have to be below main(). Another way to put it is to say that the main() is the entry door of your program. The operating system running the binary knows nothing about futures, so they have to be managed "inside the house", that is, after we entered the program. So, main() has to be synchronous.

If that is the case, how does Tokio manage to make main() async, if the top-level function cannot be async? Well, it does not. #[tokio::main] will desugar async fn main() into something like this:

fn main() {
    tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        .build()
        .unwrap()
        .block_on(async {
            // ...
        })
}

When using the attribute macro #[tokio::main], we are building a runtime below main(), a runtime that will handle the tree of futures. Why am I calling it a tree? Because a future may .await other futures. I will talk more about multiple .await calls later. For now, let us move on with this idea of handling a tree of futures.

Handling the tree of futures

Consider the example below.

#[tokio::main]
async fn main() {
    let (foo, bar) = tokio::join! { foo(), bar() };
    println!("{}{}", foo, bar);
}

async fn foo() -> &'static str {
    let listener = std::net::TcpListener::bind("0.0.0.0:8080").unwrap();
    match listener.accept(){
        Ok(_pair) => {
            println!("`foo()` is finished");
            "foo"
        },
        Err(_error) => "error"
    }
}

async fn bar() -> &'static str {
    let listener = std::net::TcpListener::bind("0.0.0.0:8081").unwrap();
    match listener.accept(){
        Ok(_pair) => {
            println!("`bar()` is finished");
            "bar"
        },
        Err(_error) => "error"
    }
}

Tip: run the code above and connect to both 0.0.0.0:8080 and 0.0.0.0:8081 using your browser and check the result in the terminal where you ran the program.

When we call foo().await, we are handing foo()'s future to the runtime scheduler, the one responsible for calling poll() on it. Futures are executed by the scheduler as part of tasks. You might think of a task as a thread that is not handled by the OS scheduler, but by the runtime scheduler (they are virtual/green threads).

This will run foo() as far as possible towards completion, which means that the executor will not preemptively stop it to run something else in its stead (as the OS does with its threads). For the Tokio scheduler, as far as a task is doing relevant work, it may keep working. In a more technical jargon, tasks run until they yield. In our example, foo() runs until it starts listening at port 8080. If you're trying to understand which part of our code is explicitly yielding the task, give up. It is not there. We don't code yields, Rust manages that for us.

After foo() yields, join!—a macro that .awaits—will call bar(), which will run until it starts listening at port 8081. At this point, as both functions have yielded, we have two futures waiting to be polled again, and they may be polled in any order. Now, imagine that foo() and/or bar() call async functions inside them, giving new tasks to the scheduler. In a scenario like this, we have a tree of futures. One important thing to understand here is that we have a "root-future" (the async book calls it “top-level future”, but I will stick with “root”); in this case, it is the future returned by that async block in main (you will find inside block_on() in the desugared example). And this is important for at least two reasons.

First, a task is responsible for a tree of futures. So, let's say we have an async fn main(). As we saw, under the hood this is a normal main() that will block_on() an async block. If inside this future we .await another future, it will be dealt by the same task, as it is part of the same tree.

Second, it points to the threshold between concurrency and parallelism. If you just .await or join! futures, you will never have two Tokio tasks running simultaneously because, at the end, our main() is .awaiting the root-future, and its node-futures are executed one after the other as part of the same task, hence in the same OS thread. In other words, your async program will have concurrency, but not parallelism.

Revisiting our example, even if ports 8080 and 8081 are accessed at the same time, foo() and bar() will be executed one after the other because they are ~~fruits~~ futures of the same tree. Sure, this is no big deal here, but if you remember that we are talking about network applications and, by doing so, extrapolate over this silly example, you will quickly see this cannot be right.

Scheduling parallel tasks

As mentioned above, main() is the entry point of our program, so everything we are doing is below it. And what we have below (what #[tokio::main] desugars to) is a runtime that was built using new_multi_thread(): a multi-threaded Tokio runtime. So far, we have been using only one of those threads; it is running our task spawned by block_on(). If we want parallelism, we need to hand our futures to the runtime itself, so they can become a "root-future" and, as such, become new tasks. To achieve such parallelism, that is, to allow the runtime to execute our tasks with a different worker of its thread pool, we got to spawn the tasks.

use std::collections::HashMap;
use std::sync::{Arc, Mutex};

#[tokio::main]
async fn main() {
    let db: Arc<Mutex<HashMap<&str, &str>>> = Default::default();
    tokio::spawn(foo(db.clone()));
    tokio::spawn(bar(db.clone()));
    handle(db).await;
}

async fn foo(db: Arc<Mutex<HashMap<&str, &str>>>) {
    let listener = std::net::TcpListener::bind("0.0.0.0:8080").unwrap();
    match listener.accept(){
        Ok(_pair) => {
            loop {
                if let Ok(mut lock) = db.try_lock(){
                    println!("`foo()` is finished");
                    lock.insert("f", "foo");
                    break;
                }
            }
        },
        Err(_error) => println!("error"),
    }
}

async fn bar(db: Arc<Mutex<HashMap<&str, &str>>>) {
    let listener = std::net::TcpListener::bind("0.0.0.0:8081").unwrap();
    match listener.accept(){
        Ok(_pair) => {
            loop {
                if let Ok(mut lock) = db.try_lock(){
                    println!("`bar()` is finished");
                    lock.insert("b", "bar");
                    break;
                }
            }
        },
        Err(_error) => println!("error"),
    }
}

async fn handle(db: Arc<Mutex<HashMap<&str, &str>>>) {
    loop {
        if let Ok(lock) = db.try_lock(){
            if lock.len() == 2 {
                println!("{}{}", lock.get("f").unwrap(), lock.get("b").unwrap());
                break;
            }
        }
    }
}

In the example above, foo() and bar() become root-futures in their own right, and as handle() is the single future within the block_on() future, we end up with three different trees of futures. That way, if we call all three functions at the “same” time, they can be executed in three different threads (assuming the runtime has these threads).

I might have went a little over the top by using Arc<Mutex<HashMap>>, since it could be dealt with in an easier manner with JoinHandle. My reasoning was that using the smart pointers made it easier to see the parallelism, as the JoinHandle looks very similar to how we use .await.

Going beyond

If you want to go above and beyond, a good place to start is to understand how Tokio employs a work-stealing technique to manage its multithreaded scheduler.

Driver

Let us reconsider our previous example. After foo() and bar() both yield, which happens once they start listening at 0.0.0.0, they return Poll::Pending. As there is still work to be done, the scheduler will not get rid of them, but will not poll() them again autonomously; it will poll() them again under request.

In this case, the source of the need to poll() them again is the access to 0.0.0.0. However, if neither foo() nor bar() are actually running, which process will pull the trigger? Something has to be running to mediate our access to 0.0.0.0 and the scheduler. That is the role of the driver, which is how Tokio call its I/O event loop.

Before diving into the driver, though, let us talk a bit more about what make it necessary: a pending future.

Pending future

Maybe this topic belongs to the scheduler, but as it is vital for an understanding of the driver, I think it also fits here.

When we poll a future, it receives a Context as an argument. Currently, this Context is just a wrapper for the &Waker. This &Waker is a reference to the Waker found within the task that called poll(). This &Waker has a method wake() that is called by the driver, so the task (that owns Waker) becomes aware that it should poll() the future once again.

The following flow is an illustrative example of how it works:

A future is .awaited.
As such, it is handed to the scheduler task that was created by block_on().
This task will poll() the future, which will do some work until it reaches the point where it has to yield; let's say it is listening at some address, as our foo() was.
Before yielding, the future clone() the &Waker received as an argument in poll(). That “binds” the future and the task.
It yields, returning Poll::Pending.
When the operating system's I/O receives a connection on that certain address, it will let the driver know.
The driver will call wake() on the &Waker stored by the future, and this will wake up the task.
The awoken task will then poll() the future again. If it returns Pending, the new Waker that was passed by this last poll() will be copied and the process restarts.

The 6th and 7th steps describes the role of the driver as an interface between the OS and the task scheduler. This means that the driver* will perform system calls to the OS, such as kqueue in BSD/macOS, IPCP in Windows or epoll in Linux (and now we are hearing more and more about io_uring, which Tokio handles as well).

* It is not really the driver that makes these system calls. The interaction is actually between the driver and mio, so it is mio who interacts with the OS. That being said, I will abstract from it here, so we can depict a simplified conversation between Tokio's driver and the OS I/O, which comprises the 6th step above.

The driver, being an event loop, will keep polling the OS using one of these system calls. Let us retrieve our foo() example. If the driver polls the OS and find out that there was a connection at 0.0.0.0:8080, it will then wake() the task for it to poll() the future.

Sure, there is a myriad of details left out. For example, how the communication via channels between the scheduler and the driver actually works? Nevertheless, I will respect the beginners tag with which I marked this post and stop here. (Even because, if I write posts for beginners, it is not only because I think we still miss more introductory content, but also because of my own current limitations; and here we are teetering on the edge of my knowledge gap 🙃).

Timer

The module tokio::time is part of the runtime and provides utilities for tracking time. I don't have much to talk about these, but I will, for the sake of completion, quote the part of the documentation that explains what this module provides:

Sleep is a future that does no work and completes at a specific Instant in time.

Interval is a stream yielding a value at a fixed period. It is initialized with a Duration and repeatedly yields each time the duration elapses.

Timeout: Wraps a future or stream, setting an upper bound to the amount of time it is allowed to execute. If the future or stream does not complete in time, then it is canceled and an error is returned.

I will stop here for today. I feel there is a lot missing, but this post is already longer than I wanted. Hopefully, we will be able to revisit some topics as we move on to talk about the other crates.

See you there!

Cover photo by Pawel Nolbert

Asynchronous Rust: basic concepts

Roger Torres (he/him/ele) — Sun, 29 Aug 2021 20:56:01 +0000

TL;DR: I will try to give an easy-to-understand account of some concepts surrounding asynchronous Rust: async, await, Future, Poll, Context, Waker, Executor and Reactor.

As with most things I write here, we already have good content related to asynchronous Rust. Let me mention a few:

The Asynchronous Programming in Rust, a.k.a. async book; incomplete, but great.
Steve's talks on Rust's Journey to async/await and on how it works.
Without Boats' proposal for await syntax (the other entries with the tags async and Future are also excellent).
Jon's stream on how Futures and async/await works.

With this amount of superb information, why writing about it? My answer here is the same for ~~almost~~ every other entry on my DEV blog: to reach an audience for which this content is still a bit too hard to grasp.

So, if you want something in a more intermediary level, go straight to the content listed above. Otherwise, let's go :)

async/.await

Asynchronous Rust (async Rust, for short) is delivered through the async/.await syntax. It means that these two keywords (async and .await) are the centerpieces of writing async Rust. But what is async Rust?

The async book states that async is a concurrent programming model. Concurrent means that different tasks will perform their activities alternatively; e.g., task A does a bit of work, hands the thread over to task B, who works a little and give it back, etc.

Do not confuse it with parallel programming, where different tasks are running simultaneously. You can combine concurrent and parallel programmin (e.g., by spawning futures), but I will not cover it here since async/.await is used to enable concurrent programming, so that is my focus here.

In short, we use the async keyword to tell Rust that a block or a function is going to be asynchronous.

// asynchronous block
async {
    // ...
}

// asynchronous function
async fn foo(){
    // ...
}

But what does it mean for a Rust program to be asynchronous? It means that it will return an implementation of the Future trait. I will cover Future in the next section; for now, it is enough to say that a Future represents a value that may or may not be ready.

We handle a Future that is returned by an async block/function with the .await keyword. Consider the silly example below:

async fn foo() -> i32 {
    11
}

fn bar() {
    let x = foo();

    // it is possible to .await only inside async fn or block
    async {
        let y = foo().await;
    };
}

In this case, x is not i32, but the implementation of the Future trait (impl Future<Output = i32> in this case). The variable y on the other hand, will be a i32: 11.

Other way to visualize this is to understand that Rust will desugar this

async fn foo() -> i32 {}

into something like this

fn foo() -> impl Future<Output=i32>{}

Of course, there is no asynchronous anything happening here. But if foo() was complex, having to wait for Mutex locks or is listening to a network connection, instead of holding the thread for the whole time, Rust would do as much progress as possible on foo() and then yields the thread to do something else, taking it back when it could do more work.

Hopefully, it will make sense after we go through concepts like Future, Poll and Wake. For now, it is enough that you have a general idea of the use of both async and await.

Be sure to read the async/.await Primer.

Futures

I think it is not an exaggeration to say that the Future trait is the heart of async Rust.

A Future is a trait that has:

An Output type (i32 in the example above).
A poll function.

poll() is a function that does as much work as it can, and then returns an enum called Poll:

enum Poll<T> {
    Ready(T),
    Pending,
}

As you can see, my description of .await and poll() kind of overlap. That's because calling .await will eventually call poll(). More on this later.

This enum is the representation of what I wrote earlier, that a Future represents a value that may or may not be ready.

The general idea behind this function is simple: when someone calls poll() on a future, if it went all the way through completion, it returns Ready(T) and the .await will return T. Otherwise, it will return Pending.

The question is, if it returns Pending, how do we get back at it, so it can keep working towards completion? The short answer is the reactor. However, we have some ground to cover before getting there.

Poll, Context, Waker, Executor and Reactor

Lots of words! But I honestly think it is easier to bundle everything together because it is easier to understand what they do in context. And to illustrate this, I came up with a simplified hypothetical scenario.

Suppose we have a Future created via async keyword. Let's remember what a Future is:

#[must_use = "futures do nothing unless you `.await` or poll them"]
pub trait Future {
    type Output;
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

I will not cover Pin here, as it is somewhat complex and not necessary to understand what is going on here.

As hinted by the code above, futures in Rust are lazy, which means that just declaring them will not make them run.

Now, let's say we run the future using .await. "Run" here means delivering it to an "executor" that will call poll() in the future.

But what is the executor? Oversimplifying, it is a schedule algorithm that will actually poll the futures. So, when you call .await, who are going to do the work is an executor.

Ok, we called .await, the future was polled and returned Ready<T>. What happens? The .await will return T and the executor will get rid of the future, so it does not get polled again.

Alternatively, if the polled future wasn't able to do all the work, it will return Pending.

After receiving Pending, the executor will not poll the future again until it is told so. And who is going to tell him? The "reactor". It will call the wake() function on the Waker that was passed as an argument in the poll() function. That allows the executor to know that the associated task is ready to move on.

But what is the reactor? It is the executor's brother. While the executor is on the Olympus, managing things, listening to ~~prayers~~ .awaits, the reactor is on the Hades, working alongside the system I/O, doing the heavy lifting. It is the reactor that will know the best time to poll that future again, and it will do so calling wake().

So, should you, just starting to read Rust async stuff, worry about how executor and the reactor work behind the scene? Not really. Why? Because when we talk about executor and reactor we are already talking about runtimes; and when we talk about runtimes we are usually talking about Tokio. In fact, calling it by the names executor and reactor is already adhering to Tokio nomenclatures. So, at the end, all you have to do is incorporate Tokio on your project. The usual way to do this is using its procedural macro before the main function:

#[tokio::main]
async fn main(){
  // your async code
}

Still about the reactor, Jon spent 45 minutes explaining this while drawing on a blackboard, and I will not pretend I can do a better job. So, if you want to dive into this level of detail, check the link above.

Wrapping up

Let us recap:

async is used to create an asynchronous block or function, making it return a Future.
.await will wait for the completion of the future and eventually give back the value (or an error, which is why it is common to use the question mark operator in .await?).
Future is the representation of an asynchronous computation, a value that may or may not be ready, something that is represented by the variants of the Poll enum.
Poll is the enum returned by a future, whose variants can be either Ready<T> or Pending.
poll() is the function that works the future towards its completion. It receives a Context as a parameter and is called by the executor.
Context is a wrapper for Waker.
Waker is a type that contains a wake() function that will be called by the reactor, telling the executor that it may poll the future again.
Executor is a scheduler that executes the futures by calling poll() repeatedly.
Reactor is something like an event loop responsible for waking up the pending futures.

Ok, there is certainly more to talk about, such as the Send and Sync traits, Pinning and so on, but I think that, for a beginner post, we had enough.

See you next time!

Cover art by TK.

Edit — Sep, 1st, 2021: I made some changes, as I realized my effort to simplify some things made them sound just wrong. This problem might still haunt the text here and there, so if you read something where I sacrificed correctness in favor of simplicity, please call me out.

Rust BDD tests with Cucumber

Roger Torres (he/him/ele) — Wed, 04 Aug 2021 14:31:01 +0000

This post is out of date. The cucumber crate version used here is obsolete. Fortunately, the new version is not only better, but also easier to use and has excellent documentation, which you can find here.

Cucumber is a tool for behavior-driven development (BDD) that uses a language called Gherkin to specify test scenarios with a syntax that is very close to natural language, using key-words such as When and Then.

Here I will not explain the particularities of neither Cucumber nor Gherkin. My goal is to just show you how to use them with Rust. If you know nothing about them, I recommend you to:

Watch this soft introduction.
Refer to the documentation, specially the Gherkin reference guide.

That being said, the examples I am using are elementary, so you should have no problem following along even if you have never seen Cucumber and Gherkin before.

Sample project

The code used in this tutorial can be found here.

To focus on how to use Cucumber with Rust, I decided to code a dummy multiplication function, so you don't get distracted by the idiosyncrasies of a particular project.

First, create a library crate:



$ cargo new --lib bdd

Then, remove the mod tests from lib.rs and code this:



pub fn mult(a: i32, b: i32) -> i32 {
    a * b
}

And that's all for lib.rs.

Setting up the manifest

The manifest (Cargo.toml) require these entries:



[[test]]
name = "cucumber"
harness = false

[dependencies]
tokio = { version = "1.9.0", features = ["rt-multi-thread", "macros", "time"] }


[dev-dependencies]
cucumber_rust = "0.9"

In [[test]] we have:

name, which is the name of the .rs file where we will code the tests, cucumber.rs in this case.
harness is set to false, allowing us to provide our own main function to handle the test run. This main function will be placed inside the .rs file specified above.

In [dependencies] we have tokio, which is an async runtime required to work with cucumber-rust (even if you are not testing anything async). You may use another runtime.

And in [dev-dependencies] we have cucumber_rust, the star of this show.

Project structure

We have to create a couple of files:

tests/cucumber.rs, where we will have the main function that will run the tests.
features/operations.feature, where we will code our test scenario using Gherkin.

At the end, we have this:



bdd
├── features
│   └── operations.feature
├── src
│   └── lib.rs
├── tests
│   └── cucumber.rs
└── Cargo.toml

Coding the test with Gherkin

This is how the operation.feature file looks like:



Feature: Arithmetic operations

  # Let's start with addition. BTW, this is a comment.
  Scenario: User wants to multiply two numbers
    Given the numbers "2" and "3"
    When the User adds them
    Then the User gets 6 as result

Here we have a context set by Given, an event described by When and an expected result expressed by Then.

Although my purpose is not to teach Cucumber, I have to say that this is not a good BDD scenario. Not so much because it is dead simple, but because it is a unit test in disguise. I have, nevertheless, kept it here because it has the virtue of making things crystal clear regarding how we are going to interpret this scenario with Rust. For best practices, check this.

Handle the scenario with Rust

From now on, we stick with ~~the mighty Crabulon~~ Rust.

World object

The first thing we need is a World, an object that will hold the state of our test during execution.

Let's start coding cucumber.rs:



use cucumber_rust::{async_trait, Cucumber, World};
use std::convert::Infallible;

pub enum MyWorld {
    Init,
    Input(i32, i32),
    Result(i32),
    Error,
}

#[async_trait(?Send)]
impl World for MyWorld {
    type Error = Infallible;

    async fn new() -> Result<Self, Infallible> {
        Ok(Self::Init)
    }
}

For now, we created an enum named MyWorld that will be our "World object", holding the data between steps. Init is its initial value.

The MyWorld object implements the trait World provided by the cucumber_rust, which in turn gives us the methods to map the steps (Given, When, Then, etc.).

The Error type is a requirement from this trait, as is the attribute #[async_trait(?Send)].

Step builder

Now it is time to actually code the interpreter. I will explain the code block by block, so it might be useful to also have the complete code open, so you don't lose sight of the whole picture.

Given



mod test_steps {
    use crate::MyWorld;
    use cucumber_rust::Steps;
    use bdd::mult;

    pub fn steps() -> Steps<MyWorld> {
        let mut builder: Steps<MyWorld> = Steps::new();

        builder.given_regex(
            // This will match the "given" of multiplication
            r#"^the numbers "(\d)" and "(\d)"$"#,
            // and store the values inside context, 
            // which is a Vec<String>
            |_world, context| {
                // With regex we start from [1]
                let world = MyWorld::Input(
                    context.matches[1].parse::<i32>().unwrap(),
                    context.matches[2].parse::<i32>().unwrap(),
                );
                world
            }
        );

        // The rest of the code will go here.
    }
}

After declaring the mod, we created our builder, a Steps struct that will store our steps.

The crate cucumber_rust provides three variations for the main Gherkin prefixes (Given, When, Then):

The "normal" one that matches fixed values (e.g. when());
The regex version that parses the regex input (e.g. when_regex()).
The async version to handle async tests, something I am not covering here (e.g. when_async()).
The async+regex, which is a combination of the last two (e.g. when_regex_async()), also not covered here.

I am using given_regex() to parse the two numbers. Remember that in operations.feature I specified this:



Given the numbers "2" and "3"

When you call a step function such as given_regex() you get a closure containing the World object and a Context. The latter have a field called matches that is a Vec<String> containing the regex matches (if you're not using a _regex step, the Vector will be empty). In this case, as I am using regex, it has three values:

[0] has the entire match, the numbers "2" and "3" in this case.
[1] has the first group, 2 in this case.
[2] has the first group, 3 in this case.

This is the regex "normal" behavior. If you are not familiar with regex, this is a good intro (thank you YouTube for holding my watch history for so long).

With these values, I return my World object now set as Input.

Before we move to when, I have two quick remarks to make:

I am not checking the unrwap() because the regex is only catching numbers with (\d). Sometimes you might want to capture everything with something like (.*) and validate the content inside your code.
If you want to change your World object (for example, if it is a struct holding multiple values and/or states), just place mut before world in the closure, and you will get a mutable object.

When



builder.when(
    "the User multiply them", 
    |world, _context|{
        match world {
            MyWorld::Input(l, r) => MyWorld::Result(mult(l,r)),
            _ => MyWorld::Error,
        }
    }
);

This one is very straightforward. I use match to get the enum inner value, multiply both inputs and return the World object with a new value.

The function mult is ratter useless, but it has a role to play here: to show you how to import what we declared within the library crate.

Then




builder.then_regex(
    r#"^the User gets "(\d)" as result$"#, 
    |world, context|{
        match world {
            MyWorld::Result(x) => assert_eq!(x.to_string(), context.matches[1]),
            _ => panic!("Invalid world state"),
        };
        MyWorld::Init
    }
);

builder

Here I use regex again to compare the value that was calculated in the Then step with the value provided by Gherkin (which is, as I said, a very suspicious BDD scenario).

At the very end, I return builder.

The substitute `main` function

After the mod, we declare our main function:



#[tokio::main]
async fn main() {
    Cucumber::<MyWorld>::new()
        .features(&["./features"])
        .steps(test_steps::steps())
        .run_and_exit()
        .await
}

Here we are:

Creating a World object
Pointing to the .feature file containing the Gherkin test.
Adding the steps defined with cucumber_rust.
Running and exiting once it is done.
The await because cucumber_rust requires the whole thing to be async.

That's it! To test it, all you have to do is to run



$ cargo test

Rust will run the main function found in the file specified in the manifest: cucumber.rs. This is the result.

I recommend you to mess with the values and run the tests, so you can see how the errors are captured.

Much more can be done with cucumber_rust. I hope this tutorial helps you get started with it.

Cover image by Roman Fox.

First steps with Rust declarative macros!

Roger Torres (he/him/ele) — Mon, 26 Jul 2021 14:09:13 +0000

Macros are one of the ways to extend Rust syntax. As The Book calls it, “a way of writing code that writes other code”. Here, I will talk about declarative macros, or as it is also called, macros by example. Examples of declarative macros are vec!, println! and format!.

The macros I will not talk about are the procedural macros, but you can read about them here.

As usual, I am writing for beginners. If you want to jump to the next level, check:

The Book's section on macros.
The chapter about macros in Rust by Example
The Little Book of Rust Macros, which is the most complete material I found about the topic (the second chapter is specially amusing).

Why do I need macros?

The actual coding start in the next section.

Declarative macros (from now on, just “macros”) are not functions, but it would be silly to deny the resemblance. Like functions, we use them to perform actions that would otherwise require too many lines of code or quirky commands (I am thinking about vec! and println!, respectively). These are (two of) the reasons to use macros, but what about the reasons to create them?

Well, maybe you are developing a crate and want to offer this feature to the users, like warp did with path!. Or maybe you want to use a macro as a boilerplate, so you don't have to create several similar functions, as I did here. It might be also the case that you need something that cannot be delivered by usual Rust syntax, like a function with initial values or structurally different parameters (such as vec!, that allows calls like vec![2,2,2] or vec![2;3]—more on this later).

That being said, I believe that the best approach is to learn how to use them, try them a few times, and when the time comes when they might be useful, you will remember this alternative.

Declaring macros

This is how you declare a macro:

macro_rules! etwas {
    () => {}
}

You could call this macro with the commands etwas!(), etwas![] or etwas!{}. There's no way to force one of those. When we call a macro always using one or the other—like parenthesis in println!("text") or square-brackets in vec![]—it is just a convention of usage (a convention that we should keep).

But what is happening in this macro? Nothing. Let's add something to make it easier to visualize its structure:

macro_rules! double {
    ($value:expr) => { $value * 2 }
}

fn main() {
    println!("{}", double!(7));
}

The left side of => is the matcher, the rules that define what the macro can receive as input. The right side is the transcriber, the output processing.

Not very important, but both matcher and transcriber could be writen using either (), [] or {}.

The matching

This will become clear later, but let me tell you right the way: the matching resembles regex. You may ask for specific arguments, fixed values, define acceptable repetition, etc. If you are familiar with it, you should have no problems picking this up.

Let's go through the most important things you should know about the matching.

Variable argument

Variable arguments begin with $ (e.g., $value:expr). Their structure is: $ name : designator.

Both $ and : are fixed.
The name follows Rust variables convention. When used in the transcriber (see below), they will be called metavariables.
Designators are not variable types. You may think of them as “syntax categories”. Here, I will stick with expressions (expr), since Rust is “primarily an expression language”. A list of possible designators can be found here.

Note: There seems to be no consensus on the name "designator". The little book calls it "capture"; The Rust reference calls it "fragment-specifier"; and you will also find people referring them as "types". Just be aware of that when jumping from source to source. Here, I will stick with designator, as proposed in Rust by example.

Fixed arguments

No mystery here. Just add them without $. For example:

macro_rules! power {
    ($value:expr, squared) => { $value.pow(2) }
}

fn main() {
    println!("{}", power!(3_i32, squared));
}

I know there are things here that I have not explained yet. I will talk about them now.

Separator

Some designators require some specific follow up. Expressions require one of these: =>, , or ;. That is why I had to add a comma between $value:expr and the fixed-value squared. You will find a complete list of follow-ups here.

Multiple matching

What if we want our macro to not only calculate a number squared, but also a number cubed? We do this:

macro_rules! power {
    ($value:expr, squared) => { $value.pow(2_i32) }
    ($value:expr, cubed) => { $value.pow(3_i32) }
}

Multiple matching can be used to capture different levels of specificity. Usually, you will want to write the matching rules from the most-specific to the least-specific, so your call doesn't fall in the wrong matching. A more technical explanation can be found here.

Repetition

Most macros that we use allow for a flexible number of inputs. For example, we may call vec![2] or vec![1, 2, 3]. This is where the matching resembles Regex the most. Basically, we wrap the variable inside $() and follow up with a repetition operator:

* — indicates any number of repetitions.
+ — indicates any number, but at least one.
? — indicates an optional, with zero or one occurrence.

Let's say we want to add n numbers. We need at least two addends, so we will have a single first value, and one or more (+) second value. This is what such a matching would look like.

macro_rules! adder {
    ($left:expr, $($right:expr),+) => {}
}

fn main() {
    adder!(1, 2, 3, 4);
}

We will work on the transcriber latter.

Repetition separator

As you can see in the example above, I added a comma before the repetition operator +. That's how we add a separator for each repetition without a trailing separator. But what if we want a trailing separator? Or maybe we want it to be flexible, allowing the user to have a trailing separator or not? You may have any of the three possibilities like this:

macro_rules! no_trailing {
    ($($e:expr),*) => {}
}

macro_rules! with_trailing {
    ($($e:expr,)*) => {}
}

macro_rules! either {
    ($($e:expr),* $(,)*) => {}
}

fn main() {
    no_trailing!(1, 2, 3);
    with_trailing!(1, 2, 3,);
    either!(1, 2, 3);
    either!(1, 2, 3,);
}

Versatility

Unlike functions, you may pass rather different arguments to macros. Let's consider the vec! macro example. For that, I will omit the transcriber.

macro_rules! vec {
    () => {};
    ($elem:expr; $n:expr) => {};
    ($($x:expr),+ $(,)?) => {};
}

It deals with three kinds of calls:

vec![], which creates an empty Vector.
vec!["text"; 10], which repeats the first value ("text") n times, where n is the second value (10).
vec![1,2,3], which creates a vector with all the listed elements.

If you want to see the implementation of the vec! macro, check Jon's stream about macros.

The transcriber

The magic happens after the =>. Most of what you are going to do here is regular Rust, but I would like to bring your attention to some specificities.

Type

When I called the exponentiation macro power!, I did this:

power!(3_i32, squared);

I had to specify the type i32 because I used the pow() function, which cannot be called on ambiguous numeric type; and as we do not define types in macros, I had to let the compiler know this information somehow. This is something to be aware when dealing with macros. Of course, I could have forced it by declaring a variable and passing the metavariable value to it and thus fixing the variable type. However, to do such a thing, we need multiple statements.

Multiple statements

To have more than one line in your transcriber, you have to use double curly brackets:

macro_rules! etwas {
                             //v --- this one
    ($value:expr, squared) => {{ 
        let x: u32 = $value;
        x.pow(2)
    }}
   //^ --- and this one
};

Easy.

Using repetition

Let us finish our adder! macro.

macro_rules! adder {
    ($($right:expr),+) => {{
        let mut total: i32 = 0;
        $( 
            total += $right;
        )+
        total
    }};
}

fn main() {
    assert_eq!(adder!(1, 2, 3, 4), 10);
}

To handle repetition, all we have to do is to place the statement we want to repeat within $()+ (the repetition operator should match, that is why I am using + here as well).

But what if we have multiple repetitions? Consider the following code.

macro_rules! operations {
    (add $($addend:expr),+; mult $($multiplier:expr),+) => {{
        let mut sum = 0;
        $(
            sum += $addend;
         )*

         let mut product = 1;
         $(
              product *= $multiplier;
          )*

          println!("Sum: {} | Product: {}", sum, product);
    }} 
}

fn main() {
    operations!(add 1, 2, 3, 4; mult 2, 3, 10);
}

How does Rust know that it must repeat four times during the first repetition block and only three times in the second one? By context. It checks the variable that is being use and figure out what to do. Clever, huh?

Sure, you can make things harder to Rust. In fact, you may turn them indecipherable, like this:

macro_rules! operations {
    (add $($addend:expr),+; mult $($multiplier:expr),+) => {{
        let mut sum = 0;        
        let mut product = 1;

        $(
            sum += $addend;
            product *= $multiplier;
        )*

          println!("Sum: {} | Product: {}", sum, product);
    }} 
}

What does “clever Rust” do with something like this? Well, one of the things it does best: it gives you a clear compile error:

error: meta-variable 'addend' repeats 4 times, but 'multiplier' repeats 3 times
  --> src/main.rs:43:10
   |
43 |           $(
   |  __________^
44 | |             sum += $addend;
45 | |             product *= $multiplier;
46 | |         )*
   | |_________^

Neat! 🦀

Expand

As mentioned earlier, macros are syntax extensions, which means that Rust will turn them into regular Rust code. Sometimes, to understand what is going ~~wrong~~ on, it is very helpful to see how rust pull that transformation off. To do so, use the following command.

$ cargo rustc --profile=check -- -Zunstable-options --pretty=expanded

This command, however, is not only verbose, but it will also call for the nightly compiler. To avoid this and get the same result, you may install cargo-expand:

$ cargo install cargo-expand

Once it is installed, you just have to run the command cargo expand.

Note: Although you don't have to be using the nightly compiler, I guess (and you may call me on this) you got to have it installed. To do so, run the command rustup instal nightly.

Look at how the macro operations! is expanded.

fn main() {
    {
        let mut sum = 0;
        sum += 1;
        sum += 2;
        sum += 3;
        sum += 4;
        let mut product = 1;
        product *= 2;
        product *= 3;
        product *= 10;
        {
            ::std::io::_print(::core::fmt::Arguments::new_v1(
                &["Sum: ", " | Product: ", "\n"],
                &match (&sum, &product) {
                    (arg0, arg1) => [
                        ::core::fmt::ArgumentV1::new(arg0, ::core::fmt::Display::fmt),
                        ::core::fmt::ArgumentV1::new(arg1, ::core::fmt::Display::fmt),
                    ],
                },
            ));
        };
    };
}

As you can see, even println! was expanded.

Export and import

To use a macro outside the scope it was defined, you got to export it by using #[macro_export].

#[macro_export]
macro_rules! etwas {
    () => {};
}

You may also export a module of macros with #[macro_use].

#[macro_use]
mod inner {
    macro_rules! a {
        () => {};
    }

    macro_rules! b {
        () => {};
    }
}

a!();
b!();

To use a macro that a crate exports, you also use #[macro_use].

#[macro_use(lazy_static)] // Or #[macro_use] to import all macros.
extern crate lazy_static;

lazy_static!{}

The example above is from The Rust Reference.

And that's all for today. There is certainly more to cover, but I will leave you with the readings I recommended earlier.

Cover image by Thom Milkovic.

Smart Pointers in Rust: What, why and how?

Roger Torres (he/him/ele) — Mon, 19 Jul 2021 15:25:22 +0000

TL;DR: I will cover some of Rust's smart pointers: Box, Cell, RefCell, Rc, Arc, RwLock and Mutex.

It seems pretty obvious that smart pointers are pointers that are... smart. But what exactly does this "smart" means? When should we use them? How do they work?

These are the questions I will begin to answer here. And that is it: the beginning of an answer, nothing more. I expect that this article will give you a "background of understanding" (something like "familiarity" with the concept) that will help you accommodate a real understanding of the topic, which will come from reading the official documentation and, of course, practice.

If you are already familiar with it, you may use this text as a list of relevant reading. Look for the "useful links" at the beginning of each section.

Index:

Box
Cell
RefCell
Rc
Arc
RwLock
Mutex

Smart points in general

As explained in The Book, pointers are variables containing an address that "points at" some other data. The usual pointer in Rust is the reference (&). Smart pointers are pointers that "have additional metadata and capabilities", e.g., they may count how many times the value was borrowed, provide methods to manage read and write locks, etc.

Technically speaking, String and Vec are also smart pointers, but I will not cover them here as they are quite common and are usually thought of as types rather than pointers.

Also note that, from this list, only Arc, RwLock, and Mutex are thread-safe.

Box<T>

Useful links: The Book; Documentation; Box, stack and heap.

What?

Box<T> allows you to store T on the heap. So, if you have, say, a u64 that would be stored on the stack, Box<u64> will store it on the heap.

If you are not comfortable with the concepts of stack and heap, read this.

Why?

Values stored on the stack cannot grow, as Rust needs to know its size at compile time. The best example of how this may affect your programming that I know is in The Book: recursion. Consider the code below (and its comments).

// This does not compile. The List contains itself,  
// being recursive and therefore having an infinite size.
enum List { 
   Cons(i32, List),
   Nil,
}
// This does compile because the size of a pointer
// does not change according to the size of the pointed value.
enum List { 
   Cons(i32, Box<List>),
   Nil,
}

Be sure to read this section on The Book's to understand the details.

On a more general note, Box is useful when your value is too big to be kept on the stack or when you need to own it.

How?

To get the value T inside Box<T> you just have to deref it.

let boxed = Box::new(11);
assert_eq!(*boxed, 11)

Cell<T>

Useful links: Module documentation; Pointer documentation.

What?

Cell<T> gives a shared reference to T while allowing you to change T. This is one of the "shareable mutable containers" provided by the module std::cell.

Why?

In Rust, shared references are immutable. This guarantees that when you access the inner value you are not getting something different than expected, as well as assure that you are not trying to access the value after it was freed (which is a big chunk of those 70% of security bugs that are memory safety issues).

How?

What Cell<T> does is provide functions that control our access to T. You can find them all here, but for our explanation we just need two: get() and set().

Basically, Cell<T> allows you to freely change T with T.set() because when you use T.get(), you retrieve Copy of T, not a reference. That way, even if you change T, the copied value you got with get() will remain the same, and if you destroy T, no pointer will dangle.

One last note is that T has to implement Copy as well.

use std::cell::Cell;
let container = Cell::new(11);
let eleven = container.get();
{
    container.set(12);
}
let twelve = container.get();
assert_eq!(eleven, 11);
assert_eq!(twelve, 12);

RefCell<T>

Useful links: The Book; Documentation.

What?

RefCell<T> also gives shared reference to T, but while Cell is statically checked (Rust checks it at compile time), RefCell<T> is dynamically checked (Rust checks it at run time).

Why?

Because Cell operates with copies, you should restrict yourself to using small values with it, which means that you need references once again, which leads us back to the problem that Cell solved.

The way RefCell deals with it is by keeping track of who is reading and who writing T. That's why RefCell<T> is dynamically checked: because you are going to code this check. But fear not, Rust will still make sure you don't mess up at compile time.

How?

RefCell<T> has methods that borrow either a mutable or immutable reference to T; methods that will not allow you to do so if this action would be unsafe. As with Cell, there are several methods in RefCell, but these two are enough to illustrate the concept: borrow(), which gets an immutable reference; and borrow_mut(), which gets a mutable reference. The logic used by RefCell goes something like this:

If there is no reference (either mutable or immutable) to T, you may get either a mutable or immutable reference to it;
If there is already a mutable reference to T, you may get nothing and got to wait until this reference is dropped;
If there are one or more immutable references to T, you may get an immutable reference to it.

As you can see, there is no way to get both mutable and immutable references to T at the same time.

Remember: this is not thread-safe. When I say "no way", I am talking about a single thread.

Another way to think about is:

Immutable references are shared references;
Mutable references are exclusive references.

It is worth to say that the functions mentioned above have variants that do not panic, but returns Result instead: try_borrow() and try_borrow_mut();

use std::cell::RefCell;
let container = RefCell::new(11);
{
    let _c = container.borrow();
    // You may borrow as immutable as many times as you want,...
    assert!(container.try_borrow().is_ok());
    // ...but cannot borrow as mutable because 
    // it is already borrowed as immutable.
    assert!(container.try_borrow_mut().is_err());
} 
// After the first borrow as mutable...
let _c = container.borrow_mut();
// ...you cannot borrow in any way.
assert!(container.try_borrow().is_err());
assert!(container.try_borrow_mut().is_err());

Rc<T>

Useful links: The Book; Module documentation; Pointer documentation; Rust by example.

What?

I will quote the documentation on this one:

The type Rc<T> provides shared ownership of a value of type T, allocated on the heap. Invoking clone on Rc produces a new pointer to the same allocation on the heap. When the last Rc pointer to a given allocation is destroyed, the value stored in that allocation (often referred to as “inner value”) is also dropped.

So, like a Box<T>, Rc<T> allocates T on the heap. The difference is that cloning Box<T> will give you another T inside another Box while cloning Rc<T> gives you another Rc to the same T.

Another important remark is that we don't have interior mutability in Rc as we had in Cell or RefCell.

Why?

You want to have shared access to some value (without making copies of it), but you want to let it go once it is no longer used, i.e., when there is no reference to it.

As there is no interior mutability in Rc, it is common to use it alongside Cell or RefCell, for example, Rc<Cell<T>>.

How?

With Rc<T>, you are using the clone() method. Behind the scene, it will count the number of references you have and, when it goes to zero, it drops T.

use std::rc::Rc;
let mut c = Rc::new(11);

{ 
    // After borrwing as immutable...
    let _first = c.clone();

    // ...you can no longer borrow as mutable,...
    assert_eq!(Rc::get_mut(&mut c), None);

    // ...but can still borrow as immutable.
    let _second = c.clone();

    // Here we have 3 pointer ("c", "_first" and "_second").
    assert_eq!(Rc::strong_count(&c), 3);
}

// After we drop the last two, we remain only with "c" itself.
assert_eq!(Rc::strong_count(&c), 1);

// And now we can borrow it as mutable.
let z = Rc::get_mut(&mut c).unwrap();
*z += 1;
assert_eq!(*c, 12);

Arc<T>

Useful links: Documentation; Rust by example.

What?

Arc is the thread-safe version of Rc, as its counter is managed through atomic operations.

Why?

I think the reason why you would use Arc instead of Rc is clear (thread-safety), so the pertinent question becomes: why not just use Arc every time? The answer is that these extra controls provided by Arc come with an overhead cost.

How?

Just like Rc, with Arc<T> you will be using clone() to get a pointer to the same value T, which will be destroyed once the last pointer is dropped.

use std::sync::Arc;
use std::thread;

let val = Arc::new(0);
for i in 0..10 {
    let val = Arc::clone(&val);

    // You could not do this with "Rc"
    thread::spawn(move || {
        println!(
            "Value: {:?} / Active pointers: {}", 
            *val+i, 
            Arc::strong_count(&val)
        );
    });
}

RwLock<T>

Useful link: Documentation.

RwLock is also provided by the parking_lot crate.

What?

As a reader-writer lock, RwLock<T> will only give access to T once you are holding one of the locks: read or write, which are given following these rules:

To read: If you want a lock to read, you may get it as long as no writer is holding the lock; otherwise, you have to wait until it is dropped;
To write: If you want a lock to write, you may get as long as no one, reader or writer, is holding the lock; otherwise, you have to wait until they are dropped;

Why?

RwLock allows you to read and write the same data from multiple threads. Different from Mutex (see below), it distinguishes the kind of lock, so you may have several read locks as far as you do not have any write lock.

How?

When you want to read a RwLock, you got to use the function read()—or try_read()—that will return a LockResult that contains a RwLockReadGuard. If it is successful, you will be able to access the value inside RwLockReadGuard by using deref. If a writer is holding the lock, the thread will be blocked until it can take hold of the lock.

Something similar happens when you try to use write()—or try_write(). The difference is that it will not only wait for a writer holding the lock, but for any reader holding the lock as well.

use std::sync::RwLock;
let lock = RwLock::new(11);

{
    let _r1 = lock.read().unwrap();
    // You may pile as many read locks as you want.
    assert!(lock.try_read().is_ok());
    // But you cannot write.
    assert!(lock.try_write().is_err());
    // Note that if you use "write()" instead of "try_write()"
    // it will wait until all the other locks are released
    // (in this case, never).
} 

// If you grab the write lock, you may easily change it
let mut l = lock.write().unwrap();
*l += 1;
assert_eq!(*l, 12);

If some thread holding the lock panics, further attempts to get the lock will return a PoisonError, which means that from then on every attempt to read RwLock will return the same PoisonError. You may recover from a poisoned RwLock using into_inner().

use std::sync::{Arc, RwLock};
use std::thread;

let lock = Arc::new(RwLock::new(11));
let c_lock = Arc::clone(&lock);

let _ = thread::spawn(move || {
    let _lock = c_lock.write().unwrap();
    panic!(); // the lock gets poisoned
}).join();

let read = match lock.read(){
    Ok(l) => *l,
    Err(poisoned) => {
        let r = poisoned.into_inner();
        *r + 1
    }
};

// It will be 12 because it was recovered from the poisoned lock
assert_eq!(read,12);

Mutex<T>

Useful links: The Book; Documentation.

Mutex is also provided by the parking_lot crate.

What?

Mutex is similar to RwLock, but it only allows one lock-holder, no matter if it is a reader or a writer.

Why?

One reason to prefer Mutex over RwLock is that RwLock may lead to writer starvation (when the readers pile on and the writer never gets a chance to take the lock, waiting forever), something the does not happen with Mutex.

Of course, we are diving into deeper seas here, so the real-life choice falls on more advanced considerations, such as how many readers you expect at the same time, how the operating system implements the locks, and so on...

How?

Mutex and RwLock work in a similar fashion, the difference is that, because Mutex does not differentiate between readers and writers, you just use lock() or try_lock to get the MutexGuard. The poisoning logic also happens here.

use std::sync::Mutex;
let guard = Mutex::new(11);

let mut lock = guard.lock().unwrap();
// It does not matter if you are locking the Mutex to read or write,
// you can only lock it once.
assert!(guard.try_lock().is_err());

// You may change it just like you did with RwLock
*lock += 1;
assert_eq!(*lock, 12);

You can deal with poisoned Mutex in the same way as you deal with poisoned RwLock.

Thank you for reading!

First steps with Docker + Rust

Roger Torres (he/him/ele) — Mon, 12 Jul 2021 11:34:12 +0000

TL;DR: We are going to install Docker and create five different containers for a Rust program, each one a little more complex than the other.

Hi! In this post, I will show you how to dockerize your Rust program. We will begin with a very simple container and will build from that to more sophisticated ones, where we take care of compile-time and image size.

I thought about just jumping to the last one instead of granularizing so much, but I prefer to be clear rather than concise with these #beginners posts. Feel free to jump whatever is too obvious for you and ask me what remained too obscure.

Install Docker

First of all, install Docker. The getting started guide has instructions for Mac OS, Linux and Windows.

Once installed, run and test it by issuing this command:



$ docker run hello-world

It should pull the hello-world image from the Docker Hub and return a text block explaining in detail what happened behind the scene.

Glossary

If you are completely new to Docker, it will help to have a clear understanding of what I mean when I use the following terms:

Docker: The application we just installed (or, to be more precise, the Docker daemon we use to deal with our images and containers);
Dockerfile: A file named Dockerfile that contains the commands that Docker will run to build the image. In a Rust project, it lies alongside the manifest, that is, the Cargo.toml file;
Image: When we run a command build we create an image that contains everything we specified in the Dockerfile. Running an image result in a container. E.g., if our Dockerfile has instructions to build and run a Web Server, the image will contain the program (that is, the Web Server itself) which will be accessible when we run the image, thus creating the container;
Base image: It is an image that we use to base ours. E.g., for us, Rust will be a base image (which we don't build manually; we download it ready to use);
Container: The result of running an image. The container is a process running on your computer that contains everything that is needed to run the application. For a better understanding, I recommend this presentation by Liz Rice.

The sample project

I am going to use a REST API that I built using warp. If you want to build it yourself, you may check this guide; if not, feel free download it here or even use your own project; I believe you will have no problem mapping the commands.

There are a few differences between the code you'll find in the guide and the one I am using here:

I am now using IP 0.0.0.0 instead of 127.0.0.1. I changed this so I don't have to tell Docker which IP to bind;

The old version has two crates: binary (main.rs) and library (lib.rs). That made things harder for Docker (and would make my explanations here too complex) so I just maintained the binary crate and moved the library crate content to a module.

The first `Dockerfile`

This and all other files are available here. You will identify them by their numbers.

I will start with a very simple version and improve upon it a few times. Here I am working with the project I mentioned above, named holodeck (so that is the name you will have to change if you're using your project):



# 1. This tells docker to use the Rust official image
FROM rust:1.49

# 2. Copy the files in your machine to the Docker image
COPY ./ ./

# Build your program for release
RUN cargo build --release

# Run the binary
CMD ["./target/release/holodeck"]

With this, I just run the command below where the Dockerfile is.



$ docker build -t holodeck .

This will create the image. To see that, you may either look at your Docker app or use the command docker images.



$ docker images
REPOSITORY  TAG       IMAGE ID       CREATED          SIZE
holodeck    latest    aad6ff7c3b4d   47 seconds ago   2.42GB

To run it, all we have to do is to issue a command like this:



$ docker run -p 8080:3030 --rm --name holodeck1 holodeck
Warp 6, Engage!

Let me expand on the parameters:

-p maps the port, so what is 3030 inside Docker (the port our warp server is using) will be accessible through 8080 outside, i.e., your machine (if you don't do this, Docker will map a random port);
--rm is here to remove the container after we close it (to visualize this, you may run without --rm and then run docker ps -a to list all the containers and then use docker rm containername to remove it).

Now it is possible to test it in localhost:8080.



$ curl --location --request POST 'localhost:8080/holodeck' \
--header 'Content-Type: application/json' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "id": 2,
    "name": "Bride Of Chaotica!"
}'

Simulation #1 created.

To stop the container:



$ docker stop holodeck1
holodeck1

If you ran your image like me, your terminal will be frozen. To avoid this, run in detached mode, by just adding the parameter -d:



$ docker run -dp 8080:3030 --rm --name holodeck1 holodeck

Great! This is fine... Until you have to build again and again (and maybe even again because you're writing a guide like this and have to tweak things often). Why is this a problem? Compile time. Every time we run docker build, Rust does the entire building process all over again; and the fact we're building for release just makes it worse.

Let's fix it.

The second `Dockerfile`

This is a more elaborated alternative:



# Rust as the base image
FROM rust:1.49

# 1. Create a new empty shell project
RUN USER=root cargo new --bin holodeck
WORKDIR /holodeck

# 2. Copy our manifests
COPY ./Cargo.lock ./Cargo.lock
COPY ./Cargo.toml ./Cargo.toml

# 3. Build only the dependencies to cache them
RUN cargo build --release
RUN rm src/*.rs

# 4. Now that the dependency is built, copy your source code
COPY ./src ./src

# 5. Build for release.
RUN rm ./target/release/deps/holodeck*
RUN cargo install --path .

CMD ["holodeck"]

I am using install, but this is the same as build, except that it places the binary on the indicated path, in this case, the WORKDIR.

This is how we avoid the compiler aeon. By building only the dependencies attached to a new program (steps 1 through 3) and then, with a new command inside the Dockerfile, build our program (commands 4 and 5), we stop Docker from ignoring the cache. Why does it work? Because every command in our Dockerfile creates a new layer, which is a modification to the image. When we run docker build, only the modified layers are updated, the rest is retrieved from the local cache. To put it in practical terms, as long as we don't change the manifest, the dependencies will not have to be rebuilt.

In my case, after a first build that took 323.6 seconds, the second one (where I just changed the main.rs) took only 33.9 seconds. Great!

However... there is another problem: image size. For example, mine has 1.65 GB, which better than the very first one, which had 2.42 GB, but still too large. Let's put a little fixin' on it.

The third `Dockerfile`

If you visited Isaac's post, you'll see he managed this by "chaining" builds by using two base images. He did everything after a first FROM rust, which is the build where everything is built, and then called another FROM rust, copying only the required files from the first build. That allows the final image to retain only these last copied files, therefore decreasing the image size.

This is how we do it:



FROM rust:1.49 as build

# create a new empty shell project
RUN USER=root cargo new --bin holodeck
WORKDIR /holodeck

# copy over your manifests
COPY ./Cargo.lock ./Cargo.lock
COPY ./Cargo.toml ./Cargo.toml

# this build step will cache your dependencies
RUN cargo build --release
RUN rm src/*.rs

# copy your source tree
COPY ./src ./src

# build for release
RUN rm ./target/release/deps/holodeck*
RUN cargo build --release

# our final base
FROM rust:1.49

# copy the build artifact from the build stage
COPY --from=build /holodeck/target/release/holodeck .

# set the startup command to run your binary
CMD ["./holodeck"]

That reduced the image size from 1.66 GB to 1.26 GB. But I promised five versions of the Dockerfile, so you already know we can do better.

The fourth `Dockerfile`

What we'll now do is to change the Rust image tag we are using. A tag is another way to say an "image variant", that is, an alternative image designed to meet certain goals. So, if we want a space-saving image, it is a tag that we're looking for.

And all that means just replacing the second FROM with:



FROM rust:1.49-slim-buster

This image tag uses Rust built upon Debian tag called buster slim to create a more compact image. Now, instead of 1.26 GB, we have 642.3 MB.

Some people might wonder why I didn't use Alpine. Well, I did, and buster-slim was 10MB smaller. But the real reason why I avoided Alpine will be clear in the next step.

The fifth (and final) `Dockerfile`

I don't think this hunt for the minimal image size is always needed; you got to have a reason.

As I do have reason (show it to you), I will do one final change that will give a really small Docker image. For this, we need an image that has no Rust whatsoever, only the binary that will be executed (that's one of the beauties of a compiled language after all).

To achieve such a thing, we use a Rust base image to build our binary and just move the binary to a Linux image without Rust. And to do that, we will use debian:buster-slim itself.

Again, regarding Alpine. I didn't use it here either for two reasons:

To run a Rust code in Alpine it has to be compiled with MUSL, which adds an extra layer of complexity to this beginner-intended post;

I am not sure that MUSL is a good option.

The result: 75.39 MB. That's a long way from 2.42 GB.
Let's make an overall comparison:

That's it! Thank you for reading so far. Bye!

Cover image by Aron Yigin

Getting started with GitHub Actions for Rust

Roger Torres (he/him/ele) — Wed, 07 Jul 2021 14:14:45 +0000

TL;DR: Create an Action on GitHub so your code gets built and tested after every push, and do all this with nothing more than a "next-next-finish".

You never used GitHub Actions and you want do do it with your Rust project; if that's the case, this might help you.

What is and Why GitHub Actions?

GitHub Actions automatize software workflow, which per se is not CI/CD (Continuous Integration/Continuous Delivery), but is used in this method.

So, which workflow are we going to automatize? Test and build. In other words, this:



$ cargo test

$ cargo build

How to do it

First, open your GitHub repository and go to Actions.

The example is from this repository that I wrote about here.

This will lead you to this GitHub proposal, which will do exactly what we're set to do here, i.e., build and test:

GitHub will preview the .yml file it will create. For this scope, you don't have to change anything (except maybe the name from "Rust" to something else — I used "test").

Commit the change clicking on the button that will appear on the right and the file will be created:

Now, if you go to Actions again, you will see GitHub creating your .yml file. For me, it took around 2 minutes.

And that's it. From now on, every time you push against the repository, GitHub will run the tests for you.

To see the result of the commit above, see the Build log here.

And that's all for today.

See ya 🙃

Cover image by Susan Q Yin

REST API Wrapper with Rust

Roger Torres (he/him/ele) — Mon, 05 Jul 2021 16:38:25 +0000

TL;DR: This is a post for beginners where I show:

How I structured the files in the project for better documentation: every file in the same folder;

How I coded a "query builder" to handle different calls without repeating code: using a generic parameter;

How I coded test cases within the documentation: nothing special, just remarks on async tests;

How I created High Order Functions (HOF) to emulate that behaviour we have in Rust's Option and Iterator, for example: created a struct with functions that return the struct itself;

How I used serde to deserialize the Json: nothing special, just some extra remarks;

How I tested it as if it were a crate without exporting it to crates.io: just add the lib.rs in the dependencies of the new project.

I build a wrapper for the Magic: The Gathering API (an SDK, by their own terms). And I did so because I wanted a personal project that offered the following possibilities:

Use the reqwest crate;
Code something for other coders (e.g.: allowing them to use HOFs, such as cards.filter().types("creature").colors("red").name("dragon")...);
To document it as a crate, including tests in the documentation;
Implement Github Actions (Edit: I did this later, you can find it here).

The reason why I chose the Magic: The Gathering (MTG) API (besides being a MTG nerd) is because it is a very simple API (it has only GET methods).

This is not a tutorial, I will just highlight the interesting choices this endeavour led me to. Also, this is for beginners; I highly doubt I will say anything new to someone who had carefully read The Book and Rust by example and toyed with async a little bit (although we never know).

The result can be found here.

The project structure

I had two choices:

I chose the left one for two reasons:

It allows the person using the crate to type use mtgsdk instead of use mtgsdk::mtgsdk
This way the documentation shows everything on the first page. Had I went for the option on the right, the docs would only show the module mtgsdk, which I found is not how the cool kids do it.

How it is (left option):

If you want to see for yourself, fork/download the repository and type cargo doc --open

How it would be (right option):

Maybe the first image and Rust by example is enough to show how each way of doing this is carried out; however, for the sake of clarity, I will say this: If you want the left one, all you have to do is to declare the mods in your lib.rs. Otherwise, you have to create a folder with your single module name, create a mod.rs file in it and use the mod and pub mod inside it, declaring only the folder name within lib.rs (in this case, lib.rs would only have pub mod mtgsdk;.

Query builder

As I said, this API only has GET methods, and there's not much to talk about how reqwest handles it, for it is pretty much just passing a URL as you would do in a curl.

I am not saying that this is all that reqwest does; it is not. I am saying that for this API we don't actually need anything else that accessing the URL and parsing the Json (more on this later).

However, instead of repeating the reqwest::get(url) inside every module, I created a query builder that receives an url and returns a Result<T, StatusCode> where T is a struct containing the data for the various calls (cards, formats, etc.).

Besides allowing me to maintain the usage of reqwest in a single spot, it also allowed me to handle the errors and just send StatusCode, so the developer using this crate would easily handle the errors. Here is the code with some additional comments.

async fn build<T>(url: String) -> Result<T, StatusCode>
where
    //This is a requirement for Serde; I will talk about it below.
    T: DeserializeOwned,
{
    let response = reqwest::get(url).await;

    // I am using match instead of "if let" or "?" 
    // to make what's happening here crystal clear
    match &response {
        Ok(r) => {
            if r.status() != StatusCode::OK {
                return Err(r.status());
            }
        }
        Err(e) => {
            if e.is_status() {
                return Err(e.status().unwrap());
            } else {
                return Err(StatusCode::BAD_REQUEST);
            }
        }
    }

    // This is where de magic (and most problems) occur.
    // Again, more on this later.
    let content = response.unwrap().json::<T>().await;

    match content {
        Ok(s) => Ok(s),
        Err(e) => {
            println!("{:?}", e);
            Err(StatusCode::BAD_REQUEST)
        }
    }
}

Pretty straightforward: the functions calling the build() function will tell which type T corresponds to, a type that will be a struct with the Deserialize trait so that reqwest's json() can do the heavy lifting for us.

Documentation tests

The documentation section in the Rust Book is pretty good. Besides reading that, I only checked some examples of how documentation is managed in the crates I use.

What I want to highlight is the insertion of tests within the docs:

That this test will be executed is something that The Book talks about, so I will not stress about it. What was specific for me is that I was testing async calls, which required two minor tweaks:

It has to be in an async block;
I could not return anything (so no await?, because it returns the error).

High Order Functions

I will not lecture about HOF, let alone explain anything about functional programming. The reason I ended up with this is because, instead of something like this Builder Pattern (this is from another wrapper for the same API)...

let mut get_cards_request = api.cards().all_filtered(
    CardFilter::builder()
        .game_format(GameFormat::Standard)
        .cardtypes_or(&[CardType::Instant, CardType::Sorcery])
        .converted_mana_cost(2)
        .rarities(&[CardRarity::Rare, CardRarity::MythicRare])
        .build(),
    );

let mut cards: Vec<CardDetail> = Vec::new();
loop {
    let response = get_cards_request.next_page().await?
    let cards = response.content;
    if cards.is_empty() {
        break;
    }
    filtered_cards.extend(cards);
}
println!("Filtered Cards: {:?}", filtered_cards);

...I wanted something like this:

let response = cards::filter()
    .game_format("standard")
    .type_field("instant|sorcery")
    .cmc(2)
    .rarity("rare|mythic")
    .all()
    .await;

println!("Filtered cards: {:?}", response.unwrap());

Why? Because as a developer I love how Option and Iterator, as well as crates such as warp, implement this, giving Rust its "functional flavour".

How to do it

The function filter() returns a struct called Where that has a vector where I keep all the filters that are going to be added.

pub struct Where<'a> {
    query: Vec<(&'a str, String)>,
}

pub fn filter<'a>() -> Where<'a> {
    Where { query: Vec::new() }
}

So, when I do something like response = mtgsdk::card::filter(), the variable response is a Where struct, and that allows me to call any function implemented inside Where, e.g.:

impl<'a> Where<'a> {
    pub fn game_format(mut self, input: &'a str) -> Self {
        self.query.push(("gameFormat", String::from(input)));
        self
    }
}

So basically, when I called filter() and then added the functions game_format(), type_field(), cmc() and rarity() I was doing this:

Created a Where struct with filter()
Called game_format() implemented inside Where, which returned the same Where
Called type_field() from the Where returned by game_format()
Called cmc() from the Where returned by type_field()
Called rarity() from the Where returned by cmc()
Called all() from the Where returned by rarity() which finally returned the vector of cards:

pub async fn all(mut self) -> Result<Vec<Card>, StatusCode> {
    let val = self.query.remove(0);
    let mut filter = format!("?{}={}", val.0, val.1);

    for (k, v) in self.query.into_iter() {
        filter = format!("{}&{}={}", filter, k, v);
    }

    let cards: Result<RootAll, StatusCode> = query_builder::filter("cards", &filter).await;

    match cards {
        Ok(t) => Ok(t.cards),
        Err(e) => Err(e),
    }
}

And that's it.

Deserialize Json

As promised.

#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Set {
    pub code: String,
    pub name: String,
    #[serde(rename = "type")]
    pub type_field: String,
    #[serde(default)]
    pub booster: Vec<Booster>,
    pub release_date: String,
    pub block: Option<String>,
    pub online_only: Option<bool>,
    pub gatherer_code: Option<String>,
    pub old_code: Option<String>,
    pub magic_cards_info_code: Option<String>,
    pub border: Option<String>,
    pub expansion: Option<String>,
    pub mkm_name: Option<String>,
    pub mkm_id: Option<u32>,
}

Few things I want to mention:

The transform tool helps quite a bit;
In case #[serde(rename_all = "camelCase")] is not sufficiently self-explanatory, it will allow a struct field like release_date to receive data from a field that the API calls releaseDate;
There are two ways to handle optional fields:
- Using Option, which I preferred because in this case the developer using the crate will have absolute certainty if the field wasn't (the Option will be None) sent or if it was just empty (it will me Some)
- Using #[serde(default)], which I used for the mandatory fields because in these cases there's no doubt that the API sent them.

Testing "as if" it was a crate

I wanted to import it in a new project, but I didn't wanted to send it to crates.io. How to do it? Like this:

In the new project's Cargo.toml I added this:

[dependencies]
mtgsdk = { path = "../mtgsdk" }
tokio = { version = "1", features = ["full"] }

And that's all. In my main.rs I just used it as if it was a crate.

use mtgsdk::cards;

#[tokio::main]
async fn main() {
    let result = cards::find(46012).await;

    if let Ok(card) = result{
        println!("{}", card.name)
    };
}

This might be helpful if you want to be truthful to what The Book calls integration testing.

See ya 🙃

Cover image by Wayne Low

Teaching as a learning method

Roger Torres (he/him/ele) — Wed, 28 Apr 2021 12:19:09 +0000

Hello everyone!

TL;DR: I am sharing the experience of using "teaching" as a "learning" method that can complement the usual approaches of "reading stuff" and "building projects".

A few weeks ago I wrote a 5-part series about using Rust + Warp to create a REST API.

During the process of writing it, I made this comment in a post about The Benefits of Project-Base Learning:

Roger Torres (he/him/ele) • Apr 11 '21

I think the decision is heavily dependent on what you already know. Someone new to programming or new to a language/technology that is wildly different from their previous experiences will have a hard time navigating through the sources required to build a project. There's a minimum required to pull that off, and I think the books, tutorials, streams, and so on are more helpful to reach this threshold. That being said, once you are past this point, I totally agree that building projects is a magnificent approach. And I would also add something else: teaching is an astonishing way to learn things, especially if you wanna go deep.

The comment was a summary of what was going on in my head during the writing of the series. Today, I decided to expand on that.

In my previous experience as an SAP Analyst, I realized there were some complicated things (like Brazilian Taxes—they're a nightmare) that I would not fully understand even after delivering several successful projects in the subject matter. Nothing strange so far. However, things became weird when clients hired me to teach it to their internal team (correctly assuming I knew about it since I was actually doing it). After going through a quick impostor syndrome crisis, I would study the topic for a few nights (because these calls were usually from one week to another) and then something magical happened: things that I used to do without knowing why I was doing them became crystal clear. Surely the previous experience helped. More than that, they were crucial because not only I had fun examples to give (always necessary to break the ice) but also because I knew that what I was saying tied to reality. In short, although I just had learned the "whys", I was already confident regarding the "hows".

Nonetheless, there was something certainly going on regarding the way I studied while preparing for these courses and I kept curious about it, wondering if I could isolate the method.

I kept this in the back of my head but wasn't until I wrote the series of posts I mentioned earlier that I was able to engage in this process of explaining something to someone while also engaging in this self-reflexive effort of paying attention to the process itself, so I could identify why it worked.

Here's what I found out.

First. I wasn't a good note-taker. I'm still not very good, but my MA in Philosophy (don't ask me why an IT guy is doing such a thing) is forcing me to get better. However, when I have to teach/post, I am forced to perform one of the most crucial steps when it comes to taking notes the right way: writing the concepts with my own words in a way it would be intelligible to someone else if I explained it to them. In other words:

It is not enough to just read, you have to write what you read;
It is not enough to just write, you must use your own words;
It is not enough to just write using your own words, it must be intelligible to other people.

Related tip: If you want to get better at taking notes, google the zettelkasten method. Even if it is not for you, it might give you some ideas.

Second. Many times while writing down what I would say in the classroom (or writing a blog post here or somewhere else), I realized I would anticipate questions that someone could ask me, even though I knew they probably wouldn't; and by doing so I was pushing myself to go deeper while looking for these answers.

It's just like that old anecdote: when you are searching for something, you stop looking as soon as you find it. And that's what I did in the projects I worked on: fixed the problem and moved on. In this scenario, however, by not knowing which specific question would be asked, I dove deeper.

Third. The ideal scenario is having all the opportunities: time to read or watch good content, real projects to work on, and a way to share it with someone; but this is not always feasible. If you don't have a project that allows you to push yourself to learn (either because you lack ideas, enthusiasm, whatever) teaching may be a good alternative. Don't get me wrong, this is not a replacement, it is an option.

Fourth. This one might be counterintuitive, especially for the workaholic folks: sometimes not having a deadline might help. If you are already motivated, the opportunity to stretch some goals, to ask one more question, to try one more thing while everything is still fresh in your head allows you to boldly go where no one has gone before.

I don't think that any teaching experience (whatever it may be, writing a blog post, showing a friend how to do something, taking notes for your future self...) can replace other methods, but it is for sure a method worth considering.

Just be careful: when you tell someone you will show them how to do something, people will trust that your content is valid, that what you're saying comes from good sources and/or experience, that your code actually works, and that you spent the time doublechecking the details and so on. That being said, if you are into this to learn (I am obviously not considering those who master what they're sharing in every way, that is, both the whys and the hows), you are not likely to ignore these steps, since you are not only the one teaching but also the first one learning.

But what about you? What do you think?

Cover image by heylagostechie

REST API with Rust + Warp 5: Beyond test utilities

Roger Torres (he/him/ele) — Wed, 14 Apr 2021 12:21:44 +0000

Last part, folks!

Time to say goodbye. Warp 5.

The code for this part is available here.

Warp's test utilities are great, but I didn't want to end this series without actually being able to serve what has been built and to curl against it.

To do so, I first created a /src/bin/main.rs file. Then, I changed Cargo.toml to add this:

[[bin]]
name = "holodeck"
path = "src/bin/main.rs"

[lib]
name = "holodeck"
path = "src/lib.rs"

As I mentioned at the beginning, it is easier to find examples of code serving and consuming the API than testing it using the test utilities; so I'm gonna fly over it.

My main.rs ended up like this:

// I made "models" and "filters" public in lib.rs
use holodeck::{ models, filters };
use warp::Filter;

#[tokio::main]
async fn main() {

    let db = models::new_db();
    let routes = filters::list_sims(db.clone())
        .or(filters::post_sim(db.clone()))
        .or(filters::update_sim(db.clone()))
        .or(filters::delete_sim(db));

    warp::serve(routes).run(([127, 0, 0, 1], 3030)).await;
}

And that's all! I just had to put the filters together using or and serve. The clones work because we're cloning an Arc, which doesn't create a new copy, just a new reference to our Mutex.

Below you will find some some curls I made.

POST

curl --location --request POST 'localhost:3030/holodeck' \
--header 'Content-Type: application/json' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "id": 1,
    "name": "The Big Goodbye"
}'

curl --location --request POST 'localhost:3030/holodeck' \
--header 'Content-Type: application/json' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "id": 2,
    "name": "Bride Of Chaotica!"
}'

PUT

curl --location --request PUT 'localhost:3030/holodeck/3' \
--header 'Content-Type: application/json' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "name": "A Fistful Of Datas"
}'

curl --location --request PUT 'localhost:3030/holodeck/3' \
--header 'Content-Type: application/json' \
--header 'Content-Type: text/plain' \
--data-raw '{
    "name": "A Fistful Of La Forges"
}'

GET

curl --location --request GET 'localhost:3030/holodeck'

curl --location --request GET 'localhost:3030/holodeck/2'

DELETE

curl --location --request DELETE 'localhost:3030/holodeck/1'

The good thing about using warp's test utilities is that when the time comes to serve what you built, there's no extra effort; no refactoring. So if you are using the tests, I strongly advise you to try some curls as well. Here are some problems that I just found when doing it:

At some point, I deleted the warp::get() from my list_sims() and as a result answering every request I made;
The serialization/deserialization in PUT wasn't working properly. As this didn't appear in the tests, I assumed serde was being so good at its job the ended up hiding a mistake I made in the code.
Delete reply had the right Status, but the wrong written message.

The End

What a journey! I really hope it ends up being useful to someone else.

Before letting you go, I would like to write a small mea culpa, to warn you about the things that are not ok regarding what was built here.

My main goal was to show the process, not the result; I didn't want to post a finished code and write a text explaining it. Although I am happy with this choice, it has some drawbacks. To make things clearer, I avoided creating and using too many functions that would compel the reader to check the previous code to make sense of it. So be aware that these things are not ok:

Code repetition is problem number one. Tests are creating the same Simulations, there are repeated use keywords everywhere and so on;
I forced the HashMap, so it never felt "natural". I don't regret it because it gave me a chance to show you the trait implementation, but oftentimes it was a bit of a hassle;
Warp provides building blocks, which means that there is more than one way of doing the same things, and naturally, some are better than others. I didn't stress this much, often going with the easiest way;
The tests are far away from actually checking everything that should be checked.

And I think that's enough. Really! Thank you for spending the time.

Live long and prosper!

🖖

REST API with Rust + Warp 4: PUT & DELETE

Roger Torres (he/him/ele) — Wed, 14 Apr 2021 12:21:26 +0000

That's it, the last methods. In the beginning, I thought it would be the end of the series, but then I realized it needed an additional post, regarding how to manually test it using curl. However, before that, there are still two more methods to be coded. But don't worry, both combined are probably easier than the single ones we handled so far.

Warp 4, Mr. Sulu.

The code for this part is available here.

The PUT method is a mix between insert and change: it creates an entry when the data is not there and updates it when it is.

This behavior is already met by our HashSet; this is exactly how the insert() function works. However, we got to know if we are inserting or changing because the status that is returned got to be different:

Code 201 when it is created
Code 200 when it is updated

Where did I get this from? It is written here.

With that in mind, I wrote this code:

#[tokio::test]
async fn try_update() {
    let db = models::new_db();
    let api = filters::update_sim(db);

    let response = request()
        .method("PUT")
        .path("/holodeck/1")
        .json(&models::Name{ new_name: String::from("The Big Goodbye")})
        .reply(&api)
        .await;

    assert_eq!(response.status(), StatusCode::CREATED);

    let response = request()
        .method("PUT")
        .path("/holodeck/1")
        .json(&models::Name{ new_name: String::from("The Short Hello")})
        .reply(&api)
        .await;

    assert_eq!(response.status(), StatusCode::OK);
}

How did I knew that 201 was StatusCode::CREATED and 200 was StatusCode::OK? Here.

As you can see, the request is made by sending the parameter id ("1", in this case). Different from GET, this parameter is mandatory. And because the id is already being sent in the URI, the body only contains the name. The reasoning behind this is also in the aforementioned rfc.

Because of this, I implemented a new struct and a new function to get the JSON body.

#[derive(Debug, Deserialize, Serialize)]
pub struct NewName{ pub name: String }

// This code is inside the mod "filters"
fn json_body_put() -> impl Filter<Extract = (models::NewName,), Error = warp::Rejection> + Clone {
    warp::body::content_length_limit(1024 * 16).and(warp::body::json())
}

This is certainly a suboptimal way of doing this. But let's move on anyway; I am saving the excuses about the poor execution of things to the last part.

Now, the filter and the handler.

// This is inside the mod "filters"
pub fn update_sim(db: models::Db) -> impl Filter<Extract = impl warp::Reply, Error = warp::Rejection> + Clone {
    let db_map = warp::any()
        .map(move || db.clone());

    warp::path!("holodeck" / u64)
        .and(warp::put())
        .and(json_body_put())
        .and(db_map)
        .and_then(handlers::handle_update_sim)
}

pub async fn handle_update_sim(id: u64, new: models::NewName, db: models::Db) -> Result<impl warp::Reply, Infallible> {
    // Replaced entry
    if let Some(_) = db.lock().await.replace(Simulation{id, name: new.name}){
        return Ok(warp::reply::with_status(
            format!("Simulation #{} was updated.\n", id), 
            StatusCode::OK,
        ));
    }

    // Create entry
    Ok(warp::reply::with_status(
        format!("Simulation #{} was inserted.\n", id), 
        StatusCode::CREATED,
    ))
}

And that's it!

Red alert

To ~~warp~~ wrap things up, the DELETE method.

As usual, the request is quite simple: it sends the id as a parameter and no body. As a response, we expect code 200 (OK) including a "representation describing the status".

#[tokio::test]
async fn try_delete() {
    let simulation = models::Simulation{
        id: 1, 
        name: String::from("The Big Goodbye!"),
    };

    let db = models::new_db();
    db.lock().await.insert(simulation);

    let api = filters::delete_sim(db);

    let response = request()
        .method("DELETE")
        .path("/holodeck/1")
        .reply(&api)
        .await;

        assert_eq!(response.status(), StatusCode::OK);
}

Hopefully, nothing about the filter implementation seems strange to you:

pub fn delete(db: models::Db) -> impl Filter<Extract = impl warp::Reply, Error = warp::Rejection> + Clone {
    let db_map = warp::any()
        .map(move || db.clone());

    warp::path!("holodeck" / u64)
        .and(warp::delete())
        .and(db_map)
        .and_then(handlers::handle_delete_sim)
}

Since it is the first time we're deleting data, the handler has a unique behavior, but also nothing very different from what has been done so far.

pub async fn handle_delete_sim(id: u64, db: models::Db) -> Result<impl warp::Reply, Infallible> {
    if db.lock().await.remove(&Simulation{id, name: String::new(),}){
        return Ok(warp::reply::with_status(
            format!("Simulation #{} was deleted", id), 
            StatusCode::OK,
        ))
    };

    Ok(warp::reply::with_status(
        format!("No data was deleted."),
        StatusCode::OK,
    ))
}

That should do...

$ cargo test

running 5 tests
test tests::try_delete ... ok
test tests::try_create ... ok
test tests::try_list ... ok
test tests::try_create_duplicates ... ok
test tests::try_update ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

And it did!

In the next episode of Engaging Warp...

Finally, the last part. We will serve what has been built and curl against it.

🖖

REST API with Rust + Warp 3: GET

Roger Torres (he/him/ele) — Wed, 14 Apr 2021 12:21:15 +0000

Welcome back! Last time we saw each other I wrote:

Next in line is the GET method, which means we'll see parameter handling and (finally) deal with this HashSet thing.

So, "let us not waste our time in idle discourse!"

Warp 3, make it so!

The code for this part is available here.

First, I needed another dependency to help me deserialize the GET return, so I changed the Cargo.toml file:

serde_json = "1.0"

Then, the time came to change try_list(). As of our last encounter, this test had only a request() and the assert_eq!. I added two things:

Before the request, I manually inserted two entries into the HashSet (I could've called POST, but since it is already being tested elsewhere, it is ok to take this shortcut);
After the request, I deserialized the HTML body and compared its content to the data I had previously inserted.

There's a chance that a few things will appear weird, but don't worry, I will go through each one of them.

use std::collections::HashSet;

#[tokio::test]
async fn try_list() {
    use std::str;
    use serde_json;

    let simulation1 = models::Simulation{
        id: 1, 
        name: String::from("The Big Goodbye"),
    };


    let simulation2 = models::Simulation{
        id: 2, 
        name: String::from("Bride Of Chaotica!"),
    };

    let db = models::new_db();
    db.lock().await.insert(simulation1.clone());
    db.lock().await.insert(simulation2.clone());

    let api = filters::list_sims(db);

    let response = request()
        .method("GET")
        .path("/holodeck")
        .reply(&api)
        .await;

    let result: Vec<u8> = response.into_body().into_iter().collect();
    let result = str::from_utf8(&result).unwrap();
    let result: HashSet<models::Simulation> = serde_json::from_str(result).unwrap();
    assert_eq!(models::get_simulation(&result, 1).unwrap(), &simulation1);
    assert_eq!(models::get_simulation(&result, 2).unwrap(), &simulation2);

    let response = request()
        .method("GET")
        .path("/holodeck/2")
        .reply(&api)
        .await;

    let result: Vec<u8> = response.into_body().into_iter().collect();
    let result = str::from_utf8(&result).unwrap();
    let result: HashSet<models::Simulation> = serde_json::from_str(result).unwrap();
    assert_eq!(result.len(),1);
    assert_eq!(models::get_simulation(&result, 2).unwrap(), &simulation2);
}

The first thing I take as deserving an explanation is the db.lock().await.insert(). The lock() gives you what's inside the Arc, and in this case, it returns a Future. Why? Because we are not using std::sync::Mutex, but tokio::sync::Mutex, which is an Async implementation of the former. That's why we don't unwrap(), but instead await, as we need to suspend execution until the result of the Future is ready.

Moving on, filters::list_sims() is now getting a parameter, which is the data it will return (which, in a real execution, would come from the HTTP body).

After the request—that remains the same—there are three lines of Bytes-handling-jibber-jabber.

Bytes is the format with which warp's RequestBuilder handles the HTML body content. It looks like a [u8] (that is, an array of the primitive u8], but it is a little bit more painful to handle. What I did with it, however, is simple. I:

Mapped its content to a Vector of u8
Moved the Vector's content to the slice
Used serde_json::from_str() function to map it to the Simulation struct inside the HashSet.

And this is one of the reasons I wanted a HashSet. As far as I know, standard Rust doesn't allow you to create a HashMap referring to a struct of two fields; that is, you cannot do that:

\\ This code is not in the project!
struct Example {
    id: u64,
    name: String,
}

type Impossible = HashMap<Example>;

And without using a struct as I did with the HashMap (as well as the cool kids did with Vector here at line 205), using serde gets... complicated (which means I have no idea how to do it).

Nonetheless, there is another reason why I wanted to stick the struct within the HashSet: it gave me the chance to implement some traits for my type.

Before diving into the traits, I would like to explain the last part of the test (which should be a different test, but the example is already too big).

The GET method can be used in three different ways:

Fetch all the entries: /holodeck
Fetch a single entry: /holodeck/:id
Fetch filtered entries: /holodeck/?search=query

This last request() using path /holodeck/2 was written to cover the second case. I did not (and will not) develop the third one.

Boldly implementing traits

If you compare the HashSet element with another, it will compare everything. That's no good if you have a key-value-pair struct. As I didn't want to use HashMap because of the aforementioned reasons, the way to go is to change this behavior, making comparisons only care about the id.

First, I brought Hash and Hasher, then I removed the Eq, PartialEq and Hash, so I could implement them myself. And the implementation was this:

use std::hash::{Hash, Hasher};

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Simulation {
    pub id: u64,
    pub name: String,
}

impl PartialEq for Simulation{
    fn eq(&self, other: &Self) -> bool {
        self.id == other.id
    }
}

impl Eq for Simulation {}

impl Hash for Simulation{
    fn hash<H: Hasher>(&self, state: &mut H){
            self.id.hash(state);
    }
}

How did I know how to do it? I just followed the documentation where it says "How can I implement Eq?". Yes, Rust docs are that good.

And what about Hash? Same thing. But it is interesting to note why I did it. HashSet requires the Hash trait, and the Hash trait demands this:

k1 == k2 -> hash(k1) == hash(k2)

That means, if the values you're comparing are equal, their hashes also have to be equal, which would not hold after the implementation of PartialEq and Eq because both values were being hashed and compared, while the direct comparison only cared about id.

99% chance that I am wrong, but I think it should not be an implication (→), but a biconditional (↔), because the way it stands if k1 == k2 is false and hash(k1) == hash(k2) is true, the implication's result is still true. But I am not a trained computer scientist and I am not sure this uses first-order logic notation. Let me know in the comments if you do.

One last addition I made below the Hash implementation was this:

pub fn get_simulation<'a>(sims: &'a HashSet<Simulation>, id: u64) -> Option<&'a Simulation>{
    sims.get(&Simulation{
        id,
        name: String::new(),
    })
}

Even though the only relevant field for comparisons is id when using methods such as get() we have to pass the entire struct, so I created get_simulation() to replace it.

Ok, back to the GET method.

Getting away with it

The functions dealing with the GET method now have to deal with two additional information, the HashSet from where it will fetch the result and the parameter that might be used.

pub fn list_sims(db: models::Db) ->  impl Filter<Extract = impl warp::Reply, Error = warp::Rejection> + Clone {
    let db_map = warp::any()
        .map(move || db.clone());

    let opt = warp::path::param::<u64>()
        .map(Some)
        .or_else(|_| async { 
            // Ok(None) 
            Ok::<(Option<u64>,), std::convert::Infallible>((None,))
        });

    warp::path!("holodeck" / ..)
        .and(opt)
        .and(warp::path::end())
        .and(db_map)
        .and_then(handlers::handle_list_sims)
}

The opt represents the optional parameter that can be sent. It gets a param, map it as an Option (i.e., Some). If it was not provided, the or_else() returns a None. The reason why there's and async there is because or_else() returns a TryFuture.

The path we are actually returning includes this opt the same way we included the db_bap. the / .. at the and of path! is there to tell the macro to not add the end() so I could add the opt. That's why there's a manual end() there soon after.

I didn't found this solution in the docs or in the examples. Actually, for some reason, most tutorials omit GET parameters. They either just list everything or use query. I found one tutorial that implemented this, but they did so by creating two filters and two handlers. It didn't felt ok, and I knew there should be a solution and that the problem was probably my searching skills; so I asked for help in warp's discord channel, and the nice gentleman jxs pointed me to the solution you saw above.

The next step was to fix the handler:

pub async fn handle_list_sims(param: u64, db: models::Db) -> Result<impl warp::Reply, Infallible> {
    let mut result = db.lock().await.clone();
    if param > 0 {
        result.retain(|k| k.id == param);
    };
    Ok(warp::reply::json(&result)) 
}

It is no longer a Matthew McConaughey handler, but still very simple. I am using retain instead of a get_simulation() because it returns a HashSet (and get would give me a models::Simulation), which is exactly what the handler must return.

$ cargo test

running 3 tests
test tests::try_create ... ok
test tests::try_create_duplicates ... ok
test tests::try_list ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

In the next episode of Engaging Warp...

We will finish the implementation by implementing the PUT and DELETE methods.

🖖

DEV Community: Roger Torres (he/him/ele)

[Rust] Tokio stack overview: Runtime

What is an asynchronous runtime?

Futures

Scheduler

Why?

Handling the tree of futures

Scheduling parallel tasks

Going beyond

Driver

Pending future

Timer

Asynchronous Rust: basic concepts

async/.await

Futures

Poll, Context, Waker, Executor and Reactor

Wrapping up

Rust BDD tests with Cucumber

Sample project

Setting up the manifest

Project structure

Coding the test with Gherkin

Handle the scenario with Rust

World object

Step builder

Given

When

Then

The substitute main function

First steps with Rust declarative macros!

Why do I need macros?

Declaring macros

The matching

Variable argument

Fixed arguments

Separator

Multiple matching

Repetition

Repetition separator

Versatility

The transcriber

Type

Multiple statements

Using repetition

Expand

Export and import

Smart Pointers in Rust: What, why and how?

Smart points in general

Box<T>

What?

Why?

How?

Cell<T>

What?

Why?

How?

RefCell<T>

What?

Why?

How?

Rc<T>

What?

Why?

How?

Arc<T>

What?

Why?

How?

RwLock<T>

What?

Why?

How?

Mutex<T>

What?

Why?

How?

First steps with Docker + Rust

Install Docker

Glossary

The sample project

The substitute `main` function

The first `Dockerfile`

The second `Dockerfile`

The third `Dockerfile`

The fourth `Dockerfile`

The fifth (and final) `Dockerfile`