DEV Community: Bernard Kolobara

Building a telnet chat server with Rust and Lunatic

Bernard Kolobara — Mon, 22 Feb 2021 14:44:20 +0000

I love the aesthetics of terminals and I’m not the only one, there is a whole subreddit dedicated to people sharing their desktops and showcasing different terminal setups. Last year I spent working on an innovative WebAssembly runtime called Lunatic. Recently we landed TCP support and I was super excited to start building real world applications with it, and what would be a better fit than a terminal based chat server with a retro vibe?

It took me around a week to build it with Rust + Lunatic and you can check out the code here. If you would like to try it out you can connect to it with:

# US
> telnet lunatic.chat
# EU
> telnet eu.lunatic.chat

While writing the server I ran into many interesting problems and would like to share here how I leveraged the power of Rust and Lunatic to overcome them.

Architecture

The reason I picked telnet is that the specification is simple enough to read through and implement in a short time. It’s a small layer on top of TCP and as mentioned before we had TCP already working. On the other hand, telnet is a really limiting protocol and I needed to get creative while building a chat application on top of it.

The first issue I encountered was the line based nature of terminals. You write a command, hit enter and the terminal prints out some text. This doesn’t go well with the UI of a chat app where
messages can come in at any time. What are you supposed to do when new text arrives and the user has already partially written her own message? Override the user's input? Print the new message after the input?

One solution would be to buffer all messages until the user hits enter and then just dump all the ones that arrived in the meantime at once, but this can’t work as we would rely on the user to keep hitting enter to read new messages.

It became clear that I needed to use some kind of terminal user interface where I render separately all the incoming messages from the user who is currently typing. It’s possible to do this by using a few extensions to the telnet protocol. Once the telnet client connects I send it the following instructions:

Don’t echo anything that the user is typing, let me be in charge of printing in the terminal.
Don’t buffer messages, send each keystroke to the server.
Report size changes of the terminal.

This allows me to construct the UI on the server and just send a sequence of terminal escape characters back to bring the user’s terminal up to date. On each keystroke or message received the UI is updated.

Massive concurrency

For this to work we need to permanently keep the telnet connection open and periodically send data through it. This is a perfect use case for Lunatic’s Processes, they are designed for massive concurrency. Each client’s connection is handled in a separate Process.

Not to be confused with Operating System processes, Lunatic’s Processes are lightweight and also known as green threads (but isolated) or go-routines in other runtimes. They are fast to create, have a small memory footprint and a low scheduling overhead. All Processes are preemptively scheduled and they can’t spend too much time running without yielding and giving others a fair share of the resources. This keeps all connections responsive in an environment where most of the time is spent waiting on I/O.

Interop with existing libraries

Luckily I could make use of existing Rust libraries and didn’t need to reinvent the wheel. I used:

Askma as a templating engine.
TUI as the rendering engine.
Chrono for date formatting.

They all compiled to WebAssembly without issues. I just needed to provide a telnet backend for TUI, but I could reuse most of the code from the termion crate (sadly it has no Windows support for now).

TUI works in a somewhat similar way to React.js, you update your state and just call a render method. It will re-render the UI and send back to the client the minimal amount of changes in the form of terminal escape characters.

State Management

A big part of programming is state management. Your application getting into a state that you couldn’t predict while writing the code is a big source of bugs, and Lunatic tries to simplify this by allowing you to isolate the state into separate processes.

From the perspective of a process, it owns the whole memory and can’t influence the memory of other processes in any way, not even by unsafe pointer dereferencing. This is a result of building
them on top of WebAssembly instances. The only way processes can talk between each other is through message passing.

This greatly simplifies reasoning about state changes. You only need to think about what state you are in and how the next message will influence the state change. It makes it a lot easier to debug once you find yourself in an undesirable state. Let’s look at a concrete code sample from the implementation:

// The server is the main coordinator between all the clients.
// It keeps track of connected clients and active channels.
// It's also in charge of assigning unique usernames to new clients.
pub fn server_process(state_receiver: Receiver<ServerMessage>) {
    let mut state = ServerState {
        clients: 0,
        channels: HashMap::new(),
    };

    let mut username_generator: i64 = 0;
    let mut all_usernames = HashSet::new();

    loop {
        match state_receiver.receive().unwrap() {
            ServerMessage::Joined(client) => {
                // Increase the number of active users
                state.clients += 1;
                // Generate a new username
                username_generator += 1;
                let username = format!("User_{}",
                                   username_generator);
                all_usernames.insert(username.clone());
                // Client specific state
                let server_info = ServerInfo {
                    clients: state.clients,
                    username,
                };
                let _ = client.send(server_info);
            }
            ServerMessage::List(client) => {
                ….
            }
            ServerMessage::ChangeName(from, to, client) => {
                if all_usernames.contains(&to) {
                    // Notify client that the name is taken
                    let _ = client.send(false);
                } else {
                    all_usernames.remove(&from);
                    all_usernames.insert(to);
                    let _ = client.send(true);
                }
            }
            ...
    }
}

We can see that the process has a few local variables to keep track of its state:

How many clients are connected.
Which channels are available.
If a new user joins what username should be assigned.
Which usernames are taken.

Afterwards the Process just runs in a loop waiting on messages. If a new client is connected the server receives a ServerMessage::Joined message. It will then update the total count of users, assign a new username to the client and send back a message notifying the client about the assigned username.

The client’s process is similarly structured, it keeps a state of the current input box for each channel and all received messages for the channels. The client’s process can receive 2 types of messages:

Keystrokes coming from the telnet connection.
New chat messages coming from all subscribed channels.

For each keystroke we update the current channel’s input box or attach the new message to the history of messages in the channel.

If we have such an architecture and run into a bug, let’s say the number of connected users shown is wrong, there is only one source of truth here and we know exactly where this information came from. We just need to figure out how we got into this state.

Other benefits

There are some not so obvious additional benefits that we get from Lunatic.

If a client’s process receives some malicious data from the telnet connection and crashes, it will only terminate the existing connection. It can’t access the state of any other Processes. In my first implementation I was often using .unwrap in the code, following Erlang’s let it crash philosophy and knowing that if I see any crashes in the logs I can always later investigate why they happened, but the application should continue running.

The message sending implementation uses Smol’s channels underneath, but you may be surprised not to see any async or .await keywords in the code. The reason for this is that Lunatic abstracts away the asynchronous code and you can just write seemingly blocking code, but it actually never blocks the underlying thread and takes full advantage of async Rust. This is a whole topic on its own so I will leave it for another blog post.

Lunatic works with any code that can compile to WebAssembly, and as I have shown earlier a lot of libraries just work out of the box. You can also link C code into your Rust application while compiling to WebAssembly. One big pain point when using C from Erlang is that you need to be extremely careful in your code, because if something crashes it will take the whole VM down. Or if you spend too much time in the C part it will block the scheduler from using the thread and endanger the responsiveness of your system. Lunatic solves both of these problems. The reduction counter is inserted before the WebAssembly code is JIT translated to machine code and will also be part of the “native” C code, allowing the scheduler to preempt it. A crash still stays isolated thanks to WebAssembly sandboxing properties.

Conclusion

In the end the chat server will remain a nice toy application and you should not use it for more serious use cases as telnet doesn’t encrypt any of the data sent to the server.

However, it’s a really good feeling to get something like this running on a runtime you have built. While developing the chat application I found a few bugs in the runtime itself, so it was totally worth creating this app . I’m really looking forward to gradually moving away from building Lunatic and building amazing applications with it. I was also positively surprised how well the chat app is working, being the first real word app built on Lunatic.

I think that we are finally at the point where WebAssembly is mature enough to be used in serious applications, and I strongly believe that WebAssembly on the backend is going to play an important role in the future.

Writing Rust the Elixir way

Bernard Kolobara — Sat, 28 Nov 2020 10:14:13 +0000

It's not a secret that I'm a big fan of Elixir, so when I started doing Rust development I tried to bring some ideas from Elixir to the world of Rust. This post describes some of the tools I'm building to bring the power of Elixir to Rust.

What makes Elixir so great?

It's hard to just pick a few of them, but I believe that Elixir's biggest advantage comes from using Erlang as the underlying virtual machine, specially from these 2 properties:

Massive concurrency
Fault tolerance

Massive concurrency

This is something hard to explain until you experience it yourself. I learned early in my career that you should never create a thread while handling a request. Threads are heavy, expensive and too many of them can bring your whole machine down. In most cases it's enough to use a thread pool, but this approach fails once the number of concurrent tasks outgrows the number of threads in the pool.

Let's look at an example: imagine a rust application that just creates 2000 threads that wake up every 100 ms and go right back to sleep.

use std::thread;
use std::time::Duration;

fn main() {
    for _ in 0..2_000 {
        thread::spawn(|| loop {
            thread::sleep(Duration::from_millis(100));
        });
    }
    thread::sleep(Duration::from_secs(1_000));
}

Even though the threads don't do anything, just running this on my MacBook forces it to reboot after a few seconds. This makes it impractical to have massive concurrency with threads. There are many solutions to this problem. The one chosen by Elixir is to abstract concurrent tasks with something called Processes. They are extremely lightweight, so even running 2 million of them doesn't present a challenge.

Massive concurrency in Rust

You can achieve amazing concurrency and performance using async Rust, but working with async Rust is not as simple as writing regular Rust code and it just doesn't provide you the same features as Elixir Processes do.

After thinking for a long time how I could make something that reassembles Elixir Processes in Rust I came up with the idea to introduce an intermediate step, WebAssembly. WebAssembly is a low level bytecode specification that Rust can target. The idea was simple, instead of compiling Rust for x86-64 you would compile it to the WASM target. From there I would build a set of libraries and a WebAssembly runtime that exposes the concept of Rust Processes. Contrary to operating system processes or threads, they are lightweight with small memory footprints, fast to create and terminate, and the scheduling overhead is low. In other languages they are also known as green threads and goroutines, but I will call them processes to stay close to Elixir's naming convention.

That was the first step towards Lunatic.

Let's look at the same Rust example, but now implemented with Lunatic. At the same time we will crank up the number of concurrent processes to 20k.

use lunatic::{Channel, Process};

fn main() {
    let channel: Channel<()> = Channel::new(0);

    for _ in 0..20_000 {
        Process::spawn((), process).unwrap();
    }

    channel.receive();
}

fn process(_: ()) {
    loop {
        Process::sleep(100);
    }
}

To run this you will need to compile this Rust code to a .wasm file first:

○ →  cargo build --release --target=wasm32-wasi

Then run it with:

○ →  lunaticvm example.wasm

Contrary to the previous example this runs without hiccups on my Late 2013 Macbook and the CPU utilisation is minimal, even if we are using 10x more concurrent tasks. Let's examine what is exactly happening here.

The processes spawned by Lunatic are actually taking full advantage of the power provided by async Rust. They are scheduled on top of a work stealing async executor, the same used by async-std. Calling Process::sleep(100) will actually invoke smol's at function.

Wait a second! How does this work without the .await keyword, you may ask yourself. Lunatic takes the same approach as Go, Erlang and the earlier implementation of Rust based on green threads. It creates a tiny stack for executing the process and grows it when your applications needs more. This is a bit less efficient than calculating the exact stack size during compile time as async Rust is doing, but a reasonable tradeoff I would say.

Now you can write regular blocking code, but the executor will take care of moving your process off the execution thread if you are waiting, so you never block a thread.

As we saw earlier, scheduling threads is a hard task for the operating system. To replace one thread that's being executed with another one, a lot of work needs to be done (including saving all the registers and some thread state). However, switching between Lunatic Processes does only the minimal amount of work possible. With an idea pioneered by the libfringe library and using some asm! macro magic, Lunatic lets the Rust compiler figure out the minimal number of registers to be preserved during context switches. This makes scheduling Lunatic processes zero-cost. On my machine usually 1ns, equivalent to a function call.

Another benefit of scheduling the Processes in user space instead of using threads is that other applications will continue running normally on your machine, even if your app misbehaves.

Now that we saw how Lunatic allows you to create applications with massive concurrency, let's look at fault tolerance.

Fault tolerance

Maybe the most known Eralng/Elixir philosophy is "let it crash". If you are building complex systems it's impossible to predict all failure scenarios. Inevitably something is going to fail in your application, but this failure should not bring down the whole thing.

Elixir Processes are completely isolated and can only communicate through messages with each other. This allows you to design your application in a way that failure stays contained inside one process and doesn't affect the rest of them.

Lunatic provides even stronger guarantees than Erlang here.
Each Lunatic process gets their own heap, stack and syscalls.

Let's look at an example of a simple TCP echo server in Lunatic:

use lunatic::{Process, net}; // Once WASI gets networking support you will be able to use Rust's `std::net::TcpStream` instead.
use std::io::{BufRead, Write, BufReader};

fn main() {
    let listener = net::TcpListener::bind("127.0.0.1:1337").unwrap();
    while let Ok(tcp_stream) = listener.accept() {
        Process::spawn(tcp_stream, handle).unwrap();
    }
}

fn handle(mut tcp_stream: net::TcpStream) {
    let mut buf_reader = BufReader::new(tcp_stream.clone());
    loop {
        let mut buffer = String::new();
        buf_reader.read_line(&mut buffer).unwrap();
        tcp_stream.write(buffer.as_bytes()).unwrap();
    }
}

This application listens on localhost:1337 for tcp connections, spawns a process to handle each incoming connection and just echoes incoming lines.

You can test it using telnet:

○ → telnet 127.0.0.1 1337
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Hello world
Hello world

The first thing you will notice is that we don't use any async or .await keywords, even though this application will fully utilise Rust's async IO under the hood.

Also, the tcp connection becomes fully encapsulated in the Process,
even if we called into unsafe C code that crashes:

fn handle(mut tcp_stream: net::TcpStream) {
    ...
    unsafe { crashing_c_function() };
    ...
}

The crash is only contained to one connection in this case. It's not possible to implement something like this in Elixir, because if a call to a C function crashes it will take the whole virtual machine with it.

Another feature exclusive to Lunatic is the possibility to limit processes' syscall access. If we replaced the previous spawn call with:

// Process::spawn_without_fs is not implemented yet.
Process::spawn_without_fs(tcp_stream, handle).unwrap();

any code called from inside the handle function would be forbidden from using syscalls for filesystem access. This works also for C dependencies, because the enforcement is happening at such a low level. It allows you to express the sandboxing requirements of a Process and to use any dependency without fear. I'm not aware of any other runtime that allows you to do this.

The future

This is just a teaser of the capabilities that Lunatic will provide. There are many more features coming. Once you have this foundation, a new world of possibilities opens up. Some of the features I'm excited about:

The ability to transparently move Processes from one machine to another. The programming model relies on processes communicating through messages and if these messages are sent locally or between different computers on a network it doesn't really matter.
Hot reloading. Now that we have the WASM bytecode as an in-between step it becomes possible to just generate new JIT machine code from it and replace it while the whole system is still running.
Running complete applications compiled to WASM as a process. One example would be redirecting file read/writes from the application to tcp streams, as we are in complete charge of syscalls. The advantage here is that you are modelling the execution environment with code.

Lunatic is still in its early days, so there is a lot of development left to do. If you are excited about it or have some ideas you would like to use Lunatic for, reach out to me over email me@kolobara.com or on twitter @bkolobara.

I also want to use this opportunity to say a big thank you to the teams working on Rust, Wasmer, Wasmtime, Lucet and waSCC. It would be impossible to build Lunatic without all the hard work put into this projects.

P.S. If you would like to learn more about the magic of Erlang and Elixir, this is one of my favorite talks about it by Saša Jurić: The Soul of Erlang and Elixir. Seriously, go and watch it!

Lunatic: Actor based WebAssembly runtime for the backend

Bernard Kolobara — Thu, 27 Aug 2020 16:33:57 +0000

I would like to share with you a project I have been working on for the last few months. Lunatic is a WASM runtime heavily inspired by Erlang and Go. I tried to take all the cool parts of these VMs and extend them with a few new features that are interesting to me. Mostly I focused on having a great VM for low latency networking applications. It's still not public, so you will need to be a bit patient before trying it out.

The main features are:

Actors are represented as WebAssembly instances.
Sandboxing and runtime permissions are on a per actor basis.
All blocking code is transformed into asynchronous automatically.

It's written in Rust using Wasmtime and Tokio, plus a custom stack switching implementation inspired by libfringe and an async stdlib implementation compatible with WASI.

How does everything fit together?

The architecture of Lunatic is quite simple. Tokio is used to schedule lightweight tasks (actors) consisting of WASM instances. Each instance is self contained (has its own stack and heap) and sandboxed, but still lightweight enough so I can spawn 300k/s tasks on my almost decade old MacBook.

In this case you can just have your web app spawn a new WASM instance for each request. Opposite of similar runtimes (Go/Erlang) you have per actor precise permissions. You can only allow files being opened from a specific folder, forbid outgoing network requests or limit memory usage. You can also use safely any existing C library inside the actor, knowing that any vulnerability or crash will just be limited to this particular request.

From the application's (that runs on top of Lunatic) perspective, all operations are blocking. You write your regular old non-async rust or C++ code, but once you compile it to WASM and run on top of Lunatic, all the blocking calls are transformed into async alternatives. Your code is simplified because you don't need to use async notations and can mix in existing C libraries with blocking syscalls, but the runtime will take care of using async alternatives instead and suspending the actors.

Having 2 million connections in your Rust application, but not needing to reach out to async code should be a nice experience :).

Using WASM as the bytecode allows Lunatic to be targeted by any language that compiles to WASM, including Rust, C/C++, Go, AssemblyScript and others. It is also possible to write different actors in different languages.

Low latency

One issue that is common across async Rust, Erlang and Go is that to assure low latency the runtime can't spend too much time on a single task. You need to yield back from this particular task for the system to re-schedule it. Or it will end up blocking a thread for a long time. This is easy if you stay inside the particular language, but really hard if you call out into C code. The advantage of Lunatic here is that all code is compiled to WASM, even C extensions and it will always periodically yield back, even if you have a tight infinite loop.

Future development

Originally I started working on a programming language, but realised I could not accomplish the things I wanted with existing VMs. So I ended up developing another runtime.

It's still early in development and I'm constantly experimenting with different features before settling down for a final design. Lately I have been playing with implementing the threading part of the standard library so you could just spawn new threads from your existing language but they will actually be abstracted as actors in Lunatic.

If you want to follow along with the development, I intend to tweet now more frequently about it.