Note: Pretty much all the code reported in this and the following articles in this series is simplified or abridged in some way and may contain inaccuracies. Readers interested in the end-result may find it at the following GitHub repo: https://github.com/gooplancton/irc-server-rust
To say that in the past few months I have fallen in love with Rust would be an understatement. My day job is backend development in Typescript, and I had never really dabbled with low-level, systems programming languages before, but I had a morbid curiosity to learn Rust, as every YouTube video I would watch about it made it look like it was the next best thing on Earth after peanut butter and jelly (which, as an Italian, I’ve never even tried nor am interested in finding out what it even is by the way it sounds).
![Apparently, they are used as spreads on a sandwich. Still does not look very appetizing to me. [CC: JefferyGoldman, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons]](https://media.dev.to/cdn-cgi/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftt8aozok6lj9q9tekj2q.jpeg)
This was five months ago. How I then went from knowing absolutely nothing about borrowing, lifetimes, smart pointers, different types of strings, futures and all the beautiful jazz of this language, to feeling somewhat comfortable using it to build interesting pieces of software is definitely a story for another day.
The thing is, I was looking for the next big challenge to tackle, and that’s when I remembered about IRC (Internet Relay Chat). I don’t know much about its history, but from what I gather it was one of the first protocols allowing for large scale communications in a fashion that resembles how modern day chat software like Slack or Discord works.
At its core the IRC protocol is very simple to understand. You join a server with a nickname, which must be unique and can be changed at a later time. While connected to a server you can join one or more channels, which work exactly like those found in Discord. To chat with other users, you can then either send messages to them directly, a la Instagram DM, or send messages to a channel, so that all users inside of it will receive them.
Like almost any other internet protocol, its inner workings are described in publicly available documents called RFCs (Request for Comments). The RFCs of interest to us for implementing IRC are RFC 1459 and RFC 2812.
If you have a look at them, you will find that it’s quite a comprehensive protocol, the specification spanning a variety of use-cases such as different levels of privacy for channels, admin users, multiple user modes and whatnot. While implementing the entirety of the specification would without a doubt be extremely challenging and stimulating, I believe that when starting any new project, it’s important to set realistic goal posts so that you don’t get overwhelmed by the complexity that naturally comes with any big challenge like this one, and can still deliver something interesting that people can read about and learn from.
The objective I set for myself for the time being was to implement enough of the IRC protocol so that I could take an IRC client like irssi, connect to my server and send messages to channels and other users.
What came out of it is a simple and neat project that showcases a bunch of interesting patterns and concepts in Rust that often come up when dealing with net servers.
Part 1: High Level Architecture
Because of Rust’s strict rules when it comes to memory access, I’ve found that when dealing with multithreaded or async applications like this one, it’s best to lay down the architecture of the various tasks and shared state upfront on paper first, before writing any code.
The centerpiece of the application, running on the main thread, would be a TcpListener that accepts incoming connections on the port 6667.
Once a connection is accepted, a new thread is spawned, which will be responsible for continuously reading and parsing incoming commands, executing the appropriate action and then writing the response back to the socket if needed.
Running alongside the main thread, there will also be a dispatcher task, which will act as a public outbox, using a Multi-Producer Single-Consumer channel as the means by which connection handler tasks can communicate with each other.
Part 1.1 Parsing the Wire Protocol
The IRC wire protocol is straightforward in nature. It’s a text protocol based on TCP with a very simple syntax. Messages are composed of multiple segments separated by spaces, with the notable exception that the beginning of the last segment of a message is instead delimited by a colon character, and thus can contain whitespace, whereas the others cannot. Each message must be followed by a CRLF sequence, i.e. the \r\n characters. Messages can have an optional header segment, which if present must be prefixed by a colon character. It is generally used to specify the sender of the message, so it is up to the server to guarantee its correctness.
The IRC protocol specifies a number of commands, which commonly appear as the first segment (after the header) of each message. If we just open a TCP socket and connect to our server using irssi, we will see the following incoming messages:
CAP LS 302
JOIN :
NICK leo
USER leonardosantangelo leonardosantangelo localhost :Leonardo Santangelo
We see here that 4 commands have been invoked:
CAP with arguments: LS and 302
JOIN with an empty string argument
NICK with a single argument: leo
USER with arguments: leonardosantangelo, leonardosantangelo, localhost and Leonardo Santangelo
To correctly handle these commands, we would need to look them up on the IRC specification, which describes how many arguments there can be, and what their meanings are, in addition to what the client expects to receive as a response. For the sake of simplicity, let’s just focus on the NICK command for now. It’s fairly obvious that it instructs the server to set a nickname for the requesting user. It is also evident that it takes just a single string parameter, which cannot contain whitespace. We can model this command as a Rust enum variant containing a struct holding the provided arguments.
pub struct NickArgs {
nickname: String
}
pub enum Command {
Nick(NickArgs)
}
By wrapping the TcpStream in a BufReader, provided by the Rust standard library, we can conveniently read the stream until we reach a newline character and store it into an owned String buffer, at which point we’ll be ready to parse the command. A naive implementation could involve defining a function parse_command which takes in a string slice and returns a Command or an Error (where we’re using the amazing anyhow crate for the sake of simplicity).
// NOTE: we defer handling headers for now...
pub fn parse_command(irc_string: &str) -> anyhow::Result<Command> {
let irc_string = irc_string.trim_end();
let first_space_idx = irc_string.find(' ');
let (command_name, arg_string) = match first_space_idx {
None => (irc_string, ""), // No arguments => empty arg string
Some(idx) => {
let arg_string = irc_string[idx + 1..].trim_end();
let command_name = irc_string[..idx].trim_end();
(command_name, arg_string)
}
};
match command_name {
"NICK" => Command::Nick(parse_nick_args(arg_string))?,
// other commands...
_ => Err(anyhow::anyhow!("unknown command: {}", command_name)),
}
}
pub fn parse_nick_args(arg_string: &str) -> anyhow::Result<NickArgs> {
// Split the argument string into an argument vec according to IRC syntax
let args: Vec<&str> = match arg_string.split_once(" :") {
None => arg_string.split(' ').collect(),
Some((first_args, last_arg)) => {
let mut args: Vec<&str> = first_args.split(' ').collect();
args.push(last_arg);
args
}
};
let mut args = args.into_iter();
// Get the next arg (in this case, the nickname)
let arg_string = args
.next()
.ok_or(anyhow::anyhow!("Not enough arguments"))?;
// Parse it to an owned String (I know it's not really elegant,
// bear with me, it will make sense later)
let nickname = arg_string.parse::<String>()?;
// Return the struct created with the shorthand assignment syntax
Ok(NickArgs { nickname })
}
While this implementation surely works fine for now, typing out each arm and each parsing function for every command definitely feels boilerplaty and unwieldy to me.
After hacking around a little bit, this is what I came up with:
// NOTE: in crate irc_parser
pub trait FromIRCString: Sized {
fn from_irc_string(irc_string: &str) -> anyhow::Result<Self>;
}
// -------------------------
// NOTE: in commands/mod.rs
mod nick;
#[derive(FromIRCString, RunCommand)]
#[command_list]
pub enum Command {
#[command_name = "JOIN"]
Join(JoinArgs),
#[command_name = "CAP"]
Capabilities(CapabilitiesArgs),
#[command_name = "NICK"]
Nick(NickArgs),
#[command_name = "USER"]
User(UserArgs),
#[command_name = "PING"]
Ping(PingArgs),
#[command_name = "PRIVMSG"]
PrivMsg(PrivMsgArgs),
#[command_name = "QUIT"]
Quit(QuitArgs),
}
// --------------------------
// NOTE: in commands/nick.rs
#[derive(FromIRCString)]
pub struct NickArgs {
nickname: String,
}
impl RunCommand for NickArgs {
// ...
}
In this implementation, I have defined a trait FromIRCString in a separate crate, which also exports the macros used to derive it for enums and structs. In my observation, I’ve found that the use case of implementing a trait on an internal struct field of each variant of an enum and then deriving the implementation for the enum itself by delegating it to the appropriate variant’s is fairly common, so I’d like to spend five minutes discussing how I went about implementing it.
The Rust language allows us to define functions that take in Rust code and produce other Rust code at compile time. These functions are called **procedural macros **and are one of the most powerful features of the language in my opinion. To implement one, we must necessarily define it in a separate crate, whose Cargo.toml file will need to contain a line specifying that the crate is going to export procedural macros, like so:
.
└── irc_server/
├── Cargo.toml
├── src/
└── irc_parser/
├── Cargo.toml
├── src/ <- FromIRCString trait here
└── irc_parser_macros/
├── Cargo.toml <- "[lib] proc-macro = true" here
└── src/lib.rs <- procedural macros here
Procedural macros come in three forms:
function-like macros, which resemble declarative macros and are invoked in much of the same way
attribute macros, which operate on existing items such as functions, structs and enums and are able to modify their inner code
derive macros, which can only be applied to structs, enums and unions and are generally used to implement traits on them, which is also why they are only able to generate additional code, and cannot modify the existing one
Assuming we had an enum such as the Command enum defined above, where each variant contains only one unnamed struct field that implements FromIRCString, a derive macro could look like this:
use proc_macro::TokenStream;
#[proc_macro_derive(FromIRCString, attributes(command_name)]
pub fn derive_from_irc_string(input: TokenStream) -> TokenStream {
let input = parse_macro_input!(input as syn::DeriveInput);
let ident = &input.ident;
match input.data {
syn::Data::Enum(enum_data) => {
impl_from_irc_string_for_commands_enum(ident, enum_data)
}
_ => panic!("Can only apply to enums")
}
}
fn impl_from_irc_string_for_commands_enum(
ident: &syn::Ident,
enum_data: syn::DataEnum,
) -> TokenStream {
// We'll see this in detail shortly
todo!()
}
As mentioned before, a procedural macro is just a function (executed at compile-time) that takes in some tokens and outputs some other tokens. Just like declarative macros must be defined through a declarative macro, macro_rules!, I find it rather interesting that procedural macros must be defined through (one of three) procedural macros, in our case, proc_macro_derive. The “arguments” accepted by proc_macro_derive are:
The name of the trait we are writing a derivation for
An optional sequence of additional attributes we can use in the body of the macro. Here we are specifying that the command_name attribute could appear in the body of the decorated item (recall that we are using it to specify the command keyword literal)
Also note that we’re making use of the amazing syn, quote and proc-macro2 crates, which is pretty much standard practice when writing macros.
syn(which stands for syntax) gives us precise types for any valid Rust code, we use it here to get the identifier (name) of the enum and the type, identifiers and attributes of its variants
The quote crate allows us to turn normal Rust code into a TokenStream, which is what the macro requires as its output type, we can use it to create snippets that we can later combine in larger expressions
The proc-macro2 crate is a wrapper around the standard proc-macro crate. We use it as the syn and quote crates are designed to work with it. Generally, we use the wrapped types as much as possible until we need to return from the macro definition, at which point we can convert back to the standard types
With this knowledge, let’s have a look at the actual implementation of the macro for enums:
pub fn impl_from_irc_string_for_commands_enum(
ident: &syn::Ident,
enum_data: syn::DataEnum,
) -> TokenStream {
// Turn each variant into a match arm
let arms = enum_data.variants
.into_iter()
.map(|variant| get_match_arm(ident, variant));
// Implement the trait
let expanded = quote! {
impl FromIRCString for #ident {
// NOTE: it is practically identical to the parse_command
// function we have implemented earlier, in fact,
// the compiler expands it to the exact same code.
fn from_irc_string(irc_string: &str) -> anyhow::Result<Command> {
let irc_string = irc_string.trim_end();
let first_space_idx = irc_string.find(' ');
let (command_name, arg_string) = match first_space_idx {
None => (irc_string, ""),
Some(idx) => {
let arg_string = irc_string[idx + 1..].trim_end();
let command_name = irc_string[..idx].trim_end();
(command_name, arg_string)
}
};
match command_name {
// Similar repetition syntax found in declarative macros
#(#arms)*
// Return an error if no command matches
_ => Err(anyhow::anyhow!(
"unknown command: {}", command_name
)
),
}
}
}
};
TokenStream::from(expanded)
}
use proc_macro2::TokenStream as TokenStream2;
fn get_match_arm(ident: &Ident, variant: Variant) -> TokenStream2 {
// Turn an enum variant into a match arm
todo!()
}
Depending on whether you have had previous exposure to procedural macros, you may find the code above a little hard to digest, but read it carefully. The only fancy thing we are doing is turning each variant of the enum into the following snippet by calling the (yet to be implemented) get_match_arm function.
#command_name => {
let args = <#args_type>::from_irc_string(arg_string).unwrap();
Ok(#ident::#variant_ident(args))
}
Note that the quote! declarative macro allows us to type normal Rust code with a twist, we can inject names and other TokenStreams by prefixing them with a # sign.
Let’s now have a look at the final piece of the puzzle, the get_match_arm function.
fn get_match_arm(ident: &Ident, variant: Variant) -> TokenStream2 {
let variant_ident = &variant.ident;
let args_type = &variant
.fields
.iter()
.next()
.expect("missing arg struct in enum variant")
.ty;
let command_name = variant
.attrs
.into_iter()
.filter_map(|attr| match attr.meta {
syn::Meta::NameValue(meta) => {
meta.path.is_ident("command_name").then_some(meta.value)
}
_ => None,
})
.next()
.expect("enum variants need to have the command_name attribute");
quote! {
#command_name => {
let args = <#args_type>::from_irc_string(arg_string).unwrap();
Ok(#ident::#variant_ident(args))
}
}
}
If you have understood the discussion above, you will have no problems understanding this function. Note that this function can panic in two cases:
The variant doest not contain an arg struct field (the equivalent of NickArgs in our earlier example)
The variant was not annotated with a command_name attribute
When a procedural macro panics, we get an error in our editor where it was expanded (note: we are also getting an error where we are calling from_irc_string on the Command because this macro could not expand to valid code, hence we did not technically implement the trait).
In fact, panicking the macro is not really the best practice. We are doing it here for the sake of brevity and getting the point across, but if we were writing a library for other people to use, this message would provide a rather poor developer experience. What we should actually do is return a syn::Error which specifies both an error message and *where *it should appear. Here’s how it would appear if we had done it the proper way:
Since we are going to be the only users of these macros, it does not really make a difference in this case, but keep it in mind should you ever want to release a library.
The attentive reader will have also noticed that we are carelessly calling from_irc_string on the args structs inside the match arms, this can only work if those types too implement FromIRCString. In fact, their implementation is provided by the same derive macro, but in this case, we will delegate it to a function called when the data property on the macro’s input is of the DataStruct variant.
pub fn impl_from_irc_string_for_command_args_struct(
ident: &syn::Ident,
struct_data: syn::DataStruct,
) -> TokenStream {
let field_assignments = struct_data.fields.iter().map(|field| {
let field_ident = field.ident.as_ref();
let field_type = &field.ty;
quote! {
let arg_string = args.next()
.ok_or(anyhow::anyhow!("Not enough arguments"))?;
let #field_ident = arg_string.parse::<#field_type>()?;
}
});
let field_idents = struct_data.fields.iter()
.map(|field| field.ident.as_ref());
let expanded = quote! {
impl irc_parser::FromIRCString for #ident {
// NOTE: Also looks very similar to the parse_nick_args function
// we had implemented earlier
fn from_irc_string(arg_string: &str) -> anyhow::Result<Self> {
let args: Vec<&str> = match arg_string.split_once(" :") {
None => arg_string.split(' ').collect(),
Some((first_args, last_arg)) => {
let mut args: Vec<&str> = first_args
.split(' ').collect();
args.push(last_arg);
args
},
};
let mut args = args.into_iter();
#(#field_assignments)*
Ok(Self {
#(#field_idents),*
})
}
}
};
TokenStream::from(expanded)
}
What we’re doing here is not particularly complex, in the final expanded code, we are first breaking up the irc_string into segments, taking into account also the final colon-delimited chunk. Then, for each field of the struct, we are defining a variable with the same name and setting it as the result of parsing the string argument into its type (it goes without saying that the type must implement FromStr). Lastly, we are constructing the struct with the field name shorthand syntax. The resulting expanded code for NickArgs would look like this:
impl irc_parser::FromIRCString for NickArgs {
fn from_irc_string(arg_string: &str) -> anyhow::Result<Self> {
let args: Vec<&str> = match arg_string.split_once(" :") {
None => arg_string.split(' ').collect(),
Some((first_args, last_arg)) => {
let mut args: Vec<&str> = first_args.split(' ').collect();
args.push(last_arg);
args
}
};
let mut args = args.into_iter();
let arg_string = args
.next()
.ok_or(anyhow::anyhow!("Not enough arguments"))?;
let nickname = arg_string.parse::<String>()?;
Ok(Self { nickname })
}
}
This was quite a deep dive into (derive) procedural macros. For the readers interested in getting to know more about this amazing tool, I highly recommend going through the procedural-macros-workshop, which provides some interesting examples about how macros can be used in practice.
I’m going to wrap this article up here for now, stay tuned for the follow-up’s in this series where we will tackle the actual implementation of the networking code using the blocking multithreading api first (Part 2 of this series), and the tokio async runtime later (Part 3).
A presto! (see you soon!)
Top comments (0)