Creating simple and fast lexer with logos

Logos

Hello guys, this is my first post. I'm writing this post cause of i really liked and enjoyed during writing the tokenizer with the logos but i couldn't what i wanted, maybe i didn't understand it very but logos is pretty good and cute project it's just like flex.

What is logos? Logos is fast and easy to use lexer generator written in rust, so if you want to make a simple lexing operation in rust you can choose logos, you can make token based lexer with it, also in this sample we'll make a simple token based lexer.

Let's get started by creating a simple project:

cargo new my_lexer
cd my_lexer

now we need to add logos dependency to our project:

cargo add logos

okay, now we're ready to start, i'm starting with creating a sample file, i want to tokenize a very very simple rust like syntax let's create a file my_example.txt under examples directory, you can set the path and filename as you want this is just a simple sample.

// examples/my_example.txt
fn main() {
   print("hello")
}

now we need to take the file path in our rust program, we can reach it from by using args and also we need to read the taken file.

// src/main.rs
use std::env;
use std::fs::read_to_string;

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() >= 2 {
        let file = &args[1];
        let ctx = read_to_string(file).expect("Should have been able to read the file");
        println!("{:?}", ctx);
    } else {
        eprintln!("pls give me a text file.");
    }
}

Okay, so now we have a simple file reader program. Now let's create some tokens.

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")]
pub enum Token<'a> {
    #[token("fn")]
    Function,
    #[token("{")]
    LeftCurlyBracet,
    #[token("}")]
    RightCurlyBracet,
    #[token("(")]
    LeftBracet,
    #[token(")")]
        RightBracet,

    #[regex(r"[a-zA-Z][a-zA-Z0-9]*")]
    Text(&'a str),
    #[regex(r#""([^"\\]|\\.)*""#, priority = 1)]
    String(&'a str),
}

What we did here? Firstly we set logos the derive, escape whitespaces, newline, eof and tab chars, after derives we just created a simple enum with our tokens with token derives.
We have:

• static tokens: fn, (, ), {, }
• text and string.

and finally we can lex the input:

// src/main.rs
use std::env;
use std::fs::read_to_string;
use logos::Logos;

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")]
pub enum Token<'a> {
    #[token("fn")]
    Function,
    #[token("{")]
    LeftCurlyBracet,
    #[token("}")]
    RightCurlyBracet,
    #[token("(")]
    LeftBracet,
    #[token(")")]
        RightBracket,

    #[regex(r"[a-zA-Z][a-zA-Z0-9]*")]
    Text(&'a str),
    #[regex(r#""([^"\\]|\\.)*""#, priority = 1)]
    String(&'a str),
}

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() >= 2 {
        let file = &args[1];
        let ctx = read_to_string(file).expect("Should have been able to read the file");
        for res in Token::lexer(&ctx) {
            match res {
                Ok(token) => println!("{:#?}", token),
                Err(e) => panic!("some error occurred {:?}", e)
            }
        }
    } else {
        eprintln!("pls give me a text file.");
    }
}

✨result✨