DEV Community

Cover image for Creating simple and fast lexer with logos
Son Penguen
Son Penguen

Posted on

Creating simple and fast lexer with logos

Logos

Hello guys, this is my first post. I'm writing this post cause of i really liked and enjoyed during writing the tokenizer with the logos but i couldn't what i wanted, maybe i didn't understand it very but logos is pretty good and cute project it's just like flex.

What is logos? Logos is fast and easy to use lexer generator written in rust, so if you want to make a simple lexing operation in rust you can choose logos, you can make token based lexer with it, also in this sample we'll make a simple token based lexer.

Let's get started by creating a simple project:

Image description

cargo new my_lexer
cd my_lexer
Enter fullscreen mode Exit fullscreen mode

now we need to add logos dependency to our project:

Image description

cargo add logos
Enter fullscreen mode Exit fullscreen mode

okay, now we're ready to start, i'm starting with creating a sample file, i want to tokenize a very very simple rust like syntax let's create a file my_example.txt under examples directory, you can set the path and filename as you want this is just a simple sample.

// examples/my_example.txt
fn main() {
   print("hello")
}
Enter fullscreen mode Exit fullscreen mode

now we need to take the file path in our rust program, we can reach it from by using args and also we need to read the taken file.

// src/main.rs
use std::env;
use std::fs::read_to_string;

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() >= 2 {
        let file = &args[1];
        let ctx = read_to_string(file).expect("Should have been able to read the file");
        println!("{:?}", ctx);
    } else {
        eprintln!("pls give me a text file.");
    }
}
Enter fullscreen mode Exit fullscreen mode

Okay, so now we have a simple file reader program. Now let's create some tokens.

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")]
pub enum Token<'a> {
    #[token("fn")]
    Function,
    #[token("{")]
    LeftCurlyBracet,
    #[token("}")]
    RightCurlyBracet,
    #[token("(")]
    LeftBracet,
    #[token(")")]
        RightBracet,

    #[regex(r"[a-zA-Z][a-zA-Z0-9]*")]
    Text(&'a str),
    #[regex(r#""([^"\\]|\\.)*""#, priority = 1)]
    String(&'a str),
}
Enter fullscreen mode Exit fullscreen mode

What we did here? Firstly we set logos the derive, escape whitespaces, newline, eof and tab chars, after derives we just created a simple enum with our tokens with token derives.
We have:

  • static tokens: fn, (, ), {, }
  • text and string.

and finally we can lex the input:

// src/main.rs
use std::env;
use std::fs::read_to_string;
use logos::Logos;

#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")]
pub enum Token<'a> {
    #[token("fn")]
    Function,
    #[token("{")]
    LeftCurlyBracet,
    #[token("}")]
    RightCurlyBracet,
    #[token("(")]
    LeftBracet,
    #[token(")")]
        RightBracket,

    #[regex(r"[a-zA-Z][a-zA-Z0-9]*")]
    Text(&'a str),
    #[regex(r#""([^"\\]|\\.)*""#, priority = 1)]
    String(&'a str),
}

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() >= 2 {
        let file = &args[1];
        let ctx = read_to_string(file).expect("Should have been able to read the file");
        for res in Token::lexer(&ctx) {
            match res {
                Ok(token) => println!("{:#?}", token),
                Err(e) => panic!("some error occurred {:?}", e)
            }
        }
    } else {
        eprintln!("pls give me a text file.");
    }
}
Enter fullscreen mode Exit fullscreen mode

✨result✨

Image description

Top comments (0)

The Most Contextual AI Development Assistant

Pieces.app image

Our centralized storage agent works on-device, unifying various developer tools to proactively capture and enrich useful materials, streamline collaboration, and solve complex problems through a contextual understanding of your unique workflow.

👥 Ideal for solo developers, teams, and cross-company projects

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay