POS-Tagger in Rust

Shubham Panchal — Sun, 14 Jan 2024 03:35:46 +0000

Hello developers!
I've build a nltk-inspired parts-of-speech tagger in Rust - postagger.rs

It is a re-write of the AveragePerceptronTagger, authored by Matthew Honnibal for nltk in Python. It uses the weights, classes and POS tags but performs the inference in Rust, with wrappers written in C and Java.

Here's the Rust API,

pub mod perceptron_tagger;
use self::perceptron_tagger::PerceptronTagger;
use serde_json ;

fn main() {
    let tagger = PerceptronTagger::new( "tagger/weights.json" , "tagger/classes.txt" , "tagger/tags.json" )  ; 
    let tags = tagger.tag( "shubham was good" ) ;
    for tag in &tags {
        println!( "{} {} {}" , tag.word , tag.tag , tag.conf ) ; 
    }
}

the C API,

#include "postagger.h"
#include <stdio.h>
#include <stdlib.h>

int main( int argc , char** argv ) {
    PerceptronTagger* tagger = tagger_create( "tagger/weights.json" , "tagger/classes.txt" , "tagger/tags.json"  )  ;
    const TagResults* results = tagger_annotate( tagger , "the quick brown fox jumps over the lazy dog" ) ; 
    for( int i = 0 ; i < results -> num_tags ; i++ ) {
        printf( "word=%s , tag=%s , conf=%f \n" , results -> tags[i].word , results -> tags[i].tag , results -> tags[i].conf ) ; 
    }
    tagger_release( tagger ) ;
}

and the Java API,

import java.util.List;
import pos.tagger.POSTagger.POSTag;

public class Main {

    public static void main( String[] args ) {
        POSTagger tagger = new POSTagger(
            "weights.json", 
            "tags.json", 
            "classes.txt"
        ) ; 
        List<POSTag> tags = tagger.tag( "the quick brown fox jumps over the lazy dog" ) ;
        for( POSTag tag : tags ) {
            System.out.println( tag.getWord() + " " + tag.getTag() ) ;
        }
    }

}

If you find the project interesting, do ⭐ star the project and consider contributing your thoughts and suggestions!

DEV Community: Shubham Panchal

POS-Tagger in Rust