DEV Community: Pawan Jain

5 NLP Libraries Everyone Should Know

Pawan Jain — Wed, 05 Jan 2022 12:49:44 +0000

In this guide, we’ll be touring the essential stack of Python NLP libraries.

These packages handle a wide range of tasks such as part-of-speech (POS) tagging, dependency parsing, document classification, topic modelling, and much more.

The fundamental aim of NLP libraries is to simplify text preprocessing

There are many tools and libraries created to solve NLP problems… but you’ll cover all the essential bases once you master a handful of them. That’s why I decided to feature the Five Python NLP libraries I’ve found to be the most useful.

But before that, you should have some basic knowledge about various components and topics of NLP

For Foundation

There are some well-known, top-notch mainstay resources for the theoretical depth of Natural Language Processing

Stanford Course — Natural Language Processing with Deep Learning
Deeplearning.ai Specialization — Natural Language Processing Specialization
Best book for FUNDAMENTALS (aka Bible of NLP) — Introduction to Natural Language Processing, Speech Recognition, and Computational Linguistics
Another Good reference book — Foundations of Statistical Natural Language Processing

1. Spacy Library

spaCy is a well-known and straightforward natural language processing library in Python. It contributes to state-of-the-art efficiency and agility and has a proactive open-source association.

Plus points:

Interface well with all major deep learning frameworks and comes with some outstanding and useful language models pre-installed
Comparatively faster because of Cython support

The best things you can do with spaCy

Part-of-speech (POS) Tagging: It is the process of appointing grammatical properties like a noun, verb, adjective, adverb, etc. to words
Entity Recognition: The primary method of labeling named entities discovered in the text into the pre-defined group
Dependency Parsing: Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.
Text Classification: Assigning categories or labels to a whole document, or parts of a document.
Sentence Boundary Detection (SBD): Finding and segmenting individual sentences.

Resources

Here is the link for there free Official Course: Advanced NLP with spaCy

More resources

A good blog post including installation process and other Spacy uses (Get started blog): Natural Language Processing With spaCy in Python
Intro to Spacy in Python (Videos) — Talks and tutorials in video format

2. NLTK Toolkit

NLTK is one of the most excellent libraries available out there to train NLP models. This library is elementary to use. It is a beginner-friendly library for NLP. It has a lot of pre-trained models and corpora, which helps us to analyze things very quickly.

Plus point: Built-in support for dozens of corpora and trained models

The best things you can do with NLTK?

Recommendation: Recommendation of content can be made based on the similarity.
Sentiment Analysis: It measures the inclination of people’s opinions through natural language processing
Wordnet[1] Support: We can use Synset to look up words in WordNet. And hence access to homonyms, Hypernyms, synonyms, definitions, family, etc. of many words
Machine Translation: Used to translate source content into target languages

Resources

Best source for learning NLTK is their book: Analyzing Text with the Natural Language Toolkit
Good collection for related articles: NLTK (Natural Language Toolkit) Tutorial in Python
Wordnet Docs — WordNet 3.0 Reference Manual

Unlike spaCy which focuses on providing software for production usage, NLTK is widely used for teaching and research — Wikipedia

3. Transformers

The transformers library is an open-source, community-based repository to train, use and share models based on the Transformer architecture[2] such as Bert[3], Roberta[4], GPT2[5], XLNet[6], etc.

The library downloads pre-trained models for Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks

Plus point: Over 32+ pre-trained models in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch. Best for deep learning

The best things you can do with Transformers

Summarization: Summarization is the task of summarizing a text / an article into a shorter text.
Translation: Task of translating a text from one language to another. 3.** Text Generation**: The goal is to create a coherent portion of text that is a continuation of the given context.
Extractive Question Answering: Task of extracting an answer from a text given a question.

Resources

Official Docs — Huggingface Transformers
Building question-answering API with BERT, HuggingFace, and AWS Lambda — Serverless BERT with HuggingFace and AWS Lambda
Learn how to fine-tune BERT for sentiment analysis — Sentiment Analysis with BERT and Transformers

4. Gensim Package

Gensim is a Python library that specializes in identifying semantic similarity between two documents through vector space modeling and topic modeling toolkit

By the way, it’s abbreviated for “Generate Similar” (Gensim) :)
Plus Point: High-level processing speed and the ability to handle large amounts of Text.

The best things you can do with Gensim

Distributed computing: It can able to run Latent Semantic Analysis and Latent Dirichlet Allocation on a cluster of computers. (Reason for handling the massive amount of data)
Document indexing: Process of associating the information with a file or specific tag allowing it to be easily retrieved later
Topic Modelling: Automatically clustering word groups and similar expressions that best defines a set of documents.
Similarity retrieval: Deals with the organization, storage, retrieval and evaluation of similar information from document repositories (here textual information)

Resources

Official API Documentation — API Reference
Official Tutorials — Core tutorials
Using Gensim LDA for hierarchical document clustering — Document Clustering with Python
Beginner tutorial for Installing, handling, etc. — Python for NLP: Working with the Gensim Library

5. Stanza

Stanza[7] is a collection of accurate and efficient tools for many human languages in one place. Starting from raw text to syntactic analysis and entity recognition, Stanza brings state-of-the-art NLP models to languages

The toolkit is built on top of the PyTorch library with support for using GPU and pre-trained neural models.

In addition, Stanza includes a Python interface to the CoreNLP Java package and inherits additional functionality from there

Plus point: It’s fast, accurate, and able to support several major languages. Suitable for product implementations

Resource: Here is the List of Python wrappers for CoreNLP

The best things you can do with Stanza

Morphological Feature Tagging: For each word in a sentence, Stanza evaluates its universal morphological features (e.g., singular/plural, 1st/2nd/3rd person, among others).
Multi-Word Token Expansion: Extending sentence split into the syntactic words as the foundation for downstream processing.

The innate characteristics of these five libraries make it a top choice for any project that relies on machine understanding of human expressions.

Thank you for reading. Don’t hesitate to stay tuned for more! Is there any other foundational or essential library? Let me know in the comments.

Footnotes

Introduction to WordNet: An On-line Lexical Database — George A. Miller & al. 1993
Attention Is All You Need — Vaswani & al., 2017
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Devlin & al., 2018
RoBERTa: A Robustly Optimized BERT Pretraining Approach — Liu & al., 2019
Language Models are Unsupervised Multitask Learners(GPT2) — Radford & al., 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding — Yang & al., 2019
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages — Peng & al., 2020

Why doesn’t Python have a main function?

Pawan Jain — Fri, 24 Jul 2020 04:58:40 +0000

Why don’t I recommend it either

There is no doubt that there is no so-called main entry function in Python, but there are often articles on the Internet that mention “Python’s main function” and “suggest to write the main function.”

Their intention may be to imitate the authentic main functions, but many people are misled (or misunderstood) and write very cumbersome code.

In this story, we will talk about Why Python does not have the main function?.

Before starting the topic, let’s answer these two questions:

What does the so-called “main function” mean?
Why do some programming languages need to force writing a main function?

Some programming languages use the main function as the execution entry of the program, such as C/C++, C#, Java, Go, Rust, etc., which have specific meanings:

This main function name is mandatory, which means that there must be a primary function.
There can be at most one main function, which means that the entry to the program is unique.
The grammar format has specific requirements, with a relatively fixed template.

Why do you have to force a main entry function?

These languages are compiled languages, which require code to be compiled into executable binaries in order for the operating system/bootloader to find The start of the program, so you need to define this one function.

Simply, there is a significant beginning that needs to be defined in a large pile of code that can be used for execution.

It is not difficult to see that the main function is an essential and indispensable organic part of those languages.
However, when we look at Python again, the situation is quite different.

Python is an interpreted language, i.e., a scripting language. The running process is from top to bottom, line by line, which means that its starting point is known.
Each .py file is an executable file, which can be used as the entry file of the entire program, implies the entry of the program is flexible, and no convention must be followed
Sometimes running the Python project, did not specify the entry file (the command line more common, such as “python -m http.server 8000”), it may be the presence of main.py file, it is in the package as a "file" to perform the operations.

In summary, it means that the scripting language Python is different from the compiled language. Whether it is at the level of a single module (that is, a .py file), or at the package level composed of multiple modules, it can choose a flexible execution method, Unlike other languages that cannot be implemented without a well-defined entry.

In other words, Python does not need to stipulate that the programmer must define a unified entry (whether it is a function or a class or something) at the syntax level.

Some people may be confused because they often see or write the following code themselves:

# main file
def main():
    ……
if __name__ == '__main__':
    main()

Isn’t this the main function of Python? I believe many of us think so!

No, It’s not

Except that the function name is “main” it does not have a half-cent relationship with the orthodox main function we introduced earlier, neither mandatory nor necessarily determining the order of program execution. Without it, it will not cause any grammatical problems.

The reason why some people want to name a “main” function is actually to emphasize its “main” status and wants to arrange it as the first function to executed artificially.

They may think that such named functions are easier to remember.

The reason why they want to write if name == 'main', may want to show that main() only runs when the current script is directly executed and doesn’t want to be run when it is imported into other modules.

However, I personally do not recommend this way of writing

The most obvious example: there are only a few dozen lines of code or just a script file that implements a simple function (a small crawler, drawing a picture with a turtle, etc.), but they’re all written in the same style as before.

It is not recommended to write if name__ == '__main' because of the following reasons.

First of all, if there is only one file because there is no possibility of export.
Secondly, if there are multiple files, it is highly discouraged to write this sentence in the entry file (main.py). Theoretically, its content shouldn’t be exported for use in other modules, since it’s the starting point.
Finally, it is also not recommended to write this judgment in non-entry files with multiple files, as the most that can be done is to write some of these test codes. Still, the test code should be separated and written in a dedicated directory or file.

Every time I see this cumbersome code without thinking, I feel uncomfortable. Why write that if statement? If possible, you should split the main function without even wrapping it into a function!

To sum up

Break inertial thinking and write authentic code. The main entry function is unique to some languages, and should not be used in Python. You should understand the characteristics of the scripting language and write a simple and elegant style.
Use main.py instead of main(). Because the program execution unit of Python is a script file, not a function or class, it is recommended to name the entry file main.py, and the internal functions are determined according to requirements.
If possible, use main.py as an entry file. This file is straightforward to use in combination with the "-m" parameter on the command line.

How do you feel about after reading this article? Do you like to use that if statement? Let me know in comments with the reason 😊