What is Text Tagging? Understanding POS and NER in SpaCy

When you want to work with text in Python, the first step is usually figuring out what you are actually looking at. That is where text tagging comes in.

In this post, we will cover the two main ways to tag text: Parts Of Speech (POS) and Named Entity Recognition (NER). We will walk through how both methods work and how you can easily implement them using the SpaCy library.

Parts Of Speech (POS)

• This way of identifying text simply finds out, by iterating through the text word by word, whether that word is a verb, noun, adverb etc. In python, we would use the SpaCy library's English models (commonly "en_core_web_sm"), to help us do this.

import spacy
nlp = spacy.load("en_core_web_sm")
spacy_doc = nlp(some_text_to_analyse)

After loading the text that we want to analyse into the Natural Language Processor,it breaks eahc word down, into individual tokens, and each token has is it's own object that has properties like text or pos_:

for token in spacy_doc : 
    print({'token' : token.text, 'pos_tag': token.pos_})

you will get something like this:

....
{'token': 'emma', 'pos_tag': 'PROPN'}
{'token': 'woodhouse', 'pos_tag': 'PROPN'}
{'token': 'handsome', 'pos_tag': 'ADJ'}
{'token': 'clever', 'pos_tag': 'ADJ'}
...

Named Entity Recognition (NER)

This goes through the same process of going through the text, but what is different about it is that it is tasked with identifying people, places, organisations etc

In python, we can find it in the same process as POS, but we iterate through the .ents property rather than the whole spacy_doc object itself. Each word/token has a .label_ property that can be used.

for word in spacy_doc.ents:
    print(word.text, word.label_)

The way it does this "magical" process is via a combination of machine learning and a really good algorithm (I think it's called the "The transition-based algorithm").

A couple of notes on the NER in SpaCy, apparently removing punctation and capitalisation from the loaded document can cause the spacy doc to recognise less entities, or rather it is less confident about calling a word a certain entity.

So that is a basic overview of text tagging using SpaCy! Both POS tagging and NER make it easy to break down raw text and pull out useful information with just a few lines of Python. Just keep in mind that things like capitalisation and punctuation really matter for these models, so leaving your text as close to its natural state as possible will give you much better results.

Let me know if you have used SpaCy for text tagging before or if you are planning to use it in your next project! See ya in the next one 👋🏻