Handle contractions in text preprocessing - NLP

Antaripa Saha — Thu, 18 Feb 2021 09:19:13 +0000

Text preprocessing is a crucial step in NLP. Cleaning our text data in order to convert it into a presentable form that is analyzable and predictable for our task is known as text preprocessing. In this article, we are going to discuss contractions and how to handle contractions in text.

What are contractions?

Contractions are words or combinations of words that are shortened by dropping letters and replacing them by an apostrophe.

Nowadays, where everything is shifting online, we communicate with others more through text messages or posts on different social media like Facebook, Instagram, Whatsapp, Twitter, LinkedIn, etc. in the form of texts. With so many people to talk to, we rely on abbreviations and shortened forms of words for texting people.

For example,

I’ll be there within 5 min. Are u not gng there? Am I mssng out on smthng? I’d like to see u near d park.

In English contractions, we often drop the vowels from a word to form the contractions. Removing contractions contributes to text standardization and is useful when we are working on Twitter data, on reviews of a product as the words play an important role in sentiment analysis.

How to expand contractions?

1. Using contractions library

First, install the library. You can try this library on Google colab as installing the library becomes super smooth.

Using pip:

!pip install contractions

In Jupyter notebook:

import sys !{sys.executable} -m pip install contractions

Code 1: For expanding contractions using contractions library

Python3

# import library 
import contractions 
# contracted text 
text = '''I'll be there within 5 min. Shouldn't you be there too?  
          I'd love to see u there my dear. It's awesome to meet new friends. 
          We've been waiting for this day for so long.'''

# creating an empty list 
expanded_words = []     
for word in text.split(): 
  # using contractions.fix to expand the shotened words 
  expanded_words.append(contractions.fix(word))    

expanded_text = ' '.join(expanded_words) 
print('Original text: ' + text) 
print('Expanded_text: ' + expanded_text)

Output:

Original text: I'll be there within 5 min. Shouldn't you be there too? I'd love to see u there my dear. It's awesome to meet new friends. We've been waiting for this day for so long.

Expanded_text: I will be there within 5 min. should not you be there too? I would love to see you there my dear. it is awesome to meet new friends. we have been waiting for this day for so long. Removing contractions before forming word vectors helps in dimensionality reduction.

Code 2: Simply using contractions.fix to expand the text.

Python3


text = '''She'd like to know how I'd done that!  
          She's going to the park and I don't think I'll be home for dinner. 
          They're going to the zoo and she'll be home for dinner.'''

contractions.fix(text)

Output:

'she would like to know how I would do that! she is going to the park and I do not think I will be home for dinner. they are going to the zoo and she will be home for dinner.'

Contractions can also be handled using other techniques like dictionary mapping, and also using pycontractions library.

You can refer to the documentation of pycontractions library for learning more about this: https://pypi.org/project/pycontractions/

Original article

DEV Community: Antaripa Saha

Handle contractions in text preprocessing - NLP

What are contractions?

How to expand contractions?

1. Using contractions library