I'm currently learning a bunch of stuff related to NLP and pre-processing it, here's a few things I learned along the way (code in python 🐍).
Lowercasing the whole doc is important
Before we even want to play with any of the NL raw data source that we have, we need to ensure that everything is in one case, in most/all cases this would be lower case. This is because we want to ensure that whatever tokenizer we use is able to treat words the same. Granted, we do want to ensure that we don't miss the context out, in terms of miss-lowercasing names or important nouns, this will be dealt a little later on with a different library.
Removal of stop words and punctuations
Stop words, depending on your goal, may not carry much meaning in the larger aspect of the data. This is why it can be good to remove it, and it can be done using the nltk (specifically the nltk.Corpus) library. It has a lexicon of stopwords that it refers to when combing through your rows. You can also remove punctuation with the combination of the re (regex) library and the use of list comprehensions in python.
Lemmatization (not stemming, that one sucks)
This is the process where we, after removal of all mentioned above, we start to look at the base meaning of every word and replace each word in the text source with those. So for example, while looping through each word and coming across the word "studying", the lemmatizer could bring back the word "study" (depending on the context, it may just bring back the same word "studying"). And so it's essentially a shredding down of the movement of the word, past, present or future tense, into a still snapshot, if that makes any sense.
Tokenization - commonly "subwords" ( pieces of words)
This is the part where we start to put the words into palatable chunks that can be digested however we want, most likely to convert them into numbers or analysing them in weird and wonderful ways. The most common reason is to simply feed text into the neural network as part of training or something else along that line.
N-Grams
What I hear most about it is that it "analyses the relationship between neighbouring words". The literal meaning of a "N-gram" is just the number of tokens in the sequence of tokens that it will analyses the relationship in. So for example, saying 1 gram (or unigram) means getting a relationship of a set of 1 word; and saying 2 gram (bigram) means a relationship of a set of 2 words. Depending on how complex the exploration of the document you want to be, you may widen the net (or in this case the "N" in N-gram to get more meaning, and analyse more tokens in relationship with each other.
Top comments (0)