DEV Community: Fatima Jannet

NLP - Semantics and Sentiment Analysis

Fatima Jannet — Wed, 04 Jun 2025 05:28:49 +0000

Overview of Semantic and Word Vector

word2vec is a two layer neural net that processes text; where input is a text corpus and output is a vector set. Word2vec’s purpose is to group the vector of similar words. It detects similarities mathematically. It is kind of a numerical distribution of vector of of similar words.

word2veq is all about training words against other words. There is 2 ways of doing this.

bag of words (CBOW)
skip-gram

They are the opposite of each other. In CBOW, given the context word, it predicts the output word. In skip-gram, it tries to predict the context word from the output word.

Whatever the method is, each of the words are going to be a vector. In spacy, each of these vectors has 300 dimensions. Training an autoencoder for word2vec yourself can take a long time on a large text corpus. Because of this, it's uncommon for people to not use built-in embedded word vectors. However, if you want to train your own autoencoder for word2vec, you can choose either fewer or more dimensions. Typically, dimensions range from 100 to 1000. The more dimensions you have, the longer the training time will be. However, with more dimensions, you can capture more context around each word, as there is more space to store information.

Semantic and Word Vector (spaCy)

Important thing to notice: Larger spacy english model contains the word2veq. The smaller versions do not contain the word vectors.

# Import spacy 
import spacy 

# Directly download via spaCy (recommended)
!python -m spacy download en_core_web_lg

# Load the model 
import spacy
nlp = spacy.load("en_core_web_lg")  

nlp('BRAC University').vector

Doc and span objects also have vectors, which are created by averaging the vectors of individual tokens. This means you can not only use word2vec but also document2vec, where the document's vector is the average of all its words.

nlp('The quick brown fox jumps over the lazy dog.').vector.shape
nlp('dog').vector.shape

Essentially, both has 300 vectors.

# Identifying similar words 
tokens = nlp('lion cat pet')

for token1 in tokens: 
  for token2 in tokens: 
    print(token1.text, token2.text, token1.similarity(token2))

The words "lion lion," "cat cat," and "pet pet" have a 100% similarity with each other. The similarity values range between zero and one. It makes sense that a word is completely similar to itself. What's interesting is that the word vectors have enough information to show that "lion" and "cat" have some similarity, with a score of 0.52. "Cat" and "pet" tend to have a high similarity because most cats are pets, so it makes sense they have a high similarity score. It also makes sense that "lion" and "pet" have a similarity of less than 0.5, as having a lion as a pet isn't common, though some people do. We can see that relationships are established just from the word vectors. Essentially, this process checks the cosine similarity between token one and token two.

Love and hate are very different words with very different meanings. However, they're often used in the same context. You either love a movie or hate a movie, love a book or hate a book. In this way, they are similar because they're frequently used in similar situations. So, words like these can often have very similar vectors.

So, remember that if words are used in a similar context, they might be similar even if they have opposite meanings in regular English.

Sometimes, it's useful to combine 300 dimensions into a Euclidean L2 norm.

nlp.vocab.vectors.shape
#OUTPUT (342918, 300)

Sometimes you may encounter words that is outside of the 342918×300 dimension’s outside. To check if a word is outside of vocabulary (token.is_oov), we have attributes to check.

# Outside of vocab attribute checking 
tokens = nlp('dog cat nargle')

for token1 in tokens: 
  print(token1.text, token1.has_vector, token1.vector_norm, token1.is_oov)

Nargel is a made-up word, so it doesn't have a vector. This means it returns a norm of 0.0. Is it outside the vocabulary? Yes, that's true. Keep in mind that common names might actually have vectors. For example, if we add karen, it does have a vector associated with it, and even some uncommon names. If I were to include my middle name, it is actually in the vocabulary, which is quite interesting.

Another thing is, vector arithmetic. You can actually calculate a new vector by adding, subtracting related vectors from these words. A famous example is: king - man + queen = queen.

# NOTE: THIS IS A PERFECT CODE BUT IT'S NOT GIVING THE EXPECTED OUTPUT.
# MOST LIKELY, ISSUES RELATED WITH THE LARGE LANGUAGE MODEL 

# King - Man + Woman = Queen. 
# Importing spatial to perform arithmetic
from scipy import spatial 

cosine_similarity = lambda vec1, vec2: 1 - spatial.distance.cosine(vec1, vec2)

# Grabbing their vector
king = nlp.vocab['king'].vector
man = nlp.vocab['man'].vector 
woman  = nlp.vocab['woman'].vector

# King - Man + Woman = Queen --> new_vector similar queen, princess, highness 

new_vector = king  - man + woman
computed_similarities = [] 

# Going all out in the entire vocabulary 
# FOR ALL WORDSIN MY VOCAB 

for word in nlp.vocab: 
  if word.has_vector: 
    if word.is_lower: 
      if word.is_alpha: #numbers 
        if word.is_stop == False:
          similarity = cosine_similarity(new_vector, word.vector)
          computed_similarities.append((word, similarity))

# Sorting in a decsending order
# Without the minus, it sorts in ascending order, showing the least similar words.
computed_similarities = sorted(computed_similarities, key = lambda item: -item[1])


# Printing the top 10 word from the tuple

# Grab the first word of the tuple, ask for it's text
# Then do it for tuple in computed_similarities
# Go through top 10, :10
print([t[0].text for t in computed_similarities[:10]])
# Expected output: ['queen', 'monarch', 'princess', 'royal', 'throne', ...]

There’s some sort of understanding of royalty related with king, and some sort of understanding of the genders here. We understand King is a man. And if we subtract the gender dimension from King and add woman to it, we get something of a royalty as well.

Sentiment Analysis

We have a tool called VADER (Valence Aware Dictionary for Sentiment Reasoning). It processes raw text data with an algorithm to determine sentiment. No pre-labeled training is needed. VADER is a model used for text sentiment analysis, and it is sensitive to polarity, indicating both positivity and negativity.

Vader is available in NLTK. We can directly use it on unlabeled text data. Vader sentiment analysis depends on a dictionary which maps the Lexical features to emotion intensities AKA sentiment score. Sentiment score is obtained by summing up the intensity of each word in a text, so like, every single word has a thing to do with sentiment score. words like love, like, enjoy, happy - all convey to positive sentiment. Vader is smart enough to understand ‘did not enjoy’ is a negative sentiment. Vader take into account of every single word. Also, Vader will mark the intensity of LOVE!!! > love. But yah, Vader can’t detect sarcasm. It’s really difficult. I mean, how it can detect if it sees positive words getting used in a negative way?

Okay, however, let’s see how to use Vader in NLTK.

Sentiment Analysis (NLTK)

import nltk

nltk.download('vader_lexicon')
from nltk.sentiment.vader import SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer()

review = "This is a good anime"
sid.polarity_scores(review) # Only one +ve word 

review = "BEST ANIME! This was the best, most AWESOME anime MADE EVER IN HISTORY!!!"
sid.polarity_scores(review) # It has many positive words, capitalized, even had '!!! marks'

review = "This was the WORST anime!! YUCK! waste of my time!"
sid.polarity_scores(review) # It has no positive word, rather bunch of negative words.

Let’s do it on amazon reviews.

Get your resources here
The whole process is written in the Colab file. There's not much to explain here, so please check out the Colab file.

That’s it! please run this model with moviereviews.tsv which is also given in the github repo.

What is Deep Learning?

Fatima Jannet — Sun, 17 Nov 2024 17:39:08 +0000

About 25-30 years ago, people didn't know what the internet was. Now, we can't imagine a day without it!

However, I’ll be giving you a quick run on what is deep learning.

Neural networks and deep learning was invented back at 60s. But they got recognition in the 80s. People started talking about them, conducted a lot of researches and so on. They thought it will change the whole world but then the hype kind of died.

Why is that? The technology to facilitate neural networks were not on the right margin. For deep learning and neural networking you need two things: enormous amount of data and strong processing power; which was not available back at that time.

Let's have look at three years: 1956, 1980, 2017.

Do you know how did storage look back in 1956?

Well, this is a hard drive, and that's 5 megabytes right there. It's on a forklift, about the size of a small room. It was being moved to another location on a plane. In 1956. A company had to pay 2,500$ of that time's money to rent that hard drive for one month, not buy it, just rent it.

In 1980, the situation had improved a little bit. Still very expensive for only 10 megabytes (which is like one photo these days)

And in 2017, we've got a 256 gigabyte SSD card for $150, which can fit on your finger.

So from 1956 to 1980, storage capacity doubled, and then it increased about 25,600 times by 2017. The time periods aren't very different, but there was a huge leap in technology. This shows that the growth isn't linear; it's exponential.

Now, for the processing: here's a chart on an algorithmic scale. If we plot hard drive costs per gigabyte, it quickly approaches zero. Now, you can get free storage on Dropbox and Google Drive.

Currently, scientists are exploring the use of DNA for storage, although it's quite expensive. It costs $7,000 to synthesize 2MB of data and another $2,000 to read it.

But have you noticed that this situation is quite similar to the early days of hard drives and planes? This is also going to improve rapidly due to the exponential growth curve. Ten or twenty years from now, everyone will be using DNA storage.

This information is from Nature. As you can see, you can store all the world's data in just 1 kilogram of DNA storage. Alternatively, you can store about 1 billion terabytes of data in just 1 gram of DNA storage.

This example shows how fast we're progressing, which is why deep learning is gaining momentum now.

The same growth applies to processing capacity, which is also increasing at an exponential rate. This is known as Moore's Law, and you've probably heard of it.

Right now, computers have surpassed the thinking ability of a rat! They have already exceeded the thinking capacity of a human brain, and by 2040 or 2045, they will surpass the combined thinking power of all humans. So, basically we're entering the era of computers that are incredibly powerful and can process things much faster than we can imagine and this is what exactly facilitating the deep learning.

Now all of this makes us question: What exactly is deep learning? What is neural networking? what is it? What is going on here?

This gentleman, Geoffrey Hinton, is known as the godfather of deep learning. He conducted research on deep learning in the 1980s and has done a lot of work in the field. He has published many research papers on deep learning. Currently, he works at Google, so much of what we will discuss comes from Geoffrey Hinton. He won the 2024 Nobel Prize in Physics

[He has many great YouTube videos where he explains things clearly, so I highly recommend checking them out!]
Image description

The idea behind deep learning is to look at the human brain. It tries to mimic the brain (neuroscience stuffs).We don't know everything about the human brain, but with the little we do know, we try to mimic it.** Why?** Because the human is one of the most powerful learning tool on this planet The way brain learns, adapts skills and apply them - we want our computers to copy that.

Here we have some neurons. These neurons are spread onto glass and observed under a microscope with some coloring. You can see what they look like. They have a body, branches, and tails. You can also see a nucleus in the middle. That's what a neuron looks like. In the human brain, there are approximately a hundred billion individual neurons in total and they are connected with each other.

So to give you a picture of this, this is what it looks like, this is an actual dissection of the human brain.
This is just to show how vast the network of neurons is. There are billions and billions of neurons all connected in your brain. We're not talking about five hundred, a thousand, or even a million. We're talking about billions of neurons who takes care of memorizing, balancing etc. And yes, that's what we're going to try to recreate.

So how do we recreate this in a computer? We create an artificial structure called an artificial neural network.

We have nodes or neurons, which are used for input values. These are the values you know about a certain situation. For example, if you're modeling something and want to make predictions, you'll need some input to begin with. This is called the input layer.

Then you have the output, which is the value you want to predict. This is called the output layer.

And in between, we have a hidden layer. In your brain, information comes through your senses like eyes and ears. It doesn't go straight to the result; it passes through many neurons first. This is why we use hidden layers in modeling the brain before reaching the output. This is the whole concept behind it: we are going to model the brain, so we need these hidden layers before the output.

Neurons are connected to the hidden layer, and the hidden layer is connected to the output value.

And then we connect everything, just like in the human brain. Connect everything, interconnect everything. That's how the input values process through all these hidden layers, just like in the human brain, and then we get an output value.

This is what deep learning is all about at a very abstract level.

I will be uploading two blogs on two deep learning models.

Artificial Neural Networks for a Business Problem
Convolutional Neural Networks for a Computer Vision task

My blogs are pretty long, so I'll upload them on Hashnode and then share another blog here with links to those two posts. Thanks for reading!

ML Chapter 7: Natural Language Processing

Fatima Jannet — Sun, 17 Nov 2024 04:18:20 +0000

Natural Language Processing (NLP) involves using machine learning models to work with text and language. The goal of NLP is to teach machines to understand spoken and written words. For example, when you dictate something into your iPhone or Android device and it converts your speech to text, that's an NLP algorithm at work.

You can also use NLP to analyze a text review and predict whether it's positive or negative. NLP can categorize articles or determine the genre of a book. It can even be used to create machine translators or speech recognition systems. In these cases, classification algorithms help identify the language. Most NLP algorithms are classification models, including Logistic Regression, Naive Bayes, CART (a decision tree model), Maximum Entropy (also related to decision trees), and Hidden Markov Models (based on Markov processes).

Small insight before starting: On the left of the Venn diagram, we have green representing NLP. On the right, we have blue representing DL. In the intersection, we have DNLP. There's a subsection of DNLP called Seq2Seq. Sequence to sequence is currently the most cutting-edge and powerful model for NLP. However, we won't discuss seq2seq in this blog. We will be covering basically the bag-of-words classification.

In this part, you will understand and learn how to:

Clean text to prepare it for machine learning models.
Create a Bag of Words model.
Apply machine learning models to this Bag of Words model. Here’s what we will be focusing on. Note: We will not discuss Seq2Seq, chatbots, or deep NLP. The materials I have used are from NLP with DL, so we will exclude the DL part.

To read the full blog: ML Chapter 7: Natural Language Processing

Calculus for ML: Derivatives Part - 01

Fatima Jannet — Thu, 16 May 2024 12:54:08 +0000

Do you know calculus plays an important role in ML? I am going to show you how derivates performs a vital role to make the most accurate model for you. Let's start !!

First let's have a look at some derivatives.

*Calculating a slope: *

slope = rise/run
= change in x axis/change in y axis
= ∆x/∆y

Here, the slope is going to be -
slope = ∆x/∆t
= x(15) - x(10) / t(15) - t(10)
= 202-122 / 15-10
= 16 m/s

Now, I ask you a question. At t = 12.5. what is the velocity? If not the exact velocity, find me a good estimate.
Let's find out the slope,

            slope = ∆x/∆t
                  = (177-155)/(13-12) = 15 m/s

But this is just a roughly value. We still don't know the exact estimate value of v at t=12.5s. To find this, we need finer and finer intervals. The more finer, the more accurate answer it is, and this is where the importance of derivate comes.

1. Derivatives and Tangents
A tangent is a line that touches a curve at only one point. The line points out the slope of the tangent to that point. A derivative of a function gives you the gradient of a tangent at a certain point on a curve.

Look, how the intervals are coming closer as we are getting smaller to smaller. This gets so close that we can't even find the difference. These are called instantaneous rate of change aka the derivatives.

2. Derivative of a line

slope = ∆y/∆x
= rise/run
= a(x+∆x)+b-(ax+b) / x+∆x-x
= a∆x/∆x = a
So, the derivative of the eq(ax+b) is the slope a.

3. Derivative of Quadratic Function :

slope = ∆f/∆x = {f(x+∆x)^2 - f(x)^2} / ∆x
Between (1,1) and (1.5, 2.25) -
∆x = 2-1 = 1
[∆f = 4-1 = 3 ; y2 - y1]
∆f = (x+∆x)^2 - (x)^2
=> (1+1)^2 - 1 => 4-1 = 3
so, slope = 3

Between (1,1) and (1.5, 2.25)-
∆x = 1.5 - 1 = 0.5
∆f = (x+∆x)^2 - (x)^2
= (1+0.5)^2 - 1 = 1.25
slope = 1.25/0.5 = 2.5

Even smaller: (1,1) and (1.25, 1.56) -
∆x = 1.25-1 = 0.25
∆f = (x+∆x)^2 - (x)^2 = (1+0,25)^2 - 1 = 0.56
slope = 0.56 / 0.25 = 2.25

Again, d/dx(f) = {f(x+∆x)^2 - f(x)^2} / ∆x
= (x^2 + 2x∆x + ∆x^2)/ ∆x = 2x + ∆x

So, f(x) = x^2 => f'(x) = 2x

4. Derivative of Cubic function

slope = ∆f/∆x = {f(x+∆x)^3 - f(x)}/∆x
If you apply the cubic formula, it will end at 3x^2 + 3x∆x + ∆x^2.
As, ∆x -> 0, so, f'(y) = 3x^2

5. Derivative of 1/x

Here, i am going to make it short -
slope = ∆f/∆x = {f(x+∆x)^(-1) - f(x)}/∆x

f(x) = x^(-1)
=> f'(x) = -x^(2)

I hope you noticed that i basically followed the rule nx^(n-1)

Calculus for ML: Derivatives. Part - 2

Fatima Jannet — Thu, 16 May 2024 12:52:09 +0000

Today I'll be talking about the Inverse function, derivative of trigonometric function, exponential and logarithms.

Inverse function
The inverse function's job is to basically undo another function. The general definition is - if you get y out of f(x), then if you put y into the inverse of f, then you will get x.

If you put x through f function and you get x^2, then you put x^2 into g function, then you get x. That's why inverse function is also known as anti-function.

Now, let's have a look at their graphs.

As we saw, they are a reflection of each other (check the axis if you have any confusion). So. the slope is inverse of one another.
Slope : ∆f/∆x = ∆y/ ∆g and, g'(y) = 1/f'(x)

Derivative of trigonometric function
Derivatives of the trigonometric functions can be found on sin(x) and cos(x). The tangent, tan(x) = sin(x)/cos(x)

f(x) = sinx
=> f'(x) = cosx

Exponentials
Do you know Euler's number? To understand exponential function we need to first understand euler's number. Euler's Number 'e' is a numerical constant used in mathematical calculations. The value of e is 2.718281828459045…so on. As it's an irrational number, e can't be expressed through ration between two integers. So, we will define it is by using the expression (1 + 1/n)^n. Let's look at this expression for several values of n.

Look here, the more we increase the number o n, the more it goes to a certain value and if we keep doing this, it gets more closer to 2.7182818284...

the function and it's derivative both are the same. Why it is the same? I'll explain it to another blog!!

Exponential function is related to many real life situations such as any kind of growth, investment, bank deposits, loans and profit, et cetera.

*Logarithms *
The logarithmic function is the inverse function of exponentiation.

The right graph was calculated by using the result for inverses.
Therefore, the derivative of logarithm is 1 over y.