Understanding Word2Vec – Part 3: Predicting the Next Word with Neural Networks

#ai #machinelearning

In the previous article, we explored how words can be converted into neural network inputs and represented numerically. Now, we will look further into how the network uses these representations to make predictions.

In order to perform backpropagation, we first need the network to make predictions.

We will use the input word to predict the next word in the phrase.

For example, consider the sentence:

“The Incredibles is great!”

If the input word is “The Incredibles”, we indicate that by putting a 1 in the The Incredibles input and 0s in all other inputs.

In this case, we want the next word “is” to have the highest output value.

Similarly, if the input word is “is”, which means the input for is is 1 and all the other inputs are 0, then we want the output corresponding to the next word “great!” to have the largest value.

And if the input word is “Despicable Me”, then we want the output corresponding to the next word “is” to have the largest value.

To make these predictions, we connect the activation functions to output nodes, and we add weights to those connections with random initial values.

Then we run the outputs through the softmax function, since we have multiple possible outputs for classification.

For training, we use the cross-entropy loss function, which allows us to perform backpropagation.

Before training, if we plug “The Incredibles” into the input and compute the outputs, the network might barely predict the next word “is” correctly.

However, when we plug “is” into the input and perform the same computation, the network fails to predict the next word “great!”.

So we need to train this neural network so that it can make better predictions.

We will explore this training process further in the next article.

Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run: