In the previous article, we have just went through the 2nd and 3rd components in LSTM, we will understand it further in this article.
Starting with the block furthest to the right, we multiply the short-term memory and the input by their respective weights:
This value, 2.03, becomes the input to the tanh activation function.
Now we plug 2.03 into the tanh function and obtain approximately 0.97.
The tanh activation function maps any input to a value between −1 and 1.
When the input to the LSTM is 1, after calculating the x-axis value, the tanh activation function produces an output close to 1.
In contrast, if the input to the LSTM were −10, then after calculating the x-axis value, the tanh activation function would produce an output close to −1.
So, based on the short-term memory and the input, we now have a potential memory of 0.97.
Next, the LSTM must decide how much of this potential memory to retain. This is done using the same method as before.
This value, 4.27, is the x-axis input to the sigmoid activation function.
When we plug this value into the sigmoid function, we obtain a y-axis value of approximately 1.0.
This means the entire potential long-term memory is retained, because multiplying it by 1 does not change it.
If the input were −10, then the percentage of potential memory to retain would be 0.
So, nothing would be added to the long-term memory.
Now, going back, we add 0.97 to the existing long-term memory:
This part is called the input gate.
Now we are ready to discuss the final stage of the LSTM, which we will explore in the next article.
Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.
Just run:
ipm install repo-name
… and you’re done! 🚀
Top comments (1)
The tanh + sigmoid split always confused me until I saw it framed as "what could be remembered" vs "how much of it to keep" — your Part 5 makes that distinction cleaner than any textbook I've read on it. Has Part 6 covered the output gate or are you saving it?