DEV Community: Rijul Rajesh

Typing Too Much While Using AI? This Tool Will Save You Hours

Rijul Rajesh — Mon, 06 Jul 2026 20:04:44 +0000

Hello, I'm Rijul. I'm building git-lrc, a micro AI code reviewer that runs on every commit. It's free and source-available on GitHub. Star git-lrc to help more developers discover the project. Do give it a try and share your feedback

Are you tired of typing the same prompts over and over again?

I definitely was.

When working with AI tools, I noticed I kept writing the same instructions repeatedly. Sometimes I'd even dig through old chats just to copy and paste a prompt I had already written dozens of times.

It isn't difficult, but it breaks your flow.

Thankfully, there's a fantastic tool that completely solves this problem: Espanso.

You can create your own shortcuts, and Espanso instantly expands them into full text anywhere on your computer.

For example, here's the built-in :date shortcut.

As soon as I type :date, it automatically replaces it with today's date.

Pretty cool, right?

Let's install it and create our own shortcuts.

Installing Espanso

Head over to espanso.org.

You'll find the installation button right on the homepage.

Installation depends on your operating system:

macOS: Install the bundle or use Homebrew.
Windows: Download and run the installer.
Linux: Choose the installation method for your distribution.

Linux users

Before installing, check whether you're running X11 or Wayland.

echo $XDG_SESSION_TYPE

My machine is running X11, so I'll install the AppImage like so.

# Create the $HOME/opt destination folder
mkdir -p ~/opt

# Download the AppImage
wget -O ~/opt/Espanso.AppImage 'https://github.com/espanso/espanso/releases/latest/download/Espanso-X11.AppImage'

# Make it executable
chmod u+x ~/opt/Espanso.AppImage

# Create the "espanso" command alias
sudo ~/opt/Espanso.AppImage env-path register

Once the installation finishes, verify that everything is working.

espanso --version

Now start Espanso.

# Required only once
espanso service register

# Start Espanso
espanso start

That's it. Espanso is now running in the background.

If you'd like more installation details, the official documentation is available here:

https://espanso.org/docs/get-started/

Witness the Magic

Let's try one of the built-in shortcuts.

I'll type:

:espanso

As soon as I finish typing, Espanso replaces it with "Hi there!"

No copy-pasting.

No scripts.

Just instant text expansion.

Now let's create our own shortcut.

Creating Your Own Shortcuts

Espanso stores its configuration in a simple YAML file.

First, locate your configuration directory.

espanso path

Mine returns:

Config: /home/rtp/.config/espanso
Packages: /home/rtp/.config/espanso/match/packages
Runtime: /home/rtp/.cache/espanso

Open the Config directory and navigate to the match folder.

I will head to /home/rtp/.config/espanso

Inside, you'll find a file named base.yml.

Open it.

You'll notice some existing shortcuts already defined.

For example:

:date inserts the current date.
:espanso expands to Hi there!

Creating your own shortcut is just as easy.

A Real-World Example

When I'm prompting AI, I frequently include this instruction:

Make only minimal, non-destructive changes to the code.

It helps prevent the AI from making unnecessary modifications that could accidentally break my software.

The problem is that I end up typing this sentence countless times every week.

Instead, I created a shortcut.

- trigger: ":mdc"
  replace: "Make only minimal, non-destructive changes to the code."

After adding it to base.yml, my configuration looks like this:

Now, whenever I type :mdc, Espanso instantly expands it into the full sentence.

It feels like magic, and after a few days of using it, you'll wonder how you ever worked without it.

Wrapping up

Espanso is one of those small tools that quietly saves you hundreds of keystrokes every day.

Whether you're writing prompts, code snippets, email templates, commit messages, or anything repetitive, custom text expansions can make your workflow noticeably faster.

Huge kudos to Federico Terzi for creating such an awesome tool.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building Word Embeddings with PyTorch and Lightning AI Part 2: Creating Labels for Next-Word Prediction

Rijul Rajesh — Sat, 04 Jul 2026 21:28:40 +0000

In the previous article, we created the training inputs for our word embedding model using one-hot encoding. In this article, we will define the labels and prepare the data for training

Let's continue by setting up the labels.

Our goal is to predict the next token given the current token.

Consider the sentence:

The Incredibles is great

If the current token is "The Incredibles", the next token is "is".

So, the correct label is the one-hot encoding for "is".

Next, if the current token is "is", the model should predict the one-hot encoding for "great".

After "great", the sentence ends. To continue training with the next sentence, we set the next token to the one-hot encoding for "Despicable Me".

Now let's convert these labels into a PyTorch tensor.

labels = torch.tensor([
    [0., 1., 0., 0.],
    [0., 0., 1., 0.],
    [0., 0., 0., 1.],
    [0., 1., 0., 0.]
])

At this point, we have finished encoding our training data.

Next, we combine the inputs and labels into a TensorDataset, and then use that dataset to create a DataLoader.

dataset = TensorDataset(inputs, labels)
dataloader = DataLoader(dataset)

Creating the Neural Network

Now let's start implementing the word embedding model.

class WordEmbedding(L.LightningModule):
    def __init__(self):
        # Initialize the weight tensors for the embedding network and loss function

    def forward(self):
        # Make a forward pass through the embedding network

    def configure_optimizers(self):
        # Configure the Adam optimizer

    def training_step(self, batch, batch_idx):
        # Calculate the loss (Cross Entropy Loss)

In the next article, we will begin implementing each of these methods one by one.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building Word Embeddings with PyTorch and Lightning AI Part 1: Setting Up the Word Embedding Model

Rijul Rajesh — Fri, 03 Jul 2026 18:56:36 +0000

In this article, we will explore how to implement word embeddings using PyTorch and Lightning AI.

The implementation is based on the same example that we used in my Word2Vec article series.

First, let's import the required modules.

import torch
import torch.nn as nn

from torch.optim import Adam
from torch.distributions.uniform import Uniform
from torch.utils.data import TensorDataset, DataLoader

import lightning as L

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

These imports are used for the following purposes:

torch – Create tensors and use helper functions.
torch.nn – Create neural network layers and weights.
Adam – Optimize the neural network using backpropagation.
Uniform – Initialize the network weights.
TensorDataset and DataLoader – Prepare and load the training data.
pandas, matplotlib, and seaborn – Analyze and visualize the results.

Creating the Training Inputs

Now let's create the word embeddings for the following two sentences:

The Incredibles is great!
Despicable Me is great!

The first step is to create a token for every unique word in the training data and connect those tokens to the word embedding network.

Suppose we want to pass "The Incredibles" through the network.

In that case, we set its corresponding input to 1 and all the other inputs to 0.

This representation, where one input is 1 and all the others are 0, is called one-hot encoding.

In PyTorch, we can represent "The Incredibles" using a four-element vector with 1 in the first position and 0 everywhere else.

Similarly, we create one-hot vectors for the remaining words.

[
    [1., 0., 0., 0.],
    [0., 1., 0., 0.],
    [0., 0., 1., 0.],
    [0., 0., 0., 1.]
]

Finally, we convert these vectors into a PyTorch tensor.

inputs = torch.tensor([
    [1., 0., 0., 0.],
    [0., 1., 0., 0.],
    [0., 0., 1., 0.],
    [0., 0., 0., 1.]
])

Now that we have created the input data, the next step is to define the labels. We will explore that in the next article.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 9: Completing the Simplified LSTM

Rijul Rajesh — Thu, 02 Jul 2026 21:04:46 +0000

In the previous article, we just saw how we can start using a more simplified version of LSTM via pytorch via nn.LSTM()

In this article, we will continue building the simplified LSTM and test how it performs.

Implementing the `forward()` Method

Let's start by implementing the forward() method.

def forward(self, input):
    input_trans = input.view(len(input), 1)
    lstm_out, temp = self.lstm(input_trans)
    prediction = lstm_out[-1]
    return prediction

In input_trans, we reshape the input so that there is one row for each data point, regardless of how many data points we have.

Next, we specify that the input should have one column, since each data point contains only a single feature.

This reshaped input is then passed to the LSTM.

The output is stored in lstm_out.

lstm_out contains the short-term memory values produced by each LSTM unit as the sequence is processed.

In our example, the sequence contains four input values, so the LSTM is unrolled four times and lstm_out contains four outputs.

Next, we extract the prediction from the final LSTM unit by selecting the last element in the sequence using the index -1.

Finally, we return this prediction.

Configuring the Optimizer

Next, let's implement configure_optimizers().

It is almost identical to the previous implementation, except that we increase the learning rate from its default value of 0.001 to 0.1.

This allows us to observe how the Adam optimizer converges to the optimal weights and biases.

def configure_optimizers(self):
    return Adam(self.parameters(), lr=0.1)

Implementing `training_step()`

Finally, we implement training_step().

This method is exactly the same as before. It calculates the loss and logs the training progress.

def training_step(self, batch, batch_idx):
    input_i, label_i = batch
    output_i = self.forward(input_i[0])
    loss = (output_i - label_i)**2

    self.log("train_loss", loss)

    if label_i == 0:
        self.log("out_0", output_i)
    else:
        self.log("out_1", output_i)

    return loss

At this point, our model contains everything it needs:

__init__()
forward()
configure_optimizers()
training_step()

Testing the Model

Let's run the model before training and check its predictions.

model = LightningLSTM()

print("\nComparing observed and predicted values")

print(
    "Company A: Observed = 0, Predicted =",
    model(torch.tensor([0., 0.5, 0.25, 1.])).detach()
)

print(
    "Company B: Observed = 1, Predicted =",
    model(torch.tensor([1., 0.5, 0.25, 1.])).detach()
)

This produces:

Comparing observed and predicted values

Company A: Observed = 0, Predicted = tensor([0.0131])
Company B: Observed = 1, Predicted = tensor([0.0102])

Training the Model

Now let's create a Lightning trainer.

trainer = L.Trainer(max_epochs=300, log_every_n_steps=2)

We train for 300 epochs and set log_every_n_steps to 2.

By default, Lightning logs every 50 steps, which is too infrequent for a small training run like this.

Next, we simply call fit().

trainer.fit(model, train_dataloaders=dataloader)

Checking the Results

Once training is complete, we check the predictions again.

Comparing observed and predicted values

Company A: Observed = 0, Predicted = tensor([0.0001])
Company B: Observed = 1, Predicted = tensor([0.9857])

The prediction for Company A is now very close to 0, and the prediction for Company B is very close to 1.

Analyzing the Training

Let's open TensorBoard once again.

Notice that the graphs have flattened out as the model converged toward the desired predictions.

With that, we have completed our exploration of LSTMs.

We built an LSTM from scratch, then implemented a much simpler version using PyTorch's built-in functionality. Along the way, we also learned how to analyze the training process using TensorBoard and how to use those insights to make training decisions.

In the next series of articles, we will explore how to implement word embeddings using PyTorch and Lightning AI.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 8: Setting Up a Simpler LSTM

Rijul Rajesh — Wed, 01 Jul 2026 19:06:51 +0000

In the previous article, we saw how easily we could continue training by adding more epochs. We also observed the improvements in the model's predictions using TensorBoard.

Let's train the model one more time to bring the predictions even closer to the desired values.

As before, we first retrieve the latest checkpoint.

path_to_best_checkpoint = trainer.checkpoint_callback.best_model_path

Next, we increase the maximum number of epochs to 5000.

trainer = L.Trainer(max_epochs=5000)

Then we resume training by calling fit() with the checkpoint.

trainer = L.Trainer(max_epochs=5000)

trainer.fit(
    model,
    train_dataloaders=dataloader,
    ckpt_path=path_to_best_checkpoint
)

Once training is complete, we can print the predictions again.

print("\nComparing observed and predicted values")

print(
    "Company A: Observed = 0, Predicted =",
    model(torch.tensor([0., 0.5, 0.25, 1.])).detach()
)

print(
    "Company B: Observed = 1, Predicted =",
    model(torch.tensor([1., 0.5, 0.25, 1.])).detach()
)

This gives the following output:

Comparing observed and predicted values

Company A: Observed = 0, Predicted = tensor(0.0004)
Company B: Observed = 1, Predicted = tensor(0.9672)

As you can see, both predictions are now much closer to their target values.

Next, let's open TensorBoard and inspect the updated training graphs.

Notice that all of the graphs have become much flatter, indicating that the model has nearly converged.

The predictions for Company A and Company B are now very close to their target values of 0 and 1, respectively.

At this point, we have successfully trained our LSTM model.

Now let's look at an even simpler way to build an LSTM using PyTorch's built-in nn.LSTM module.

We will start by creating another class.

class LightningLSTM(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(input_size=1, hidden_size=1)

The input_size specifies the number of features in each input.

In our example, each day contains only a single feature, which is the stock price of a company.

The hidden_size specifies the number of values in the hidden state.

For this example, we use a hidden size of 1, since we ultimately want the model to predict a single value for Day 5.

Next, we will implement the forward() method, which works a little differently when using nn.LSTM. We will explore that in the next article.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 7: Resuming Training with Checkpoints

Rijul Rajesh — Tue, 30 Jun 2026 20:44:51 +0000

In the previous article, we used TensorBoard to analyze the training process. Based on the graphs, we concluded that the model had not fully converged and could benefit from additional training epochs.

Let's continue with that in this article.

One of the advantages of Lightning is that we can continue training without starting from scratch.

This is possible because Lightning automatically saves checkpoints during training.

These checkpoints allow us to resume training from where we left off and continue optimizing the model.

Getting the Checkpoint

First, we need to find the path to the latest checkpoint.

path_to_best_checkpoint = trainer.checkpoint_callback.best_model_path

Here, best_model_path gives us the path to the latest checkpoint that Lightning has saved.

Increasing the Number of Epochs

Now we create a new trainer and increase the maximum number of epochs to 3000.

trainer = L.Trainer(max_epochs=3000)

Instead of starting from the beginning, we resume training from the saved checkpoint.

trainer.fit(
    model,
    train_dataloaders=dataloader,
    ckpt_path=path_to_best_checkpoint
)

By specifying ckpt_path, Lightning continues training from the saved checkpoint instead of initializing the model again.

Checking the Updated Predictions

Now let's print the predictions once again.

print("\nComparing observed and predicted values")

print(
    "Company A: Observed = 0, Predicted =",
    model(torch.tensor([0., 0.5, 0.25, 1.])).detach()
)

print(
    "Company B: Observed = 1, Predicted =",
    model(torch.tensor([1., 0.5, 0.25, 1.])).detach()
)

This produces the following output:

Comparing observed and predicted values

Company A: Observed = 0, Predicted = tensor(0.0009)
Company B: Observed = 1, Predicted = tensor(0.9423)

The prediction for Company A has moved even closer to the target value of 0.

Similarly, the prediction for Company B has moved closer to the target value of 1.

Comparing the TensorBoard Graphs

Let's look at TensorBoard again.

Company A

After training for more epochs, the prediction has moved closer to the desired value of 0.

Company B

Similarly, the prediction for Company B has moved closer to the desired value of 1.

Training Loss

The train_loss graph also shows that the loss has decreased further after the additional training.

Although the model has improved, we can still train it for more epochs to refine the predictions even further.

In the next article, we will continue improving the model and also explore how Lightning can simplify LSTM implementations using PyTorch's built-in nn.LSTM() module.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 6: Analyzing Training with TensorBoard

Rijul Rajesh — Mon, 29 Jun 2026 19:31:30 +0000

In the previous article, we trained our model and checked the outputs, In this article we will take that analysis a step further by using TensorBoard

Launching TensorBoard

First, install TensorBoard if you haven't already.

pip install tensorboard

Then start TensorBoard by running:

tensorboard --logdir=lightning_logs/

We will be doing the analysis based on the lightning logs.

Once the command runs, TensorBoard will start a local server, usually available at http://localhost:6006.

Open it in your browser and navigate to the Scalars section.

Analyzing the Training Loss

Scroll down until you find the train_loss graph.

This graph plots:

X-axis: Training steps
Y-axis: Loss value

As training progresses, the loss steadily decreases toward zero.

Notice that the loss drops very quickly during the early stages of training. As the model improves, the rate of decrease slows down, since there is less error left to correct.

The loss still hasn't reached zero, which suggests that the model could benefit from additional training.

Analyzing Company A Predictions

Next, let's look at the out_0 graph.

This graph tracks the predictions for Company A throughout training.

Our target prediction for Company A is 0.

Initially, the prediction moves to higher values before gradually flattening out. Training for more epochs may help push the prediction closer to the desired value.

Analyzing Company B Predictions

Now let's examine the out_1 graph.

This graph shows how the predictions for Company B change during training.

Our target prediction for Company B is 1.

The prediction has started to level off around 0.5, which indicates that the model has not fully learned the desired output. Increasing the number of training epochs may help improve this prediction.

From these TensorBoard graphs, we can see that the model is learning, but it has not fully converged yet.

In the next article, we will increase the number of training epochs and see how the model's predictions improve.

Building LSTMs with PyTorch and Lightning AI Part 5: Improving Predictions Through Training

Rijul Rajesh — Sat, 27 Jun 2026 20:18:21 +0000

In the previous article, we ran our model and checked how accurate its predictions were.

In this article, we will train the model.

First, we create the training data.

inputs = torch.tensor([
    [0., 0.5, 0.25, 1.],
    [1., 0.5, 0.25, 1.]
])

These represent the stock prices for Days 1 through 4 for both companies.

Next, we create the labels, which are the values we want the LSTM to predict.

labels = torch.tensor([0., 1.])

Here, we want the LSTM to predict:

0 for Company A
1 for Company B

Now we combine the inputs and labels into a TensorDataset called dataset.

dataset = TensorDataset(inputs, labels)
dataloader = DataLoader(dataset)

As we discussed in previous articles, DataLoaders are useful because:

They make it easy to access the data in batches.
They can shuffle the data at the beginning of each epoch.
They allow us to use a small subset of the data when we want to quickly debug the training process.

Next, we create a Lightning trainer.

trainer = L.Trainer(max_epochs=2000)

Here, we tell Lightning to train the model for a maximum of 2,000 epochs.

During training, backpropagation is used to optimize all the trainable weights and biases in the LSTM.

To begin training, we simply call the trainer's fit() method.

trainer.fit(model, train_dataloaders=dataloader)

Once training is complete, we can print the predictions just as we did before.

print("\nComparing observed and predicted values")

print(
    "Company A: Observed = 0, Predicted =",
    model(torch.tensor([0., 0.5, 0.25, 1.])).detach()
)

print(
    "Company B: Observed = 1, Predicted =",
    model(torch.tensor([1., 0.5, 0.25, 1.])).detach()
)

This produces the following output:

Comparing observed and predicted values

Company A: Observed = 0, Predicted = tensor(0.0003)
Company B: Observed = 1, Predicted = tensor(0.9287)

As you can see, the predictions have improved significantly after training. The model now produces values that are much closer to the expected outputs.

In the next article, we will explore TensorBoard to analyze what happened during training.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 4: Training Step and Initial Predictions

Rijul Rajesh — Fri, 26 Jun 2026 19:20:44 +0000

In the previous article, we finished the LSTM cell, explored the forward method and the Adam optimizer for the model.

In this article, we will explore the training_step() function, and try to run the model without training.

The training_step() function takes a batch of training data from one of the two companies, along with the index of that batch.

It then uses the forward() function to make a prediction for that training example.

def training_step(self, batch, batch_idx):
    input_i, label_i = batch
    output_i = self.forward(input_i[0])
    loss = (output_i - label_i)**2

Next, it calculates the loss, which is the squared residual between the predicted value and the observed value.

We can also log the loss to easily track how it changes during training.

Lightning provides the log() function for this purpose. It automatically stores the logs in a lightning_logs directory.

We can log other values as well, such as the predictions for Company A and Company B.

Finally, we return the loss.

def training_step(self, batch, batch_idx):
    input_i, label_i = batch
    output_i = self.forward(input_i[0])
    loss = (output_i - label_i)**2

    self.log("train_loss", loss)

    if label_i == 0:
        self.log("out_0", output_i)
    else:
        self.log("out_1", output_i)

    return loss

So far, we have implemented the following:

Initialized the weight and bias tensors.
Implemented the LSTM calculations in lstm_unit().
Created the forward() method to perform a forward pass through the unrolled LSTM.
Configured the Adam optimizer using configure_optimizers().
Calculated and logged the training loss using training_step().

Now let's try using the model.

model = LSTMByHand()

print("\nComparing observed and predicted values")

print(
    "Company A: Observed = 0, Predicted =",
    model(torch.tensor([0., 0.5, 0.25, 1.])).detach()
)

print(
    "Company B: Observed = 1, Predicted =",
    model(torch.tensor([1., 0.5, 0.25, 1.])).detach()
)

Here, we pass a tensor containing the stock prices for Days 1 through 4. The model then predicts the value for Day 5.

The model returns both the prediction and its associated computation graph. We call .detach() to remove the computation graph and retrieve only the prediction.

Running the code produces the following output:

Comparing observed and predicted values
Company A: Observed = 0, Predicted = tensor(-0.2321)
Company B: Observed = 1, Predicted = tensor(-0.2360)

The prediction for Company A is reasonably close to the observed value.

However, the prediction for Company B is quite far from the expected value.

In the next article, we will train the model to improve these predictions.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 3: Finishing the LSTM Cell

Rijul Rajesh — Wed, 24 Jun 2026 19:05:42 +0000

In the previous article, we started with the creation of LSTM cell.

In this article we will continue building the LSTM Unit as well as create the forward pass and the optimizer.

Creating the Short-Term Memory

In this stage, we create the updated short-term memory and determine what percentage of it should be sent to the output.

First, we calculate the output percentage:

output_percent = torch.sigmoid(
    (short_memory * self.wo1) +
    (input_value * self.wo2) +
    self.bo1
)

Here:

wo1 is the weight associated with the current short-term memory.
wo2 is the weight associated with the current input value.
bo1 is the bias term.

The sigmoid function produces a value between 0 and 1, representing the percentage of information that should be passed to the output.

Next, we use this percentage to scale the new short-term memory.

We first apply the tanh activation function to the updated long-term memory, and then multiply the result by output_percent.

updated_short_memory = torch.tanh(updated_long_memory) * output_percent

Finally, we return the updated long-term and short-term memory values:

return [updated_long_memory, updated_short_memory]

At this point, our lstm_unit() function is complete.

def lstm_unit(self, input_value, long_memory, short_memory):

    long_remember_percent = torch.sigmoid(
        (short_memory * self.wlr1) +
        (input_value * self.wlr2) +
        self.blr1
    )

    potential_remember_percent = torch.sigmoid(
        (short_memory * self.wpr1) +
        (input_value * self.wpr2) +
        self.bpr1
    )

    potential_memory = torch.tanh(
        (short_memory * self.wp1) +
        (input_value * self.wp2) +
        self.bp1
    )

    updated_long_memory = (
        (long_memory * long_remember_percent) +
        (potential_remember_percent * potential_memory)
    )

    output_percent = torch.sigmoid(
        (short_memory * self.wo1) +
        (input_value * self.wo2) +
        self.bo1
    )

    updated_short_memory = (
        torch.tanh(updated_long_memory) * output_percent
    )

    return [updated_long_memory, updated_short_memory]

Now that we have implemented the LSTM unit, the next step is to create the forward() method that performs a forward pass through the unrolled LSTM.

For this example, the input will be the stock prices from the previous four days.

First, we initialize the long-term and short-term memory values:

def forward(self, input):
    long_memory = 0
    short_memory = 0

Next, we process each day's stock price through the LSTM unit:

def forward(self, input):

    long_memory = 0
    short_memory = 0

    day1 = input[0]
    day2 = input[1]
    day3 = input[2]
    day4 = input[3]

    long_memory, short_memory = self.lstm_unit(
        day1, long_memory, short_memory
    )

    long_memory, short_memory = self.lstm_unit(
        day2, long_memory, short_memory
    )

    long_memory, short_memory = self.lstm_unit(
        day3, long_memory, short_memory
    )

    long_memory, short_memory = self.lstm_unit(
        day4, long_memory, short_memory
    )

    return short_memory

Here, the same LSTM unit is reused for each day's input. As each value is processed, the long-term and short-term memory are updated and carried forward to the next step.

After the fourth day, we return the final short-term memory, which serves as the output of the LSTM.

Now that we have a forward() method capable of performing a forward pass through the unrolled LSTM, we are ready to configure the optimizer.

This is straightforward:

def configure_optimizers(self):
    return Adam(self.parameters())

This tells Lightning to use the Adam optimizer to train all trainable parameters in the model.

In the next article, we will explore the training_step() method, which is responsible for calculating the loss during training.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 2: Starting the LSTM Unit Implementation

Rijul Rajesh — Mon, 22 Jun 2026 18:30:25 +0000

In the previous article, we began building the LSTM by defining the class and initializing the weights and biases.

In this article, we will continue by implementing the lstm_unit() function.

The lstm_unit() function requires three inputs:

The current input value
The current long-term memory value
The current short-term memory value

def lstm_unit(self,input_value,long_memory,short_memory):

Stage 1: Determine How Much Long-Term Memory to Remember

We use the parameters defined in the __init__() method to determine what percentage of the existing long-term memory should be retained.

For this, we create long_remember_percent:

def lstm_unit(self,input_value,long_memory,short_memory):
    long_remember_percent = torch.sigmoid(
        (short_memory * self.wlr1) +
        (input_value * self.wlr2) +
        self.blr1
    )

Here:

The short-term memory is multiplied by its associated weight.
The input value is multiplied by its associated weight.
Both results are added together along with the bias.
The final sum is passed through the sigmoid activation function.

The sigmoid function produces a value between 0 and 1, which represents the percentage of long-term memory that should be remembered.

Stage 2: Determine Potential Long-Term Memory

Next, we calculate the percentage of new information that could potentially be added to long-term memory.

For this, we create potential_remember_percent:

potential_remember_percent = torch.sigmoid(
    (short_memory * self.wpr1) +
    (input_value * self.wpr2) +
    self.bpr1
)

This uses calculations similar to the previous step.

Stage 2: Calculate Potential Memory

We also need to calculate the candidate memory value itself.

For this, we use the tanh activation function:

potential_memory = torch.tanh(
    (short_memory * self.wp1) +
    (input_value * self.wp2) +
    self.bp1
)

The tanh function produces values between -1 and 1, allowing the LSTM to store both positive and negative information.

Updating the Long-Term Memory

Now we can update the long-term memory.

We do this by:

Keeping the percentage of the existing long-term memory that should be remembered.
Adding the percentage of the new potential memory that should be stored.

updated_long_memory = (
    (long_memory * long_remember_percent) +
    (potential_remember_percent * potential_memory)
)

This gives us the updated long-term memory value for the LSTM.

In the next article, we will continue with the third stage of the LSTM, where we create the updated short-term memory and determine what percentage of the long-term memory should be sent to the output.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Building LSTMs with PyTorch and Lightning AI Part 1: First Steps with LSTMs

Rijul Rajesh — Sun, 21 Jun 2026 18:25:33 +0000

In this article, we will explore how to implement an LSTM using PyTorch and Lightning.

For more details about LSTMs, there is a separate series of articles available here.

Imports

To begin, we first import the required modules.

import torch
import torch.nn as nn
import torch.nn.functional as F

Introducing a New Optimizer

We also introduce a new optimizer:

from torch.optim import Adam

Adam is used to fit the neural network to the data.

It works similarly to SGD, but in practice, Adam often converges faster and adapts the learning rate more effectively.

Lightning and Data Utilities

Next, we continue with the remaining imports:

import lightning as L
from torch.utils.data import TensorDataset, DataLoader

Defining the LSTM Model

We define the neural network by creating a Lightning module.

class LSTMByHand(L.LightningModule):
    def __init__(self):
        # Create and initialize weight and bias tensors

    def lstm_unit(self, input_value, long_memory, short_memory):
        # LSTM computations

    def forward(self, input):
        # Forward pass through the unrolled LSTM

    def configure_optimizers(self):
        # Configure Adam optimizer

    def training_step(self, batch, batch_idx):
        # Compute loss and log training progress

Initializing the Model

Now let’s implement the __init__ method.

This is where we initialize all weights and biases.

class LSTMByHand(L.LightningModule):
    def __init__(self):
        super().__init__()

        mean = torch.tensor(0.0)  # Mean of the normal distribution
        std = torch.tensor(1.0)   # Standard deviation

        # -------------------------
        # Forget Gate (l = "lr")
        # -------------------------
        self.wlr1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.wlr2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.blr1 = nn.Parameter(torch.tensor(0.0), requires_grad=True)

        # -------------------------
        # Input Gate (p = "pr")
        # -------------------------
        self.wpr1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.wpr2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.bpr1 = nn.Parameter(torch.tensor(0.0), requires_grad=True)

        # -------------------------
        # Cell Candidate (p)
        # -------------------------
        self.wp1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.wp2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.bp1 = nn.Parameter(torch.tensor(0.0), requires_grad=True)

        # -------------------------
        # Output Gate (o)
        # -------------------------
        self.wo1 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.wo2 = nn.Parameter(torch.normal(mean=mean, std=std), requires_grad=True)
        self.bo1 = nn.Parameter(torch.tensor(0.0), requires_grad=True)

Why Use Normal Distribution?

Unlike earlier examples, we initialize weights using a normal distribution.

Before moving further, let’s understand what that means.

What is a Normal Distribution?

Imagine measuring the heights of a large group of people:

Most people are around the average height
Very tall and very short people are rare

When plotted, this forms a symmetric bell-shaped curve.

This is called a normal distribution.

Key Properties

The center represents the most common values
The curve is symmetric
The tails represent rare values

Mean and Standard Deviation

Mean → the average value
Standard deviation → how spread out the values are

Small vs Large Standard Deviation

Small Standard Deviation

Values are tightly clustered around the mean
Example: Class A scores mostly between 55–65

Large Standard Deviation

Values are widely spread
Example: Class B scores range from 20–90

In Our Code

We use:

Mean = 0
Standard deviation = 1

Also, all parameters have requires_grad=True, meaning they will be trained during backpropagation.

Next, we will explore the lstm_unit function and how the LSTM actually processes information step by step.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

DEV Community: Rijul Rajesh

Typing Too Much While Using AI? This Tool Will Save You Hours

Installing Espanso

Linux users

Witness the Magic

Creating Your Own Shortcuts

A Real-World Example

Wrapping up

Building Word Embeddings with PyTorch and Lightning AI Part 2: Creating Labels for Next-Word Prediction

Creating the Neural Network

Building Word Embeddings with PyTorch and Lightning AI Part 1: Setting Up the Word Embedding Model

Creating the Training Inputs

Building LSTMs with PyTorch and Lightning AI Part 9: Completing the Simplified LSTM

Implementing the forward() Method

Configuring the Optimizer

Implementing training_step()

Testing the Model

Training the Model

Checking the Results

Analyzing the Training

Building LSTMs with PyTorch and Lightning AI Part 8: Setting Up a Simpler LSTM

Building LSTMs with PyTorch and Lightning AI Part 7: Resuming Training with Checkpoints

Getting the Checkpoint

Increasing the Number of Epochs

Checking the Updated Predictions

Comparing the TensorBoard Graphs

Company A

Company B

Training Loss

Building LSTMs with PyTorch and Lightning AI Part 6: Analyzing Training with TensorBoard

Launching TensorBoard

Analyzing the Training Loss

Analyzing Company A Predictions

Analyzing Company B Predictions

Building LSTMs with PyTorch and Lightning AI Part 5: Improving Predictions Through Training

Building LSTMs with PyTorch and Lightning AI Part 4: Training Step and Initial Predictions

Building LSTMs with PyTorch and Lightning AI Part 3: Finishing the LSTM Cell

Creating the Short-Term Memory

Building LSTMs with PyTorch and Lightning AI Part 2: Starting the LSTM Unit Implementation

Stage 1: Determine How Much Long-Term Memory to Remember

Stage 2: Determine Potential Long-Term Memory

Stage 2: Calculate Potential Memory

Updating the Long-Term Memory

Building LSTMs with PyTorch and Lightning AI Part 1: First Steps with LSTMs

Imports

Introducing a New Optimizer

Lightning and Data Utilities

Defining the LSTM Model

Initializing the Model

Why Use Normal Distribution?

What is a Normal Distribution?

Key Properties

Mean and Standard Deviation

Small vs Large Standard Deviation

Small Standard Deviation

Large Standard Deviation

In Our Code

Implementing the `forward()` Method

Implementing `training_step()`