DEV Community

Super Kai (Kazuya Ito)
Super Kai (Kazuya Ito)

Posted on • Edited on

1

ReLU and LeakyReLU in PyTorch

Buy Me a Coffee

*Memos:

ReLU() can get the 0D or more D tensor of the zero or more values computed by ReLU function from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

  • The 1st argument for initialization is inplace(Optional-Default:False-Type:bool): *Memos:
    • It does in-place operation.
    • Keep it False because it's problematic with True.
  • The 1st argument is input(Required-Type:tensor of int or float).
  • You can also use relu() with a tensor.

Image description

import torch
from torch import nn

my_tensor = torch.tensor([8, -3, 0, 1, 5, -2, -1, 4])

relu = nn.ReLU()
relu(input=my_tensor)
my_tensor.relu()
# tensor([8, 0, 0, 1, 5, 0, 0, 4])

relu
# ReLU()

relu.inplace
# False

relu = nn.ReLU(inplace=True)
relu(input=my_tensor)
# tensor([8, 0, 0, 1, 5, 0, 0, 4])

my_tensor = torch.tensor([[8, -3, 0, 1],
                          [5, 0, -1, 4]])
relu = nn.ReLU()
relu(input=my_tensor)
# tensor([[8, 0, 0, 1],
#         [5, 0, 0, 4]])

my_tensor = torch.tensor([[[8, -3], [0, 1]],
                          [[5, 0], [-1, 4]]])
relu = nn.ReLU()
relu(input=my_tensor)
# tensor([[[8, 0], [0, 1]],
#         [[5, 0], [0, 4]]])

my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
                          [[5., 0.], [-1., 4.]]])
relu = nn.ReLU()
relu(input=my_tensor)
# tensor([[[8., 0.], [0., 1.]],
#         [[5., 0.], [0., 4.]]])
Enter fullscreen mode Exit fullscreen mode

LeakyReLU() can get the 0D or more D tensor of the zero or more values computed by LeakyReLU function from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

  • The 1st argument for initialization is negative_slope(Optional-Default:0.01-Type:float). *It's applied to negative input values.
  • The 2nd argument for initialization is inplace(Optional-Default:False-Type:bool): *Memos:
    • It does in-place operation.
    • Keep it False because it's problematic with True.
  • The 1st argument is input(Required-Type:tensor of float).

Image description

import torch
from torch import nn

my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])

lrelu = nn.LeakyReLU()
lrelu(input=my_tensor)
# tensor([8.0000, -0.0300, 0.0000, 1.0000, 5.0000, -0.0200, -0.0100, 4.0000])

lrelu
# LeakyReLU(negative_slope=0.01)

lrelu.negative_slope
# 0.01

lrelu.inplace
# False

lrelu = nn.LeakyReLU(negative_slope=0.01, inplace=True)
lrelu(input=my_tensor)
# tensor([8.0000, -0.0300, 0.0000, 1.0000, 5.0000, -0.0200, -0.0100, 4.0000])

my_tensor = torch.tensor([[8., -3., 0., 1.],
                          [5., -2., -1., 4.]])
lrelu = nn.LeakyReLU()
lrelu(input=my_tensor)
# tensor([[8.0000, -0.0300, 0.0000, 1.0000],
#         [5.0000, -0.0200, -0.0100, 4.0000]])

my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
                          [[5., -2.], [-1., 4.]]])
lrelu = nn.LeakyReLU()
lrelu(input=my_tensor)
# tensor([[[8.0000, -0.0300], [0.0000, 1.0000]],
#         [[5.0000, -0.0200], [-0.0100, 4.0000]]])
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more