PReLU and ELU in PyTorch

#python #pytorch #prelu #elu

*Memos:

My post explains Leaky ReLU, PReLU and FReLU.
My post explains ELU, SELU and CELU.
My post explains heaviside() and Identity().
My post explains ReLU() and LeakyReLU().
My post explains SELU() and CELU().
My post explains GELU() and Mish().
My post explains SiLU() and Softplus().
My post explains Tanh() and Softsign().
My post explains Sigmoid() and Softmax().

PReLU() can get the 1D or more D tensor of the zero or more values computed by PReLU function from the 1D or more D tensor of zero or more elements as shown below:

*Memos:

The 1st argument for initialization is num_parameters(Optional-Default:1-Type:int): *Memos:
- It's the number of learnable parameters.
- It must be 1 <= x.
The 2nd argument for initialization is init(Optional-Default:0.25-Type:float). *It's the initial value for learnable parameters.
The 3rd argument is device(Optional-Default:None-Type:str, int or device()): *Memos:
- If it's None, get_default_device() is used. *My post explains get_default_device() and set_default_device().
- device= can be omitted.
- My post explains device argument.
The 4th argument is dtype(Optional-Default:None-Type:dtype): *Memos:
- If it's None, get_default_dtype() is used. *My post explains get_default_dtype() and set_default_dtype().
- dtype= can be omitted.
- My post explains dtype argument.
The 1st argument is input(Required-Type:tensor of float).
You can also use prelu() with a tensor.

import torch
from torch import nn

my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])

prelu = nn.PReLU()
prelu(input=my_tensor)
# tensor([8.0000, -0.7500, 0.0000, 1.0000,
#         5.0000, -0.5000, -0.2500, 4.0000],
#        grad_fn=<PreluKernelBackward0>)

my_tensor.prelu(weight=torch.tensor(0.25))
# tensor([8.0000, -0.7500, 0.0000, 1.0000,
#         5.0000, -0.5000, -0.2500, 4.0000])

prelu
# PReLU(num_parameters=1)

prelu.num_parameters
# 1

prelu.init
# 0.25

prelu.weight
# Parameter containing:
# tensor([0.2500], requires_grad=True)

prelu = nn.PReLU(num_parameters=1, init=0.25)
prelu(input=my_tensor)
# tensor([8.0000, -0.7500, 0.0000, 1.0000,
#         5.0000, -0.5000, -0.2500, 4.0000],
#        grad_fn=<PreluKernelBackward0>)

prelu.weight
# Parameter containing:
# tensor([0.2500], requires_grad=True)

my_tensor = torch.tensor([[8., -3., 0., 1.],
                          [5., -2., -1., 4.]])
prelu = nn.PReLU()
prelu(input=my_tensor)
# tensor([[8.0000, -0.7500, 0.0000, 1.0000],
#         [5.0000, -0.5000, -0.2500, 4.0000]],
#        grad_fn=<PreluKernelBackward0>)

prelu.weight
# Parameter containing:
# tensor([0.2500], requires_grad=True)

prelu = nn.PReLU(num_parameters=4, init=0.25, device=None, dtype=None)
prelu(input=my_tensor)
# tensor([[8.0000, -0.7500, 0.0000, 1.0000],
#         [5.0000, -0.5000, -0.2500, 4.0000]],
#        grad_fn=<PreluKernelBackward0>)

prelu.weight
# Parameter containing:
# tensor([0.2500, 0.2500, 0.2500, 0.2500], requires_grad=True)

my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
                          [[5., -2.], [-1., 4.]]])
prelu = nn.PReLU()
prelu(input=my_tensor)
# tensor([[[8.0000, -0.7500], [0.0000, 1.0000]],
#         [[5.0000, -0.5000], [-0.2500, 4.0000]]],
#        grad_fn=<PreluKernelBackward0>)

prelu.weight
# Parameter containing:
# tensor([0.2500], requires_grad=True)

prelu = nn.PReLU(num_parameters=2, init=0.25)
prelu(input=my_tensor)
# tensor([[[8.0000, -0.7500], [0.0000, 1.0000]],
#         [[5.0000, -0.5000], [-0.2500, 4.0000]]],
#        grad_fn=<PreluKernelBackward0>)

prelu.weight
# Parameter containing:
# tensor([0.2500, 0.2500], requires_grad=True)

ELU() can get the 0D or more D tensor of the zero or more values computed by ELU function from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

The 1st argument for initialization is alpha(Optional-Default:1.0-Type:float). *It's applied to negative input values.
The 2nd argument for initialization is inplace(Optional-Default:False-Type:bool): *Memos:
- It does in-place operation.
- Keep it False because it's problematic with True.
The 1st argument is input(Required-Type:tensor of float).

import torch
from torch import nn

my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])

elu = nn.ELU()
elu(input=my_tensor)
# tensor([8.0000, -0.9502, 0.0000, 1.0000, 5.0000, -0.8647, -0.6321, 4.0000])

elu
# ELU(alpha=1.0)

elu.alpha
# 1.0

elu.inplace
# False

elu = nn.ELU(alpha=1.0, inplace=True)
elu(input=my_tensor)
# tensor([8.0000, -0.9502, 0.0000, 1.0000, 5.0000, -0.8647, -0.6321, 4.0000])

my_tensor = torch.tensor([[8., -3., 0., 1.],
                          [5., -2., -1., 4.]])
elu = nn.ELU()
elu(input=my_tensor)
# tensor([[8.0000, -0.9502, 0.0000, 1.0000],
#         [5.0000, -0.8647, -0.6321, 4.0000]])

my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
                          [[5., -2.], [-1., 4.]]])
elu = nn.ELU()
elu(input=my_tensor)
# tensor([[[8.0000, -0.9502], [0.0000, 1.0000]],
#         [[5.0000, -0.8647], [-0.6321, 4.0000]]])