Dropout in PyTorch

Buy Me a Coffee☕

*Memos:

• My post explains Dropout Layer.
• My post explains manual_seed().
• My post explains requires_grad.

Dropout() can get the 0D or more D tensor of the zero or more elements randomly zeroed or multiplied from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

• The 1st argument for initialization is p(Optional-Default:0.5-Type:float): *Memos:
  • It's the probability of an element to be zeroed.
  • It must be 0 <= x <= 1.
• The 2nd argument for initialization is inplace(Optional-Default:False-Type:bool):
  • It does in-place operation.
  • Keep it False because it's problematic with True.
• The 1st argument is input(Required-Type:tensor of float): *Memos:
  • It must be the 0D or more D tensor of zero or more elements.
  • The tensor's requires_grad which is False by default is not set to True by Dropout().

import torch
from torch import nn

tensor1 = torch.tensor([8., -3., 0., 1., 5., -2.])

tensor1.requires_grad
# False

torch.manual_seed(7)

dropout1 = nn.Dropout()
tensor2 = dropout1(input=tensor1)
tensor2
# tensor([16., -0., 0., 2., 10., -4.])

tensor2.requires_grad
# False

dropout1
# Dropout(p=0.5, inplace=False)

dropout1.p
# 0.5

dropout1.inplace
# False

torch.manual_seed(7)

dropout2 = nn.Dropout()
dropout2(input=tensor2)
# tensor([32., -0., 0., 4., 20., -8.])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.5, inplace=False)
dropout(input=tensor1)
# tensor([16., -0., 0., 2., 10., -4.])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.8)
dropout(input=tensor1)
# tensor([40., -0., 0., 0., 25., -0.])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.3)
dropout(input=tensor1)
# tensor([11.4286, -0.0000, 0.0000, 1.4286, 7.1429, -2.8571])

my_tensor = torch.tensor([[8., -3., 0.],
                          [1., 5., -2.]])
torch.manual_seed(7)

dropout = nn.Dropout()
dropout(input=my_tensor)
# tensor([[16., -0., 0.],
#         [2., 10., -4.]])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.8)
dropout(input=my_tensor)
# tensor([[40., -0., 0.],
#         [0., 25., -0.]])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.3)
dropout(input=my_tensor)
# tensor([[11.4286, -0.0000, 0.0000],
#         [1.4286, 7.1429, -2.8571]])

my_tensor = torch.tensor([[8.], [-3.], [0.],
                          [1.], [5.], [-2.]])
torch.manual_seed(7)

dropout = nn.Dropout()
dropout(input=my_tensor)
# tensor([[16.], [-0.], [0.], [2.], [10.], [-4.]])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.8)
dropout(input=my_tensor)
# tensor([[40.], [-0.], [0.], [0.], [25.], [-0.]])

torch.manual_seed(7)

dropout = nn.Dropout(p=0.3)
dropout(input=my_tensor)
# tensor([[11.4286], [-0.0000], [0.0000], [1.4286], [7.1429], [-2.8571]])

my_tensor = torch.tensor([[[8., -3., 0.],
                           [1., 5., -2.]]])
torch.manual_seed(7)

dropout = nn.Dropout()
dropout(input=my_tensor)
# tensor([[[16., -0., 0.],
#          [2., 10., -4.]]])