The layers for Neural Network in PyTorch

A layer is a collection of nodes to do a specific task.

Basically, a Neural Network(NN) consists of 3 layers as shown below:

• Input Layer:

  • is the 1st layer which accepts data and pass it to a hidden layer.

• Hidden Layer:

  • is the layer between an input and output layer.
  • can be zero or more hidden layers in a neural network.

• Output layer:

  • is the last layer which holds a result.

*There is the Single Layer Neural Network or Perceptron which only has an input and output layer without hidden layer.

And, there are popular kinds of layers as shown below:

*Memos:

• These layers below can be used as an input, hidden or output layer.
• Some layers are also called Neural Network.

(1) Fully-connected Layer:

• connects every neuron in one layer to every neuron in the next layer.
• is also called Linear Layer, Dense Layer or Affine Layer.
• is Linear() in PyTorch.

(2) Convolutional Layer:

• can make data stand out by extracting the features from the data with filters(kernels). *Extracting the features from the data also downsamples and reduce the data to reduce computation.
• is used for image processing.
• is Conv1d(), Conv2d() or Conv3d() in PyTorch.

*The Neural Network with Convolutional Layer is called Convolutional Neural Network(CNN).

(3) Transposed Convolutional Layer:

• can upsample data.
• is used for image processing.
• is similar to Deconvolutional Layer but not the same.
• is ConvTranspose1d(), ConvTranspose2d() or ConvTranspose3d() in PyTorch.

(4) Pooling Layer:

• can downsample and reduce data to reduce overfitting and computation. *The way of downsampling data is different from Convolutional Layer.
• is used for image processing.
• is MaxPool1d(), MaxPool2d() or MaxPool3d() in PyTorch.

(5) Recurrent Layer:

• can remember the output of the previous step to use it in the next step.
• is used for NLP(Natural Language Processing). *NLP is the technology which enables computers to understand and communicate with human language.
• is also called Recurrent Neural Network(RNN).
• is RNN() in PyTorch.

(6) LSTM(Long Short-Term Memory) layer:

• can remember the output of the previous steps to use them in the next step while Recurrent Layer can remember the output of only the previous step.
  • is the improved version of Recurrent Layer.
  • has more ability than Recurrent Layer.
  • is slower to train data than Recurrent Layer because the computation is more complex.
  • is used for NLP.
  • is LSTM() in PyTorch.

(7) Batch Normalization layer:

• can normalize each feature within a mini-batch of samples(layers) to be similar scale to accelerate(speed up) training.
• is unstable with small batch sizes, then it leads to increased train time.
• is not good with RNN.
• is BatchNorm1d(), BatchNorm2d() or BatchNorm3d() in PyTorch.

(8) Layer Normalization:

• can normalize all features within each sample(layer) to be similar scale to accelerate(speed up) training.
• is the improved version of Batch Normalization layer.
• is stable with small batch sizes, then it doesn't lead to increased train time.
• is good with RNN.
• is LayerNorm() in PyTorch.

(9) Dropout Layer:

• can reduce overfitting by randomly dropping out nodes during training.
• is Dropout() in PyTorch.

(10) Embedding Layer:

• can convert categorical data to numerical data.
• is used for NLP.
• is Embedding() in PyTorch.