pytutorial/pytorch/layers

PyTorch: Layers

{:.no_toc}

* TOC {:toc}

Goal

Layers, layers, everywhere. If you understand PyTorch layers then you understand most of PyTorch. Well, and torch.tensor...

Questions to David Rotermund

Open Book & Website recommendation If you don't know what these layers mean then don't be ashamed and check out these resources: 

Dive into Deep Learning https://d2l.ai/

as well as corresponding Youtube channel from Alex Smola

torch.tensor

Before we can use layers we need to understand torch.tensor a bit more. You can understand them as a np.ndarray with additional properties and options. Most of it is like with numpy arrays. However, the tensor can shelter an additional gradient!

There are official tutorials about tensors:

• Tensors
• Introduction to Torch’s tensor library However, I feel that they are a bit useless.

Let's be very blunt here: If you don't understand the basics of Numpy ndarray then you will be lost. If you know the usual stuff about np.ndarrays then you understand most of torch.tensor too. You only need some extra information, which I will try to provide here. 

The torch.tensor data types (for the CPU) are very similar to the Numpy. e.g. np.float32 -> torch.float32

There and back again (np.ndarray <-> torch.tensor)

Let us convert x_np: np.ndarray into a torch.Tensor:

x_torch: torch.Tensor = torch.tensor(x_np)

And we convert it back into a torch.Tensor:

x_np: np.ndarray = x_torch.detach().numpy()

or if x_torch is on the GPU:

x_np : np.ndarray= x_torch.cpu().detach().numpy()

• Tensor.detach() : Get rid of the gradient.
• Tensor.numpy() : "Returns self tensor as a NumPy ndarray. This tensor and the returned ndarray share the same underlying storage. Changes to self tensor will be reflected in the ndarray and vice versa."
• torch.tensor() : Put stuff in there and get a torch.tensor out of it.
• Tensor.type() : numpy's .astype()

GPU interactions

torch.tensor.to  : tensors and layers can life on different devices e.g. CPU and GPU. You need to move them where you need then. Typically only objects on the same device can interact. 

device_gpu = torch.device("cuda:0")
device_cpu = torch.device("cpu")

something_on_gpu =  something_on_cpu.to(device_gpu)
something_on_cpu =  something_on_gpu.to(device_cpu)
something_on_cpu =  something_on_gpu.cpu()

• torch.cuda.is_available() : Is there a cuda gpu in my system?
• torch.cuda.get_device_capability() : If you need to know what your gpu is capable of. You get a number and need to check here what this number means.  
• torch.cuda.get_device_name() : In the case you forgot what GPU you bought or if you have sooo many GPUs in your system and don't know which is what.
• torch.cuda.get_device_properties() : The two above plus how much memory the card has.
• torch.Tensor.device : Where does this tensor life?

Dimensions 

• torch.squeeze() : get rid of (a selected) dimension with size 1
• torch.unsqueeze() : add a dimension of size 1 at a defined position

Layers (i.e. basic building blocks)

Obviously there are a lot of available layer. I want to list only the relevant ones for the typical daily use.

Please note the first upper case letter. If the thing you want to use doesn't have a first upper case letter then something is wrong and it is not a layer.

I will skip the following layer types (because you will not care about them):

• Sparse Layers
• Vision Layers
• Shuffle Layers
• DataParallel Layers (multi-GPU, distributed)
• Distance Functions

In the following I will mark the relevant layers.

Convolution Layers

torch.nn.Conv1d	Applies a 1D convolution over an input signal composed of several input planes.
torch.nn.Conv2d	Applies a 2D convolution over an input signal composed of several input planes.
torch.nn.Conv3d	Applies a 3D convolution over an input signal composed of several input planes.
torch.nn.ConvTranspose1d	Applies a 1D transposed convolution operator over an input image composed of several input planes.
torch.nn.ConvTranspose2d	Applies a 2D transposed convolution operator over an input image composed of several input planes.
torch.nn.ConvTranspose3d	Applies a 3D transposed convolution operator over an input image composed of several input planes.
torch.nn.LazyConv1d	A torch.nn.Conv1d module with lazy initialization of the in_channels argument of the Conv1d that is inferred from the input.size(1).
torch.nn.LazyConv2d	A torch.nn.Conv2d module with lazy initialization of the in_channels argument of the Conv2d that is inferred from the input.size(1).
torch.nn.LazyConv3d	A torch.nn.Conv3d module with lazy initialization of the in_channels argument of the Conv3d that is inferred from the input.size(1).
torch.nn.LazyConvTranspose1d	A torch.nn.ConvTranspose1d module with lazy initialization of the in_channels argument of the ConvTranspose1d that is inferred from the input.size(1).
torch.nn.LazyConvTranspose2d	A torch.nn.ConvTranspose2d module with lazy initialization of the in_channels argument of the ConvTranspose2d that is inferred from the input.size(1).
torch.nn.LazyConvTranspose3d	A torch.nn.ConvTranspose3d module with lazy initialization of the in_channels argument of the ConvTranspose3d that is inferred from the input.size(1).
torch.nn.Unfold	Extracts sliding local blocks from a batched input tensor.
torch.nn.Fold	Combines an array of sliding local blocks into a large containing tensor.

Pooling layers

torch.nn.MaxPool1d	Applies a 1D max pooling over an input signal composed of several input planes.
torch.nn.MaxPool2d	Applies a 2D max pooling over an input signal composed of several input planes.
torch.nn.MaxPool3d	Applies a 3D max pooling over an input signal composed of several input planes.
torch.nn.MaxUnpool1d	Computes a partial inverse of MaxPool1d.
torch.nn.MaxUnpool2d	Computes a partial inverse of MaxPool2d.
torch.nn.MaxUnpool3d	Computes a partial inverse of MaxPool3d.
torch.nn.AvgPool1d	Applies a 1D average pooling over an input signal composed of several input planes.
torch.nn.AvgPool2d	Applies a 2D average pooling over an input signal composed of several input planes.
torch.nn.AvgPool3d	Applies a 3D average pooling over an input signal composed of several input planes.
torch.nn.FractionalMaxPool2d	Applies a 2D fractional max pooling over an input signal composed of several input planes.
torch.nn.FractionalMaxPool3d	Applies a 3D fractional max pooling over an input signal composed of several input planes.
torch.nn.LPPool1d	Applies a 1D power-average pooling over an input signal composed of several input planes.
torch.nn.LPPool2d	Applies a 2D power-average pooling over an input signal composed of several input planes.
torch.nn.AdaptiveMaxPool1d	Applies a 1D adaptive max pooling over an input signal composed of several input planes.
torch.nn.AdaptiveMaxPool2d	Applies a 2D adaptive max pooling over an input signal composed of several input planes.
torch.nn.AdaptiveMaxPool3d	Applies a 3D adaptive max pooling over an input signal composed of several input planes.
torch.nn.AdaptiveAvgPool1d	Applies a 1D adaptive average pooling over an input signal composed of several input planes.
torch.nn.AdaptiveAvgPool2d	Applies a 2D adaptive average pooling over an input signal composed of several input planes.
torch.nn.AdaptiveAvgPool3d	Applies a 3D adaptive average pooling over an input signal composed of several input planes.

Padding Layers

torch.nn.ReflectionPad1d	Pads the input tensor using the reflection of the input boundary.
torch.nn.ReflectionPad2d	Pads the input tensor using the reflection of the input boundary.
torch.nn.ReflectionPad3d	Pads the input tensor using the reflection of the input boundary.
torch.nn.ReplicationPad1d	Pads the input tensor using replication of the input boundary.
torch.nn.ReplicationPad2d	Pads the input tensor using replication of the input boundary.
torch.nn.ReplicationPad3d	Pads the input tensor using replication of the input boundary.
torch.nn.ZeroPad1d	Pads the input tensor boundaries with zero.
torch.nn.ZeroPad2d	Pads the input tensor boundaries with zero.
torch.nn.ZeroPad3d	Pads the input tensor boundaries with zero.
torch.nn.ConstantPad1d	Pads the input tensor boundaries with a constant value.
torch.nn.ConstantPad2d	Pads the input tensor boundaries with a constant value.
torch.nn.ConstantPad3d	Pads the input tensor boundaries with a constant value.

Non-linear Activations (weighted sum, nonlinearity)

torch.nn.ELU	Applies the Exponential Linear Unit (ELU) function, element-wise, as described in the paper: Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs).
torch.nn.Hardshrink	Applies the Hard Shrinkage (Hardshrink) function element-wise.
torch.nn.Hardsigmoid	Applies the Hardsigmoid function element-wise.
torch.nn.Hardtanh	Applies the HardTanh function element-wise.
torch.nn.Hardswish	Applies the Hardswish function, element-wise, as described in the paper: Searching for MobileNetV3.
torch.nn.LeakyReLU	Applies the element-wise function:
torch.nn.LogSigmoid	Applies the element-wise function:
torch.nn.MultiheadAttention	Allows the model to jointly attend to information from different representation subspaces as described in the paper: Attention Is All You Need.
torch.nn.PReLU	Applies the element-wise function:
torch.nn.ReLU	Applies the rectified linear unit function element-wise:
torch.nn.ReLU6	Applies the element-wise function:
torch.nn.RReLU	Applies the randomized leaky rectified liner unit function, element-wise, as described in the paper:
torch.nn.SELU	Applied element-wise...
torch.nn.CELU	Applies the element-wise function...
torch.nn.GELU	Applies the Gaussian Error Linear Units function:
torch.nn.Sigmoid	Applies the element-wise function:...
torch.nn.SiLU	Applies the Sigmoid Linear Unit (SiLU) function, element-wise.
torch.nn.Mish	Applies the Mish function, element-wise.
torch.nn.Softplus	Applies the Softplus function
torch.nn.Softshrink	Applies the soft shrinkage function elementwise:
torch.nn.Softsign	Applies the element-wise function:...
torch.nn.Tanh	Applies the Hyperbolic Tangent (Tanh) function element-wise.
torch.nn.Tanhshrink	Applies the element-wise function:
torch.nn.Threshold	Thresholds each element of the input Tensor.
torch.nn.GLU	Applies the gated linear unit function

Non-linear Activations (other)

torch.nn.Softmin	Applies the Softmin function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range [0, 1] and sum to 1.
torch.nn.Softmax	Applies the Softmax function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range [0,1] and sum to 1.
torch.nn.Softmax2d	Applies SoftMax over features to each spatial location.
torch.nn.LogSoftmax	Applies the LogSoftmax
torch.nn.AdaptiveLogSoftmaxWithLoss	Efficient softmax approximation as described in Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou.

Normalization Layers

torch.nn.BatchNorm1d	Applies Batch Normalization over a 2D or 3D input as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
torch.nn.BatchNorm2d	Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
torch.nn.BatchNorm3d	Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
torch.nn.LazyBatchNorm1d	A torch.nn.BatchNorm1d module with lazy initialization of the num_features argument of the BatchNorm1d that is inferred from the input.size(1).
torch.nn.LazyBatchNorm2d	A torch.nn.BatchNorm2d module with lazy initialization of the num_features argument of the BatchNorm2d that is inferred from the input.size(1).
torch.nn.LazyBatchNorm3d	A torch.nn.BatchNorm3d module with lazy initialization of the num_features argument of the BatchNorm3d that is inferred from the input.size(1).
torch.nn.GroupNorm	Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization
torch.nn.SyncBatchNorm	Applies Batch Normalization over a N-Dimensional input (a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .
torch.nn.InstanceNorm1d	Applies Instance Normalization over a 2D (unbatched) or 3D (batched) input as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
torch.nn.InstanceNorm2d	Applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
torch.nn.InstanceNorm3d	Applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization.
torch.nn.LazyInstanceNorm1d	A torch.nn.InstanceNorm1d module with lazy initialization of the num_features argument of the InstanceNorm1d that is inferred from the input.size(1).
torch.nn.LazyInstanceNorm2d	A torch.nn.InstanceNorm2d module with lazy initialization of the num_features argument of the InstanceNorm2d that is inferred from the input.size(1).
torch.nn.LazyInstanceNorm3d	A torch.nn.InstanceNorm3d module with lazy initialization of the num_features argument of the InstanceNorm3d that is inferred from the input.size(1).
torch.nn.LayerNorm	Applies Layer Normalization over a mini-batch of inputs as described in the paper Layer Normalization
torch.nn.LocalResponseNorm	Applies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension.

Recurrent Layers

torch.nn.RNNBase	Base class for RNN modules (RNN, LSTM, GRU).
torch.nn.RNN	Applies a multi-layer Elman
torch.nn.LSTM	Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence.
torch.nn.GRU	Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence.
torch.nn.RNNCell	An Elman RNN cell with tanh or ReLU non-linearity.
torch.nn.LSTMCell	A long short-term memory (LSTM) cell.
torch.nn.GRUCell	A gated recurrent unit (GRU) cell

Transformer Layers

torch.nn.Transformer	A transformer model.
torch.nn.TransformerEncoder	TransformerEncoder is a stack of N encoder layers.
torch.nn.TransformerDecoder	TransformerDecoder is a stack of N decoder layers
torch.nn.TransformerEncoderLayer	TransformerEncoderLayer is made up of self-attn and feedforward network.
torch.nn.TransformerDecoderLayer	TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network.

Linear Layers

torch.nn.Identity	A placeholder identity operator that is argument-insensitive
torch.nn.Linear	Applies a linear transformation to the incoming data
torch.nn.Bilinear	Applies a bilinear transformation to the incoming data
torch.nn.LazyLinear	A torch.nn.Linear module where in_features is inferred.