linalg.norm() in PyTorch

Buy Me a Coffee☕

linalg.norm() can get the 0D or more D tensor of the zero or more elements computed with norm from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

• linalg.norm() can be used with torch but not with a tensor. *A tensor can use torch.norm which is deprecated but not linalg.norm().
• The 1st argument with torch is input(Required-Type:tensor of float or complex). *A complex tensor returns a float tensor even if dtype=torch.complex64 is set to linalg.norm() which is a bug.
• The 2nd argument with torch is ord(Optional-Default:None-Type:int, float or str): *Memos:
  • It sets the order of norm.
  • fro(Frobenius norm) or nuc(Nuclear norm) can be selected for str.
• The 3rd argument with torch is dim(Optional-Default:None-Type:int or tuple or list of int). The length of tuple or list of int must be 2.
• The 4th argument with torch is keepdim(Optional-Default:False-Type:bool). *My post explains keepdim argument.
• There is out argument with torch(Optional-Default:None-Type:tensor): *Memos:
  • out= must be used.
  • My post explains out argument.

• There is dtype argument with torch(Optional-Default:None-Type:dtype):
  *Memos:

  • If it's None, it's inferred from input.
  • dtype= must be used.
  • My post explains dtype argument.

• If dim is int, a vector norm is computed.

• If dim is tuple or list of int, a matrix norm is computed.

• If ord and dim are None, the input tensor is flattened to 1D and the L2 norm of the resulting vector will be computed.

• If ord is not None and dim is None, input must be 1D or 2D tensor.

• ord supports the following order of norms:
  *Memos:

  • inf can be torch.inf, float('inf'), etc:
  • For vector, there are also L3 norm(ord=3), L4 norm(ord=4) etc.

ord	Norm for matrix	Norm for vector
None(default)	Frobenius norm	L2 norm
'fro'	Frobenius norm	Not supported
'nuc'	Nuclear norm	Not supported
inf	max(sum(abs(x), dim=1))	max(abs(x))
-inf	min(sum(abs(x), dim=1))	min(abs(x))
0	Not supported	L0 norm(sum(x != 0))
1	max(sum(abs(x), dim=0))	sum(abs(x)^{ord})^{(1/ord)}(L1 norm)
-1	min(sum(abs(x), dim=0))	Same as above
2	The largest singular value of SVD(Singular Value Decomposition)	Same as above(L2 norm)
-2	The smallest singular value of SVD	Same as above
Other int or float	Not supported	Same as above

import torch
from torch import linalg

my_tensor = torch.tensor([-3., -2., -1., 0., 1., 2., 3., 4.])

linalg.norm(input=my_tensor) # L2 norm
# tensor(6.6332)

linalg.norm(input=my_tensor, ord=0) # L0 norm
# tensor(7.)

linalg.norm(input=my_tensor, ord=1) # L1 norm
# tensor(16.)

linalg.norm(input=my_tensor, ord=-1)
# tensor(0.)

linalg.norm(input=my_tensor, ord=2) # L2 norm
# tensor(6.6332)

linalg.norm(input=my_tensor, ord=-2)
# tensor(0.)

linalg.norm(input=my_tensor, ord=torch.inf)
# tensor(4.)

linalg.norm(input=my_tensor, ord=-torch.inf)
# tensor(0.)

my_tensor = torch.tensor([[-3., -2., -1., 0.],
                          [1., 2., 3., 4.]])
linalg.norm(input=my_tensor) # L2 norm
# tensor(6.6332)

linalg.norm(input=my_tensor, ord=1)
# tensor(4.)

linalg.norm(input=my_tensor, ord=-1)
# tensor(4.)

linalg.norm(input=my_tensor, ord=2)
# tensor(5.8997)

linalg.norm(input=my_tensor, ord=-2)
# tensor(3.0321)

linalg.norm(input=my_tensor, ord=torch.inf)
# tensor(10.)

linalg.norm(input=my_tensor, ord=-torch.inf)
# tensor(6.)

linalg.norm(input=my_tensor, ord='fro') # Frobenius norm
# tensor(6.6332)

linalg.norm(input=my_tensor, ord='nuc') # Nuclear norm
# tensor(8.9318)

my_tensor = torch.tensor([[[-3., -2.], [-1., 0.]],
                          [[1., 2.], [3., 4.]]])
linalg.norm(input=my_tensor) # L2 norm
# tensor(6.6332)

linalg.norm(input=my_tensor, ord=0, dim=2) # L0 norm
linalg.norm(input=my_tensor, ord=0, dim=-1) # L0 norm
# tensor([[2., 1.],
#         [2., 2.]])

linalg.norm(input=my_tensor, ord=1, dim=2) # L1 norm
linalg.norm(input=my_tensor, ord=1, dim=-1) # L1 norm
# tensor([[5., 1.],
#         [3., 7.]])

linalg.norm(input=my_tensor, ord=-1, dim=2)
linalg.norm(input=my_tensor, ord=-1, dim=(2,))
linalg.norm(input=my_tensor, ord=-1, dim=-1)
linalg.norm(input=my_tensor, ord=-1, dim=(-1,))

# tensor([[1.2000, 0.0000],
#         [0.6667, 1.7143]])

linalg.norm(input=my_tensor, ord=2, dim=2) # L2 norm
linalg.norm(input=my_tensor, ord=2, dim=(2,)) # L2 norm
linalg.norm(input=my_tensor, ord=2, dim=-1) # L2 norm
linalg.norm(input=my_tensor, ord=2, dim=(-1,)) # L2 norm
# tensor([[3.6056, 1.0000],
#         [2.2361, 5.0000]])

linalg.norm(input=my_tensor, ord=-2, dim=2)
linalg.norm(input=my_tensor, ord=-2, dim=(2,))
linalg.norm(input=my_tensor, ord=-2, dim=-1)
linalg.norm(input=my_tensor, ord=-2, dim=(-1,))

# tensor([[1.6641, 0.0000],
#         [0.8944, 2.4000]])

linalg.norm(input=my_tensor, ord=torch.inf, dim=2)
linalg.norm(input=my_tensor, ord=torch.inf, dim=(2,))
linalg.norm(input=my_tensor, ord=torch.inf, dim=-1)
linalg.norm(input=my_tensor, ord=torch.inf, dim=(-1,))
# tensor([[3., 1.],
#         [2., 4.]])

linalg.norm(input=my_tensor, ord=-torch.inf, dim=2)
linalg.norm(input=my_tensor, ord=-torch.inf, dim=-1)
linalg.norm(input=my_tensor, ord=-torch.inf, dim=(2,))
linalg.norm(input=my_tensor, ord=-torch.inf, dim=(-1,))
# tensor([[2., 0.],
#         [1., 3.]])

linalg.norm(input=my_tensor, ord='fro', dim=(1, 2)) # Frobenius norm
# tensor([3.7417, 5.4772])

linalg.norm(input=my_tensor, ord='nuc', dim=(1, 2)) # Nuclear norm
# tensor([4.2426, 5.8310])

my_tensor = torch.tensor([[[-3.+0.j, -2.+0.j], [-1.+0.j, 0.+0.j]],
                          [[1.+0.j, 2.+0.j], [3.+0.j, 4.+0.j]]])
linalg.norm(input=my_tensor, dtype=torch.complex64) # L2 norm
# tensor(6.6332)