Dot vs Matrix vs Element-wise multiplication in PyTorch

*My post explains the functions and operators for Dot and Matrix multiplication and Element-wise calculation in PyTorch.

<Dot multiplication(product)>

• Dot multiplication is the multiplication of 1D tensors(arrays).
• The rule which you must follow to do dot multiplication is the number of the rows of A and B tensor(array) must be 1 and the number of the columns must be the same.

   <A>         <B>
[a, b, c] x [d, e, f] = ad+be+cf
1 row       1 row
3 columns   3 columns

[2, 7, 4] x [6, 3, 5] = 53
                 (2x6)+(7x3)+(4x5)
  [2, 7, 4]
   x  x  x
  [6, 3, 5]
      ||
 [12, 21, 20]
 12 + 21 + 20
      ||
      53

In PyTorch with @, dot() or matmul():

import torch

tensor1 = torch.tensor([2, 7, 4])
tensor2 = torch.tensor([6, 3, 5])

torch.dot(tensor1, tensor2)
tensor1.dot(tensor2)
tensor1 @ tensor2
torch.matmul(tensor1, tensor2)
tensor1.matmul(tensor2)
# tensor(53)

*Memos:

• dot() can multiply 1D tensors by dot multiplication
• @ or matmul() can multiply 1D or more D tensors by dot or matrix multiplication.

In NumPy with @, dot() or matmul():

import numpy

array1 = numpy.array([2, 7, 4])
array2 = numpy.array([6, 3, 5])

numpy.dot(array1, array2)
array1.dot(array2)
array1 @ array2
numpy.matmul(array1, array2)
# 53

*Memos:

• dot() multiply 0D or more D arrays by dot or matrix multiplication. *dot() is basically used to multiply 1D arrays.
• @ or matmul() can multiply 1D or more D arrays.

<Matrix multiplication(product)>

• Matrix multiplication is the multiplication of 2D or more tensors(arrays). *Either of 2 operands can be a 1D tensor(array) but not both of them.
• The rule which you must follow to do matrix multiplication is the number of the columns of A tensor(array) must match the number of the rows of B tensor(array).

2D tensors(arrays):

    <A>            <B>
[[a, b, c], x [[g, h, i, j], = [[ag+bk+co, ah+bl+cp, ai+bm+cq, aj+bn+cr],
 [d, e, f]]    [k, l, m, n],    [dg+ek+fo, dh+el+fp, di+em+fq, dj+en+fr]]
               [o, p, q, r]]
2 rows         (3) rows
(3) columns    4 columns

[[2, 7, 4], x [[5, 0, 8, 6], = [[35, 58, 59, 69],
 [6, 3, 5]]    [3, 6, 1, 7],    [44, 38, 96, 67]]              
               [1, 4, 9, 2]]   [[2x5+7x3+4x1, 2x0+7x6+4x4, 2x8+7x1+4x9, 2x6+7x7+4x2]
                                [6x5+3x3+5x1, 6x0+3x6+5x4, 6x8+3x1+5x9, 6x6+3x7+5x2]]

In PyTorch with @, matmul() or mm(). *mm() can multiply 2D tensors by matrix multiplication:

import torch

tensor1 = torch.tensor([[2, 7, 4], [6, 3, 5]])
tensor2 = torch.tensor([[5, 0, 8, 6], [3, 6, 1, 7], [1, 4, 9, 2]])

tensor1 @ tensor2
torch.matmul(tensor1, tensor2)
tensor1.matmul(tensor2)
torch.mm(tensor1, tensor2)
tensor1.mm(tensor2)
# tensor([[35, 58, 59, 69], [44, 38, 96, 67]])

In NumPy with @, matmul() or dot():

import numpy

array1 = numpy.array([[2, 7, 4], [6, 3, 5]])
array2 = numpy.array([[5, 0, 8, 6], [3, 6, 1, 7], [1, 4, 9, 2]])

array1 @ array2
numpy.matmul(array1, array2)
numpy.dot(array1, array2)
array1.dot(array2)
# array([[35, 58, 59, 69], [44, 38, 96, 67]])

A 1D and 3D tensor(array). *B 3D tensor(array) has 3 2D tensors(arrays) which have 2 rows and 4 columns each:

  <A>             <B>
[a, b]   x   [[[c, d, e, f], = [[(ac+bg), (ad+bh), (ae+bi), (af+bj)],
               [g, h, i, j]],   [(ak+bo), (al+bp), (am+bq), (an+br)],
              [[k, l, m, n],    [(as+bw), (at+bx), (au+by), (av+bz)]]
               [o, p, q, r]],
              [[s, t, u, v],
               [w, x, y, z]]]
1 row         (2) rows
(2) columns   4 columns

[2, 7]   x   [[[6, 3, 5, 2], = [[47,  6, 66, 32],
               [5, 0, 8, 4]],   [20, 68, 65, 35],
              [[3, 6, 1, 0],    [42, 39, 21, 69]]
               [2, 8, 9, 5]],  [[2x6+7x5, 2x3+7x0, 2x5+7x8, 2x2+7x4],
              [[7, 2, 0, 3],    [2x3+7x2, 2x6+7x8, 2x1+7x9, 2x0+7x5],
               [4, 5, 3, 9]]]   [2x7+7x4, 2x2+7x5, 2x0+7x3, 2x3+7x9]]

In PyTorch with @ or matmul():

import torch

tensor1 = torch.tensor([2, 7])
tensor2 = torch.tensor([[[6, 3, 5, 2], [5, 0, 8, 4]],
                        [[3, 6, 1, 0], [2, 8, 9, 5]],
                        [[7, 2, 0, 3], [4, 5, 3, 9]]])
tensor1 @ tensor2
torch.matmul(tensor1, tensor2)
tensor1.matmul(tensor2)
# tensor([[47, 6, 66, 32], [20, 68, 65, 35], [42, 39, 21, 69]])

In NumPy with @, matmul() or dot():

import numpy

array1 = numpy.array([2, 7])
array2 = numpy.array([[[6, 3, 5, 2], [5, 0, 8, 4]],
                      [[3, 6, 1, 0], [2, 8, 9, 5]],
                      [[7, 2, 0, 3], [4, 5, 3, 9]]])
array1 @ array2
numpy.matmul(array1, array2)
numpy.dot(array1, array2)
array1.dot(array2)
# array([[47, 6, 66, 32], [20, 68, 65, 35], [42, 39, 21, 69]])

<Element-wise multiplication(product)>

• Element-wise multiplication is the multiplication of 0D or more D tensors(arrays).
• The rule which you must follow to do element-wise multiplication is 2 tensors(arrays) must have the same number of rows and columns.

1D tensors(arrays):

[a, b, c] x [d, e, f] = [ad, be, cf]
1 row       1 row 
3 columns   3 columns

[2, 7, 4] x [6, 3, 5] = [12, 21, 20]
                       [2x6, 7x3, 4x5]
 [2, 7, 4]
  x  x  x 
 [6, 3, 5]
     ||
[12, 21, 20]

In PyTorch with * or mul(). ** or mul() can multiply 0D or more D tensors by element-wise multiplication:

import torch

tensor1 = torch.tensor([2, 7, 4])
tensor2 = torch.tensor([6, 3, 5])

tensor1 * tensor2
torch.mul(tensor1, tensor2)
tensor1.mul(tensor2)
# tensor([12, 21, 20])

In NumPy with * or multiply(). ** or multiply() can multiply 0D or more D arrays by element-wise multiplication.

import numpy

array1 = numpy.array([2, 7, 4])
array2 = numpy.array([6, 3, 5])

array1 * array2
numpy.multiply(array1, array2)
# array([12, 21, 20])

2D tensors(arrays):

[[a, b, c], x [[g, h, i], = [[ag, bh, ci],
 [d, e, f]]    [j, k, l]]    [dj, ek, fl]]
2 rows        2 rows
3 columns     3 columns

[[2, 7, 4], x [[5, 0, 8], = [[10, 0, 32],
 [6, 3, 5]]    [3, 6, 1]]    [18, 18, 5]]
                            [[2x5 7x0 4x8, 
                             [6x3 3x6 5x1]]
  [[2, 7, 4], [6, 3, 5]]
    x  x  x    x  x  x
  [[5, 0, 8], [3, 6, 1]]
            ||
[[10, 0, 32], [18, 18, 5]]

In PyTorch with* or mul():

import torch

tensor1 = torch.tensor([[2, 7, 4], [6, 3, 5]])
tensor2 = torch.tensor([[5, 0, 8], [3, 6, 1]])

tensor1 * tensor2
torch.mul(tensor1, tensor2)
tensor1.mul(tensor2)
# tensor([[10, 0, 32], [18, 18, 5]])

In NumPy with * or multiply():

import numpy

array1 = numpy.array([[2, 7, 4], [6, 3, 5]])
array2 = numpy.array([[5, 0, 8], [3, 6, 1]])

array1 * array2
numpy.multiply(array1, array2)
# array([[10, 0, 32], [18, 18, 5]])