PyTorch :: Performing Derivatives on Tensors

Hello everyone,

Especially to those who fear from maths :P

In this you will learn how to perform derivatives on tensors using pytorch. Let me tell you, once you will learn, it will be fun for you.

Basically the derivatives are used for generating the parameters in the neural network. For now you will learn the basics of derivatives.

Suppose you have a function $f(x) = x^2$ , so according to calculus, the derivative of $f(x)$ would be $\frac{d f(x)}{x} = 2x^1$

Let's see how can we do this and find the value of where $x=9$

Read more about it's usage: https://medium.com/@alfonsollanes/why-do-we-use-the-derivatives-of-activation-functions-in-a-neural-network-563f2886f2ab

import torch

x = torch.tensor(9.0, requires_grad=True)
print(x)

tensor(9., requires_grad=True)

This time I am setting requires_grad=True, this means we are using pytorch autograd engine to compute the derivatives. Derivatives are calculated by tracing the graph (enabled when requires_grad=True) from root to leaf by using chain rule (back propagation).

Note Only Tensors of floating point dtype can require gradients

For more information you can read this wonderful post: https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95

Also you can read the official documentation regarding autograd on pytorch's website: https://pytorch.org/docs/stable/notes/autograd.html

Moving on, let's calculate $x^2$

y = x**2
print(y)

tensor(81., grad_fn=<PowBackward0>)

If you would see, this time the pytorch has attached a gradient function with the tensor.

To calculate the derivative, you need to call Tensor.backward() method.

And use .grad property on the leaf tensor (here it's $x$)

y.backward()
print(x.grad)

tensor(18.)

Isn't it look like a magic to you? Let's calculate the derivative of another complex equation.

$$f(x) = \sin ( \frac{x}{2 \sqrt 2} )$$

Where $x = 100$

If you would simplify the denominator $2 \sqrt 2$ , it would be $2^{\frac{3}{2}}$

x = torch.tensor(100., requires_grad=True)

y = torch.sin(x / (2 ** 1.5) )

print(x)
print(y)

y.backward()
print(x.grad)

tensor(100., requires_grad=True)
tensor(-0.7158, grad_fn=<SinBackward>)
tensor(-0.2469)

If you want to test it, you can go to https://www.derivative-calculator.net/ and type in the above expression.

You can also calculate the partial derivatives in pytorch. For example you have $f(u,v) = uv + u^2 + v^3$ , the derivative $\frac{d f(u,v)}{du} = 2u + v$ and $\frac{d f(u,v)}{dv} = 3v^2 + u$

Let's calculate the partial derivative using pytorch where $u = 2, v = 4$

u = torch.tensor(2., requires_grad=True)
v = torch.tensor(4., requires_grad=True)

f = u**2 + v**3 + u*v

print(u)
print(v)
print(f)

f.backward()
print(u.grad)
print(v.grad)

tensor(2., requires_grad=True)
tensor(4., requires_grad=True)
tensor(76., grad_fn=<AddBackward0>)
tensor(8.)
tensor(50.)

If you have doubts or any idea reach me out via following sources

• Email: [e-mail] (Recommended)
• Twitter: @tbhaxor
• GitHub: @tbhaxor
• LinkedIn: @gurkirat--singh
• Instagram: @_tbhaxor_