One of the best ways to understand Deep Learning is by learning concepts rather than frameworks, so occasionally I take a piece of code written in TensorFlow and convert it to PyTorch or vice-versa. On a high level, there isn't much difference between these two libraries.
# Sequential model in both Pytorch and Tensorflow model = keras.Sequential( [ layers.conv2d(20,5, activation="relu"), layers.conv2d(64,5, activation="relu"), ] ) model = nn.Sequential( nn.Conv2d(1,20,5), nn.ReLU(), nn.Conv2d(20,64,5), nn.ReLU() )
They both implement Artificial Neural Networks the same way. Thanks to Keras we can say that TensorFlow is quite beginner-friendly. Pytorch can be quite intimidating at first partly because you have to define your own training loop. This is great if you are a researcher because you can easily take control of the entire process. With TensorFlow, you can also define your own training loop too which is pretty much the same as PyTorch. As you go deeper into both frameworks you may start seeing the difference between both of them. If you read any article or listen to an expert on deep learning they usually say stuff like PyTorch is more pythonic but how.
Originally there was a huge difference between TensorFlow and PyTorch even on a high level. In Tensorflow 1.x in other to train a neural network you have to create a computational graph then place it within a session
import tensorflow as tf # Tensorflow 1.x x = tf.constant([1., 2., 3.]) # Define a tensor x y = tf.constant([4., 5., 6.]) # Define a tensor y add_op = tf.add(x,y) # Add operation with tf.Session() as sess: #create a session result = sess.run(add_op) # run the session print(result) # [ 5.,7., 9.,]
I still remember my first introduction to TensorFlow 1.x it was so terrifying that I had to abandon it and use Keras instead. This wasn't pythonic at all it felt like you were coding in Java and not python, you couldn't test your code without compiling it. Luckily Tensorflow 2.x fixed this issue and introduced Eager Execution which enables programmers to run ML code in a more pythonic way.
#Tensorflow 2.x x = tf.constant([1., 2., 3.]) # Define a tensor x y = tf.constant([4., 5., 6.]) # Define a tensor y result = tf.add(x,y) # Add operation print(result) # [5,7,9]
There is no need of running your code within sessions everything happened automatically. This is what made PyTorch and TensorFlow become similar in many ways.
import torch x = torch.tensor([1,2,3]) # Define a tensor x y = torch.tensor([4,5,6]) # Define a tensor y result = torch.add(x,y) # Add operation print(result) # [5,7,9]
Pytorch never made programmers create a computational graph and run it within sessions you can say PyTorch always ran eagerly.
Apart from its name, PyTorch feels more pythonic here's why. Pytorch is basically Numpy with only two differences, it runs on the GPU and can perform automatic differentiation. If you're already familiar with NumPy using PyTorch will be very intuitive.
import torch import numpy as np torch.tensor([1,2,3]) np.array([1,2,3]) tf.constant([1,2,3]) # OUTPUT: [1,2,3] torch.ones(5) np.ones(5) tf.ones(5) # OUTPUT: [1, 1, 1, 1, 1] torch.zeros(5) np.zeros(5) # OUTPUT:[0, 0, 0, 0, 0] t = torch.tensor([[1,2,3],[4,5,6]]) n = np.array([[1,2,3], [4,5,6]]) t.flatten() n.flatten() # OUTPUT:[1, 2, 3, 4, 5, 6] # No TF equivalent np.arange(4) torch.arange(4) tf.range(4) # slightly different from numpy # OUTPUT: [0,1,2,3]
In general, all three libraries have the same functionalities but PyTorch and NumPy are basically the same. One way TensorFlow differs from NumPy/PyTorch is in TensorFlow tensors are immutable. They are like strings in python.
n = np.array([1,2,3,4]) p = torch.tensor([1,2,3,4]) t = tf.constant([1,2,3,.4]) print(n) # output: 1 print(p) # output: 1 print(t) # output: 1 n = 2 # n = [2,2,3,4] p = 2 # p = [2,2,3,4] t = 2 # TypeError
There are ways around this in TensorFlow. You will have to define a
tf.variable instead of
tf.constant but still it is not very straightforward like PyTorch, NumPy, or regular python list. This is just one of the ways that set TensorFlow apart from the others.
Pytorch is a Deep learning library built from the ground up to be very python-centric. If you are familiar with NumPy already PyTorch will feel natural. Tensorflow on the other hand is a bit eccentric in the way it performs operations that some might say are not pythonic.