DEV Community: Aravind Ramalingam

Do You Visualize DataLoaders for Deep Neural Networks?

Aravind Ramalingam — Thu, 03 Jun 2021 18:12:16 +0000

A lot of factors can affect the performance of a Deep Neural Network. Let’s see how we can set up a PyTorch pipeline for Image augmentation (with help of Albumentations Python library), and use visualization to find any potential issue before it costs you time and money.

In software development when something is wrong, usually we get an error, but with data science this is not the case. If the model is not performing well, then the general approach is to alter the model architecture or tune hyperparameters and train more. Yes, these are good options, but making sure that the data is correct should be the priority.

From the television series The Newsroom1:

The first step in solving a problem is to recognize that it does exist.

Difference in train vs test data is the single biggest reason for low performing models. Image augmentations help to fight overfitting and improve the performance of deep neural networks for computer vision tasks. In this post, we will cover the following:

Overview of Computer Vision Tasks
Computer Vision Pipeline
Image Augmentation using Albumentations
Visualize DataLoaders

If you are familiar with Computer Vision tasks and Pytorch, then feel free to skip the first two.

Overview of Computer Vision Tasks

Common types of Computer Vision (CV) tasks are:

Image Classification
Image Segmentation
Object Detection

If you are not familiar with the difference between them, then check out the post - Image Classification vs. Object Detection vs. Image Segmentation by. Image augmentations help to make the model generalize better for all 3 types CV tasks.

Computer Vision Pipeline

Both Input and Target data has to go through Dataset and DataLoader before being passed on to the model for training. It is better to visualize the output of the DataLoader. By this way, we can also identify when there is an issue with the Dataset definition. In relation to the above data pipeline diagram, only Target data type and Transformations applied differs based on CV task type.

Image Augmentation using Albumentations

Albumentations is a fast and flexible image augmentation library. It supports both PyTorch and Keras. Torchvision library is good but when it comes to Image Segmentation or Object Detection, it requires a lot of effort to get it right. To elaborate, Pixel-level and Spatial-level are two types of image augmentations. The way of applying transformations to input data and target label differs for these two types.

Pixel-level augmentations:

Changes the values of pixels of the original image, but they don't change the output mask. Image transformations such as changing brightness or contrast of adjusting values of the RGB-palette of the image are pixel-level augmentations. We modify the input image by adjusting its brightness, but we keep the output mask intact.

Spatial-level augmentations:

Changes both the image and the mask. When you apply image transformations such as mirroring or rotation or cropping a part of the input image, you also need to apply the same transformation to the output label to preserve its correctness.

Brownie Points

Ability to add image augmentations to any computer vision pipeline with minimal effort.
Syntax for transforms declaration is very similar to Torchvision.
Lets you set the required probabilities and the magnitude of values for each transformation.

An example definition of an augmentation pipeline is as follows:

Still not convinced? Then check out the article - Why you need a dedicated library for image augmentation.

Visualize DataLoaders

Why?

Right now, probably would be wondering if Albumentations is so great then why do we need to visualize the DataLoaders. I'm glad you asked, Albumentation still requires user input, and we are not that great at providing correct values2. Any rational person should think about the type of augmentation and whether it is applicable to a particular dataset or not but often this is not the case.

It is better to visualize just to avoid any surprises.

When Rotation transformation is applied to the digit 9 from MNIST dataset, it can be transformed to 6, but the label would still say 9. See how easy it is to screw up data augmentation.?

Original Image - 9	Transformed Image - 9

How?

Torchvision functions like make_grid and draw_bounding_boxes are quite handy, but they are not end to end. So we will write sort of wrappers which take in the iter object of the DataLoaders and plot the Input and Target data values.

Notes about the below code:

PyTorch Tensor expects image of shape (C x H x W) which is the reverse when compared to NumPy array shape (H x W x C).
Image data must be torch tensor of float data type to train the model, whereas for plotting it should be of type uint8.
It is best practice to avoid multiple initialization of the iter object for DataLoaders when possible as it is very time-consuming. This is the reason for taking the iter object as function parameter instead of DataLoader itself.

Once again, The focus is on the output of DataLoaders, if you are interested in the overall data pipeline then check out this Colab Notebook

Image Classification

Dataset 👉 CIFAR10

Transforms 👉 Resize ➡️ Rotate ➡️ Horizontal Flip

Image Segmentation

Dataset 👉 CAMVID

Transforms 👉 Resize ➡️ Rotate ➡️ Horizontal Flip

Object Detection

Dataset 👉 Penn-Fudan Database for Pedestrian Detection

Transforms 👉 Resize ➡️ Horizontal Flip ➡️ Brightness & Contrast ➡️ Shift, Scale & Rotate

Conclusion

In Computer Vision, it is relatively easier to grasp what the model is doing. How it does something is a different story, but with enough experiments we can sort of guess what is working and what is not. The key is to figure this out with minimal number of experiments. The more confident we are in the data pipeline setup, more time to run various experiments to improve the model performance.

Side Notes:

[1]: I believe that the quote from The Newsroom is adopted from the below qoute by Zig Ziglar, but I am not too sure.

You cannot solve a problem until you acknowledge that you have one and take responsibility for solving it.

[2]: I am aware of AutoAlbument but I am yet to give it a go.

Stamp Detection using Computer Vision and Python

Aravind Ramalingam — Fri, 21 May 2021 20:38:34 +0000

Background

A friend of mine reach out and asked me whether I could write a program to detect the number of Rubber stamps in an image. Apparently, these invoice receipts will be categorized based on the number of stamps on them. Initially, I thought of building a Deep Learning Segmentation model, but soon I realized that it is not worth the effort.

The images are generated in a controlled environment so few computer vision algorithms should do the trick. To illustrate the computer vision algorithms used in detecting the stamps, I will be using a sample image downloaded from Google as the original image is company property. The goal is to identify two stamps in the sample image.

Solution

High level solution steps are:

Read the image.
Blur & detect the edges.
Find all contours and remove the smaller contours.
Fill the area inside contours & Close the blobs.
Filter the stamps.

Before we start let us import the necessary packages.

1. Read the image

Read the color image using imread function. To display the image we will use Matplotlib. Matplotlib expects the color image channels to be of the order RGB, but OpenCV reads the image as BGR, so we will write a helper function for the conversion.

2. Blur & detect the edges

First, we need to convert the image to grayscale using cvtColor function. Then, we will use bilateralFilter to reduce noise in the image. Bilateral filter is preferred over Gaussian because it preserves the edges much better. Finally, we will use canny edge detector to threshold the image and detect edges. The normal canny edge detector requires two threshold parameters which is hard to tune so we will use the one from Zero-parameter, automatic Canny edge detection with Python and OpenCV

3. Find all contours and remove the smaller contours

findContours function can find all contours in the image. The outer most contours are good enough for our use case so we will use the retrieval mode RETR_EXTERNAL. CHAIN_APPROX_NONE mode is preferred as we don't want to lose any point on the contour due to approximation. To remove the unwanted smaller contours, we can filter the contours by area.

Total nr of contours found: 408

4. Fill the area inside contours & Close the blobs

Instead of working on the original binary image, we will draw the top contours on a image with black background and use this as base. Any disconnect in the contours are easier to identify when fill the area inside the contours using drawContours function.

As suspected, the top stamp has a thin black line passing through it. We need to close this blob so that the top stamp is considered as one contour instead of two different ones. Morphological Closing operation is perfect for achieving this.

5. Filter the stamps

To isolate the stamp contours, we can identify all the contours from the latest binary image and filter for contours with more than 5 points as the stamp is an ellipse.

Bonus - Highlight the identified stamps

For demo of this program, wouldn't it be cool if we can highlight only the stamped area of the image? Since we agree that it is indeed cool, let us see how we can achieve that. The steps involved are:

Duplicate the original image and blur the entire image.
Loop through the blurred image and for the points on or inside the image (using pointPolygonTest to check) we replace it with pixel values from the output image. We are using pixel values from the output image because we need the blue lines drawn over the stamps.

Conclusion

Yep, that's it for this post. You can access this notebook from here.

Automate Gist Creation From Markdown File Using Python

Aravind Ramalingam — Tue, 11 May 2021 12:33:19 +0000

Markdown's code block is great for writing code snippets. However, when you try to publish or share it via blogging platform there are some shortcomings. Gist can help to bridge the gap. Gist makes your blog look better & easier to maintain. In this post, Let us see how automated gist creation can improve your technical blogging experience.

Gist Intro

Gist is a GitHub repository where you can store & share code/data with others. A single gist can store multiple files. Syntax highlighting is supported based on the file extension type, so be careful when you name the file. Gists can be public or secret. Do note that secret gists aren't private, meaning anyone with the URL would be able to view the code/data.

Why to use gist?

Supports GitHub repository functionalities like version control, fork & clone.
Ability to subscribe, star and comment makes gist easy to collaborate with others.
Gist can be shared in multiple ways without any dependency or installation.

Why use gist in a blog?

My current workflow for blogging is as follows:

Use Jupyter notebook to write the blog.
Download the notebook as markdown file.
Make edits to markdown file before posting to platforms like Medium & Dev to

There are many manual processes involved in step 3, which I'm planning to address in the next few weeks. My biggest concern right now is how the code snippet looks in different platforms. To be more specific:

No easy way to make changes to code snippets after publishing
Medium loses syntax highlighting
Unable to highlight specific lines in a long code snippet in both the platforms

Medium	Gist

Gist helps to solve all these issues. Also as a bonus, All code snippets from the same blog can be grouped together via gist. This makes it easier to refer/access (if needed).

Gist Creation From Markdown File

The goal is to create a gist for the markdown file where every code snippet will be a separate file.

First we will download the token for gist creation like mentioned in this post and store it in a file named - create_gist_token.txt
We can read the markdown file using python and use regular expression to find all the code blocks. Code block for python is created by wrapping the code between `` in the markdown file. The flag re.S makes sure that the matching includes newline as well
As mentioned earlier, each code snippet is going to be a separate file in a gist, so we need a name for each one. To keep it simple, we will take the first comment as the name of the file.
The description of the gist will be the first header of the markdown file. Typically, this will be the name of the blog, so it should be sufficient. Now we can put all the data together and make the API call.
As Dev to supports liquid text we need special formatting for gist embedding when compared to Medium. In both the platforms, we need to specify the file so that the entire gist is not displayed.
Loop through all the uploaded files and replace them with corresponding gist embeddings in the markdown file. I have included the embeddings for both the platforms, feel free to modify it as per your needs.
Rewiring the script to make it easier to run from command line.

Takeaway

Gist is quite useful and addresses many of the pet peeves in blogging. We have managed to put together a python script which can reduce the manual effort required in publishing a blog. Feel free to reach out to me via Twitter or comments on your thoughts about this post.

Digit Emoji Predictor: Build UI for Deep Learning model in Notebook

Aravind Ramalingam — Fri, 30 Apr 2021 12:03:13 +0000

Build a digit emoji predictor in Jupyter notebook by passing data between JavaScript & Python. Typically, JavaScripts are used for data visualization in notebooks, but it can also be used for prototyping front-end/UI for deep learning models.

Why the hell build UI in notebook?

Short Answer:

Share the Deep Learning model notebook with colleagues from different teams like Business, Data Science, Front End developer, DevOps for their opinion before the real software development takes place.

Jupyter notebook is meant for quick prototyping and everyone knows this but what many misses is we can also do a quick prototype of UI as well. Many data scientists are quick to start building the big deep learning model with whatever little data they have without even thinking about what they are developing and for whom 🤷‍♂️. Trust me, the first model you are satisfied with (based on the metrics like accuracy) won't be deployed into production for various reasons like slow inference time, no on-device support, memory hogging blah blah.

Even experienced project/product managers sometimes won't know what they really want so first try to get the workflow or pipeline agreement from all the players involved, even with this some things will slip through the crack. "I am data scientist, My job is only to build ML models is not the right mindset, at least in my opinion." A tad bit of knowledge in Business, Web development, DevOps can help you save a ton of time in trying to debug, or explain why the model/workflow you built is the best for business. Better to be jack of all trades here. BTW, this does not mean that you should take over the responsibilities of your colleagues. Just try to get a sense of what is happening outside your circle. In this post, Let me share an example of how creating a simple UI for input & output for deep learning model can help you see the cracks in your workflow and increase productivity.

Application

If you are not familiar with MNIST dataset then just know that you can build a Digit Emoji Predictor as the dataset contains gray-scale images of hand-drawn digits, from zero through nine. So our UI is going to let the user draw a digit & display the corresponding emoji.

Okay, I am somewhat convinced to build UI but now what?

Firstly, we will decide what we are going to build? Instead of creating from scratch let us try to recreate Digit Recognizer in notebook without using Django. The use case we have chosen is great because we can't simply use pywidgets, this forces us to dig deep into the world of HTML & Javascript. In the longer run this route has more benefits because of unlimited capability, free resource and more importantly closer to the actual front-end. Ofcourse if all you want is a slider then just roll with pywidgets. We will first setup the back-end and then get back to the UI.

Digit Emoji Recognizer

I have trained the CNN Deep learning model using the pytorch, we will just download the model & weights from here. This is a simple model with 3 Conv blocks and 2 Fully connected layer. The Transformations used for this model are Resize & ToTensor. Upon calling the predict function on the image, we will get the predicted digit.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import transforms

# Digit Recognizer model definition
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 32, kernel_size=5)
        self.conv3 = nn.Conv2d(32,64, kernel_size=5)
        self.fc1 = nn.Linear(3*3*64, 256)
        self.fc2 = nn.Linear(256, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(F.max_pool2d(self.conv3(x),2))
        x = F.dropout(x, p=0.5, training=self.training)
        x = x.view(-1,3*3*64 )
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return x



# Predict the digit given the tensor
def predict(image, modelName = 'mnist_model.pth'):

    # Resize before converting the image to Tensor
    Trfms = transforms.Compose([transforms.Resize(28), transforms.ToTensor()])

    # Apply transformations and increase dim for mini-batch dimension
    imageTensor = Trfms(image).unsqueeze(0)

    # Download model from URL and load it
    model = torch.load(modelName)

    # Activate Eval mode
    model.eval()

    # Pass the image tensor by the model & return max index of output
    with torch.no_grad():
        out = model(imageTensor)
        return int(torch.max(out, 1)[1])

We need a helper function to convert BASE64 Image (canvas area in which the user is going to draw the digit) to PIL Image.

import re, base64
from PIL import Image
from io import BytesIO

# Decode the image drawn by the user
def decodeImage(codec):

    # remove the front part of codec
    base64_data = re.sub('^data:image/.+;base64,', '', codec)

    # base64 decode
    byte_data = base64.b64decode(base64_data)

    # Convert to bytes
    image_data = BytesIO(byte_data)

    # Convert to image & convert to grayscale
    img = Image.open(image_data).convert('L')
    return img

Now let us combine the two functions namely decodeImage and predict as a separate function. This way is easier because the output of the function (emoji html code) can be directly passed to the HTML tag via JavaScript. Hang on for a bit if you are confused, things will get much easier to follow in the next section.

# Decode the image and predict the value
def decode_predict(imgStr):

    # Decode the image
    image = decodeImage(imgStr)

    # Declare html codes for 0-9 emoji
    emojis = [ 
            "&#48;&#65039;&#8419;",
             "&#49;&#65039;&#8419;",
             "&#50;&#65039;&#8419;",
             "&#51;&#65039;&#8419;",
             "&#52;&#65039;&#8419;",
             "&#53;&#65039;&#8419;",
             "&#54;&#65039;&#8419;",
             "&#55;&#65039;&#8419;",
             "&#56;&#65039;&#8419;",
             "&#57;&#65039;&#8419;"
    ]

    # Call the predict function
    digit = predict(image)

    # get corresponding emoji
    return emojis[digit]

UI

The HTML & CSS part are quite straight forward, the JavaScript lets the user to draw inside the canvas and call functions via the buttons Predict & Clear. The Clear button cleans up the canvas and sets the result to 🤔 emoji. We will setup the Predict button JS function in the next section.

from IPython.display import HTML

html = """
<div class="outer">
        <! -- HEADER SECTION -->
    <div> 
        <h3 style="margin-left: 30px;"> Digit Emoji Predictor &#128640; </h3> 
        <br>
        <h7 style="margin-left: 40px;"> Draw a digit from &#48;&#65039;&#8419; - &#57;&#65039;&#8419;</h7>
    </div>

    <div>
           <! -- CANVAS TO DRAW THE DIGIT -->
        <canvas id="canvas" width="250" height="250" style="border:2px solid; float: left; border-radius: 5px; cursor: crosshair;">
        </canvas>

            <! -- SHOW PREDICTED DIGIT EMOJI-->
        <div class="wrapper1"> <p id="result">&#129300;</p></div>

            <! -- BUTTONS TO CALL DL MODEL & CLEAR THE CANVAS -->
        <div class="wrapper2">
            <button type="button" id="predictButton" style="color: #4CAF50;margin:10px;">  Predict </button>  
            <button type="button" id="clearButton" style="color: #f44336;margin:10px;">  Clear </button>  
        </div>
    </div>
</div>
"""


css = """
<style>
    .wrapper1 {
      text-align: center;
      display: inline-block;
      position: absolute;
      top: 82%;
      left: 25%;
      justify-content: center;
      font-size: 30px;
    }
    .wrapper2 {
      text-align: center;
      display: inline-block;
      position: absolute;
      top: 90%;
      left: 19%;
      justify-content: center;
    }

    .outer {
        height: 400px; 
        width: 400px;
        justify-content: center;
    }

</style>
"""


javascript = """

<script type="text/javascript">
(function() {
    /* SETUP CANVAS & ALLOW USER TO DRAW */
    var canvas = document.querySelector("#canvas");
    canvas.width = 250;
    canvas.height = 250;
    var context = canvas.getContext("2d");
    var canvastop = canvas.offsetTop
    var lastx;
    var lasty;
    context.strokeStyle = "#000000";
    context.lineCap = 'round';
    context.lineJoin = 'round';
    context.lineWidth = 5;

    function dot(x, y) {
        context.beginPath();
        context.fillStyle = "#000000";
        context.arc(x, y, 1, 0, Math.PI * 2, true);
        context.fill();
        context.stroke();
        context.closePath();
    }

    function line(fromx, fromy, tox, toy) {
        context.beginPath();
        context.moveTo(fromx, fromy);
        context.lineTo(tox, toy);
        context.stroke();
        context.closePath();
    }
    canvas.ontouchstart = function(event) {
        event.preventDefault();
        lastx = event.touches[0].clientX;
        lasty = event.touches[0].clientY - canvastop;
        dot(lastx, lasty);
    }
    canvas.ontouchmove = function(event) {
        event.preventDefault();
        var newx = event.touches[0].clientX;
        var newy = event.touches[0].clientY - canvastop;
        line(lastx, lasty, newx, newy);
        lastx = newx;
        lasty = newy;
    }
    var Mouse = {
        x: 0,
        y: 0
    };
    var lastMouse = {
        x: 0,
        y: 0
    };
    context.fillStyle = "white";
    context.fillRect(0, 0, canvas.width, canvas.height);
    context.color = "black";
    context.lineWidth = 10;
    context.lineJoin = context.lineCap = 'round';
    debug();
    canvas.addEventListener("mousemove", function(e) {
        lastMouse.x = Mouse.x;
        lastMouse.y = Mouse.y;
        Mouse.x = e.pageX - canvas.getBoundingClientRect().left;
        Mouse.y = e.pageY - canvas.getBoundingClientRect().top;
    }, false);
    canvas.addEventListener("mousedown", function(e) {
        canvas.addEventListener("mousemove", onPaint, false);
    }, false);
    canvas.addEventListener("mouseup", function() {
        canvas.removeEventListener("mousemove", onPaint, false);
    }, false);
    var onPaint = function() {
        context.lineWidth = context.lineWidth;
        context.lineJoin = "round";
        context.lineCap = "round";
        context.strokeStyle = context.color;
        context.beginPath();
        context.moveTo(lastMouse.x, lastMouse.y);
        context.lineTo(Mouse.x, Mouse.y);
        context.closePath();
        context.stroke();
    };

    function debug() {
        /* CLEAR BUTTON */
        var clearButton = $("#clearButton");
        clearButton.on("click", function() {
            context.clearRect(0, 0, 250, 250);
            context.fillStyle = "white";
            context.fillRect(0, 0, canvas.width, canvas.height);

            /* Remove Result */
            document.getElementById("result").innerHTML = "&#129300;";
        });
        $("#colors").change(function() {
            var color = $("#colors").val();
            context.color = color;
        });
        $("#lineWidth").change(function() {
            context.lineWidth = $(this).val();
        });
    }
}());

</script>

"""


HTML(html + css + javascript)

Great! We are able to draw inside the canvas and upon clicking the Clear button, canvas is cleaned up. The tricky part starts when the user clicks the Predict button as we have do the following:

JS to Python 👉 Grab the drawing inside the canvas and pass it to python

    /* PASS CANVAS BASE64 IMAGE TO PYTHON VARIABLE imgStr*/
    var imgData = canvasObj.toDataURL();
    var imgVar = 'imgStr';
    var passImgCode = imgVar + " = '" + imgData + "'";
    var kernel = IPython.notebook.kernel;
    kernel.execute(passImgCode);

Python to JS 👉 Set the predicted emoji as the value of HTML element (#result)

    /* CALL PYTHON FUNCTION "decode_predict" WITH "imgStr" AS ARGUMENT */
    function handle_output(response) {
        /* UPDATE THE HTML BASED ON THE OUTPUT */
        var result = response.content.data["text/plain"].slice(1, -1);
        document.getElementById("result").innerHTML = result;
    }
    var callbacks = {
        'iopub': {
            'output': handle_output,
        }
    };
    var getPredictionCode = "decode_predict(imgStr)";
    kernel.execute(getPredictionCode, callbacks, { silent: false });

Linking all together.

predictJS = """
<script type="text/javascript">

/* PREDICTION BUTTON */
$("#predictButton").click(function() {
    var canvasObj = document.getElementById("canvas");
    var context = canvas.getContext("2d");

    /* PASS CANVAS BASE64 IMAGE TO PYTHON VARIABLE imgStr*/
    var imgData = canvasObj.toDataURL();
    var imgVar = 'imgStr';
    var passImgCode = imgVar + " = '" + imgData + "'";
    var kernel = IPython.notebook.kernel;
    kernel.execute(passImgCode);

    /* CALL PYTHON FUNCTION "decode_predict" WITH "imgStr" AS ARGUMENT */
    function handle_output(response) {
        /* UPDATE THE HTML BASED ON THE OUTPUT */
        var result = response.content.data["text/plain"].slice(1, -1);
        document.getElementById("result").innerHTML = result;
    }
    var callbacks = {
        'iopub': {
            'output': handle_output,
        }
    };
    var getPredictionCode = "decode_predict(imgStr)";
    kernel.execute(getPredictionCode, callbacks, { silent: false });
});
</script>

"""
HTML(predictJS)

We have both good & bad news, the good thing is data connection between Python and Javascript works but the prediction values are all wrong (mapped to same value - 8). Let us compare the canvas image passed to python and MNIST dataset image.

Canvas Image	MNIST Image

It is very clear now that we need to invert the image because the model was trained on the dataset with black background and white strokes.

# Invert the image as model expects black background & white strokes
image = PIL.ImageOps.invert(image)

import PIL.ImageOps

# Decode the image and predict the value
def decode_predict(imgStr):

    # Decode the image
    image = decodeImage(imgStr)

    # Invert the image as model expects black background & white strokes
    image = PIL.ImageOps.invert(image)

    # Declare html codes for 0-9 emoji
    emojis = [ 
            "&#48;&#65039;&#8419;",
             "&#49;&#65039;&#8419;",
             "&#50;&#65039;&#8419;",
             "&#51;&#65039;&#8419;",
             "&#52;&#65039;&#8419;",
             "&#53;&#65039;&#8419;",
             "&#54;&#65039;&#8419;",
             "&#55;&#65039;&#8419;",
             "&#56;&#65039;&#8419;",
             "&#57;&#65039;&#8419;"
    ]

    # Call the predict function
    digit = predict(image)

    # get corresponding emoji
    return emojis[digit]

Awesome! It works.

Conclusion

Now you can quickly share this notebook with everyone and ask for their opinion. Don't forget to mention the caveat that this is an early prototype. Without much effort, we sort of replicated the functionalities of a web app. The biggest plus is we never left the comfort of notebook and potentially bridged the gap between Training & Test data.

Hopefully I managed to convince you that basic web developer skills are really helpful for a data scientist. The notebook can be accessed from here. Feel free to reach out via comments or Twitter.

Do You Really Understand Try & Finally in Python?

Aravind Ramalingam — Mon, 19 Apr 2021 12:30:12 +0000

In python, try and except blocks are often used by programmers for handling any exception or unhappy scenarios. finally clause is very under appreciated & can be better utilized. Let us check out how final-block works.

Deep dive

No matter what happened previously, the final-block is executed once the code block is complete and any raised exceptions handled. Even if there's an error in an exception handler or the else-block and a new exception is raised, the code in the final-block is still run.

This quote from the python documentation is absolutely correct but the execution behavior is little tricky when try and finally blocks are encapsulated within a function which has a return statement. Let me explain with examples. See if you could guess the output of the following functions.

Example 1:

# Both the try & final blocks have print statements and function returns value from final-block
def example_1():
    try:
        val = 1
        print(f"Print: Try Block - {val}")
    finally:
        val = val + 1
        print(f"Print: Finally Block - {val}")
        return f"Return: Finally Block - {val}"

example_1()

Function example_1 is simple and straight, first the try-block gets executed and then final-block. The variable val has value 1 in try-block and gets updated to 2 in final-block.

Example 2

# The try block has return statement & final block has only print statement
def example_2():
    try:
        val = 1
        print(f"Print: Try Block - {val}")
        return f"Return: Try Block - {val}"
    finally:
        val = val + 1
        print(f"Print: Finally Block - {val}")

example_2()

Function example_2 is where things get a bit tricky, the return statement in try-block is executed after the final-block but the value of the variable val returned is not affected by the changes made in the final-block.

Example 3

# Both the try & final blocks have return statements
def example_3():
    try:
        val = 1
        print(f"Print: Try Block - {val}")
        return f"Return: Try Block - {val}"
    finally:
        val = val + 1
        print(f"Print: Finally Block - {val}")
        return f"Return: Finally Block - {val}"

example_3()

Output of the function example_3 is easy to guess. When the return statement in final-block is executed, the function exits so the return statement in try-block is never executed.

Take Away

try & finally blocks are not quite as straight forward as one might think, especially when they are returning values from a function. For more such interesting tidbits follow me via Twitter. The takeaways from this post are:

Where you put the return statement is going to make a difference.
Even though the return statement from try-block is executed after the final-block, value of the variable returned won't be affected by alteration made in the final-block.

Write Better Python Functions (Using Type Dispatch)

Aravind Ramalingam — Tue, 13 Apr 2021 14:21:14 +0000

Writing a function in python is easy but writing a good function is not that easy. Most python programmers are often not aware of how to write clear and concise functions, let alone using Type Dispatch. Let us see how we can write better functions in python using Type Dispatch.

Getting Started

Before we get into what is Type Dispatch? or How Type dispatch can make you a better python programmer, let us get some basics out of the way like What is a good python function? To be honest, I would rather let Jeff Knupp explain it.

I truly believe that only way to really understand how something works is by breaking it into granular pieces and putting it back together or building it from scratch. Let us take a very well defined use case and write a function for it from scratch. By scratch, I mean like starting with one to one translation of the pseudo code to python code without thinking about any best practices. Then by analyzing what works & what didn't, we can build on top of this. We are going to repeat this process until we run out of ideas to improve the code. So this post is going to be not one, but many versions of a function for the same use case.

If you are wondering Why? Because we are going to unlearning everything we know about writing a function and question each line of code as we write it. By this way, we can clearly see how each piece fits. Rather than blindly following a check list of items about writing a function, we will create our own check list. In case this post missed something that is of value, then please let me know I will gladly update it.

We will try to write the best or at least my best python function for a particular use case in 4 iterations. Each iteration is going to be an improvement over the last one, with clear objective declared upfront. Getting better at anything is an iterative process, depending on where you are in the spectrum of python proficiency, one of the 4 iterations is going to resonate more with you than others. Next step of this post is understanding the use case. Buckle up, it's going to be a very interesting journey.

Use Case

Briefing

Convert the given image input into pytorch tensor. Yes, that's it. Quite simple right?

Folks with data science background can skip this part and jump to next section, as I am going to give a brief introduction about python libraries needed for this use case.

Pillow is a that adds support for opening, manipulating, and saving many different image file formats.
Numpy stands for Numerical Python. Numpy is the core library for scientific computing in Python. It provides a high-performance object, and tools for working with these arrays.
PyTorch is an open source, it provides two major functions namely computing (like NumPy) with strong acceleration via & Deep neural networks built on a type-based automatic differentiation system. Pytorch can run the code using CPU if there is no CUDA supported GPU available but this will be way slower for training a ML model.

If you wondering why or to whom this use case is useful? Most Machine Learning applications (Face detection, OCR) with image data as input are using these libraries. This use case is considered data preprocessing in ML world.

How do we go about this?

The steps involved are very straight forward. Open the image using Pillow library, and convert it to numpy array & transpose it before converting to Tensor. BTW, Gentle reminder - This post is about writing better python functions and not a data pre-processing tutorial for deep learning so if it's intimidating, don't worry about it. You will feel more comfortable, once we start coding.

Couple of points to take note about the process steps before we move on.

PIL image when converted to Numpy array, it takes shape - (Height, Width, # of Channels). In Pytorch the image tensor must be of the shape - (# of Channels, Height, Width) so we are going to transpose the array before converting it to tensor.
For experienced Pytorch users who are wondering, why the hell are we not using torch functions like ToTensor or transform methods and skip Numpy? Yep, you are right, we can absolutely do that but to drive the point of this post home better, I have consciously decided to take the longer route.

Hiccup

Sorry, the use case is not that simple. There is a small hiccup, The function should support multiple input data types, namely str/Path, Pillow(Image) and Numpy(Ndarray).

The process step defined in the above image only supports the data types str/Path as input. Therefore we need to add support for Pillow(Image) and Numpy(Ndarray) types, but if we think about it, these data types are intermediate steps in converting the image file to torch tensor. So there is really no additional step from the above image, we just have to duplicate the steps defined for str/Path data types and alter few initial steps to support Pillow(Image) and Numpy(Ndarray) data types as input for our function.

Process steps for 2 additional Workflows are as follows.

After comparing the 3 images it is very clear that:

Image data type's process steps are subset of the original process steps.
Array data type's process steps are subset of the Image data type process steps.

Solution

First 2 iterations of the function is going to be about the behavior implementation whereas last two iterations are all about code refactoring. If you are one of those readers who likes to read the ending of a book first (I am not judging), you can jump straight to section with title - Iteration 4.

Iteration 1

Objective:

Make it work. As simple as that, as we are starting from scratch it is best just to focus on getting the basic feature or functionality work. We don't need all the bells and whistles. For this iteration we will focus on only implementing Use Process Steps and not worry about Hiccup.

Code:

# Import required libraries
import numpy as np
import torch
from PIL import Image as PILImage

# Set Torch device to GPU if CUDA supported GPU is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Function to convert image to Torch tensor
def Tensor(inp):

    # Read image from disk using PIL Image
    img = PILImage.open(inp).convert('RGB')

    # Convert the image to numpy ndarray
    imgArr = np.asarray(img, np.uint8)

    # Rearrange the shape of the array so that it is pytorch compatible
    imgArr = imgArr.transpose(2, 0, 1)

    # Convert Numpy array to Torch Tensor and move it to device
    return torch.from_numpy(imgArr).to(device)

What have we done?

We managed to translate the process steps into code. The number of lines in code almost matches the number of boxes in the diagram.

Quick Check:

We need a sample image to test our function so we will download one.

import urllib.request 

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Python-logo-notext.svg/200px-Python-logo-notext.svg.png"
filename = "sample.png"
urllib.request.urlretrieve(url, filename)

Now, let us take our function for a test drive.

# Pass the sample image through our function
imgTensor = Tensor(filename)

# Check whether the output of the function is a Tensor
assert isinstance(imgTensor, torch.Tensor), "Not a Tensor"

Looks like we managed to nail the basic functionality. Awesome!

What can we improve?

We are yet to implement multiple data type input functionality i.e support for data types like PIL Image, Ndarray.
Also we should definitely improve name of the function, it doesn't really say anything about the function.

Iteration 2

Objective:

When we add support for more than one data type, we need to be very careful because without proper validation, things can go south real quick and from end user point of view the error messages won't make any sense. So in this iteration we will:

Implement multiple data type support and validations for these types.
Make the function name more useful.

Code:

# Import Libraries
import numpy as np
import torch
from PIL import Image as PILImage
from pathlib import Path, PurePath

# Set Torch device to GPU if CUDA supported GPU is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Function to convert image to Torch tensor
def from_multiInput_toTensor(inp):

    # Input type - str/ Path, then read from disk & convert to array
    if isinstance(inp, (str, PurePath)):
        try: 
            image = PILImage.open(inp).convert('RGB')
            imageArray = np.asarray(image.copy(), np.uint8)
        except Exception as error:
            raise error

    # Input type - PIL Image, then we convert it to array      
    elif isinstance(inp, PILImage.Image):
        imageArray = np.asarray(inp, np.uint8)

    # Input type - ndarray, then assign it to imageArray variable  
    elif isinstance(inp, np.ndarray):
        imageArray = inp

    # Raise TypeError with input type is not supported
    else: 
        raise TypeError("Input must be of Type - String or Path or PIL Image or Numpy array")

    # Rearrange shape of the array so that it is pytorch compatible
    if imageArray.ndim == 3: imageArray = imageArray.transpose(2, 0, 1)

    # Convert Numpy array to Torch Tensor and move it to device
    return torch.from_numpy(imageArray).to(device)

What have we done?

We have managed to implement all the functionalities required for this use case. Since ndarray is the last data type before the output data type - tensor, we convert the input first to ndarray for every supported data type and transpose it at the end, right before converting it to tensor. By adopting this style of coding, we are able to avoid having to convert the input to tensor & return the value for every supported data type. Instead we do it only once at the end.

With the help of isinstance function, we are able to identify the data type of the input and notify the users by raising TypeError with appropriate error message if any unsupported data type is passed. Often with io operations, something can go wrong easily so if the data type is str or Path, we read the image file inside try & except block and let the user know about the error (if any).

Quick Check:

Before we proceed, we will write a helper function to check whether two tensors are same or not.

# Test if two torch tensors are same or not
def is_same_tensor(tensor1, tensor2):
    assert torch.eq(tensor1, tensor2).all(), "The Tensors are not equal!"
    return True

When writing a test function, better to throw a proper error and not simple print function which can get buried in between other messages. We will also check a couple of things and ensure that the updated version is all good.

1. Is the output of the functions from last two iterations the same?

# Verify that the output of two versions are same or not
is_same_tensor(Tensor(filename), from_multiInput_toTensor(filename))

2. Is the support for multiple data types working?

# Check the support for Path
path = Path(Path.cwd(), filename).resolve()
is_same_tensor(Tensor(filename), from_multiInput_toTensor(path))

# Check the support for PIL Image
image = PILImage.open(filename).convert('RGB')
is_same_tensor(Tensor(filename), from_multiInput_toTensor(image))

# Check the support for Ndarray
imageArray = np.asarray(image, np.uint8)
is_same_tensor(Tensor(filename), from_multiInput_toTensor(imageArray))

3. Are the validations working?

# Validate whether an error is thrown when user passes wrong file
from_multiInput_toTensor('test.png')

# Validate whether an error is thrown when input type is list
from_multiInput_toTensor([filename])

Great! We didn't break anything. Now let us move on with refractoring as we are done with behavior implementation.

What can we improve?

With the goal of adding support for multiple data types, we sort of wrote a Spaghetti code. The life cycle of the variable imageArray is confusing at the least, this type of code design is not very intuitive and can make life hell for the person who is going to maintain this code base. Of course, for our simple use case this is not case but in the spirit of writing better function, let us avoid this type of coding style.
The function name is a lot better but still not ideal. Why? Because it still does not clearly state what the input type for the function is. Letting the user play trail & error with TypeError is definitely not the right way of coding.
As the name of the function suggest it takes many inputs which means we have lot of moving parts in one function. Long run this may lead to many unintended side effects.

Iteration 3

Objective:

After taking a closer look at the points from last section, it is very clear that we need to break the function into 3 smaller ones. i.e one function for each data type.

Code:

# Import Libraries
import numpy as np
import torch
from PIL import Image as PILImage
from pathlib import Path, PurePath

# Change numpy array to torch tensor
def numpy_ToImageTensor(imageArray):

    # if not type - ndarray then raise error
    if not isinstance(imageArray, np.ndarray):
        raise TypeError("Input must be of Type - Numpy array")

    # Set Torch device to GPU if CUDA supported GPU is available
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    # Transpose the numpy array before converting it to Tensor
    if imageArray.ndim == 3: imageArray = imageArray.transpose(2, 0, 1)
    return torch.from_numpy(imageArray).to(device)

# Change image to torch tensor
def pil_ToImageTensor(image):

    # if not type - PIL Image then raise error
    if not isinstance(image, PILImage.Image):
        raise TypeError("Input must be of Type - PIL image")

    # Convert the image to numpy 
    imageArray = np.array(image)

    # Return output of numpy_ToImageTensor function
    return numpy_ToImageTensor(imageArray)

# Change image file to torch tensor
def file_ToImageTensor(file):

    # if not input - string or Path then raise error
    if not isinstance(file, (str, PurePath)):
        raise TypeError("Input must be of Type - String or Path")

    # Read the image from disk and raise error (if any)
    try: 
        image = PILImage.open(file).convert('RGB')
    except Exception as error:
        raise error

    # Return output of pil_ToImageTensor function
    return pil_ToImageTensor(image)

What have we done?

This is good progress, we have removed the spaghetti code and added modularity. Now it is like we have a linear chain of three functions where two of them are calling another one, which makes it easier to understand and maintain.

file_ToImageTensor -> pil_ToImageTensor -> numpy_ToImageTensor

Modularity makes adding new feature quite straight forward and easy to test as we have to alter only one specific function. Some examples for functions to modify for hypothetical feature request:

file_ToImageTensor - Ability to read black & white image as well.
pil_ToImageTensor - Resize the image before converting it to Tensor.

Please do note that even though you didn't touch the rest of code base for the above stated examples, it is best to test other functions as well to avoid any surprise.

Modular design also paved the way for better function names and parameter names as the input type have become more specific.

	From	To
Function Name:	from_multiInput_toTensor	file_ToImageTensor pil_ToImageTensor numpy_ToImageTensor
Parameter Name:	inp	file image imageArray

These subtle changes can a go long way in making the code more user friendly. Honestly at this point, I feel like I am preaching, so I will try to wrap this quickly.

Quick Check:

This post is getting too long for my own liking, so I am not going to test negative or different data type scenarios again and only check for the original use case. You really shouldn't skip testing after refractoring.

is_same_tensor(from_multiInput_toTensor(filename), file_ToImageTensor(filename))

What can we improve?

Looks like we are following the same pattern of checking data type with isinstance function and raising TypeError for every function, which is a kind of code repetition as well.
Okay, we have to do something about the function names. I am pretty sure no one will remember these names, definitely not my future self.

Iteration 4

Objective:

Improve the documentation and avoid code repetition with the help of Type Dispatch.

Type Dispatch:

Type dispatch allows you to change the way a function behaves based upon the input types it receives. This is a prominent feature in some programming languages like Julia & Swift. All we have to do is add a decorator typedispatch before our function. Probably, it is easier to demonstrate than to explain.

Example for Type Dispatch:

Function definitions:

from fastcore.all import *
from typing import List

# Function to multiply two ndarrays
@typedispatch
def multiple(x:np.ndarray, y:np.ndarray ): 
    return x * y

# Function to multiply a List by an integer
@typedispatch
def multiple(lst:List, x:int): 
    return [ x*val for val in lst]

Calling 1st function:

x = np.arange(1,3)
print(f'x is {x}')

y = np.array(10)
print(f'y is {y}')

print(f'Result of multiplying two numpy arrays: { multiple(x, y)}')

Calling 2nd function:

x = [1, 2]
print(f'x is {x}')

y = 10
print(f'y is {y}')

print(f'Result of multiplying a List of integers by an integer: {multiple(x, y)}')

Life of a programmer can be made so much better if they don't have to come up with different function names with no change in purpose (for various data types). If this doesn't encourage you to use Type Dispatch whenever possible then I don't know what will 🤷

We will be using fastcore package for implementing Type Dispatch to our use case. For more details on fastcore and Type Dispatch, check this awesome blog by Hamel Husain. Also check out fastai, which inspired me to write this post.

Code

# Import Libraries
import numpy as np
import torch
from PIL import Image as PILImage
from pathlib import Path, PurePath
from fastcore.all import *

@typedispatch
def to_imageTensor(arr: np.ndarray) -> torch.Tensor:
    """Change ndarray to torch tensor.

    The ndarray would be of the shape (Height, Width, # of Channels)
    but pytorch tensor expects the shape as 
    (# of Channels, Height, Width) before putting 
    the Tensor on GPU if it's available.

    Args:
        arr[ndarray]: Ndarray which needs to be 
        converted to torch tensor

    Returns:
        Torch tensor on GPU (if it's available)   
    """

    # Set Torch device to GPU if CUDA supported GPU is available
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    # Transpose the array before converting to tensor
    imgArr = arr.transpose(2, 0, 1) if arr.ndim == 3 else arr
    return torch.Tensor(imgArr).to(device)

@typedispatch
def to_imageTensor(image: PILImage.Image) -> torch.Tensor:
    """Change image to torch tensor.

    The PIL image cast as numpy array with dtype as uint8,
    and then passed to to_imageTensor(arr: np.ndarray) function
    for converting numpy array to torch tensor.

    Args:
        image[PILImage.Image]: PIL Image which 
        needs to be converted to torch tensor

    Returns:
        Torch tensor on GPU (if it's available)   

    """
    return to_imageTensor(np.asarray(image, np.uint8))

@typedispatch
def to_imageTensor(file: (str, PurePath)) -> torch.Tensor:
    """Change image file to torch tensor.

    Read the image from disk as 3 channels (RGB) using PIL, 
    and passed on to to_imageTensor(image: PILImage.Image)
    function for converting Image to torch tensor.

    Args:
        file[str, PurePath]: Image file name which needs to
        be converted to torch tensor

    Returns:
        Torch tensor on GPU (if it's available)

    Raises:
        Any error thrown while reading the image file,
        Mostly FileNotFoundError will be raised.

    """
    try: 
        img = PILImage.open(file).convert('RGB')
    except Exception as error:
        raise error
    return to_imageTensor(img)

@typedispatch
def to_imageTensor(x:object) -> None:
    """For unsupported data types, raise TypeError. """
    raise TypeError('Input must be of Type - String or Path or PIL Image or Numpy array')

What have we done?

By utilizing the Type dispatch functionality, we managed to use the same name for all 3 functions, and each one's behavior is differentiated by their input type. The function name is also shortened with the removal of input type. This makes the function name easier to remember.

By calling the function name, we can see what are the different input types supported by the function. Fastcore by default expects two input parameters, since we have only one it assigns second one as an object. The second parameter will not have any impact on the function behavior.

to_imageTensor
# OUTPUT
(ndarray,object) -> to_imageTensor
(Image,object) -> to_imageTensor
(str,object) -> to_imageTensor
(PurePath,object) -> to_imageTensor
(object,object) -> to_imageTensor

With help of inspect module, we can access the input & output types of a particular function.

import inspect
inspect.signature(to_imageTensor[np.ndarray])

The docstrings implemented in this iteration makes the code more readable. The docstring of a particular input type can be accessed by calling doc along with the input type.

print(to_imageTensor[np.ndarray].__doc__)

As discussed in the last section, we managed to move the TypeError message to a separate function with input type as object.

Quick Check:

Still working!

is_same_tensor(file_ToImageTensor(filename), to_imageTensor(filename))

Validating error message when unsupported data type is passed to the function.

to_imageTensor([filename])

What can we improve?

This is it. I believe we are done here.

Closing Thoughts

Did we really write the best function for this use case? I think so, but don't take my word. You can find the notebook version of this post here. If you like to add anything to this post you, feel free to reach out to me via Twitter or the comments section. I will gladly update this post based on your comments.

Even though Type Dispatch was the focal point when I started writing this post, soon I realized that the code evolution process is equally important too. So I have decided to include that as well. I hope you enjoyed this journey of writing a python function.

Credits

Giphy

Icons:

Video Card icon by IconBros
zwicon

Use case for Python Tuple

Aravind Ramalingam — Fri, 26 Mar 2021 09:42:08 +0000

Most Python programmers are guilty (me included) of always using List without even considering Tuple at all. Surely for some use cases, Tuple should be better right? Keep reading for one such case.

Tuple is like List but immutable so it has lesser in built methods. Tuples are faster than Lists, but it is noticeable only when size of the tuple is considerably large. For a refresher on Tuples, check out this blog

Caveat

A general rule of thumb is to use Tuple only when the data will or should not change.



# Create Tuple with two elements
tup = (1, [2])

# Re-assign first element
tup[0] = 3

This does not mean you really can’t change a tuple. If the underlying tuple element is mutable then we can change it. This could lead to unintended side effects. For example, children/inherited class can modify a tuple element initialised by the parent class.



# Add element to existing List element inside Tuple
tup[1].append(3)

print(tup)

Use Case

Personally, I am more into Deep Learning and have to deal with image files a lot. Image files have many popular formats like jpg, png and svg etc. We often have a situation where we have to find all files of certain types from a folder. There are many ways of going about this. We will explore few options.

Option 1: Using List

The filtering based on the extension type is done with the help of endswith method. Since the endswith method does not accept list we are looping through all the extension elements. The code looks readable and takes about 2 microseconds which is not bad, but for every extension type, we have to run through all the files one more time. I have a suspicion that we can do much better.



# List of files in the folder

all_files = ['cat.jpg', 'dog.png', 'report.docx', 'sales.csv']

# List of image file extensions

img_exts_lst = ['jpg','png']

# Filter only image files using List

img_files = [file for file in all_files for ext in img_exts_lst if file.endswith(ext)]

print(img_files) # ['cat.jpg', 'dog.png']

Option 2: Using Tuple

Almost similar code but we are able to pass the entire Tuple as parameter to the endswith method. This removes the extra loop and makes the code even shorter. The code now runs about 2 times faster. To run your own experiments use this notebook as base.



# List of files in the folder

all_files = ['cat.jpg', 'dog.png', 'report.docx', 'sales.csv']

# Tuple of image file extensions

img_exts_tup = ('jpg','png')

# Filter only image files using Tuple

img_files = [file for file in all_files if file.endswith(img_exts_tup)]

print(img_files) # ['cat.jpg', 'dog.png']

Conclusion

Some methods like startswith and endswith accept Tuple as parameter, if the data does not change then better to use Tuple or convert List to Tuple before passing data to the method. Tuples make the code look more elegant and run faster. Also, don’t forget about the caveat.

Hopefully this post helps you to be more open to tuples. If you have any questions or thoughts, feel free to reach out via comments down here or Twitter.