DEV Community: Erol

3D Brain Segmentation with UNET (BraTS19)

Erol — Thu, 11 Feb 2021 11:01:22 +0000

Project Link: https://github.com/ierolsen/Brain-Tumor-Segmentation-BraTS-2019/blob/main/subset_BraTS19_colab.ipynb

INTRODUCTION

In this project, I aim to work with 3D images and UNET models. Before I couldn’t have any chance to work with them thus I don’t have any idea what they are. But this project will be so educational for me. As well I aim to make practice in algorithms.

Download Dataset:

Dataset Link: https://ipp.cbica.upenn.edu/categories/brats2020

In order to download the dataset, first, you must sign up. After sign up, your account will be reviewed within 3 days. Then you can send a request to download data, after this step you will get an email (if I am not wrong) that contains REGISTRATION_STATUS.txt. In this txt file, you can find train and validation dataset links.

READ and OPEN IMAGES

First I started with uploading data using by glob, glob is one of the awesome libraries of Python. Using that I can get paths. But defining glob, I used recursive=True which means reach subfiles.

After reading the data, now I need to convert to array. In order to do that, I will use skimage.io but first, it must be installed, thus:

pip install skimage

Now, I can read the data but first I will go on step-by-step. I meant I will show pic step-by-step, after all I will define a function to read data. I import skimage.io but I will use it as io whatever, using by io.imread() I can easily read the pic. But this function contains a really important parameter called plugin that means which kind of pics read, I’d rather choose simpleitk due to work on medical images. This simpleitk is used for medical images. But first I need to install that using this short magical word :

pip install SimpleITK

After installation, I am reading the pic :

As you can see the image is 3D, that’s great because I am working on 3D images for the first time.

Now I will visualize this 3D image axes-by-axes using by the traditional way (using matplotlib)

The image that was reviewed is not segmented. This time I will read and visualize the segmented image. For that, I will use skimage.io again.

As I read my data, now I will visualize this segmented img as 3 axes:

To Array

I need to convert all images to the array. To do that, I will define a function and then I will explain what it is.

This function makes you think. But no worries, I will explain now. Probably the second loop makes you think, so I will start there.

range(60, 130):
In data, there are no significant images before 60 and after 130. All significant images are between 60 and 130, so that’s the reason why I only make them contain 60 and 130. Also, you can find some images that are before 60 and after 130:

As you can see above, the tumor is not certain. Thus there is no meaning to contain all data. Also if all images are contained, it makes my pc force. Unfortunately, my pc doesn’t have enough space. Anyway...

Also, I want to explain what np.expand_dims() works.
I am working on 3D images, therefore I need to work with U-Net. If you work with U-Net I need 3 input dimensions that are width, height, and channel. But the images are 3D and also there is no channel.

But using by np.expand_dims() I can add one more size to axis=0

As you can see, now the image has a channel.

If I give a short summary,

First I define a function
I define an empty list called img_list that will be used for appending images in the end.
Using by glob I reach the locations
Then using by random library, I shuffled them
I define a loop due to open images.
After opening images made standardization and convert them float32
Then I created another for loop due to work on limited images.
In Axial (transverse) plane totally of 155 images are available. But I only wanted to work them in between 60 and 130. Now there are 70 images.
I applied np.expand_dims() deu to add one more size that is called a channel.
And finally I appended them to the image list that was defined before, and then it is returned as a np.array()!

That’s all...

To Array for Segmented Images

1 - Non-enhancing Tumor

2 - Edema

4 - Enhancing Tumor

To understand clearly, I will visualize step-by-step tumor areas. But first I will show you up the segmented image again.

In the image above, you can see the layer by layer of Non-enhancing Tumor, Edema, and Enhancing Tumor.

First I will start with all area tumors, for that, I will equal all values 1 that do not equal 0. Thus only all areas of the tumor will appear clearly.

After that, I will go on with a Non-enhancing Tumor that is marked by 1. To make it show up, I will equal all area 0, except for those that equal 1.

And now I can visualize Edema, in order to do that, I need to cover (equal zero) Non-enhancing Tumor (1) and Enhancing Tumor (4). For that, if values equal 1 and 4, I will equal them 0. But if other values do not equal zero, I will equal them 1. These are what I’m expecting.

Then lastly, I can visualize Enhancing Tumor for that, Only I will cover those that do not equal 4. I meant I will only show up values that do not equal 4.

Now it is time to define a function that turns images into array.

Also, we need the original segmented image and I labeled it as 0. This original segmented image will be used for comparative.

But also, I can not use these functions because of the empty size of my computer. When I try I got Memory Error. Maybe next day, I can figure out that. I hope so.

GREAT NEWS:

Finally, I figured out that! Instead of jupyter notebook, I am using Google Colab. That’s the best way actually, even probably I will use it in my all projects :)

Anyway, I want to talk about how to upload data to colab, etc.

First, you must connect your Gmail account and go to the drive section. After this step, you must upload your data into drive. That’s all! Then open a colab notebook and start to code! I don’t know if there is another way, actually I didn’t searched for that because I was annoyed by my pc.

But this time, I face another problem. When I run the code that converts to array train images, google colab file gives a run time error. But when I run the code that converts to array for segmentation dataset, there is no problem, it works and I don’t face any error.

Also, I fixed up these errors :) what I did is resize all images. Their sizes were 240,240, after resize they have 120,120 sizes.

U-NET MODEL

Before defining the model, I will define Flair, T2, and seg images. Also during training, I will use Flair + T2. The point is Flair and T2 will be added each self. The reason why I will add two images called Flair and T2 is that these two images complete their deficiency. For example, we can see clearly Edema in Flair but in T2, we can see the center of the Tumor clearly.

Explanations

K.set_image_data_format('channels_first')

The shape of images are like that : (2, 120, 120),

if images are like that: (120, 120, 2) then I would use this code

K.set_image_data_format('channels_last')

Dice Coefficient

When we use accuracy in U-NET models, we can not realize that if the model was trained or not. Thus we use Dice Coefficient. When we put images on top of each other, Dice Coefficient gives differences of pixels. In U-NET Models, input and output are images !! so therefore we should analyze the difference of pixels.

NEWS:

Unfortunately for 2 days, I have been trying to solve some UNET errors.

The first of them was about the TensorFlow version. In my environment, I’ve been using 1.13.1 tf version since 2017. Then my solution was, using Colab again. Because Colab uses 2.4.1 tf version.

Another and last error was that:

ValueError: A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 512, 14, 14), (None, 512, 15, 15)]

When I googled that, what I found was related by the size of images. I’ve learned that when you work with Neural Networks, the sizes of your images must be divided by 32. Then I changed the sizes as (128,128)

Then using GPU of Colab I’ve trained my model in the 15th epochs.

PREDICTION

In order to test my model, I will randomly choose an image and then I will make it predicted by the model.
BUT first I need to expand the size of my image. Because, during training, the model worked with 4 dimensions, thus I will use np.expand_dims() again.
First, its size was (2, 128, 128) after expand, its size is (1, 2, 128, 128)

Actually, all these parts were the easy part. Now it’s time to switch to the hard part!
Now I will find out a solution for segmenting separately Enhancing Tumor and center of Tumor.

In the above images, as you can see Non-enhancing Tumor and Enhancing Tumor images seem really less in the pictures. So what my aim is to crop useless areas of these images.

For that I will apply these steps:

I will get coordinates of Tumor Centers of Non-enhancing Tumor and Enhancing Tumor using by Numpy
Then I will be cropping images using these coordinates as the center point.
These cropping steps, all cropped images will be input for the UNET model for training.
After all, I will add the new output to the original image using the coordinate of the cropped image.

So how can I find the center point of Tumors? The answer is that using by np.where()
np.where gives coordinates as a list for 2 axes (x,y)

According to these results, when I calculate center point, what I need to do is that (min_value + max_value) / 2 gives me center point for 2 axes.

CROP

Now I can crop it but first I will create a matrix that has (64,64) sizes. After this matrix, which has only zeros, I will add T1ce image. But an important point is when I add them, I have to add 64/2 and also I have to subtract 64/2 these operations provide that the image will be middle of the matrix.

After T1ce image, I will do it same steps for the segmentation image.

In order to explain my steps, I only happened to all these steps just one image that is 280 images of the dataset.

It’s time to define a function that applies all these steps for all images!

FUNCTION FOR CROPPING

As you can see in the above image, I defined my function. Also, I don’t think that I need to explain what they are because I believe that I mentioned them in a nice way!

After applying these functions, you can realize that all images are got bigger and they are more clear.

In the next step, all images will be turned into np.array by for loop.

BUT unfortunately, when I use this loop for all images, I get an error again, I checked it meticulously but then I realized that there is no error in code. Because, in order to check that, I’ve created a subset that has about 217 images (36 files of HGG), and when I use them, the code works! That means there is no error in code, the problem is on images. But I don’t know why. Next few days, probably I can work on that why.

The error is:

After using subset (a part of all dataset of HGG images)

As you can see, there is no change in code but it works for images that are in subset.

After this loop, all segmentation images will be in the center of the frame and also they look bigger, more clear!

UNET MODEL FOR CROPPED IMAGES (SEGMENTATION)

Now, I will train a model again for cropped images

The model doesn’t obtain well results because of that I worked with a subset (less dataset) but next days, if I solve the error, I will upload that again.

CONCLUSION

In this great project, I’ve learned tons of useful knowledge about algorithms, computer vision, and above all, 3D images and UNET Models. The other important thing that I learned is absolutely fixing error :) they took a few days but I solved all of them! These projects are the best way to improve myself, facing new errors and solving them.

But actually, I can not say this project ends here because I have some idea about that I have to update this project but I can not because of my exams. Unfortunately, my exams are approaching so I need to study.

Some ideas that I have to add to this project:

Improving crop function. I determined the size as (64, 64) but if the tumor is bigger than this size what will happen? The tumor can not fit in the frame. So I have to define a new function that contains some simple conditions according to tumor size.

Anyways maybe I can improve this function too.

GitHub Link: https://github.com/ierolsen/Brain-Tumor-Segmentation-BraTS-2019/blob/main/subset_BraTS19_colab.ipynb

Diabetic Retinopathy Detection

Erol — Thu, 14 Jan 2021 17:52:34 +0000

Diabetic Retinopathy Detection

In this article, I worked on Diabetic Retinopathy Detection that was a competition on Kaggle.
Link: https://www.kaggle.com/c/aptos2019-blindness-detection

This competition contains 4 different labels, these are

0 - No DR

1 - Mild

2 - Moderate

3 - Severe

4 - Proliferative DR

When we check the images, we can realize that we should apply some operations before training. These operations will be used for destroying the black areas, making clear blood vessels.

In order to do these steps, first I will upload images and check them.

Above, I read the first 20 images, because today there are 3662 in the training dataset, I just want to take a look at the first step.

As is seen, the second image has a huge black area, we don't want it, cause it avoids learning, and also it makes images smaller. Therefore I will crop these black areas.

BUT for now, I will go on just the first image, after complete these steps in the first image, I will define a function that makes apply these steps to all datasets.

To crop the image, first I need to convert it to GRAYSCALE because I will apply THRESHOLD and in order to apply that, the image should be GRAY.

Before THRESHOLD, I need to apply Median Blurring because my aim is destroying the black area and applying the Blurring operation, it makes the contour of image clarity.

Well, now I am ready to apply THRESHOLD, it is so simple:

Now I have an image that has a clear contour.

To find Contour I will use a method call cv2.findContours() When we use this method, it returns an array

After getting the list of coordinates, I will find max and min values for X and Y axes.

Now I have coordinates for cropping operations. So I can crop it. To crop an image there is no method actually, I just use slicing.

The image has been cropped! BUT as you can see, its sizes are changed due to crop. So now, I need to resize it again.

That's all.

Now We are in the important part called CLAHE

First I want to start with a question, what is CLAHE?

Contrast-Limited Adaptive Histogram Equalization (CLAHE) is a variant of adaptive histogram equalization in which the contrast amplification is limited.

In order to apply, CLAHE image should be LAB format, for that, I will convert the image according to this method cv2.COLOR_RGB2LAB

I applied this method easily but as you can see it has 3 channel, they are:

L => Lightning
A => Green - Red
B => Blue - Yellow

In these channels, I need to reach L layers, for that, I will use this method cv2.split() and as a result of this method, it returns 3 parameters (L, A, B)

Above the image, it has 2 sizes now.

Now I can apply CLAHE. But before that, first I want to show their Histograms

as you can see, it doesn't have a Symmetrical Distribution.

BUT now I will apply CLAHE and then you can check the new Histogram.

now the image has a more Symmetrical Distribution.

If the Histogram Graphs doesn't have a meaning, you can check the before/after images:

Now that I have applied CLAHE I can move on to other steps,

The other step is MERGING, before I split L, A, B channels, now it's time to MERGE them. In order to merge them, I will use cv2.merge()

Now I have a really good looking image but actually, I shouldn't say good looking because it has a noise (Salt-Peper). The best way to destroy them is by using BLURRING OPERATION*

It is a little better now.

The other important, even the most important step is making blood vessels more clearly.

To do that, First I will apply BLURRING with a huge kernel size because it makes the image more blur, just the blood vessels will appear, after that, I will have a background that has appeared blood vessels.

And Secondly I will apply a masking using cv2.addWeighted()

Here, I want to explain some parameters.

cv2.addWeighted(src1=med_blur, alpha=1, src2=back_ground, beta=-1, gamma=255)

I want to use BLUR IMAGE 100%, for that aplha=1
also, beta=-1 means, reverse all values of pixels. Gamma=255 is for reverse image color. as you can see it turned white. And thus, only blood vessels appear.

Now I will apply to mask to the image. For that, I will use cv2.bitwise_and() We can say that it merges two images.

If we compare before/after versions:

After these steps, I can see that works! now I will define a function in a python file because I used jupyter notebook for that I can see the result immediately. Also in next, I will implement Transfer Learning Models. For that Pycharm is my favorite IDE.

The function is here: https://github.com/ierolsen/Detect-Diabetic-Retinopathy/blob/main/src/image_preprocessing.py

After applied function:

from tqdm import tqdm

path = "data\\datasets\\"
files = os.listdir(path + "train_images")


def preprocessing(files):
    """
    This function returns images prepared for training
    """
    img_list = []
    for i in tqdm(files):
        image = cv2.imread(path + 'train_images\\' + i)
        image = cv2.resize(image, (400, 400))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        copy = image.copy()
        copy = cv2.cvtColor(copy, cv2.COLOR_RGB2GRAY)
        blur = cv2.GaussianBlur(copy, (5, 5), 0)

        thresh = cv2.threshold(blur, 10, 255, cv2.THRESH_BINARY)[1]

        # CONTOUR DETECTION
        contour = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        contour = contour[0][0]
        contour = contour[:, 0, :]

        # GET COORDINATES
        x1 = tuple(contour[contour[:, 0].argmin()])[0]
        y1 = tuple(contour[contour[:, 1].argmin()])[1]
        x2 = tuple(contour[contour[:, 0].argmax()])[0]
        y2 = tuple(contour[contour[:, 1].argmax()])[1]

        #Crop Images Again to Destroy Black Area
        x = int(x2 - x1) * 4 // 50
        y = int(y2 - y1) * 5 // 50

        # THRES FOR CROPPED IMAGES
        copy2 = image.copy()
        if x2 - x1 > 100 and y2 - y1 > 100:
            copy2 = copy2[y1 + y: y2 - y, x1 + x: x2 - x]
            copy2 = cv2.resize(copy2, (400, 400))

        # LAB
        lab = cv2.cvtColor(copy2, cv2.COLOR_RGB2LAB)
        l, a, b = cv2.split(lab)

        # CLAHE - Contrast-Limited Adaptive Histogram Equalization
        clahe = cv2.createCLAHE(clipLimit=5.0, tileGridSize=((8, 8)))
        cl = clahe.apply(l)

        # MERGING LAB
        merge = cv2.merge((cl, a, b))
        final_img = cv2.cvtColor(merge, cv2.COLOR_LAB2RGB)
        med_blur = cv2.medianBlur(final_img, 3)
        back_gorund = cv2.medianBlur(final_img, 37)

        # MASK FOR BLEEDING VEIN
        mask = cv2.addWeighted(med_blur, 1, back_gorund, -1, 255)
        final = cv2.bitwise_and(mask, med_blur)
        img_list.append(final)

    return img_list


img_list = preprocessing(files=files)

Now, I can transform my labels to One Hote Encoding, For that:

But here there is a really important situation. Images are not Multi-Class they are Multi-Label In the same image, there are more probabilities. I mean if the patient's scala is 3 (it means "Severe") also the patient has 0,1 and 2 scalas. Because these symptoms correlated with each self.
*Summary If the patient has a scala 2 problem, we can see scala 0, scala 1, and scala 2, as told before they are correlated with each self.
In this situation labels shoul be like that:

If scala 4 => [1, 1, 1, 1, 1]
If scala 3 => [1, 1, 1, 1, 0]
If scala 2 => [1, 1, 1, 0, 0]
If scala 1 => [1, 1, 0, 0, 0]

In order to convert that I will use a simple for loop and I will use np.logical_or() method. This method computes the truth value of x1 OR x2 element-wise.

np.logical_or(1,1)
>> True

np.logical_or(0,1)
>> True

np.logical_or(0,0)
>> False

To convert to Multi-Label:

After this steps:

And now I am ready to define the transfer learning model but first I need to transform my images into arrays. It is so simple with this method np.array()

So, one more step to define the model, it's time to split data:

Now I will define a Generator to generate more images for training. Because the model has to face images in different ways. It makes the model better!

Maybe you realize that I didn't use rescale. Because the transfer learning model called EfficientNetB5 I will use, makes images rescale itself. So I don't need to do that.

Now it's time to define the transfer learning model. But first, the model must be downloaded:

pip install git+https://github.com/qubvel/segmentation_models

After installation I will create Neural Network:

In the transfer learning model called EfficientNetB5 I chose include_top=False because I don't want to use the last part of layers, also I've changed the input size as (240,240,3) first it was (400,400,3) but on my pc, it doesn't work because of the RAM :)

Then I compile my model like that and also I define a callback for Learning Rate

Finally, I am in the training step:

After 5 epochs of training, as you can see we got really reasonable accuracy.

Actually I would make the model better but my pc doesn't have enough space and RAM.
.

GitHub: https://github.com/ierolsen/Detect-Diabetic-Retinopathy

Trex Game with CNN

Erol — Thu, 10 Dec 2020 18:52:33 +0000

Trex Game with CNN

In this article, I created a model that plays Trex Game. I need to explain that, this model is not a Reinforcement Learning (RL) model. It is a simple CNN model that predicts objects and keyboard actions.

Content:

1) Getting Data

2) Train a CNN Model

3) Test the Trained Model in Game

Getting Data

First I started with getting data. In order to do that, I used some libraries;

import keyboard
import uuid
import time
from PIL import Image
from mss import mss

Using the library called keyboard I saved my action for example "up", "down" and "right".
To import that, first, the library must be installed:

pip install keyboard

Using this library called mss, I can record my screen for the game. When I run all lines, I'll switch the game screen and the model that is trained with the data I got, will predict action according to images. Also, this library (mss) helps me to cut off some areas in the screen. Thus, the model can only focuses on determined area. But at the first, this library must be installed like others. For that:

pip install mss

After import libraries, I will set coordinates of the game, because in the screen there are a lot of useless stuff, in order to remove them I will determine some coordinates using Paint :)

mon = {"top":370,
       "left":700,
       "width":200,
       "height":145}

Using the library called mss, I can easily cut off the area that I want the model to see only. Therefore I defined mss

sct = mss()

Now, I will create a function that I will use for recording.

i = 0
def record_screen(record_id, key):
    global i # I will use this i inside and outside of the function. (I have an other i)

    i += 1
    print(f"{key}, {i}") #key: char of keyboard, i: num of press for char
    img = sct.grab(mon)
    im = Image.frombytes("RGB", img.size, img.rgb)
    im.save(f"data/img/{key}_{record_id}_{i}.png")

After this record function, I need to define an other function for the exit. When I want to exit in recording, I will press "esc" and then the exit function will be called.
For that:

is_exit = False

def exit():
    global is_exit
    is_exit = True

keyboard.add_hotkey("esc", exit)

After all, I can set last stuff. Here I will define

record_id = uuid.uuid4()
while True:

    if is_exit: break

    try:
        if keyboard.is_pressed(keyboard.KEY_UP):
            record_screen(record_id=record_id, key="up")
            time.sleep(0.1)

        elif keyboard.is_pressed(keyboard.KEY_DOWN):
            record_screen(record_id=record_id, key="down")
            time.sleep(0.1)

        elif keyboard.is_pressed("right"):
            record_screen(record_id=record_id, key="right")
            time.sleep(0.1)

    except RuntimeError: continue

Now, I am ready to get data in game.

2) Train a CNN Model

In this content, I will train a CNN model using the data I got. The model will predict keyboard actions according to images. For example, if the model recognizes a cactus, the action will be "UP", or if the model recognizes a bird, the action will be "DOWN" or "UP". In order to take place that, first I start with that import libraries.

import glob
import os
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from PIL import Image
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
from sklearn.model_selection import train_test_split
import seaborn as sns
import matplotlib.pyplot as plt

after libraries, I'll define path of images:

imgs = glob.glob("data/img/*.png")

I'll resize my images

width = 125
height = 50

Before training, I need to apply some operations like resize, normalization.
Here our labels are keyboard actions like "UP", "DOWN" "RIGHT". To reach this actions:
First of all I need to find file names, this line gives me that:

filename = os.path.basename(img)

OUTPUT: "down_022f78bc-435f-4978-8524-ff1ea1a40d9a_1.png"

Then I'll use split method and separate them as "_", after that the first index will be our label.

label = filename.split("_")[0]

OUTPUT: "down"

Also, you can check the record_screen function to how to save the images in the first section.
After determined labels, I'll resize and normalization images

im = np.array(Image.open(img).convert("L").resize((width, height)))

Here is the all collectively steps what I told above;

X = [] # images ("cactus", "bird")
y = [] # labels ("up", "right", "down")

for img in imgs:

    filename = os.path.basename(img)
    label = filename.split("_")[0] # up, down, right
    im = np.array(Image.open(img).convert("L").resize((width, height)))
    im = im/255 # normalization
    X.append(im)
    y.append(label)

For slipt data as train and test, I must convert them to array

X = np.array(X)
X = X.reshape(X.shape[0], width, height, 1)

Now, I will apply firstly Label Encoding and then One Hot Encoding to Y (labels).
Thus, firstly labels will be numeric for example;
UP --> 0
DOWN --> 1
RIGHT --> 2
After Label Encoding I will apply One Hot Encoding
Thus, labels will be Binary Value, for example,
0 --> 000
1 --> 010
2 -->001
In order to do that, I define a function:

def one_hot_labels(values):

    # Label Encoding -> One Hot Encoding

    label_encoder = LabelEncoder()
    integer_encoded = label_encoder.fit_transform(values)
    integer_encoded = integer_encoded.reshape(len(integer_encoded), 1)

    onehot_encoder = OneHotEncoder(sparse=False)
    onehot_encoded = onehot_encoder.fit_transform(integer_encoded)

    return onehot_encoded

# One Hote Encoding
Y = one_hot_labels(y)

Using X and Y, I will split my data in 0.25 rate, the part of %75 will be train, other %25 will be test size.

# train test split
train_X, test_X, train_y, test_y = train_test_split(X, Y , test_size = 0.25, random_state = 2)

After Split Data, I will create Convolutional Neural Network

# CNN Model
model = Sequential()
model.add(Conv2D(32, kernel_size = (3,3), activation = "relu", input_shape = (width, height, 1)))
model.add(Conv2D(64, kernel_size = (3,3), activation = "relu"))
model.add(MaxPooling2D(pool_size = (2,2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation = "relu"))
model.add(Dropout(0.4))
model.add(Dense(3, activation = "softmax"))

model.compile(loss="categorical_crossentropy",
              optimizer="Adam",
              metrics=["acc"])

# training            
model.fit(train_X, train_y, epochs = 35, batch_size = 64)

After training, I will evaluate Train and Test score

score_train = model.evaluate(train_X, train_y)
print("Training Score: %", score_train[1]*100)

score_test = model.evaluate(test_X, test_y)
print("Test Score: %", score_test[1]*100)

To use trained model in the game, I need to save it.

# save weights
open("trex_model.json","w").write(model.to_json())
model.save_weights("trex_weight.h5")

and finally, I have a Trained CNN Model.

3) Test the Trained Model in Game

In this section, I will test my model in the game as real-time.

For that, first I will import my libraries

from keras.models import model_from_json
import numpy as np
from PIL import Image
import keyboard
import time
import os
from mss import mss

For screen, I'll set up my sizes

mon = {"top":370,
       "left":700,
       "width":200,
       "height":145}

sct = mss()

# size of images
width = 125
height = 50

Now, I can upload my Trained CNN Model

# load model
model = model_from_json(open("trex_model.json", "r").read())
model.load_weights("trex_weight.h5")

I will determine my labels again because the model predicts images, I make it choose in this list according to output of the model. I do not touch the keyboard, the model does.

#down:0, right:1, up:2
labels = ["Down", "Right", "Up"]

framerate_time = time.time()
counter = 0
i = 0
delay = 0.4
key_down_pressed = False
while True:

    img = sct.grab(mon)
    im = Image.frombytes("RGB", img.size, img.rgb)
    im2 = np.array(im.convert("L").resize((width, height)))
    im2 = im2 / 255  # normalization

    X = np.array([im2])
    X = X.reshape(X.shape[0], width, height, 1)
    r = model.predict(X)

    result = np.argmax(r)

    if result == 0: #down: 0
        keyboard.press(keyboard.KEY_DOWN)
        key_down_pressed = True

    elif result == 2: #up: 2

        if key_down_pressed:
            keyboard.release(keyboard.KEY_DOWN)
            time.sleep(delay)

        keyboard.press(keyboard.KEY_UP)

        if i < 1500:
            time.sleep(0.3)

        elif 1500 < i and i < 5000:
            time.sleep(0.2)

        else:
            time.sleep(0.17)

        keyboard.press(keyboard.KEY_DOWN)
        keyboard.release(keyboard.KEY_DOWN)

    counter += 1

    if (time.time() - framerate_time) > 1:

        counter = 0
        framerate_time = time.time()

        if i <= 1500:
            delay -= 0.003

        else:
            delay -= 0.005

        if delay < 0:
            delay = 0

        print("----------------")
        print(f"Down: {r[0][0]}\nRight: {r[0][1]}\nUp: {r[0][2]}")

        i += 1

Now, I need to explain what this while loop does.

First, I got an area according to the pixel (mon), and then I converted it. After Converting, I applied resize and normalization

    img = sct.grab(mon)
    im = Image.frombytes("RGB", img.size, img.rgb)
    im2 = np.array(im.convert("L").resize((width, height)))
    im2 = im2 / 255  # normalization

After these steps, I turned it array and reshaped it. Because the input is known for model.
The np.argmax() returns the indices of the maximum values along an axis. I use it due to find the label maximum probability.

    X = np.array([im2])
    X = X.reshape(X.shape[0], width, height, 1)
    r = model.predict(X)

    result = np.argmax(r)

If the result is 0 it means DOWN, using keyboard library, model will press DOWN.
And after that, I changed it as key_down_pressed = True

    if result == 0: #down: 0
    keyboard.press(keyboard.KEY_DOWN)
    key_down_pressed = True

If the result is 2 it means UP, again using keyboard library model will press UP
Here, i<1500 is random number. (1500 is frame)

    elif result == 2: #up: 2

        if key_down_pressed:
            keyboard.release(keyboard.KEY_DOWN)
            time.sleep(delay)

        keyboard.press(keyboard.KEY_UP)

        if i < 1500:
            time.sleep(0.3)

        elif 1500 < i and i < 5000:
            time.sleep(0.2)

        else:
            time.sleep(0.17)

After that, the dinosaur downs again but I must release it because, if I don't, the dinosaur keeps to stay down.

        keyboard.press(keyboard.KEY_DOWN)
        keyboard.release(keyboard.KEY_DOWN)

Now here, I set up delay

    counter += 1

    if (time.time() - framerate_time) > 1:

        counter = 0
        framerate_time = time.time()

        if i <= 1500:
            delay -= 0.003 # 3 msec

        else:
            delay -= 0.005 # 5 msec

        if delay < 0:
            delay = 0

        print("----------------")
        print(f"Down: {r[0][0]}\nRight: {r[0][1]}\nUp: {r[0][2]}")

        i += 1

You can find the project link:
https://github.com/ierolsen/Trex-Game-with-CNN

Introduction to Watershed Algorithm

Erol — Thu, 26 Nov 2020 17:32:43 +0000

Previously, I worked on Object Detection algorithm and I make detected coordinaters, rotations and dimension of objects. This project is about Watershed. Basically Watershed algoritm is used for seperating the other objects in image. After apply this algorithm, I need to apply some Low-Pass Filtering and Morphological Operations deu to determine area and edge of objects.

In Watershed Algoritm, I will apply this techniques:

1- Median Blurring
2- Gray Scale
3- Binary Threshold
4- Opening
5- Distance Transform
6- Threshold for foreground
7- Dilation for enlarging image to use for background
8- Connected Components
9- Watershed
10- Find and Draw Contours around of objects

I'll explain these subjects again when I explain code cells.

First of all I must import my libraries I'll use:

import cv2
import numpy as np
import matplotlib.pyplot as plt

After that, I need to read my image I want to apply Watershed:

coin = cv2.imread("data/coins.jpg")

plt.figure(), plt.title("Original"), plt.imshow(coin), plt.axis("off");

By the way, the semicolon (;) which is in end of plt.axis("off") is a trick for ingoring some inputs of matplotlib.

To remove the noises which are on image I will apply one of Low-Pass Filtering methods called MedianBlurring

coin_blur = cv2.medianBlur(src=coin, ksize=13)

plt.figure(), plt.title("Low Pass Filtering (Blurring)"), plt.imshow(coin_blur), plt.axis("off");

After that, I need to convert color of image to Gray

coin_gray = cv2.cvtColor(coin_blur, cv2.COLOR_BGR2GRAY)

plt.figure(), plt.title("Gray Scale"), plt.imshow(coin_gray, cmap="gray"), plt.axis("off");

Using Binary Threshold I will make image specific between coins and background

ret, coin_thres = cv2.threshold(src=coin_gray, thresh=75, maxval=255, type=cv2.THRESH_BINARY)

plt.figure(), plt.title("Binary Threshold"), plt.imshow(coin_thres, cmap="gray"), plt.axis("off");

As you can see after Thresholding it is almostly clear between coins and background.

Now, I will try to draw these Contours

contour, hierarchy = cv2.findContours(image=coin_thres.copy(), mode=cv2.RETR_CCOMP, method=cv2.CHAIN_APPROX_SIMPLE)

for i in range(len(contour)):

    if hierarchy[0][i][3] == -1: # external contour
        cv2.drawContours(image=coin,contours=contour,contourIdx=i, color=(0,255,0), thickness=10)

plt.figure(figsize=(7,7)), plt.title("After Contour"), plt.imshow(coin, cmap="gray"), 
plt.axis("off");

But as you all can see it doesn't work, I couldn't seperate their edge.

I will use other method called Watershed for seperating and draw their edge. First method is not Watershed

Before Watershed, I need to apply all of this techniques. For that:

# read data
coin = cv2.imread("data/coins.jpg")
plt.figure(), plt.title("Original"), plt.imshow(coin), plt.axis("off");

# Blurring
coin_blur = cv2.medianBlur(src=coin, ksize=15)
plt.figure(), plt.title("Low Pass Filtering (Blurring)"), plt.imshow(coin_blur), plt.axis("off");

# Gray Scale
coin_gray = cv2.cvtColor(coin_blur, cv2.COLOR_BGR2GRAY)
plt.figure(), plt.title("Gray Scale"), plt.imshow(coin_gray, cmap="gray"), plt.axis("off");

# Binary Threshold
ret, coin_thres = cv2.threshold(src=coin_gray, thresh=65, maxval=255, type=cv2.THRESH_BINARY)
plt.figure(), plt.title("Binary Threshold"), plt.imshow(coin_thres, cmap="gray"), 
plt.axis("off");

First I need to remove connetion between coins. Almostly every coins connetc other self, therefor using Opening from Morphological Operations, I will Open them

kernel = np.ones((3,3), np.uint8)

opening = cv2.morphologyEx(coin_thres, cv2.MORPH_OPEN, kernel=kernel, iterations=2)

plt.figure(), plt.title("Opening"), plt.imshow(opening, cmap="gray"), plt.axis("off");

To romevo the connection of between coins I will use Distance Transform.
After that I can see Distance between objects (coins)

dist_transform = cv2.distanceTransform(src=opening, distanceType=cv2.DIST_L2, maskSize=5)

plt.figure(), plt.title("Distance Transform"), plt.imshow(dist_transform, cmap="gray"), plt.axis("off");

After finding distance, to find image which is in foreground what it is, I'll minimizing that using Threshold.

ret, sure_foreground = cv2.threshold(src=dist_transform, thresh=0.4*np.max(dist_transform), maxval=255, type=0)

plt.figure(), plt.title("Fore Ground"), plt.imshow(sure_foreground, cmap="gray"), plt.axis("off");

To find background what it is, I will enlarging image using Dilate.

sure_background = cv2.dilate(src=opening, kernel=kernel, iterations=1) #int

sure_foreground = np.uint8(sure_foreground) # change its format to int

And now, I can Subtrack them eachothers (BackGround - ForeGround) so that, the image can be more understandable. Here is the result of Opened and Dilated image

unknown = cv2.subtract(sure_background, sure_foreground)

plt.figure(), plt.title("BackGround - ForeGround = "), plt.imshow(unknown, cmap="gray"), plt.axis("off");

After these steps, I need to find Markers for giving inputs for Watershed algorithm. And now I'll provide Connection between Components.

ret, marker = cv2.connectedComponents(sure_foreground)

marker = marker + 1

marker[unknown == 255] = 0 # White area is turned into Black to find island for watershed

plt.figure(), plt.title("Connection"), plt.imshow(marker, cmap="gray"), plt.axis("off");

After that, now I can apply Watershed Algorithm and I can make segmentation

marker = cv2.watershed(image=coin, markers=marker)

plt.figure(), plt.title("Watershed"), plt.imshow(marker, cmap="gray"), plt.axis("off");

As a last step, I'll find and Draw Contours around of Coins.

contour, hierarchy = cv2.findContours(image=marker.copy(), mode=cv2.RETR_CCOMP, method=cv2.CHAIN_APPROX_SIMPLE)

for i in range(len(contour)):

    if hierarchy[0][i][3] == -1:
        cv2.drawContours(image=coin,contours=contour,contourIdx=i, color=(255,0,0), thickness=3)

plt.figure(figsize=(7,7)), plt.title("After Contour"), plt.imshow(coin, cmap="gray"), plt.axis("off");

You can find the notebook here: https://github.com/ierolsen/Object-Detection-with-OpenCV/blob/main/7-watershed.ipynb

Also other things about Object Detection: https://github.com/ierolsen/Object-Detection-with-OpenCV

Object Detection with Color

Erol — Thu, 26 Nov 2020 17:31:35 +0000

Object Detection with Color

In this project, I've worked on object detection, tried to find coordinates, width, height of object (in this project object is blue things).

Using HSV color range which is determined as Lower and Upper, I detected colorful object. Here I prefered blue objects.

# blue HSV
blueLower = (84,  98,  0)
blueUpper = (179, 255, 255)

When I got the color range, I set capture size and then I read the capture.

First I apply Gaussian Blurring for decreasing the noises and details in capture.

#blur
blurred = cv2.GaussianBlur(imgOriginal, (11,11), 0)

After Gaussian Blurring, I convert that into HSV color format.

# HSV
hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV)

To detect Blue Object, I define a mask.

# mask for blue
mask = cv2.inRange(hsv, blueLower, blueUpper)

After mask, I have to clean around of masked object. Therefor I apply first Erosion and then Dilation

# deleting noises which are in area of mask
mask = cv2.erode(mask, None, iterations=2)
mask = cv2.dilate(mask, None, iterations=2)

After removing noises, the Contours have to be found

contours, _ = cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
center = None

If the Contours have been found, I'll get the biggest contour due to be well.

# get max contour
c = max(contours, key=cv2.contourArea)

The Contours which are found have to be turned into rectangle deu to put rectangle their around. This cv2.minAreaRect() function returns a rectangle which is smallest to cover the area of object.

rect = cv2.minAreaRect(c)

In the screen, I want to print the information of rectangle, therefor I need to reach its inform.

((x,y), (width, height), rotation) = rect
s = f"x {np.round(x)}, y: {np.round(y)}, width: {np.round(width)}, height: {np.round(height)}, rotation: {np.round(rotation)}"

Using this rectangle I found, I want to get a Box. In the next, I will use this Box for drawing Rectangle.

# box
box = cv2.boxPoints(rect)
box = np.int64(box)

Image Moment is a certain particular weighted average (moment) of the image pixels' intensities.
To find Momentum, I use Max. Contour named as "c". After that, I find Center point.

# moment
M = cv2.moments(c)
center = (int(M["m10"] / M["m00"]), int(M["m01"] / M["m00"]))

Now, I will draw the center which is found.

# point in center
cv2.circle(imgOriginal, center, 5, (255, 0, 255), -1)

After Center Point, I draw Contour

# draw contour
cv2.drawContours(imgOriginal, [box], 0, (0, 255, 255), 2)

I want to print coordinators etc. in the screen

# print inform
cv2.putText(imgOriginal, s, (25, 50), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255, 255, 255), 2)

And Final:

Project Link: https://github.com/ierolsen/Object-Detection-with-OpenCV/blob/main/4-object-detection-with-color.py

You can find more in my GitHub: https://github.com/ierolsen/Object-Detection-with-OpenCV