Multithreading to Memory leak! Why?

Nayan Das — Thu, 23 Mar 2023 19:10:31 +0000

Everything has it's own pros and cons! Even your most favourite language like Python. One of the most known feature of Python is GIL which stands for "Global Interpreter Lock". It is a mechanism that is used in the CPython interpreter to synchronize the execution of threads so that only one thread can execute Python bytecode at a time. This means that even on a multi-core processor, only one thread can execute Python code at a time.

Back then GIL was made in order to simplify the implementation of the CPython interpretor. The GIL make it easier to write thread-safe CPython extensions, but it also limits the parallelism of Python programs and became performance bottleneck for CPU-bound programs.

Then how is this related to memory leak while multithreading? Why am I talking about GIL? Nice question!

Before getting into it we need to understand when we should use multithreading or multiprocessing or asyncio in Python. When a task is CPU-bound, multiprocessing should be used, when it is fast and I/O-bound with a small number of connections, multithreading should be used, and when it is slow and I/O-bound with a large number of connections, asyncio should be used.

Here is a program to demonstrate multithreading in python:

# Multithreading.py
from threading import Thread
import time

def func(item):
    time.sleep(item)

start = time.perf_counter()
batch_processes = []

for i in list(range(1,13)):
    p = Thread(target=func, args=(i,))
    batch_processes.append(p)
    p.start()
for p in batch_processes:
    p.join()
finish = time.perf_counter()
print (f'Finished in {round(finish-start, 2)} second(s)')

# OUTPUT
Finished in 12.01 second(s)

Now, how can GIL affects the memory leak in Python. To be concise it is not properly memory leak. When running any I/O-bound Python programmes, such as any API calls, writing to and reading from disc creating connection to web servers or database clients, the GIL will always be released, allowing other threads to run Python code during this time. This is because these programmes spend the majority of their time waiting for I/O operations to complete.

But suppose you are using multithreading and loading a tone of heavy libraries while creating multiple connections with different apis or databases as well as storing them can lead to memory leak because at that time there will be a situation when the thread will be sleeping for 1 or something seconds and the GIL will get released and it will start the next thread in the mean time which will repeat the whole process of loading heavy libraries and creating connections with APIs and databases and it will take up the memory ones again and the memory will not release until the process inside the thread is completed.

Same if we run any highly cpu bounded task which also include some I/O bound task in multithreading it will also take up extra memory due to incomplete threads when GIL get released at any point of time which will led to high memory leak.

This is what I faced and while researching more about this topic of GIL and multithreading I concluded the above information. Please be sure to share any difficulties you have encountered while working with multiple threads in Python.

AtlasHackathon: Explore With Us (A Web App using Django, MongoDB Atlas)

Nayan Das — Mon, 10 Jan 2022 19:01:10 +0000

Overview of My Submission

Explore With Us is a vacation destination catalog website build with Django, Bootstrap and MongoDB Atlas.
I created this webapp to take part in MongoDB Atlas Hackathon.

Technologies I used:

HTML, CSS, JavaScript and Bootstrap 4.
Django
MongoDB Atlas
- I used MongoDB Atlas to store the database by creating a cluster on MongoDB
- I used the Atlas search feature to implement a fast, relevance-based full-text search for the destinations that I have in my database.

How I used Atlas Search feature:

I used MongoDB Atlas search to implement the autocomplete feature in my "Search Destination" Search Bar.
First I made a search index for my collection "main_destination"

The Search Index I used:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "name": [
        {
          "foldDiacritics": false,
          "maxGrams": 7,
          "minGrams": 3,
          "tokenization": "edgeGram",
          "type": "autocomplete"
        }
      ]
    }
  }
}

And then in backend I created Connect.py in main folder to connect with the cluster for searching the index from views.py search action. using pymongo package built for python to connect with MongoDB, as djongo does not support aggregate function.

So that once the user starts searching particular destination it will suggest them destinations automatically.

views.py>search :

This is how the function looks like:

add_result = collection.aggregate(
           [
               {
                   "$search": {
                       "autocomplete": {
                           "query": address,
                           "path": "name",
                           "fuzzy": {
                               "maxEdits": 2
                           }
                        }
                   }
               }
           ]
      )

This function build a pipeline that searches using the address parameter automatically from the collection as defined in MongoDB Atlas search index as I demoed at the top in my demo video.

How you will see this project

STEP 1:

Fork the project repository and clone it using:

git clone https://github.com/mr-palindrome/Explore-With-Us.git

The Django skeleton is look like this:

STEP 2:

Install virtualenv then create a virtual environment test:

pip3 install virtualenv
virtualenv test
source test/bin/activate

Install all the dependencies using:

pip3 install -r requirements.txt

STEP 3:

Create a Django SECRET_KEY using following command in python3 terminal:

python3
>>> from django.core.management.utils import get_random_secret_key  
>>> get_random_secret_key()

Create a .env file in explore_with_us folder to store all the environment and secret keys, like Django SECRET_KEY, mongoDB connection string.

But, for demonstration purpose you don't have to create your own cluster, I have provided my own connection string to access all the data present in my Atlas cluster database(AtlasHackathon>Project_data)

Now store the Django SECRET_KEY and my connection string in .env, it will look like:

SECRET_KEY = <django SECRET_KEY>
HOST = mongodb+srv://user_demo:demo123@atlashackathon.nhdkl.mongodb.net/myFirstDatabase?retryWrites=true&w=majority

STEP 4:

Now you can see the website in your localhost:8000 using:

python3 manange.py runserver

Submission Category:

The category is "Choose Your Own Adventure". It could be "E-Commerce Creation" but this is not an e-commerce store or any product catalog.

Link to my Code

mr-palindrome / Explore-With-Us

Explore With Us

Overview

Explore With Us is a vacation destination catalog website build with Django, Bootstrap and MongoDB Atlas I created this website to take part in MongoDB Atlas Hackathon hosted by DEV.to

Tech I used:

HTML,CSS,JS and Bootstrap 4
Django
MongoDB Atlas
- I used MongoDB Atlas to store the database by creating a cluster on [MongoDB](https://www.mongodb.com/cloud/atlas/register)
- I used the Atlas search feature to implement a fast, relevance-based full-text search for the destinations that I have in my database.

How I used Atlas Search feature:

I used MongoDB Atlas search to implement the autocomplete feature in my "Search Destination" Search Bar So that users can easily find their suitable destination within few seconds without scrolling First I made a search index for my collection "main_destination".

The Search Index I used:

{
  "mappings": {
    "dynamic": false
    "fields": {
      "name": [
        {
          "foldDiacritics": false,
          "maxGrams": 7,
          "minGrams": 3,
          "tokenization":

…

View on GitHub

Additional Resources / Info

This was my first time using MongoDB as the database for a website which was a really great experience of using NoSQL specially MongoDB Atlas, I'm also planning to use it in my future projects.
I learned about pymongo package and djongo availabe for python. I took help from How to Use Django with MongoDB document.
Initially I started researching about Atlas search and how to use them using the Atlas Search Docs.

And another resource that help me to build my frontend was Bootstrap 4.

DEV Community: Nayan Das

Multithreading to Memory leak! Why?

AtlasHackathon: Explore With Us (A Web App using Django, MongoDB Atlas)

Overview of My Submission

Technologies I used:

How I used Atlas Search feature:

How you will see this project

STEP 1:

STEP 2:

STEP 3:

STEP 4:

Submission Category:

Link to my Code

mr-palindrome / Explore-With-Us

Explore With Us

Overview

Tech I used:

How I used Atlas Search feature:

Additional Resources / Info