Posted on Feb 19

Concurrency and Parallelism in Python – Threads, Multiprocessing, and Async Programming

#python #programming #concurrency #async

Modern applications often need to perform multiple tasks simultaneously to improve efficiency. Python provides three primary ways to achieve this:

✅ Threads – Lightweight tasks running in the same process
✅ Multiprocessing – Running tasks in parallel using multiple CPU cores
✅ Async Programming – Handling I/O-bound operations efficiently

Understanding when and how to use these techniques will help you write more efficient Python programs. Let’s dive in! 🚀

1️⃣ Understanding Concurrency vs. Parallelism

🔹 Concurrency means handling multiple tasks at the same time but not necessarily executing them simultaneously.
🔹 Parallelism means executing multiple tasks simultaneously by utilizing multiple CPU cores.

Feature	Concurrency	Parallelism
Execution	Tasks start and pause efficiently	Tasks run simultaneously
CPU Cores Used	Usually single-core	Uses multiple CPU cores
Suitable For	I/O-bound tasks	CPU-bound tasks

2️⃣ Using Threads in Python (Concurrency for I/O-Bound Tasks)

Threads allow multiple operations to run concurrently within a single process.
Python provides the threading module for creating and managing threads.

✅ Basic Thread Example


import threading

def print_numbers():
    for i in range(5):
        print(i)

thread = threading.Thread(target=print_numbers)  
thread.start()  # Start the thread
thread.join()   # Wait for the thread to complete

🔥 Key Takeaways:

✅ Threads are useful for I/O-bound tasks (e.g., network requests, file operations).
✅ Python threads run on a single CPU core due to the Global Interpreter Lock (GIL).
✅ Use join() to ensure a thread finishes before proceeding.

3️⃣ Multiprocessing in Python (Parallelism for CPU-Bound Tasks)

For CPU-intensive tasks, Python provides the multiprocessing module to run code on multiple CPU cores.

✅ Basic Multiprocessing Example


import multiprocessing

def square_number(n):
    print(n * n)

process = multiprocessing.Process(target=square_number, args=(5,))
process.start()  
process.join()  # Ensures process completes before moving forward

🔥 Why Use Multiprocessing?

✅ Bypasses the GIL, allowing true parallel execution.
✅ Ideal for CPU-heavy tasks like image processing and data analysis.

4️⃣ Using ThreadPoolExecutor for Simpler Thread Management

The concurrent.futures module provides a simpler way to manage threads.


from concurrent.futures import ThreadPoolExecutor

def fetch_data(site):
    print(f"Fetching data from {site}")

sites = ["site1.com", "site2.com", "site3.com"]

with ThreadPoolExecutor(max_workers=3) as executor:
    executor.map(fetch_data, sites)  # Run tasks concurrently

🔥 Key Benefits of ThreadPoolExecutor:

✅ Manages threads efficiently.
✅ Easier than manually starting/stopping threads.

5️⃣ Using ProcessPoolExecutor for Multiprocessing

Similar to ThreadPoolExecutor but runs tasks in separate processes.


from concurrent.futures import ProcessPoolExecutor

def square(n):
    return n * n

numbers = [1, 2, 3, 4, 5]

with ProcessPoolExecutor() as executor:
    results = executor.map(square, numbers)

print(list(results))

✅ When to Use?

ThreadPoolExecutor → Best for I/O-bound tasks.
ProcessPoolExecutor → Best for CPU-bound tasks.

6️⃣ Async Programming in Python (asyncio)

For high-performance I/O-bound applications, Python provides async programming using asyncio.

✅ Basic Async Example


import asyncio

async def say_hello():
    print("Hello, World!")
    await asyncio.sleep(1)  # Simulates an I/O operation
    print("Goodbye!")

asyncio.run(say_hello())

🔥 Why Use Async Programming?

✅ Handles thousands of tasks concurrently.
✅ Best for network requests, web scraping, and database queries.
✅ Uses event loops to avoid blocking operations.

7️⃣ Combining asyncio with aiohttp for Async HTTP Requests

A common use case for async programming is making multiple API calls concurrently.


import asyncio
import aiohttp

async def fetch(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

async def main():
    urls = ["https://example.com", "https://example.org"]
    tasks = [fetch(url) for url in urls]
    results = await asyncio.gather(*tasks)
    print(results)

asyncio.run(main())

✅ Key Benefits:

Faster than traditional requests module for multiple API calls.

Non-blocking execution improves performance.

8️⃣ Choosing the Right Approach for Your Application

Scenario	Best Approach
Making multiple API calls	asyncio with aiohttp
Reading large files	Threads (ThreadPoolExecutor)
CPU-intensive computations	Multiprocessing (ProcessPoolExecutor)
Running background tasks	Threads

🔹 Conclusion

✅ Threads → Best for I/O-bound tasks (network requests, file handling).
✅ Multiprocessing → Best for CPU-heavy tasks (data processing, machine learning).
✅ Async Programming → Ideal for high-performance I/O tasks (web scraping, API calls).

Mastering these concepts helps write efficient and scalable Python programs. 🚀