DEV Community: Abhinav Pasham

ASYNC CONTEXT MANAGERS

Abhinav Pasham — Fri, 07 Aug 2026 12:06:25 +0000

Why Does Python Need Async Context Managers?

Understanding Resource Pooling with a Fake Database Connection

INTRODUCTION

While learning asyncio, I came across something called an Async Context Manager.

Initially, I thought it was just another way of writing try...finally.

Then I started learning about database connection pools.

That's when another question came to my mind.

Every request needs a database connection. Does every request create a new connection?

If that's true, creating and destroying database connections for every request would be expensive.

So how do frameworks like FastAPI and libraries like asyncpg manage thousands of requests efficiently?

This led me to two important concepts:

Resource Pooling
Async Context Managers (async with)

In this article, I'll explain the problem they solve and how they work together.

What You Will Learn

Why resource pooling exists
Why creating database connections repeatedly is expensive
How connection pools work
Why Python introduced async context managers
How async with works internally
Building a fake resource pool

Prerequisites

Before reading this article, you should understand:

Coroutines
Event Loop
await
Classes
asyncio

The Problem

Suppose you're building a backend.

Whenever a request arrives,

it needs to query the database.

Initially, I thought the flow looked like this.

Request

↓

Create Database Connection

↓

Execute Query

↓

Close Connection

Seems perfectly fine.

Now imagine

1000 users send requests simultaneously.

Request 1

Request 2

Request 3

...

Request 1000

If every request creates a brand new database connection,

the server has to

Open a network connection
Authenticate the user
Allocate memory
Establish communication

for every request.

Creating database connections is expensive.

So another question came into my mind.

Instead of creating new connections every time, why can't we reuse existing ones?

That's exactly why Resource Pooling exists.

What is Resource Pooling?

Instead of creating a connection for every request,

the application creates a small number of reusable connections.

Imagine

Pool

Connection 1

Connection 2

Connection 3

Now,

when a request arrives,

it doesn't create a new connection.

It simply borrows one.

Request

↓

Take Connection 2

↓

Execute Query

↓

Return Connection 2

↓

Pool

The next request can reuse the same connection.

The Next Question

After understanding connection pools,

another question came to my mind.

Suppose I borrow a connection.

How do I make sure it's always returned to the pool?

Imagine this code.

conn = await pool.acquire()

await conn.execute(query)

await pool.release(conn)

Looks fine.

But what if

await conn.execute(query)

raise Exception()

The exception occurs before

await pool.release(conn)

The connection is never returned.

After enough requests,

every connection remains occupied.

Eventually,

the pool becomes empty.

New requests have no available connections.

This is called a connection leak.

Python's Solution

Python introduced

async with

Instead of writing

conn = await pool.acquire()

try:
    await conn.execute(query)
finally:
    await pool.release(conn)

we simply write

async with pool as conn:

    await conn.execute(query)

Much cleaner.

More importantly,

the connection is always returned,

even if an exception occurs.

How Python Achieves This

When Python sees

async with pool as conn:

it automatically performs

Acquire Connection

↓

Execute Block

↓

Release Connection

Internally,

Python converts it into

conn = await pool.__aenter__()

try:

    await conn.execute()

finally:

    await pool.__aexit__()

This means

__aenter__() acquires the resource,

while

__aexit__() releases it.

Building a Fake Resource Pool

Let's build a simple version.

class FakeConnection:

    async def execute(self, query):

        print(f"Executing: {query}")

        await asyncio.sleep(2)

        print("Query Completed")

Now we create a resource pool.

class ResourcePool:

    async def __aenter__(self):

        print("Acquiring Connection...")

        await asyncio.sleep(1)

        self.connection = FakeConnection()

        print("Connection Acquired")

        return self.connection

    async def __aexit__(self, exc_type, exc, tb):

        print("Releasing Connection...")

        await asyncio.sleep(1)

        print("Connection Released")

Using it becomes very simple.

async with ResourcePool() as conn:

    await conn.execute(
        "SELECT * FROM users"
    )

    print("Working with database...")

Execution Flow

Program Starts

↓

async with

↓

__aenter__()

↓

Acquire Connection

↓

Return Connection

↓

Execute Query

↓

Exit async with

↓

__aexit__()

↓

Return Connection

↓

Program Ends

Real-world Use Cases

Resource pooling is used almost everywhere.

Database Connections (asyncpg)
HTTP Client Sessions (aiohttp)
Redis Connections
WebSocket Connections
File Handles

Instead of creating expensive resources repeatedly,

applications reuse them.

Advantages

Reuses expensive resources
Improves performance
Prevents connection leaks
Automatic cleanup
Cleaner code
Easier error handling

Conclusion

Initially, I thought every request created and destroyed its own database connection.

Later I learned that this approach doesn't scale.

Instead, applications maintain a pool of reusable connections.

But simply reusing connections isn't enough.

We also need a reliable way to return them back to the pool, even when something goes wrong.

That's exactly why Python provides Async Context Managers.

Once I understood that async with automatically acquires and releases resources, the idea of connection pooling became much easier to understand.

In the next article, we'll explore another interesting concept:

How can a coroutine produce values one at a time while still performing asynchronous operations?

That's where Async Generators come in.

asyncio.Queue

Abhinav Pasham — Thu, 06 Aug 2026 19:19:33 +0000

Why Does Python Need asyncio.Queue?

INTRODUCTION

After understanding asyncio.Semaphore and asyncio.Lock, I started wondering how real backend systems handle thousands of incoming requests.

A semaphore limits how many coroutines can execute simultaneously.

A lock protects shared data.

But another question came to my mind.

If all the workers are busy, where do the remaining tasks wait?

Imagine thousands of users uploading images or placing orders simultaneously. The workers can't process everything instantly, so there has to be a mechanism that stores the incoming work until a worker becomes available.

That's exactly the problem asyncio.Queue solves.

In this article, I'll explain why Python introduced asyncio.Queue, the problem it solves, and how it is used in real-world asynchronous applications.

What You Will Learn

Why asyncio.Queue exists
The Producer-Consumer problem
How Queue works internally
FIFO (First In First Out)
Practical implementation
Real-world backend examples

Prerequisites

Before learning asyncio.Queue, you should understand:

Coroutines
Event Loop
asyncio.Semaphore
asyncio.Lock

The Problem

Suppose you're building an image processing service.

Whenever a user uploads an image, it needs to

Compress the image
Generate a thumbnail
Store it in cloud storage

Initially I thought every upload could be processed immediately.

await process_image(image)

Now imagine 500 users uploading images at the same time.

Image 1

Image 2

Image 3

...

Image 500

The workers can only process a few images simultaneously.

So another question came to my mind.

If every worker is already busy, what happens to the remaining images?

Should we reject them?

Should they disappear?

Obviously not.

They need a place to wait.

That's where Queue comes in.

What is asyncio.Queue?

asyncio.Queue temporarily stores tasks until a worker becomes available.

Think of it as a waiting room.

Instead of immediately processing every task,

new tasks are placed inside the queue.

Whenever a worker finishes its current task,

it picks the next task from the queue.

How Python Achieves This

Imagine three workers.

Worker 1

Worker 2

Worker 3

Ten jobs arrive.

Job1

Job2

Job3

...

Job10

Initially

Worker1 ← Job1

Worker2 ← Job2

Worker3 ← Job3

The remaining jobs don't disappear.

Instead,

Queue

↓

Job4

Job5

Job6

Job7

Job8

Job9

Job10

When Worker2 finishes,

Worker2

↓

Gets Job4

↓

Queue becomes

Job5

Job6

Job7
...

The queue automatically provides the next available job.

FIFO (First In First Out)

asyncio.Queue follows FIFO.

This means

First Job Entered

↓

First Job Processed

Example

Queue

↓

Job1

Job2

Job3

Worker executes

job = await queue.get()

Now

Worker receives

↓

Job1

Queue becomes

Job2

Job3

Practical Example

import asyncio

queue = asyncio.Queue()

async def producer():

    for i in range(1,6):

        print(f"Added Job {i}")

        await queue.put(i)

async def consumer():

    while True:

        job = await queue.get()

        print(f"Processing Job {job}")

        await asyncio.sleep(2)

        queue.task_done()

async def main():

    producer_task = asyncio.create_task(producer())

    consumer_task = asyncio.create_task(consumer())

    await producer_task

    await queue.join()

    consumer_task.cancel()

asyncio.run(main())

Output

Added Job 1

Added Job 2

Added Job 3

Processing Job 1

Processing Job 2

Processing Job 3

Notice that the producer keeps adding work,

while the consumer processes one job at a time.

queue.task_done()

Initially I didn't understand why we call

queue.task_done()

The job has already been removed from the queue.

So why call another method?

The answer is that removing a job from the queue doesn't mean the work has finished.

It only means the worker has accepted the job.

Only after processing completes,

the worker calls

queue.task_done()

to inform the queue that the job has been completed.

queue.join()

Suppose you want the program to wait until every job has finished.

Instead of manually checking every worker,

Python provides

await queue.join()

It waits until every job that was added using

queue.put()

has been marked as completed using

queue.task_done()

What if Queue Didn't Exist?

Suppose workers are busy.

New jobs keep arriving.

Without a queue,

New Job

↓

??

Where should the job go?

The producer has to wait until a worker becomes free.

Or even worse,

the application may reject incoming work.

A queue acts as a temporary buffer between producers and consumers.

Real-world Use Cases

Image Processing

Upload Image

↓

Queue

↓

Compress Image

↓

Generate Thumbnail

↓

Store in Cloud

The user receives a response immediately,

while the background worker processes the image.

Email Service

When thousands of users register,

emails aren't sent immediately.

Instead,

Register User

↓

Queue Email

↓

Background Worker

↓

Send Email

Order Processing

Customer Places Order

↓

Queue

↓

Worker

↓

Generate Invoice

↓

Update Inventory

Log Processing

Servers continuously generate logs.

Instead of processing every log instantly,

logs are pushed into a queue,

and background workers process them.

Advantages of Queue

Stores work temporarily
Decouples producers and consumers
Prevents losing incoming tasks
Makes applications scalable
Smoothly handles traffic spikes

Conclusion

Initially, I thought workers would process every task immediately.

But real applications receive requests much faster than workers can process them.

Instead of rejecting work or making users wait,

Python provides asyncio.Queue, which temporarily stores incoming tasks until workers become available.

Once I understood that Queue acts like a waiting room between producers and consumers, the entire Producer-Consumer pattern became much easier to understand.

In the next article, we'll learn how Python automatically acquires and releases asynchronous resources using Async Context Managers (async with).

ASYNCIO.LOCK

Abhinav Pasham — Thu, 06 Aug 2026 18:20:48 +0000

Why Does Python Need asyncio.Lock?

INTRODUCTION

After understanding asyncio.Semaphore, I thought I had learned everything required to control multiple coroutines.

A semaphore limits how many coroutines can execute simultaneously.

Then another question came to my mind.

If Python's event loop executes only one coroutine at a time, why do we even need a Lock?

Initially, I assumed a lock was unnecessary because there was only one thread.

But after experimenting with shared variables, I realized that even though only one coroutine executes at a particular instant, multiple coroutines can still interfere with each other.

In this article, I'll explain the problem that led to asyncio.Lock, how it works, and why almost every backend application uses it.

What You Will Learn

Why asyncio.Lock exists
What is a race condition
What is a critical section
How Lock works internally
Practical examples
Real-world backend use cases

Prerequisites

Before learning asyncio.Lock, you should understand:

Coroutines
Event Loop
await
asyncio.Semaphore

The Problem

Suppose we have a shared variable.

counter = 0

Now imagine two coroutines trying to increment it.

async def increment():

    global counter

    temp = counter

    await asyncio.sleep(1)

    counter = temp + 1

Initially I expected the final value to become

because two coroutines are incrementing the counter.

But that wasn't what happened.

Let's See What Actually Happens

Initially

counter = 0

Now Coroutine A starts executing.

Read counter

↓

temp = 0

↓

await

The coroutine reaches await.

The event loop suspends it and starts another coroutine.

Now Coroutine B executes.

Read counter

↓

temp = 0

↓

await

Notice something interesting.

Both coroutines have already read

counter = 0

Now Coroutine A resumes.

counter = 1

Then Coroutine B resumes.

counter = 1

The final value becomes

instead of

This is called a Race Condition.

Why Did This Happen?

Initially I blamed the Event Loop.

Later I realized,

the Event Loop didn't do anything wrong.

Its job is simply to switch between coroutines whenever they reach an await.

The real problem was that both coroutines were modifying the same data.

The Critical Section

The block of code that accesses or modifies shared data is called the Critical Section.

temp = counter

await asyncio.sleep(1)

counter = temp + 1

If multiple coroutines execute this block, the shared data can become inconsistent.

So this block should only be executed by one coroutine at a time.

Python's Solution

Python introduced

asyncio.Lock()

A Lock ensures that only one coroutine can execute the critical section at a time.

If another coroutine tries to enter,

it simply waits until the lock is released.

How Lock Works

Imagine only one key exists.

Coroutine A

↓

Acquires Lock 🔒

↓

Critical Section

↓

Releases Lock

↓

Coroutine B acquires Lock

Unlike Semaphore,

a Lock always allows only one coroutine inside.

Practical Example

import asyncio

lock = asyncio.Lock()

counter = 0

async def increment():

    global counter

    async with lock:

        temp = counter

        await asyncio.sleep(1)

        counter = temp + 1

async def main():

    await asyncio.gather(
        increment(),
        increment()
    )

    print(counter)

asyncio.run(main())

Output

Now both coroutines execute safely because only one coroutine can enter the critical section.

Why async with?

Initially I wondered why we write

async with lock:

instead of

await lock.acquire()

...

lock.release()

The answer became clear when I introduced an exception.

await lock.acquire()

raise Exception()

lock.release()

The lock is never released.

Every other coroutine waits forever.

Using

async with lock:

Python automatically releases the lock,

even if an exception occurs.

What if Lock Didn't Exist?

Imagine an online banking application.

Current Balance

₹1000

User A

Deposit ₹500

User B

Withdraw ₹200

Without a Lock,

both operations may read the same balance before updating it.

The final balance becomes incorrect.

The same problem occurs in

Inventory systems
Payment gateways
Order processing
Shared counters
Database updates

Real-world Use Cases

Banking Systems

Updating account balances safely.

Inventory Management

Preventing two customers from purchasing the last available product.

Order Processing

Generating unique order numbers.

Shared Cache

Updating shared cache values safely.

Advantages of Lock

Prevents race conditions
Protects shared resources
Maintains data consistency
Automatically releases resources with async with
Makes concurrent applications reliable

Conclusion

Initially I thought that because asyncio uses only one thread, race conditions couldn't happen.

But after understanding how the event loop switches between coroutines at every await, I realized that multiple coroutines can still interfere with shared data.

asyncio.Lock doesn't make the program asynchronous.

It simply makes shared resources safe by ensuring that only one coroutine executes the critical section at a time.

Once I understood the problem it solves, using a Lock became much more intuitive than simply remembering its syntax.

In the next article, we'll answer another question that came to my mind while learning asyncio.

If workers are busy processing tasks, where should newly arriving tasks wait?

That's where asyncio.Queue comes in.

Semaphore

Abhinav Pasham — Thu, 06 Aug 2026 18:14:46 +0000

Why Does Python Need asyncio.Semaphore?

python

INTRODUCTION

While learning asyncio, I understood coroutines, the event loop, and asyncio.gather(). I learned that multiple coroutines can run concurrently by pausing at await, allowing the event loop to switch between them.

After that, one question immediately came to my mind.

If I create 1000 coroutines, will all of them execute at the same time?

Initially, I thought that would be a good thing because more concurrency means better performance. But later I realized that too much concurrency can actually become a problem.

In this article, I'll explain why asyncio.Semaphore exists, the problem it solves, and where it's used in real-world backend applications.

What You Will Learn

Why asyncio.Semaphore exists
The problem it solves
What happens without a semaphore
How semaphore works internally
Real-world use cases
Practical implementation

Prerequisites

Before learning asyncio.Semaphore, you should understand:

Coroutines
Event Loop
await
asyncio.create_task()
asyncio.gather()

The Problem

After learning asyncio.create_task() and asyncio.gather(), I wrote something like this.

import asyncio

async def worker(task_id):
    print(f"Task {task_id} started")
    await asyncio.sleep(2)
    print(f"Task {task_id} finished")

async def main():

    tasks = []

    for i in range(1000):
        tasks.append(asyncio.create_task(worker(i)))

    await asyncio.gather(*tasks)

asyncio.run(main())

Initially I thought,

"Great! Now all my tasks are running concurrently."

But then another question came into my mind.

What if every task calls the same database or the same external API?

async def worker():

    await call_api()

Now imagine creating 1000 coroutines.

1000 Coroutines

↓

1000 API Requests

Most APIs have rate limits.

Databases also have connection limits.

Even if there is no limit, sending thousands of requests simultaneously wastes resources and increases load on the backend.

So the problem isn't creating many coroutines.

The problem is allowing all of them to access the same resource simultaneously.

This is exactly the problem asyncio.Semaphore solves.

What is asyncio.Semaphore?

A semaphore limits how many coroutines can execute a particular block of code at the same time.

For example,

semaphore = asyncio.Semaphore(3)

means,

At most 3 coroutines are allowed inside the protected block simultaneously.

If a fourth coroutine arrives, it doesn't fail.

It simply waits until one of the running coroutines finishes.

How Python Achieves This

Think of a semaphore as a collection of permission tokens.

Semaphore(3)

↓

Permit

Permit

Permit

Initially, three permits are available.

Now suppose six coroutines arrive.

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

The execution flow becomes

Task 1 gets Permit ✓

Task 2 gets Permit ✓

Task 3 gets Permit ✓

Task 4 waits

Task 5 waits

Task 6 waits

When one task finishes,

Task 2 finishes

↓

Permit Released

↓

Task 4 gets Permit

The semaphore doesn't stop creating coroutines.

It only limits how many can execute a particular block simultaneously.

Practical Example

import asyncio

semaphore = asyncio.Semaphore(3)

async def worker(task_id):

    async with semaphore:

        print(f"Task {task_id} started")

        await asyncio.sleep(2)

        print(f"Task {task_id} finished")

async def main():

    tasks = []

    for i in range(10):
        tasks.append(asyncio.create_task(worker(i)))

    await asyncio.gather(*tasks)

asyncio.run(main())

Output (order may vary)

Task 0 started
Task 1 started
Task 2 started

Task 1 finished

Task 3 started

Task 2 finished

Task 4 started

Notice something interesting.

Although we created 10 tasks, only 3 tasks were allowed to execute inside the semaphore block at any moment.

The remaining tasks simply waited for their turn.

What if Semaphore Didn't Exist?

Suppose you're downloading 1000 images.

Without a semaphore,

Download Image 1

Download Image 2

Download Image 3

...

Download Image 1000

Every download starts immediately.

Now imagine the same thing happening for

API requests
Database queries
File uploads

Your application can easily overload the server or hit API rate limits.

Instead of controlling concurrency manually, Python provides asyncio.Semaphore.

Real-world Use Cases

API Rate Limiting

Suppose an external API only allows 20 requests at a time.

Instead of sending hundreds of requests simultaneously,

semaphore = asyncio.Semaphore(20)

ensures only twenty requests are processed concurrently.

Database Connections

Most databases don't allow unlimited active connections.

Instead of allowing every coroutine to access the database simultaneously,

a semaphore limits the number of concurrent database operations.

File Downloads

Suppose you need to download 500 images.

Without limiting concurrency,

every download starts immediately.

Using

asyncio.Semaphore(10)

only ten downloads happen simultaneously, reducing network congestion.

Advantages of Semaphore

Prevents resource exhaustion
Prevents API rate-limit errors
Controls concurrency easily
Reduces unnecessary load on databases
Makes applications more stable

Conclusion

Initially, I thought creating more coroutines would always improve performance.

Later I realized that the real problem wasn't creating coroutines—it was allowing too many of them to access the same resource simultaneously.

asyncio.Semaphore solves this by limiting how many coroutines can enter a particular block of code at the same time.

Once I understood the problem it solves, the syntax became much easier to remember.

In the next article, we'll look at another question that came to my mind while learning asyncio.

If only one coroutine executes at a time, why do we still need asyncio.Lock?

ASYNC SYNCHRONIZATION IN PYTHON

Abhinav Pasham — Thu, 06 Aug 2026 17:05:33 +0000

INTRODUCTION

While learning Python's asyncio, I understood coroutines and the event loop. Coroutines can run concurrently by pausing at await, allowing the event loop to switch between them.

But after understanding this, I had a few questions:

If I create 1000 coroutines, will all of them access the database simultaneously?
If two coroutines update the same variable, won't they overwrite each other's changes?
If producers generate work faster than consumers process it, where does that work wait?

These questions led me to Python's asynchronous synchronization primitives. They are not used to make coroutines asynchronous—they are used to coordinate asynchronous coroutines safely.

In this article, I'll explain why Semaphore, Lock, and Queue exist, the problems they solve, and how they are used in real-world backend applications.

What You Will Learn
Why synchronization primitives exist
Why the event loop alone is not enough
The problem solved by asyncio.Semaphore
The problem solved by asyncio.Lock
The problem solved by asyncio.Queue
Internal working of each concept
Real-world backend examples
How these concepts work together
Prerequisites

Before reading this article, you should understand:

Coroutines
asyncio
Event Loop
await
asyncio.create_task()
asyncio.gather()
The Problem

When I first learned asyncio, I thought:

Since Python uses only one thread with the event loop, why do we even need synchronization?

Initially this made sense because only one coroutine executes at a particular instant.

But later I realized something important.

A coroutine can pause whenever it reaches an await.

Coroutine A

↓

Reads shared data

↓

await

↓

Coroutine B starts executing

↓

Modifies same data

↓

Coroutine A resumes

Now both coroutines are working on the same resource.

Similarly,

1000 Coroutines

↓

All call the same API

↓

Server overloaded

Or,

1000 Jobs

↓

Only 5 workers

↓

Where should remaining jobs wait?

These problems cannot be solved by the event loop itself.

They require synchronization.

Why Async Synchronization Exists

The event loop schedules coroutines.

Synchronization primitives coordinate coroutines.

These are completely different responsibilities.

Event Loop

↓

Decides WHO executes

Synchronization

↓

Controls HOW they execute

Python provides three important synchronization primitives.

Semaphore

↓

Limit concurrent access

Lock

↓

Protect shared resources

Queue

↓

Store and distribute work
asyncio.Semaphore
The Problem

Suppose you create 1000 coroutines.

Each coroutine calls an external API.

Coroutine1

Coroutine2

Coroutine3

...

Coroutine1000

Without any limit,

all 1000 requests may start together.

This can:

overload the backend
exceed API rate limits
consume unnecessary memory
Why Semaphore Exists

A semaphore limits how many coroutines are allowed to execute a particular section simultaneously.

Semaphore(3)

↓

Permit

Only three coroutines may enter.

The remaining coroutines wait.

Internal Working
Task1 enters

↓

Permit Count = 2

Task2 enters

↓

Permit Count = 1

Task3 enters

↓

Permit Count = 0

Task4 arrives

↓

Wait

Task2 finishes

↓

Permit released

↓

Task4 enters

The semaphore is automatically released when execution leaves the async with block.

Example
import asyncio

semaphore = asyncio.Semaphore(3)

async def worker(task_id):
async with semaphore:
print(f"Task {task_id} started")
await asyncio.sleep(2)
print(f"Task {task_id} finished")
Real-world Example

Suppose your backend downloads images.

1000 Images

↓

Semaphore(20)

↓

Only 20 downloads happen simultaneously.

This prevents excessive resource usage.

What if Semaphore didn't exist?
1000 Tasks

↓

1000 API Calls

↓

Rate Limit

↓

Failures
asyncio.Lock
The Problem

Imagine two coroutines updating the same bank balance.

Balance = ₹1000

Coroutine A

Read Balance

↓

Add ₹500

↓

Write Balance

Coroutine B

Read Balance

↓

Subtract ₹200

↓

Write Balance

Both read the same value before either writes it.

One update overwrites the other.

This is called a race condition.

Critical Section

A critical section is a block of code that accesses or modifies shared resources and therefore should only be executed by one coroutine at a time.

Why Lock Exists

A lock ensures only one coroutine executes the critical section at a time.

Other coroutines wait until the lock is released.

Internal Working
Coroutine1 acquires lock

↓

Critical Section

↓

Coroutine2 waits

↓

Coroutine1 finishes

↓

Lock Released

↓

Coroutine2 enters
Example
lock = asyncio.Lock()

async def update_balance():

async with lock:

    balance = await get_balance()

    balance += 500

    await save_balance(balance)

Real-world Example

Inventory Management

Product Quantity = 1

Two users purchase simultaneously.

Without Lock,

both may successfully purchase the same product.

With Lock,

only one coroutine updates the inventory at a time.

What if Lock didn't exist?
Coroutine A

↓

Read

↓

await

↓

Coroutine B

↓

Modify

↓

Coroutine A resumes

↓

Incorrect Data
asyncio.Queue
The Problem

Imagine customers placing orders faster than chefs can prepare them.

Orders

↓

Only two chefs are available.

Where should the remaining orders wait?

Why Queue Exists

A queue temporarily stores work until workers become available.

It follows the FIFO (First In, First Out) principle.

Internal Working
Producer

↓

queue.put()

↓

Queue

↓

queue.get()

↓

Consumer

↓

queue.task_done()
Example
queue = asyncio.Queue()

await queue.put("Order1")

job = await queue.get()

queue.task_done()
Real-world Example

Image Processing

User uploads image

↓

Queue

↓

Background Worker

↓

Compress Image

↓

Generate Thumbnail

↓

Upload

The user gets an immediate response while background workers process the image.

What if Queue didn't exist?
1000 Jobs

↓

Workers Busy

↓

Jobs Lost

The producer must continuously wait for a worker to become free.

How These Three Work Together

In a real backend system, these primitives are often used together.

User Request

↓

Queue

↓

Worker

↓

Semaphore

↓

Lock

↓

Database

Each primitive solves a different problem.

Queue stores incoming work.
Semaphore limits concurrent processing.
Lock protects shared data.
Advantages
Semaphore
Prevents resource exhaustion
Controls concurrency
Helps respect API rate limits
Lock
Prevents race conditions
Protects shared resources
Ensures data consistency
Queue
Buffers incoming work
Implements producer-consumer architecture
Decouples producers from workers
Conclusion

Initially, I thought the event loop alone was enough because only one coroutine executes at a time. But while learning more, I realized that asynchronous applications face a different set of problems:

Too many coroutines may compete for limited resources.
Multiple coroutines may modify shared data.
Producers and consumers may run at different speeds.

These problems are solved by asyncio.Semaphore, asyncio.Lock, and asyncio.Queue.

Understanding why these synchronization primitives exist is much more valuable than simply memorizing their syntax, because once the problem is clear, the solution becomes intuitive.

Python Concurrency: GIL, Threading, Multiprocessing and Asyncio

Abhinav Pasham — Wed, 05 Aug 2026 05:57:14 +0000

Introduction

When I started learning concurrency in Python, the first thing that confused me was why there are so many different ways to execute multiple tasks.

We have threading, multiprocessing, and asyncio.

At first, they all seemed to solve the same problem: doing multiple things at once.

But then I realized that they don't solve exactly the same problem.

The real question is:

What is preventing our program from finishing faster?

Sometimes the CPU is busy doing calculations.

Sometimes the CPU is barely doing anything because the program is waiting for an API, database, file, or network response.

Once I understood this difference, threading, multiprocessing, the GIL, and asyncio started connecting naturally.

What You Will Learn

Why concurrency is needed
Concurrency vs parallelism
I/O-bound vs CPU-bound tasks
Why threading exists
What the GIL changes
Why multiprocessing is needed
Why asyncio exists even though we already have threading
Event loop
Coroutines
async and await
Tasks and Futures
asyncio.gather()
asyncio.wait()
asyncio.wait_for()
Cancellation
Cooperative concurrency

Where Does the Problem Start?

Imagine I have three API requests.

Each request takes around 5 seconds to return a response.

If I execute them normally:

request_a()
request_b()
request_c()

the first request starts and waits for its response.

Only after it finishes does the second request start.

Then the third.

So roughly:

A → wait 5 sec → finish

              B → wait 5 sec → finish

                                C → wait 5 sec → finish

Total time:

5 + 5 + 5 = 15 seconds

But something feels wrong here.

While A is waiting for the server, our CPU doesn't need to spend those entire 5 seconds calculating something for A.

A lot of that time is simply waiting.

So why can't we use that waiting time to make progress on B?

This is where concurrency starts becoming useful.

Concurrency

Concurrency means multiple tasks can make progress during overlapping periods of time.

Instead of:

A → WAIT → finish

         B → WAIT → finish

                      C → WAIT → finish

we want something closer to:

A → start → WAIT ─────────────→ finish
↓
B → start → WAIT ────→ finish
↓
C → start → WAIT → finish

When A cannot make progress because it is waiting, something else can make progress.

This does not necessarily mean A and B are executing CPU instructions at exactly the same moment.

That leads to another concept I initially mixed up with concurrency.

Concurrency vs Parallelism

Concurrency is about handling multiple tasks over overlapping time.

Parallelism means multiple tasks are literally executing at the same time.

Imagine one CPU core.

It can do something like:

A → execute
A → wait

B → execute
B → wait

A → resume

That's concurrency.

Now imagine multiple CPU cores:

Core 1 → Task A

Core 2 → Task B
Now A and B can actually execute simultaneously.

That's parallelism.

So I started thinking about it like this:

Concurrency
→ multiple tasks make progress over overlapping time

Parallelism
→ multiple tasks actually execute at the same time

But now another question comes.

How do I know whether my program needs concurrency or parallelism?

For that, I first need to know what kind of work my program is doing.

I/O-Bound vs CPU-Bound

This distinction connects almost everything else.

Suppose I make an API request.

The CPU might spend a tiny amount of time sending the request and processing the response.

But most of the time looks like:

Send request
↓
WAIT
WAIT
WAIT
WAIT
↓
Response

This is an I/O-bound task.

Examples include:

API requests
Database queries
Network operations
File operations

The important part is:

Most of the time is spent waiting.

Now consider:

total = 0

for i in range(100_000_000):
total += i * i

Here the CPU is constantly calculating.

CALCULATE
CALCULATE
CALCULATE
CALCULATE

There isn't much waiting time.

This is CPU-bound work.

Examples include heavy calculations, some data processing, image processing, and computation-heavy algorithms.

Now the problem becomes much clearer.

For I/O-bound work, I want to make use of the time where one operation is waiting.

For CPU-bound work, I want more actual CPU execution power.

This is where threading enters.

Why Threading?

Imagine:

API Request A → waiting

API Request B → ready to start

There is no reason for B to wait just because A is waiting for the network.

With threading, we can have multiple threads inside the same process.

Python Process
│
├── Thread A
├── Thread B
└── Thread C

Suppose Thread A makes an API request.

Thread A
↓
send request
↓
WAIT

While A is blocked waiting for I/O, another thread can make progress.

Thread A → WAITING

Thread B → RUNNING

This makes threading useful for many I/O-bound problems.

The threads also share the memory of their process, which makes communication between them relatively straightforward, although shared mutable data can create synchronization problems.

At this point I had another thought.

If threads can execute my work, and my laptop has multiple CPU cores, why don't I just use threads for CPU-heavy calculations too?

That's where the GIL becomes important.

The GIL

In standard GIL-enabled CPython, there is something called the Global Interpreter Lock.

Consider:

ONE Python Process
↓
CPython Interpreter
↓
GIL
↓
┌──────┼──────┐
↓ ↓ ↓
T1 T2 T3

Multiple threads can exist.

But for execution of Python bytecode in that interpreter, the threads contend for the GIL.

This means that creating four threads for four heavy pure-Python calculations doesn't normally mean those four threads will execute Python bytecode simultaneously across four CPU cores.

This was an important correction to my original thinking.

I initially thought:

If the threads are working with different variables, why should Python stop them?

But the GIL isn't locking only the variables I created.

It is part of CPython's interpreter/runtime design and historically simplifies and protects important internal object and memory-management operations.

So even if:

Thread A → variable x

Thread B → variable y

both threads still execute within the same interpreter and contend for its GIL.

This explains why threading is often excellent for I/O-bound work but isn't normally the solution for getting multi-core parallelism from CPU-heavy pure-Python code.

So now there is another problem.

How do we actually use multiple CPU cores for Python calculations?

Why Multiprocessing?

Instead of creating multiple threads inside one process, we can create multiple processes.

Process A

Process B

Process C

Each process normally has its own address space and Python interpreter state.

Conceptually:

Process A
→ Interpreter A
→ GIL A

Process B
→ Interpreter B
→ GIL B

Now the operating system can potentially schedule them on different CPU cores.

Core 1 → Process A

Core 2 → Process B

This gives us actual CPU parallelism when hardware resources are available.

That's why multiprocessing makes sense for CPU-bound pure-Python work.

But multiprocessing introduces its own trade-off.

Threads Share Memory, Processes Don't Normally Share It

Suppose:

count = 0

With threads:

ONE PROCESS

Shared count
↑
┌───┴───┐
↓ ↓
T1 T2


Both threads are inside the same process and can access its memory.

With multiprocessing:

Parent Process
count = 0

Process A
count = 0

Process B
count = 0


If A changes its `count` to 1 and B changes its `count` to 1, that doesn't automatically mean:

Parent count = 2


The processes normally have separate address spaces.

So multiprocessing gives us:


CPU parallelism
        ↓
Separate processes
        ↓
More isolation
        ↓
But sharing data becomes more complicated


Now I understood the basic split:


I/O-bound
→ Threading

CPU-bound
→ Multiprocessing


But then I reached another question.

**If threading already handles I/O-bound work, why does Python have asyncio?**

---

# Why Asyncio If Threading Already Exists?

Imagine a server handling a very large number of network connections.

With a thread-per-operation design, we might imagine:

Connection 1 → Thread 1
Connection 2 → Thread 2
Connection 3 → Thread 3
...

This can work, but threads aren't free.

The operating system has to manage them, schedule them, switch between them, and allocate resources for them.

And the interesting part is that many network operations spend most of their time doing this:

WAITING


So I started thinking:

**Do I really need a large number of OS threads just to manage a large number of waiting operations?**

Asyncio gives us another model.

Instead of relying on one thread per async operation, an event loop, commonly running on one thread, can coordinate many asynchronous tasks.

text id="al7b5p"
ONE THREAD
↓
EVENT LOOP
↓
┌────┼────┐
↓ ↓ ↓
A B C


This is where the **event loop** becomes important.

---

# What Does the Event Loop Actually Do?

Suppose we have:

Task A
Task B
Task C


One event-loop thread can only execute one piece of Python code at an instant.

So something needs to keep track of:

Who can run?

Who is waiting?

Whose I/O has become ready?

Who should resume?


That's the job of the event loop.

Imagine A starts:

Event Loop
    ↓
Task A


A sends an API request and cannot continue until the response arrives.

Instead of blocking the event-loop thread, A can suspend.

Now:

A → WAITING
B → READY
C → READY


The event loop can run B.

Later, when A's awaited operation becomes ready:
A → READY


the event loop can eventually resume A.

So I started thinking of the event loop as a **coordinator**.

It keeps async work moving based on what is ready and what is waiting.

But this creates another problem.

**How can a Python function stop in the middle and continue later?**

That's where coroutines enter.

---

# Coroutines

A normal function roughly behaves like:

START
 ↓
execute
 ↓
execute
 ↓
RETURN


For asyncio, we need something capable of:

START
 ↓
execute
 ↓
SUSPEND
 ↓
something else runs
 ↓
RESUME
 ↓
execute
 ↓
RETURN


That's the important idea behind a coroutine.

In Python:

async def download():
    print("Starting")

    await asyncio.sleep(2)

    print("Finished")

`async def` defines a coroutine function.

When we call it:


download()

we get a coroutine object representing that asynchronous operation.

The coroutine can later be awaited or scheduled.

Now we need something that allows the coroutine to suspend.

That's `await`.

---

# What Does `await` Really Mean?

Consider:


data = await get_data()
At first, I interpreted `await` as:

> Wait here until the result comes.

That is correct from the **coroutine's point of view**, but it misses the most important part.

The coroutine cannot continue past that line until `get_data()` is ready.

But if the awaited operation needs to wait and supports asynchronous suspension, the coroutine can give control back to the event loop.

Task A
  ↓
await get_data()
  ↓
result isn't ready
  ↓
A SUSPENDS
  ↓
Event Loop gets control
  ↓
runs another ready task


Later:

get_data becomes ready
        ↓
Task A becomes ready
        ↓
Event Loop resumes A
        ↓
data = result


This was the point where asyncio started making sense to me.

`await` doesn't mean:

text id="84k0f5"
Stop the entire program.


It means more like:

> I cannot continue until this awaited operation is ready. If I need to suspend, let the event loop make progress elsewhere.

But there is another important trap.

---

# `async` Doesn't Automatically Mean Concurrent

Suppose:

python id="8yr9go"
async def main():
await task_a()
await task_b()


If A takes 3 seconds and B takes 2 seconds, B isn't even reached until the first `await` completes.

So:

text id="s8z1op"
A → 3 seconds → finish
↓
B → 2 seconds → finish


Total is around 5 seconds.

So just writing:

python id="5s46wq"
async def


doesn't automatically make everything concurrent.

Sometimes we want to tell the event loop:

> Schedule this coroutine so it can make progress independently while I do other async work.

This leads to **Tasks**.

---

# Tasks

Suppose:

python id="pivjwp"
task_a()


creates a coroutine object.

Now:

python id="e3cjbs"
asyncio.create_task(task_a())


schedules that coroutine with the running event loop as a Task.

Think:

text id="mt0pf2"
Coroutine
↓
create_task()
↓
Task
↓
scheduled with Event Loop


Now we can do:

python id="52ks84"
a = asyncio.create_task(task_a())
b = asyncio.create_task(task_b())

await a
await b


Both A and B are scheduled before we wait for their completion.

So if A suspends:

text id="w55qv8"
A → await → suspend


B can make progress.

This gives us concurrent async execution.

---

# Then What Is a Future?

After understanding Tasks, Futures sounded complicated, but the underlying idea is simple.

A Future represents:

**A result that may not exist yet but should become available later.**

Imagine:

text id="v1u6m7"
API request
↓
Future
↓
PENDING
↓
response arrives
↓
DONE
↓
result available


If I write:

python id="q53g1v"
result = await future


and the Future isn't complete, my coroutine can suspend.

When the Future becomes complete, the coroutine can resume with its result or receive its exception.

A useful relationship is:

text id="rz0pvz"
Future
→ represents an eventual result

Task
→ schedules/drives a coroutine
and also represents its eventual result


In asyncio, a Task is a specialized kind of Future.

---

# What If I Have Many Async Operations?

Suppose I have:

text id="rg2yio"
A
B
C


and I want all of them to run concurrently and then collect their results.

I could manually create and await Tasks.

But asyncio gives us:

python id="fdvev2"
results = await asyncio.gather(
task_a(),
task_b(),
task_c()
)


Conceptually:

text id="shludv"
A ──────────────┐
B ──────────┐ │
C ──────┐ │ │
↓ ↓ ↓
all complete
↓
results


So I think of `gather()` as:

**Run/await these async operations concurrently and collect their results together.**

But sometimes I don't want to simply wait for everything.

Maybe I want more control.

---

# `asyncio.wait()`

Imagine:

text id="1rbsr4"
A → 10 sec
B → 2 sec
C → 6 sec


Maybe my requirement is:

> As soon as one finishes, tell me what is finished and what is still pending.

That's where `wait()` is useful.

python id="l82zyr"
done, pending = await asyncio.wait(
tasks,
return_when=asyncio.FIRST_COMPLETED
)


After around 2 seconds:

text id="f0qq5y"
DONE
→ B

PENDING
→ A
→ C


So the difference becomes clearer:

text id="nv8vgo"
gather()
→ collect results from async operations

wait()
→ give me control over done/pending tasks


But what if my problem isn't multiple tasks?

What if one operation is simply taking too long?

---

# `asyncio.wait_for()`

Suppose:

python id="wuy3st"
await get_data()


normally takes a few seconds.

But if the server has a problem, I don't want my program waiting indefinitely.

I can say:

python id="3uswb8"
await asyncio.wait_for(
get_data(),
timeout=5
)


Now I'm giving the operation a maximum waiting time.

If it doesn't finish within that timeout, `wait_for()` normally cancels the awaited operation and raises a timeout exception.

So:

text id="4hq10d"
wait()
→ completion control over tasks

wait_for()
→ timeout around an awaitable


This naturally brings us to cancellation.

---

# Cancellation

Suppose a download is running:

text id="wm16rk"
Downloading...
Downloading...
Downloading...


and the user clicks Cancel.

We can request cancellation of a Task:

python id="0vqtaj"
task.cancel()


But asyncio cancellation is **cooperative**.

It isn't best understood as Python violently killing arbitrary code at a random CPU instruction.

Cancellation is delivered through asyncio's task machinery, typically by raising `CancelledError` in the task at an appropriate point.

That allows the coroutine to clean up.

For example:

python id="5mr1rk"
async def download():

try:
    await get_file()

except asyncio.CancelledError:
    print("Cleaning up...")
    raise


This might allow us to close a connection, remove a temporary file, or release some resource before stopping.

And this finally connects to one of the most important ideas behind asyncio.

---

# Cooperative Concurrency

Why is this called **cooperative concurrency**?

Because async tasks need to cooperate with the scheduler.

Imagine:

text id="78r1ks"
Task A
↓
runs
↓
await
↓
gives control back
↓
Event Loop
↓
Task B


A reaches a point where it cannot make progress and allows the event loop to run other work.

Then B does the same.

text id="nn3etb"
A → run → await ───────────→ resume
↓
B → run → await ─────→ resume


But consider:

python id="3ib4ap"
async def calculate():

while True:
    do_heavy_calculation()


There is no useful asynchronous suspension point.

Even though the function says:

python id="dl81dq"
async def


the CPU-heavy code can occupy the event-loop thread and prevent other async tasks from getting a chance to run.

This is why `asyncio` isn't a replacement for multiprocessing.

They solve different problems.

---

# Connecting Everything

This is the flow that finally made the entire topic clear to me.

Start with the program:

text id="gwhicf"
My program is slow
↓
Why?


There are two major possibilities:

text id="qx4c7o"
PROGRAM
↓
Where is time spent?
↓
┌────────┴────────┐
↓ ↓
WAITING CALCULATING
↓ ↓
I/O-BOUND CPU-BOUND


If the program is mostly waiting:

text id="0e3rd0"
I/O-bound
↓
Need concurrency
↓
Threading OR Asyncio


Threading:

text id="r1hj7k"
Multiple OS threads
inside a process


Asyncio:

text id="28ktn5"
Event loop
↓
coordinates many
async operations
↓
coroutines suspend
using await


If the program is mostly calculating:

text id="74sdq7"
CPU-bound
↓
Threads in standard
GIL-enabled CPython
don't normally give us
multi-core Python-bytecode
parallelism
↓
Multiprocessing
↓
Separate processes
↓
Multiple CPU cores


And inside asyncio itself:

async def
   ↓
Coroutine
   ↓
scheduled as Task
   ↓
Event Loop
   ↓
Run
   ↓
await
   ↓
Suspend
   ↓
Run other ready work
   ↓
Resume

Once I stopped learning threading, multiprocessing, GIL, asyncio, event loop, coroutines, and await as isolated definitions and instead followed the problem each concept solves, the entire topic became much easier to understand.

The main lesson I took from Python concurrency is:

Don't first ask, "Should I use threading, multiprocessing, or asyncio?"

First ask:

"What is my program doing most of the time — waiting or calculating?"

Once that is clear, choosing the right concurrency model becomes much easier.

LEGB RULES

Abhinav Pasham — Wed, 22 Jul 2026 15:56:25 +0000

LEGB RULES IN PYTHON

INTRODUCTION

While learning Python, I often heard people say, "Python follows the LEGB rule while searching for variables." At first, I memorized what LEGB stood for, but I never understood why Python needs this rule or how it actually searches for variables. Once I understood the search order, it became much easier to understand concepts like closures, nested functions, and decorators.

In this article, I'll explain the LEGB rule in a simple way.

What You Will Learn

Why Python needs the LEGB rule
The four scopes in Python
How Python searches for variables
What happens when a variable isn't found
How LEGB is related to closures

Prerequisites

Before learning LEGB, you should understand:

Variables
Functions
Nested functions
Variable scope

The Problem

Imagine you write a program like this:

x = 100

def outer():
    x = 50

    def inner():
        print(x)

    inner()

outer()

There are two variables named x.

Now Python has one question:

Which x should I print?

Should it print:

100 (global)?
50 (inside outer)?
Or something else?

Without a rule, Python wouldn't know which variable to use.

To solve this problem, Python follows the LEGB Rule.

What is LEGB?

LEGB is the order Python follows while searching for a variable.

It stands for:

L → Local
E → Enclosing
G → Global
B → Built-in

Python checks these scopes one by one until it finds the variable.

How Python Searches

Whenever Python sees a variable, it searches like this:

Need variable

↓

Local Scope

↓

Found?

↓

Yes → Use it

↓

No

↓

Enclosing Scope

↓

Found?

↓

Yes → Use it

↓

No

↓

Global Scope

↓

Found?

↓

Yes → Use it

↓

No

↓

Built-in Scope

↓

Found?

↓

Yes → Use it

↓

No

↓

NameError

Python stops searching as soon as it finds the variable.

Understanding Each Scope

1. Local Scope (L)

The local scope contains variables created inside the current function.

def greet():
    name = "Abhinav"
    print(name)

greet()

Output:

Abhinav

Here, Python first looks inside greet() and finds name.

2. Enclosing Scope (E)

The enclosing scope exists when one function is inside another function.

def outer():
    message = "Hello"

    def inner():
        print(message)

    inner()

outer()

Output:

Hello

message isn't inside inner(), so Python checks the enclosing function (outer) and finds it there.

3. Global Scope (G)

Variables created outside all functions belong to the global scope.

city = "Hyderabad"

def show():
    print(city)

show()

Output:

Hyderabad

Python doesn't find city inside show(), so it checks the global scope.

4. Built-in Scope (B)

Python already provides many built-in functions.

numbers = [3, 7, 2]

print(len(numbers))

Output:

We never created len(), but Python finds it in the built-in scope.

Complete Example

x = "Global"

def outer():
    x = "Enclosing"

    def inner():
        x = "Local"
        print(x)

    inner()

outer()

Output:

Local

Search order:

Looking for x

↓

Local → Found

↓

Print "Local"

↓

Stop Searching

Python never checks the enclosing or global scopes because it already found the variable locally.

What Happens If Python Doesn't Find the Variable?

def greet():
    print(name)

greet()

Output:

NameError: name 'name' is not defined

Python searches:

Local

↓

Enclosing

↓

Global

↓

Built-in

↓

Not Found

↓

NameError

Connection to Closures

Closures work because of the Enclosing scope.

def outer():
    x = 10

    def inner():
        print(x)

    return inner

func = outer()
func()

Output:

When inner() looks for x:

Local

↓

Not Found

↓

Enclosing

↓

Found x = 10

↓

Print 10

Closures rely on Python remembering the enclosing scope even after the outer function has finished executing.

Advantages of the LEGB Rule

Removes ambiguity when multiple variables have the same name.
Makes variable lookup predictable.
Supports nested functions.
Enables closures.
Forms the foundation for decorators.
Helps organize code by separating local and global data.

Conclusion

The LEGB rule is simply Python's variable lookup mechanism. Whenever a variable is used, Python searches in this order:

Local → Enclosing → Global → Built-in

As soon as the variable is found, Python stops searching. If it isn't found in any of these scopes, Python raises a NameError.

Understanding LEGB is important because many advanced Python concepts—including nested functions, closures, and decorators—depend on it.

CLOSURES

Abhinav Pasham — Tue, 21 Jul 2026 06:01:27 +0000

INTRODUCTION

While learning Python, I understood functions, nested functions, and scopes. But when I came across closures, I had one question: Why does Python even need closures? It felt like just another language feature until I understood the problem closures solve. In this article, I will try to make you understand better about closures.

What You Will Learn

Why closures Exist? -The problem they solve
What happens without closures
How closures work internally
Real-world use cases -How closures lead to decorators

Prerequisites

Before learning closures, you should understand:

-Functions
-Nested functions
-Variable scope (LEGB)
-Returning functions
-First-class functions

The Problem

Think, when u write the nested functions and u want to use the variable in the inner function which exists in the outer function after the outer finishes its execution so here we cannot access that variable.
Generally, the local scope disappears when the function finishes its execution.

Here to solve this problem it leads to the concept of closures.

What is closure?

A closure is a function that remembers and can access variables from its enclosing scope even after the enclosing function has finished execution.

def outer():
x = 10

def inner():
    print(x)

return inner

func = outer()
func()
Explanation: As we saw in the above example, the outer function finishes its execution, but the inner function can still access the variable x. This is possible because of closures. Python preserves the variables from the enclosing scope that the inner function depends on, allowing it to access them even after the outer function has returned. This behavior follows the Enclosing scope in Python's LEGB rule.

How python achieves this?

`
outer()

↓

Creates x

↓

Creates inner()

↓

inner references x

↓

Python stores x along with inner

↓

outer() finishes

↓

inner still has x
`

EXAMPLE

def outer():
x = 10

def inner():
    print(x)

return inner

func = outer()
func()
Explanation: Here initially the the function objects are initialized then the outer function is referenced and stored in the func variable here when u stored it it will be called and executes here it initializes x to 10 then it returns the inner function still here the inner function didnt called then when it returns and stored in the func now it is referring to the function inner then the func() is called then the inner funcition executes and it prints x here it will try to print the x while initially checking the local here the local variable is not present so it will check to the enclosing scope and prints it.

What if closures doesn't exist?

Without Closures:
outer()

↓

Returns

↓

x destroyed

↓

inner()

↓

NameError

Connection to decorators

A decorator works because
wrapper()

↓

Creates inner()

↓

inner remembers func

↓

Returns inner
Without Clousures:
func()

would disappear

Decorators are built on top of closures.

Advantages of Closures

Examples:

Preserve state
Data hiding
Avoid global variables
Function factories
Callback functions
Foundation for decorators