DEV Community: Ja'far Khakpour

Django Ninja: The Sweet Spot Between Django's Simplicity and FastAPI's Power

Ja'far Khakpour — Tue, 02 Jun 2026 17:13:40 +0000

My first experience using Django REST Framework (DRF) in a project was almost a decade ago. I've defeneded DRF philosophy it in code reviews, taught it to juniors, and built production systems that still run today. But in recent years, after trying FastAPI in different projects, I started feeling bored. Not of Django. I still love Django. I felt it from the verbose boilerplate, layers of abstraction that solved problems I no longer had amde me ask myself if I need this much complexity in one place?

Then I found Django Ninja. And after two years of using it in production, I finally understand what I was missing.

This post isn't about "DRF is dead" — it's not. It's about understanding where each tool shines, and why Ninja has become my default for most new projects.

Let me walk you through the landscape honestly.

1. Django in 2026

Let's start with something we can all agree on: Django is still fantastic. Here's why I still reach for Django first:

"Batteries included" actually matters. Authentication, admin interface, sessions, CSRF protection, security middleware — it's all there, tested, and working. No stitching together 14 Flask extensions or FastAPI plugins.
The admin interface is a superpower. For internal tools, MVP dashboards, or giving non-technical staff data access, nothing else comes close (I know there are alternatives for other frameworks, but still none feels like Django's Admin Panel).
The ORM is genuinely productive. To be honest, I love SQLAlchemy, and I hate some weird behavior of Django ORM, but no one can argue how well it integrates with the whole framework.
The ecosystem is enormous. I can put here a never ending list of great apps and modules for anything you need in your project.

But Django's built-in API tooling is not good.

If you've ever tried to build a real API using only JsonResponse and manual request parsing, you know the pain:

Manual validation everywhere
No automatic documentation
Routing feels repetitive
Error handling is ad hoc

That's why we reached for DRF in the first place. But DRF was built in a different age.

2. DRF: A Decade-Old Masterpiece Showing Its Age

Let me be clear: DRF is not bad code. It's brilliant code written for a different world.

The Era DRF Was Built For (2011–2015)

Back then, the hot API trends were:

RESTful purity — Hypermedia as the engine of application state (HATEOAS), resource-oriented design, proper HTTP verbs for everything
Hypermedia APIs — HyperlinkedRelatedField was a feature, not a footnote
Browsable API — inspired by Django admin, it was revolutionary for debugging
Class-based views — because function-based views were "too simple" those days!
XML support — yes, that's how old we're talking.

What DRF Did Right

Standardized REST APIs on Django — before DRF, everyone rolled their own. It standardized a lot of concepts in Dajngo ecosystem.
Built a mature ecosystem — django-rest-auth, drf-spectacular, drf-nested-routers, etc.
Introduced serializers — which, love them or hate them, were a solid abstraction for their time
Gave us ViewSets + routers — made CRUD APIs almost automatic

What DRF Struggles With Today

No official async support — and no announced plans. Django added async in version 3.0 (2019), then 3.1, then 4.0, etc. But DRF? Still sync-only. The workarounds are fragile.
OpenAPI docs require third-party packages — drf-yasg (not very active at maintenance), drf-spectacular (good but still extra work). No built-in solution.
Serializers are slow and verbose — pure Python, lots of reflection, and writing Meta classes for every model gets tedious.
ViewSets + routers are magical — great when they work, a nightmare when you need to debug why your destroy action isn't honoring permissions.
The browsable API is less useful now — frontends are separate. Postman, Swagger UI, and generated clients are the modern workflow.

DRF is like a well-maintained 2015 luxury car. Comfortable, known, reliable. Great engine, but terrible gas mileage compared to modern hybrids. Steering system is fascinating, but the navigation is outdated.

That doesn't mean throw it away. But maybe stop buying new ones.

3. FastAPI: The Shiny New Toy That Makes You Question Everything

Around 2018, FastAPI exploded onto the scene. And for good reason.

What FastAPI got right:

Async by default — before Django even had solid async support
Pydantic validation — type hints that actually validated your data at runtime
Automatic OpenAPI docs — Swagger UI and ReDoc with zero configuration
Incredible performance — as fast as Node or Go for many workloads
Dependency injection — an elegant, testable, and intuitive design pattern

I built several microservices with FastAPI. I enjoyed every minute of it.

But here's what nobody tells you about leaving Django for FastAPI:

Batteries not included. You reach for SQLAlchemy — which is excellent, but different. And now you're also reaching for Alembic (migrations), and something for admin, and something for auth, and something for...
No built-in authentication system. You'll piece together OAuth2, JWT, sessions, or reach for fastapi-users (good library, but not Django's django.contrib.auth).
You're building half of Django yourself. That's fine for a small microservice. For a maintained large application? That's technical debt disguised as "lightweight."

The honest take: FastAPI is fantastic for microservices, APIs that need pure async, or greenfield projects where you don't need Django's ecosystem. But if you already have Django — or you love Django's "batteries" — leaving hurts.

What if you could have FastAPI's ergonomics inside Django world? Enter Django Ninja.

4. Django Ninja: What It Is and Why It Matters

Django Ninja was created around 2020 — explicitly inspired by FastAPI but built for Django, not against it.

The core idea is to keep Django's ORM, admin, auth, and ecosystem. Replace only the API layer with something modern.

You have Pydantic validation in APIs, type hints everywhere, native async support, automatic OpenAPI docs and other modern features.

Key distinction I missed for too long: Django Ninja alone is not a REST framework. It's a modern HTTP layer. You add REST conventions on top if you want them (more on that in next sections).

Now let me explain why each feature matters in practice.

5. Let's Dive Deeper Into Key Features

5.1 Framework Speed (Request/Overhead)

Django Ninja isn't as fast as FastAPI, but it's meaningfully faster than DRF — and for most Django projects, that's the comparison that matters. DRF's request/response cycle passes through parsers, renderers, and content negotiators on every request, and its serializers do heavy runtime introspection in pure Python. Ninja skips most of that by default, and hands validation to Pydantic v2, whose core is written in Rust. Your database is still your real bottleneck, but lower validation overhead means better latency and a framework that doesn't fight you when you need async.

DRF's serializers do a lot of work — field introspection, validation, object transformation
DRF's request/response cycle passes through multiple layers (parsers, renderers, negotiators)
Ninja does less by default — you opt into complexity

5.2 Pydantic vs. DRF Serializers: Speed + Syntax

Speed first:

Pydantic v2's validation core is written in Rust. DRF serializers are pure Python with heavy use of getattr, hasattr, and runtime introspection. The result is better performance for validation, which is even higher for complex nested validation.

This matters when you're validating arrays of objects or request bodies with 20+ fields.

And the syntax: the real win for developer happiness. DRF serializers are much more verbose compared to Pydantic scheams.

What I actually care about: Less code. Fewer files to jump between. Validation that lives with the schema. Autocomplete everywhere because Pydantic uses standard Python types. It is much easier to get how Pydantic works compared to DRF serializers and steep leanirng curve of this tool.

5.3 Where Django Ninja Sits on the Abstraction Spectrum

Think of it as a an spectrum: plain Django with JsonResponse at one end, full DRF with ViewSets and routers at the other, and Django + Ninja sitting cleanly in the middle. Most APIs don't actually need a full REST framework — they need validation, docs, and async. That's Ninja on its own, with no Meta classes, no serializer/view split, no router ceremony. The mistake I made for years was reaching for DRF by default and carrying all that weight before knowing if I needed it.

I used to reach for DRF by default. Now I ask: "Do I actually need ViewSets? Browsable API? Hyperlinked relationships?" 80% of the time, the answer is no.

5.4 Ninja + CRUD / Extra: When You Want RESTful Convenience (This Is the Real DRF Alternative)

If you genuinely want DRF-style RESTful conventions (ModelViewSet, automatic CRUD, router registration), don't build it yourself. Use these add-ons:

Django Ninja CRUD

ModelViewSet-like functionality on top of Ninja
Auto-generates endpoints: GET /items, POST /items, GET /items/{id}, PUT /items/{id}, DELETE /items/{id}
Pydantic schemas auto-generated from your Django models
Permission control per action (similar to DRF's permission_classes)

Django Ninja Extra (My Preference)

Class-based controllers
Service layer injection — similar to FastAPI's dependency injection, but inside Django
Separating API handling codes from business logic using service based controllers and dependency injection
Advanced pagination, filtering, throttling out of the box
OpenAPI extended features
What DRF doesn't have: Native async, service-level DI, cleaner separation between HTTP layer and business logic

In DRF, business logic tends to bleed into views — one database call becomes a caching layer becomes an external API call, and suddenly your view is untestable. Ninja Extra's DI lets you put that logic in a service class and inject it into your controller, so the controller handles HTTP and nothing else. In tests you swap the service for a mock; in production you swap it for one that hits a cache first — the controller never changes. It's the clean separation DRF gestures toward but never quite delivers.

So the real comparison is: DRF vs. Ninja CRUD/Extra for RESTful conventions. Vanilla Ninja is a different (often better) choice for non-RESTful APIs.

6. Progressive Adoption: Ninja's Undervalued Superpower

Most "better" tools force a choice: rewrite everything or stay stuck.

Django Ninja gives you a third path: coexistence and gradual migration.

Real Example From My Last Project

We had a 30,000-line DRF codebase. Full rewrite? Impossible. Here's what we did in a large project:

1) Install Ninja alongside DRF (no breaking changes):

# urls.py
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('old_drf_urls')),      # Existing DRF endpoints
    path('ninja/api/', include('myapp.ninja_router')), # New Ninja based endpoints with a ninja/ prefix on paths
]

2) All new features written in Ninja:

Same authentication (request.user works)
Same models (Django ORM unchanged)
Same middleware (CORS, security, sessions — all work)
Same testing tools (django.test.Client, force_login)

Result: Zero friction. The team learned Ninja on real tasks, not tutorials.

3) Gradually migrate high-traffic DRF endpoints:

Pick one endpoint (e.g., GET /api/v1/users/profile)
Write Ninja version with identical response structure
Deploy both versions
Update client application to use Ninja URL
Remove DRF version after 2 weeks of stability

End result after 2 months: 80% of traffic served by Ninja, DRF endpoints still running but deprecated. No downtime. No big bang. No anxious weekends debugging a full rewrite.

This Works So Seamlessly, you don't need to migrate ORM, migrations, auth handlers, tests or admin panel. You also don't need to touch your deployment plan!

Ninja is not a migration — it's an upgrade path that respects your existing investment.

7. Conclusion: My 2026 Rule of Thumb

After two years of using Ninja in production, here's my honest decision table:

Scenario	My Choice
New Django project, simple API (most cases)	Django Ninja (vanilla)
New Django project, CRUD-heavy (admin panel, internal tool)	Django Ninja + CRUD
New Django project, complex REST + service layer	Django Ninja Extra
Existing DRF project, adding new endpoints	Add Ninja alongside, don't touch old code
Existing DRF project with budget to modernize	Incremental migration to Ninja (endpoint by endpoint)
Team deeply tied to DRF patterns, no willingness to change	Stay with DRF — tooling isn't worth team friction
Non-Django project (microservice, pure async, no admin needed)	FastAPI or Litestar
Legacy DRF project with HATEOAS or browsable API requirements	DRF (Ninja doesn't have these by default)

The Bottom Line

Django Ninja respects Django. It doesn't ask you to leave the Dajngo ecosystem. It just asks you to write better APIs with less frustration.

And the progressive adoption path means you can start today — on one endpoint — without any risk.

I'm not saying drop DRF from your projects. It's a really wonderful and robust framework. I'm saying: next time you start a new endpoint, try Ninja. See how it feels.

Most of my colleagues didn't want to switch. Now they refuse to go back.

Have you tried Django Ninja? Still on DRF? Made the jump to FastAPI and never looked back? Let me know in the comments — I genuinely enjoy hearing how other teams are solving these same problems.

And if you found this useful, consider sharing it with someone who's still writing ModelSerializer and wondering if there's a better way.

Python and Memory Management, Part 3: Call Stack

Ja'far Khakpour — Sun, 08 Feb 2026 19:19:38 +0000

In our previous sctions about Python's memory management, we discovered how Python allocates and manages objects in memory ( PyObjects, Aernas and Allocation Pools). Now let's explore what happens when those objects come to life through code execution.
In this part, we will talk about another important section of application memory: the stack memory and how it is handled in a Python programs. We are going to talk about the bridge between static memory and dynamic execution: PyFrame object as a fundamental unit of Python's call stack.

From Memory to Execution: Connecting PyObjects to PyFrames

Before we dive into Python's frame system, we need to clear up a critical distinction that often confuses people coming from languages like C or C++. Python operates with two fundamentally different stack concepts that serve entirely different purposes, and confusing them can leads to misunderstanding.

Memory Layout of a Program

From the operating system's perspective, a process's memory is organized into several distinct sections:

1. Text (Code Segment)
It stores the executable instructions (Compiled machine instructions - machine code - from your application). This memory block is read-only, fixed size, and it could be shared between processes when possible.
This segment is loaded into memory when the executable file loaded, and OS would not allow to change content of this section in the runtime.

2. Data Segment
This section stores global and static variables of the program. Variables in this section live throughout program execution.

3. Heap Segment
This is the section that is used for dynamic memory allocation. This memory is what you allocate using malloc() and free it using free() in C code. The heap is shared by all functions and modulesin a process.
Python use a different memory section for storing normal variables in memory (e.g. it uses a special method named pymalloc() instead of malloc() which manages allocation of objects in anonymous segments of application memory). This section was the subject of previous parts of this post.

4. Stack
This segment stores function call information and local variables, function parameters and function return value addresses. It has automatic allocation/deallocation of memory via simple pointer arithmetic in machine codes (It is very fast). It has a LIFO structure with fixed-size per each thread (typically 1-8MB per thread on Linux).
But Python is not using this memory for manageing call stack. It use a different memory structure and manages call stack by itself.

In fact, not only Heap and Stack segments, but also code segment is also managed differently in Python.

Code as object

From last sections, we noticed that modules and functions are stored in memory sections that Python stores normal variables (like strings). In Python every function is just an object. Every executable code can be interpreted as a AST (Abstract Syntac Tree) and also execute every AST in the runtime (Not writing a code and executin in another process, we can just run them inside the same process). This is because Python is an interpreted language. When you run a .py file, the actual code segment of your code is Python executable, and your code is part of data this interpreter is working with.
For a function, you can see it's AST structure with a readable format:

import ast
import inspect

source_code = """
def func(a: int, b: int) -> int:
    "This function adds two numbers"
    return a + b
"""

print("Function source code:")
print(source_code)
print("="*80)
print("Equivalent AST Syntax:")
print(ast.dump(ast.parse(source_code), indent=4))

You can use formatted dump output of above code to recreate the same function in runtime:


import ast
from ast import *

module_ast = Module(
    body=[
        FunctionDef(
            name='func',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(
                        arg='a',
                        annotation=Name(id='int', ctx=Load())
                    ),
                    arg(
                        arg='b',
                        annotation=Name(id='int', ctx=Load())
                    )
                ],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]
            ),
            body=[
                Expr(
                    value=Constant(value='This function adds two numbers')
                ),
                Return(
                    value=BinOp(
                        left=Name(id='a', ctx=Load()),
                        op=Add(),
                        right=Name(id='b', ctx=Load())
                    )
                )
            ],
            decorator_list=[],
            returns=Name(id='int', ctx=Load()))],
    type_ignores=[]
)


ast.fix_missing_locations(module_ast)
code_obj = compile(module_ast, '<runtime>', 'exec')
print("code objet:", code_obj)
namespace = {}
exec(code_obj, namespace)

dynamic_func = namespace['func']

print(f"{dynamic_func(1, 3)=}")
print(f"{id(dynamic_func)=:x}")

Now, we have a fully functioning function dynamic_func() which interpreter is not aware of until it runs the code. But after execution, you can see the function is working like a normal function.

Python Stack as Object

But this is not the only thing you can access as object, you can even access call stack in Python:

import inspect

def recursive_function(n, depth=0):
    """Demonstrate stack frames with recursion"""
    current_frame = inspect.currentframe()
    print(f"Depth: {depth} Frame ID: {id(current_frame)}, Previous Frame ID: {id(current_frame.f_back)}")

    if n > 0:
        recursive_function(n-1, depth=depth+1)
    else:
        demonstrate_frames(depth)

    print(f"Depth {depth}: Reurning from Frame ID: {id(current_frame.f_back)} , Local n: {n}")



def demonstrate_frames(depth):
    print("="*80)
    frame = inspect.currentframe()
    print(f"Frame details:")
    print(f"Current Line number: {frame.f_lineno}")
    print(f"Filename: {frame.f_code.co_filename}")
    print(f"Function name: {frame.f_code.co_name}")
    print(f"Code object:\t{frame.f_code}")
    print(f"Local variables: {list(frame.f_locals)}")
    print(f"Depth variable: {frame.f_locals['depth']}")
    print(f"Global variables: {list(frame.f_globals)}")


    print("="*80)
# Call the recursive function
recursive_function(2)

On top of this output, you can see frame objects and each frame, previous frame which it has been called from:

Depth: 0 Frame ID: 140421115686208, Previous Frame ID: 140421113260096
Depth: 1 Frame ID: 140421115686448, Previous Frame ID: 140421115686208
Depth: 2 Frame ID: 140421115687168, Previous Frame ID: 140421115686448

Frame stack in python is actually a linked list of frames starting from curent frame and groing back by a link to previous stack (f_back attribute). Frame object keeps track of code execution, context and variables.

In the output of this code, you can see different objects assigned to this frame, like function name, current line number, builtins, globals and locals variables. You can see all attributes of frame object here.

As seen above, call stack is a LIFO which has implemented as a linked list, not an array (like OS managed stack):

import gc
import sys

all_frame_ids = set()

def track_frame_allocation():
    """Track how frames are allocated and garbage collected"""
    def create_frames():
        """Create several frames to observe allocation patterns"""
        def level_3():
            frame = sys._getframe()
            print(f"  level 3: frame at 0x{id(frame):x}, function: {frame.f_code.co_name}")
            all_frame_ids.add(id(frame))
            return 3

        def level_2():
            frame = sys._getframe()
            print(f"  level 2: frame at 0x{id(frame):x}, function: {frame.f_code.co_name}")
            all_frame_ids.add(id(frame))
            return 2 + level_3()

        def level_1():
            frame = sys._getframe()
            print(f"  level 1: frame at 0x{id(frame):x}, function: {frame.f_code.co_name}")
            all_frame_ids.add(id(frame))
            return 1 + level_2()

        level_1()

    # Run multiple times to see if frames are reused
    print("=== FRAME ALLOCATION PATTERNS ===")

    for i in range(3):
        print(f"\nRun {i + 1}:")
        create_frames()
        # Force garbage collection and free memory used for frame objects
        gc.collect()

    print(f"\nUnique frame addresses across all runs: {len(all_frame_ids)}")

track_frame_allocation()

when you run this code, you get this output:

=== FRAME ALLOCATION PATTERNS ===

Run 1:
  level 1: frame at 0x7fb2b1ff8110, function: level_1
  level 2: frame at 0x7fb2b1ff9be0, function: level_2
  level 3: frame at 0x7fb2b1f34b80, function: level_3

Run 2:
  level 1: frame at 0x7fb2b1ff9a40, function: level_1
  level 2: frame at 0x7fb2b1ff8110, function: level_2
  level 3: frame at 0x7fb2b1f34a00, function: level_3

Run 3:
  level 1: frame at 0x7fb2b1ff9a40, function: level_1
  level 2: frame at 0x7fb2b1ff8110, function: level_2
  level 3: frame at 0x7fb2b1f34a00, function: level_3

Unique frame addresses across all runs: 5

As you can see, a memory address used for one iteration, could be easily reused in next iteration (but memory address order may change).
In a code execution a frame is released as soon as function call ends, and it return the output value, but this is not always true for generators and coroutines:

import gc
import inspect


def gen_numbers(n):
    i = 1
    while True:
        while i <= n:
            ## yield i, and receive value sent by generator.send() as x
            x = yield i
            if x == -1:
                ## generaot is called top level function (`level3()`)
                print(f"Inner loop done. show stack at level {i}:")
                stack = inspect.stack()
                for frame in stack:
                    print(f"  frame at 0x{id(frame[0]):x}, function: {frame[3]}")
                i = 1
            else:
                i += 1


def create_frames(generator):
    """Create several frames to observe allocation patterns"""
    def level_3(generator):
        ## You can send any value to the generator when iterating over it
        ## and next(generator) is just equivalent to generator.send(None)
        return generator.send(-1)

    def level_2(generator):
        return next(generator) + level_3(generator)

    def level_1(generator):
        return next(generator) + level_2(generator)

    level_1(generator)


# Run multiple times to see if frames are reused
print("=== FRAME ALLOCATION PATTERNS ===")
generator = gen_numbers(3)
for iteration in range(3):
    print(f"\nIteration {iteration + 1}:")
    create_frames(generator=generator)
    # Force garbage collection and check frame objects
    gc.collect()

if you rune this code, you will see something like this:

=== FRAME ALLOCATION PATTERNS ===

Run 1:
inner loop done. show stack at level 3:
  frame at 0x7f0944582c50, function: gen_numbers
  frame at 0x7f0944344a90, function: level_3
  frame at 0x7f0944535180, function: level_2
  frame at 0x7f0944535000, function: level_1
  frame at 0x7f0944340880, function: create_frames
  frame at 0x7f09443407c0, function: <module>

Run 2:
inner loop done. show stack at level 3:
  frame at 0x7f0944582c50, function: gen_numbers
  frame at 0x7f0944344720, function: level_3
  frame at 0x7f0944537dc0, function: level_2
  frame at 0x7f0944534a00, function: level_1
  frame at 0x7f0944340040, function: create_frames
  frame at 0x7f09443407c0, function: <module>

Run 3:
inner loop done. show stack at level 3:
  frame at 0x7f0944582c50, function: gen_numbers
  frame at 0x7f0944344a90, function: level_3
  frame at 0x7f0944535180, function: level_2
  frame at 0x7f0944535000, function: level_1
  frame at 0x7f0944340880, function: create_frames
  frame at 0x7f09443407c0, function: <module>

At each try, Address per each frame IDs for functions change, but there are two addresses not changing: root frame (<module>), which is obvious, and the generator function's frame at top of the stack. That's because the same generator (and it's frame) is used until end of code. This feature in python does not comply with standard C-like stack's LIFO srtucture, and this is exactly the point! Python behaves as a stacked language for it's normal function calls, but when you work with generators and coroutines, it behaves like a stackless language. This is a huge benefit for Python, it makes lazy evaluation, pipelines and chaining actions much easier using generators.

The same is going on for coroutines in a different way. Coroutines have smaller memory footprint compared to threads and processes. Processes are heavier than other two, and threads also have context switch costs (return control to OS, OS decides which thread to start execution (OS scheduling), store current registers and load the other thread's registers to start it's stack, and repeat), which you can add cost of Python Interpreter's Mutex (GIL) to this flow. This costs much more than switching control on PyFrames which happens inside Python interpreter. This allows the program to try smarter behavior when scheduling async tasks.

Take this sample code:

import time
import asyncio
import os
from argparse import ArgumentParser
from concurrent.futures import ThreadPoolExecutor, as_completed
import resource

N_TASKS = 10_000
IO_DELAY = 0.1
MAX_CONCURRENCY = 500

class PerformanceMonitor:
    """Simple performance monitoring context manager"""

    def __enter__(self):
        # Time measurements
        self.start_perf = time.perf_counter()

        # Resource measurements
        self.ru_start = resource.getrusage(resource.RUSAGE_SELF)

        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        # Get final measurements
        self.end_perf = time.perf_counter()
        self.ru_end = resource.getrusage(resource.RUSAGE_SELF)

    def perf_counter_time(self):
        return self.end_perf - self.start_perf
    def user_time(self):
        return self.ru_end.ru_utime - self.ru_start.ru_utime
    def sys_time(self):
        return self.ru_end.ru_stime - self.ru_start.ru_stime
    def cpu_time(self):
        return self.user_time() + self.sys_time()
    def cpu_percent(self):
        return (self.cpu_time() / self.perf_counter_time() * 100) if self.perf_counter_time() > 0 else 0
    def max_rss_kb(self):
        return self.ru_end.ru_maxrss / 1024.
    def voluntary_switches(self):
        return self.ru_end.ru_nvcsw - self.ru_start.ru_nvcsw
    def involuntary_switches(self):
        return self.ru_end.ru_nivcsw - self.ru_start.ru_nivcsw
    def page_faults(self):
        return self.ru_end.ru_majflt - self.ru_start.ru_majflt
    def page_reclaims(self):
        return self.ru_end.ru_minflt - self.ru_start.ru_minflt

    def print_results(self):
        print(
            "",
            f"{'='*50}",
        # Time metrics
            f"TIME METRICS:",
            f"  CPU time:          {self.cpu_time():.6f}s",
            f"       User:         {self.user_time():.6f}s",
            f"       System:       {self.sys_time():.6f}s",
            f"  CPU Usage Percent: {self.cpu_percent():.1f}%",
        # Memory
            "",
            f"  Max RSS:                          {self.max_rss_kb():,} MB",
        # Context switches
            f"  Voluntary Context switch:         {self.voluntary_switches():,}",
            f"  Involuntary Context switch:       {self.involuntary_switches():,}",

        # Page faults
            "",
            f"  Major (I/O):       {self.page_faults():,}",
            f"  Minor:             {self.page_reclaims():,}",
            sep="\n"
        )


def sync_stack():
    time.sleep(IO_DELAY)
    return 1


async def make_async_stack():
    await asyncio.sleep(IO_DELAY)
    return 1

def run_threads():
    print(f"Running in THREAD mode")

    with ThreadPoolExecutor(max_workers=MAX_CONCURRENCY) as executor:
        futures = [executor.submit(sync_stack) for _ in range(N_TASKS)]
        for f in as_completed(futures):
            f.result()

async def run_async():
    print(f"Running in ASYNC mode ")

    sem = asyncio.Semaphore(MAX_CONCURRENCY)

    async def limited_task():
        async with sem:
            await make_async_stack()

    tasks = [asyncio.create_task(limited_task()) for _ in range(N_TASKS)]
    await asyncio.gather(*tasks)

def run_noop():
    print("Doing Nothing! Return...")

def run_benchamrk(mode: str):
    print(
        "________________________________________",
        f"PID: {os.getpid()}",
        "Config:",
        f"  Tasks: {N_TASKS}, ",
        f"  Concurrency: {MAX_CONCURRENCY}, ",
        f"  Delay: {IO_DELAY}s, ",
        "﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌﹌",
        sep="\n"
    )

    if mode == "threads":
        run_threads()

    if mode == "async":
        asyncio.run(run_async())

    if mode == "none":
        run_noop()

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument(
        "mode",
        choices=["threads", "async", "none"],
    )
    arguments = parser.parse_args()
    mode = arguments.mode

    with PerformanceMonitor() as mon:
        run_benchamrk(mode)
    mon.print_results()

This code runs a task concurrently using threads and async loops:

python concurrent_benchmark.py none # nothing to do. just start the app and immediately stop it
python concurrent_benchmark.py threads # run a very simple task which simulates IO wait in threads
python concurrent_benchmark.py async # simulate IO wait in async coros

There is not big diference between execution time of two version and most of the time threaded execution end a little faster in my computer. But numbers we gather using PerformanceMonitor show some interesting differences:

1. CPU Time

CPU execution time has some meaningful difference between asnc and threads:

## Threads
  CPU time:          0.744787s
       User:         0.561942s
       System:       0.182845s
  CPU Usage Percent: 35.5%

## Async
  CPU time:          0.235827s
       User:         0.223514s
       System:       0.012313s
  CPU Usage Percent: 11.2%

Amount of time Threaded benchmark use CPU is around ~3x Async benchmark. But there is one more interesting metric in these stats: System time for Threads is ~15x more than Async, because in threaded mode, the application has to release control to OS, so OS switches execution flow to another thread. But in Async mode, application itself is switching between coroutines and no need to ask OS to schedule them.
You can see these results here:

## Threads
  Voluntary Context switch:         30,351
  Involuntary Context switch:       50

## Async
  Voluntary Context switch:         54
  Involuntary Context switch:       16

Voluntary switch happens when the application is voluntary releasing control. This can be for switching threads, or application do not have anything to do at the moment (all tasks are some simple sleeps, no much to do with CPU at when they are executed).

There is one more metric in this benchmark, Memory Usage:

## No-OP
  Max RSS:                          19.820 MB

## Threads
  Max RSS:                          74.512 MB

## Async
  Max RSS:                          35.043 MB

RAM needed for this script to just load and do nothing, is ~20MB. When running script using Async coros, it takes ~15MB more space, but Threads takes ~60MB more RAM space. And I have to mention, this test scenario was very simple case, but adding more CPU heavy works and a very deep stack can show worse performance in threads.

Over this series, we’ve traced Python’s memory architecture from the ground up (exploring PyObjects, the distinction between heap and stack, and the inner workings of PyFrames). These foundational concepts reveal why Python behaves the way it does, and more importantly, they illuminate the profound efficiency of generators and coroutines.

In mastering these tools and concepts, we don't just write better Python code, we also learn to think in harmony with underlying software and hardware layers, transforming our understanding of performance from an abstract concern into an intuitive mindset.

Python and Memory Management, Part 2: Heap Memory

Ja'far Khakpour — Sat, 07 Feb 2026 21:08:11 +0000

In previous section, I talked about how objects are stored in memory and how they are stored in memory.
In this part we will talk about memeory regions and how Python allocate/deallocate memory.

Python's Memory Sections: Where Objects Live?

When you run a Python program, the operating system allocates several memory regions. In Linux, you can use this code to see regions of the python script you create (you need to save it as a file and execute it):

#!/usr/bin/env python3
import os, sys

def main():
    pid = os.getpid()
    with open(f"/proc/{pid}/maps") as f:
        for line in f:
            line = line.strip()
            parts = line.split()
            if len(parts) < 5: continue

            s, e = [int(x,16) for x in parts[0].split('-')]
            print(f"{line:70}")

            for name, obj in vars.items():
                if s <= id(obj) < e:
                    print(f"  >>>>> Found: {name} ({id(obj):#x}) in this block")

# Objects to track
vars = {
    "True": True,
    "small number": 4,
    "large number": 2**123,
    "small string": "a", 
    "large string": "a"*2000,
    "int class": int, 
    "sys module": sys,
    "main function": main,
}
if __name__ == "__main__":    
    main()

This code is showing memory maps from /proc//maps. It is showing each memory block and if any of values in vars dictionary are stored in any of these blocks. These memory blocks have different types:

1. Executable Code Region (Read-Only)

This is the code (instruction) section of program, which OS creates a read-only and private (r--p permission) memory block and some of them are executable code (r-xp):

00400000-0041f000 r--p 00000000 08:11 545993 /usr/bin/python3.14
0041f000-006d2000 r-xp 0001f000 08:11 545993 /usr/bin/python3.14
006d2000-00945000 r--p 002d2000 08:11 545993 /usr/bin/python3.14
...
7f3d6c6ab000-7f3d6c6d1000 r--p 00000000 08:11 541316 /usr/lib/x86_64-linux-gnu/libc.so.6
7f3d6c6d1000-7f3d6c827000 r-xp 00026000 08:11 541316 /usr/lib/x86_64-linux-gnu/libc.so.6
...

These sections are related to Python binary code, or libraries (shared objects) used by Python Interpreter.

2. Data Segment (Read-Write Data)

what we see here is a memory block from Python Executabele which contains: True, 4 (immutable number which is initialized in memory when Python starts), and type int. These are part of python executable on the file system:

00946000-00a85000 rw-p 00545000 08:11 545993 /usr/bin/python3.14
  >>>>> Found: True (0x95a0a0) in this block
  >>>>> Found: small number (0xa5bb08) in this block
  >>>>> Found: int class (0x958cc0) in this block

3. Heap Region (Dynamic Allocation)

OS managed heap of the application. Objects allocated with C's malloc are placed here. Generally, object with memory requrement more than 512 bytes are located here.

1039c000-10449000 rw-p 00000000 00:00 0                                  [heap]
  >>>>> Found: large string (0x10406840) in this block

4. Anonymous Memory Mappings (Python's pymalloc)

These include memory blocks used for storing data in RAM called Arena. Arenas have a spceific structure and help manage small objects (<512 bytes) easier than OS managed heap. They are like pre-allocated memory spaces Python reserve for future usage. (For memory management in python, there are some posts like this post which explain memory structure in detail, for Arena, pool based allocation and why it is efficient, look at this series)
You can find a lot of object types here:

7f75e62bf000-7f75e6400000 rw-p 00000000 00:00 0                       
  >>>>> Found: small string (0x7f75e62ec730) in this block
  >>>>> Found: main function (0x7f75e62d85e0) in this block
  >>>>> Found: large number (0x7f75e67e3f30) in this block
  >>>>> Found: sys module (0x7f75e67a2ca0) in this block

5. Stack Region (Function Calls)

Used for C function call stack (not Python objects!). Python objects are never stored here - only C frames and local C variables.

7fff20372000-7fff20394000 rw-p 00000000 00:00 0 [stack]

6. Kernel Interface Regions

These are OS related regions. They allow the application talk to OS faster without need to context switch in CPU, and they are out of this post's scope.

7fff203b8000-7fff203bc000 r--p 00000000 00:00 0 [vvar]
7fff203bc000-7fff203be000 r-xp 00000000 00:00 0 [vdso]

Now we have an idea on how Python allocates memory for a variable. In next section we will talk about how these variables are garbase collected in Python.

How Python Manages Garbage Collection with Reference Counting

We already know that every Python object has a counter tracking how many references point to it. You remember PyObjects from first part of these series:

// CPython's PyObject structure
typedef struct _object {
    Py_ssize_t ob_refcnt;    // Reference count field
    PyTypeObject *ob_type;   // Type pointer
    // ... type-specific data
} PyObject;

Let's play with this concept a little:

import sys
import weakref

class Node:
    name = None
    child = None
    def __init__(self, Name):
        self.name = Name
    def __repr__(self):
        return f"Node: {self.name}"
a = Node('a') # reference count = 1
b = Node('b')
a_ref = weakref.ref(a) # weak reference keeps the reference, but does not increase refcount
b_ref = weakref.ref(b)
a.child = b
b.child = a # reference count = 2

print("All refs a, b, and a_ref asigned", f"{sys.getrefcount(a_ref())=}") # 2 + 1 x argument to getrefcount = 3
del a
print("After varialble a deleted", f"{sys.getrefcount(a_ref())=}") # reference count = 1
del b
print("After both varialble deleted", f"{sys.getrefcount(a_ref())=}") # reference count = 0, object freed. But Refcount is still 1!
print("After both varialble deleted", f"{sys.getrefcount(b_ref())=}") # same results

What is happening here? After deleting objects a and b, we still see a reference pointing to the object? Add these two lines to bottom of above code to see what is still refering to a:

import gc # Garbage Collector Interface Module
print(f"Referres to a: {gc.get_referrers(a_ref())}")

These references are a.child -> b and b.child -> a references. we have deletd a, but reference a.child exists, so b will be alive, and for the same reason b exists in memory.

How does Python handle such cases? Well, Python scans all objects in memory and marks objects as reachable/unreachable. This algorithm starts from some root variables which Python is sure about their existence (like global variables and local variables), and searches for values in memory referred by them to mark as reachable, and repets this lookup for these reachable objects. At the end, every object not tagged as reachable in this algorithm will be garbage collected.

This algorithm is not possible to run after every assignment, or after each refcount becoming zero. So, Python can't run this algorithm after any single assigment in python. Interpreter runs it after reaching a specific amount of assigments as the threshold:

import gc

print(f"GC generations: {gc.get_threshold()}") # Default: (700, 10, 10), but you can change them

There are three numbers:

Genration 0: New objects created in memory and not scanned for garbage collection yet. These objects will be scanned after every 700 assigments in the code
Generation 1: Objects that survived one collection, These objects will be scanned again after every 10 Generation 0 collection
Generation 2: Objects that survived multiple collections, they will be scnanned after every 10 Generation 1 scan (by default) This partitioning is based on Generational Hypothesis: Young objects are more likely to become garbage quickly, and old objects that survive tend to stay alive longer

Now you know why after deleting a and b in the Python code above, Weak reference was still able to access the object, because Mark-and-Sweep algorithm (search algorithm used by garbage collector) was going to start working after 700 assignments, and before this threshold reaches, a.node was still there in memory, and b_ref weak reference was able to access this data in memory.

Try to force garbage collection:

import sys
import weakref

class Node:
    child = None

a = Node()
b = Node()
b_ref = weakref.ref(b)
a.child = b
b.child = a

print("All refs a, b, and a_ref asigned", f"{sys.getrefcount(a_ref())=}") # a_ref() returns value of a, if it's till in memory, or None if it has been deleted
del a
print("After varialble a deleted", f"{sys.getrefcount(a_ref())=}")
del b
print("After both varialble deleted", f"{sys.getrefcount(b_ref())=}")
n_collected = gc.collect(2) # force Python to collect generations 0, 1 and 2 values
print(f"collected {n_collected} objects") # collected 2 objects

The GIL's Role in Garbage Collection

The Global Interpreter Lock (GIL) is crucial for thread-safe garbage collection, particularly for reference counting. Without the GIL, reference counting can cause race condition and memory corruption and break the garbage collection. Actually, GIL guarantees all operations in Python code is atomic (it always has an expected behavior).

I hope you enjoyed reading this post. Now we know how does Python (CPython to be specific) decides to collect memory slot and free it. Python's memory management is a fascinating blend of simplicity (everything is an object) and sophistication (multiple allocators, immortal objects, generational GC). By understanding these internals, you can understand Python code's behavior better than before.

Python and Memory Management, Part 1: Objects

Ja'far Khakpour — Sat, 07 Feb 2026 21:06:02 +0000

You’ve heard it before: in Python, everything is an object. Integers, strings, even the functions you write, all are objects. It’s what gives the language its clean, consistent feel.

But here’s what nobody tells you upfront: this elegant design hides a memory management puzzle that can make your code behave in surprising ways.
To truly grasp how Python handles memory under the hood (and to write faster, more reliable code) you need to move past understanding language and take a look into CPython internal to understand what Python code does under the hood.

First-Class Objects in Python: Everything is an Object

To truly understand how Python manages memory, we need to start from distinction between three fundamental operations, ==, is, and id(). These three tools reveal different layers of Python's object model and memory management.

The equality operator

The == operator checks value equality: whether two objects contain the same data. When you write a == b, Python compares the actual values stored in the objects. This is a deep comparison that might involve calling special methods like eq():

# == compares values
a = [1, 2, 3]
b = [1, 2, 3]
print(a == b) # True - same content

Under the hood, == can trigger more complicated operations. In lists, it compares equality of each element, and for custom objects, it calls custom __eq__ methods.

class SensorValue:
    def __init__(self, value, tolerance):
        self.value = int(value)
        self.tolerance = int(tolerance)

    def __eq__(self, other):
        # Special behavior: equality within tolerance
        return abs(self.value - other.value) <= min(
            self.tolerance,
            other.tolerance
        )

v1 = SensorValue(100, 3)
v2 = SensorValue(97, 5)
v3 = SensorValue(96, 10)

print(f"v1 == v2: {v1 == v2}") # True both have a value in each other's tolerance range
print(f"v1 == v3: {v1 == v3}") # False

The is Operator: Object Identity

The is operator checks object identity - whether two variables reference the exact same thing in memory:

# `is` compares object identity
a = [1, 2, 3]
b = a  # b references the same list object
c = [1, 2, 3]  # New list with same contents

print(a is b)  # True - same object
print(a is c)  # False - same values, but different objects

When you write a is b, Python compares the memory addresses of the objects.
The is operator is lightning fast because it simply compares two pointers (memory addresses). This is why it is better to check None, True, and False with is, not ==. They're singleton objects and is would always do what == can do for us.

The id() Function and Memory Address of Data

The id() function returns the virtual memory address of an object as an integer. This is the raw pointer value that is compares internally:

a = [1, 2, 3]
b = a
print(f"{a is b=}") # True
print(f"{id(a) == id(b)=}")  # True - equivalent to 'a is b'
print(f"{id(a)=}") # inetger value as memory address of a
print(f"id(a)=0x{id(a):x}") ## Hexadecimal value of variable address

We are going to use these addresses throughout this journey through this post to investigate CPython behavior.

Exploring memory Pointers

Let's start with a simple test to understand id() better:

x = 4
y = 2**2  # Both calculate to 4

print(f"{x is y=}")  # True - they're the same object!
print(f"{id(x)=}")
print(f"{id(y)=}") # same as X

Wait, why are x and y the same object? Because Python caches small integers (-5 to 256) as singleton objects to optimize performance. When you create the integer 4, Python doesn't allocate new memory. It returns a reference to the pre-existing "4" object.

Now let's try with larger numbers:

x, y = 2*123, 2**123
print(f"{x is y=}")  # False - different objects!
print(f"{hex(id(x))=}")
print(f"{hex(id(y))=}")

But what happens with assignment?

x = y = 2**123
print(f"{x is y=}")  # True!

print(f"{hex(id(x))=}") 
print(f"{hex(id(y))=}")  # Same address!

Why are they different now? Numbers outside the -5 to 256 range aren't cached. Each calculation creates a new PyLongObject in memory.
When you write x = y = 2**123, Python evaluates 2**123 once, creates one object, and makes both x and y point to it. This is different from calculating the same value twice.

Understanding Reference Counting

At the core of Python’s memory management is reference counting. A simple but powerful mechanism that tracks how many references point to each object in memory. Every time you assign a variable, pass an argument, or store an object in a data structure, Python increments that object’s internal reference counter. When a reference is removed or goes out of scope, the count decreases. Once it reaches zero, the memory is immediately reclaimed.
Let's dive deeper into reference counting mechanism:

import sys
x = y = 2**123
z = 2**123
print(f"{x is y=}")  # True - same object
print(f"{sys.getrefcount(x)=}")  # 3 references
print(f"{sys.getrefcount(y)=}")  # Also 3
print(f"{x is z=}")  # False - different object!
print(f"{sys.getrefcount(z)=}")  # 2 references

Why does x have refcount 3 while z has 2? Let's break it down:

x and y reference the same object (+2)
The getrefcount() call creates a temporary reference (+1) = Total 3
z references a different object (+1)
getrefcount() creates a temporary reference (+1) = Total 2

The Mystery of Small Integer Reference Counts

Try this for small integers, and you'll discover another interesting point:

x = 4
print(f"{sys.getrefcount(x)=}")  # Not 2 or 3, but 3221225472?!

That's not a real reference count! Let's investigate further:

for x in range(5):
    print(f"for {x=}:")
    print(f"    {sys.getrefcount(x)=}")
    print(f"    {id(x)=}")

You'll notice that integers 0-4 all have the same "refcount": 3221225472 which if you convert them to hex and binary format:

x = 4
print(f"{sys.getrefcount(x)=:x}")  # 0xc0000000
print(f"{sys.getrefcount(x)=:b}")  # 11000000000000000000000000000000

Oh! This looks like a binary flag, not a reference count!! Indeed, 0xC0000000 is a special marker value that Python uses for immortal objects.

Immortal Objects: The Python Optimization You Never Knew About

Python has an optimization tweak: certain objects are designated as "immortal" - they live for the entire program's lifetime and never get garbage collected. These include:

Small integers (-5 to 256)
None
True, False
Empty tuples By the way, you can check any value being immortal using sys._is_immortal(Python 3.14+). For these objects, Python doesn't bother with reference counting. Instead, it sets their refcount field to a special flag value (0xC0000000 in many versions) that means "this object never dies."

CPython has also a kind of special behavior when you define a string. It cachces some strings and make them interned object which has a similar behavior to Immortal objects. You can read about them at sys.intern documentation. Just note that Python impliciltly makes some string interned, and this may show some unexpected behavior on is operator for strings.

How Python Objects are Stored in Memory Bytes?

We already know that id() return virtual memory address. In out prevous example of sequence of small numbers, there was another interesting sequence of small integers memeory addresses:

for i in range(5):
    print(f"id({i+1}) - id({i}) = {id(i+1) - id(i)}")

Are these 32 bytes, the allocates memory for a integer in Python?
Lets check size of int objects in memory:

import sys
print(f"{sys.getsizeof(1)=}") #sys.getsizeof(1)=28!!

What are these 28-bytes memory size and 32-byte gaps? id() returns the memory address of the object, and small integers are allocated in a contiguous array, each taking exactly 32 bytes. Looking into CPython documentation, you will find out content of this 32 bytes memory are as:

8 bytes for reference count (Which is the magical 0xC0000000 number in case of small -immortal- numbers)
8 bytes for type pointer (Pointer to type object)
8 bytes for size field (Remember that Python ineger can have any length! it is not just int32 or int64 number!)
4 bytes for the actual digit (This number can grow and allocate more than 32-bits if the number is large)
4 bytes padding for alignment (memory blocks must have a length as multiplication of 8)

The PyObject: Python's Fundamental Building Block

Lets break the above memory block to see what CPython does under the hood. At the C code level, every Python object starts with a PyObject header:

typedef struct _object {
    Py_ssize_t ob_refcnt;    // Reference count (or immortal flag)
    PyTypeObject *ob_type;   // Pointer to type object
} PyObject;

Now, if you take a look into Int object definition, you will see this struct (cpython/Include/cpython/longintrepr.h in CPython source code):

typedef struct _PyLongValue {
    uintptr_t lv_tag; /* Number of digits, sign and flags */
    digit ob_digit[1]; /* Numerical data of integer. Array of unit32 (4-bytes) numbers */
} _PyLongValue;

struct _longobject { /* yes, Integer is called Long Ineteger in C-level API */
    PyObject_HEAD /* PyObject reference */
    _PyLongValue long_value;
};

PyObject part is an important part which memory manager use for retrieval and garbage collection of this object in memory. So, now we know how is an object stored in memory. Let's talk about how is it allocated and accessed.

Python's Garbage Collection: From Refcounting to Generational Garbage Collection

The Foundation: Reference Counting

As mentioned earlier, at the heart of Python's memory management is reference counting. Every Python object has a counter tracking how many references point to this memory block. As discussed in first section:

typedef struct _object {
    Py_ssize_t ob_refcnt;    // Reference count field
    PyTypeObject *ob_type;   // Type pointer
    // ... type-specific data
} PyObject;

You can try it in Python via sys.getrefcount() which shows number of references to value +1 (one for the argument pased to getrefcount(...)):

import sys

x = [1, 2, 3]
print(f"Initial refcount: {sys.getrefcount(x)=}") # 2 (one for x and one for the argument passed to getrefcount)
y = x 
print(f"Assignintng y <- x: {sys.getrefcount(x)=}") # 3
z = x
print(f"Assignintng z <- x: {sys.getrefcount(x)=}") # 4

del y
print(f"De-Assigning y: {sys.getrefcount(x)=}") # 3
z = None
print(f"De-Assigning z: {sys.getrefcount(x)=}") # 2
del x
print("Deleted x")

Refcount allows Python to easily manage the memory. When refrence count of an object reaches 0, Python releases the memory.
But there is an issue:

import sys
import weakref

class Node:
    def __init__(self, child):
        self.child = child

a = Node(None) # reference count = 1
b = Node(None)
a_ref = weakref.ref(a) # weak reference keeps the reference, but does not increase refcount
a.child = b
b.child = a # reference count = 2
a_ref = weakref.ref(a) # reference count = 2

print("All refs a, b, and a_ref asigned", f"{sys.getrefcount(a_ref())=}") # 2 + 1(argument to getrefcount) = 3
del a
print("After varialble A deleted", f"{sys.getrefcount(a_ref())=}") # 2, which means real reference count = 1
del b
print("After varialble B deleted", f"{sys.getrefcount(a_ref())=}") # reference count = 0, object freed. But still accessible!

Circular reference is not possible to handle with a simple refcount. This kind of referencing needs to scan objects in memory and collect deleted circular references.
This kind of garbage collection is expensive and it is not possible to do after any value assignment or dissociation.
Python runs garbage collection process in cycles and different values in memory have different cycles. This means object in memeory will not be removed immediately (this is why we are able to access the memory using a_ref() even though all references are deleted).

The Generational Garbage Collector

Python's solution to circular references is the generational garbage collector. It operates on the principle of generational hypothesis:

Most objects die young (80-90% of objects)
Old objects rarely reference young objects

Python uses three generations:

import gc

print(f"GC generations: {gc.get_threshold()}") # (700, 10, 10) by default

These numbers are for:

Generation 0: New objects created in memory. These objects are collected after every 700 allocations.
Generation 1: Objects that survived one Gen 0 collection. Collected every 10 Gen 0 collections.
Generation 2: Objects that survived multiple collections. Collected every 10 Gen 1 collections.

This is why after deleting all references to variable a, we were able to access through weak reference a_ref: Generation 0 garbage collection was not started yet.