DEV Community: Olanrewaju Mark

Stop Confusing npm and npx: Here's What Every Developer Should Know

Olanrewaju Mark — Wed, 19 Aug 2026 18:34:22 +0000

npm and npx: A Simple Guide for Everyone

If you have ever worked with JavaScript or Node.js, you have probably seen the words "npm" and "npx" many times. They look similar, but they do different things. This article will explain both in simple terms, so anyone can understand them, even if you are new to coding.

What is npm?

npm stands for Node Package Manager. It comes installed automatically when you install Node.js on your computer.

Think of npm as a store. In this store, you can find small pieces of code called packages (or libraries) that other developers have already written. Instead of writing everything from scratch, you can download these packages and use them in your own project.

For example, if you need a package to format dates in JavaScript, you do not have to write that code yourself. Someone has probably already built it and shared it on npm. You just download it and use it.

npm has three main parts:

A website (npmjs.com) where you can search for packages.
A command line tool that lets you install and manage packages from your terminal.
A registry, which is a huge online database that stores all these packages.

Why do developers use npm?

Here are the main reasons:

It saves time. You do not need to build every tool or function from zero.
It keeps projects organized. npm tracks every package your project uses, along with the exact version.
It makes sharing easy. If you build something useful, you can publish it on npm so others can use it too.
It handles updates. npm can help you update packages when new versions come out.

Getting started with npm

To check if npm is installed on your computer, open your terminal and type:

npm -v

This will show you the version number. If you see a version, npm is ready to use.

Starting a new project

When you start a new project, you usually run:

npm init

This command asks you a few questions (like project name and version) and then creates a file called package.json. This file is very important. It works like a summary sheet for your project. It lists the project's name, version, and all the packages it depends on.

If you want to skip the questions and just create the file with default values, use:

npm init -y

Installing a package

To add a package to your project, use:

npm install package-name

For example, to install a popular package called axios (used for making web requests), you would type:

npm install axios

After running this, two things happen:

A folder called node_modules is created. This folder holds the actual code of the packages you installed.
Your package.json file is updated to list axios as a dependency.

You will also see a file called package-lock.json. This file locks the exact versions of every package (and their smaller dependencies) so that your project works the same way on any computer.

Installing all packages for a project

If you download a project that already has a package.json file but no node_modules folder, you can install everything it needs with just:

npm install

npm will read the package.json file and download all the listed packages automatically.

Removing a package

If you no longer need a package, you can remove it with:

npm uninstall package-name

Updating packages

To update your packages to newer versions, you can run:

npm update

What is npx?

Now let us talk about npx. npx stands for Node Package Execute. It comes bundled with npm, so if you have npm, you already have npx too.

npx solves a different problem. Sometimes you do not want to install a package permanently. You just want to run it one time.

Without npx, if you wanted to use a tool like create-react-app (a tool that sets up a new React project for you), you would need to install it globally on your computer first:

npm install -g create-react-app

This has some downsides:

It takes up space on your computer.
The tool can become outdated if you do not update it often.
You end up with many global packages you rarely use.

With npx, you can run the same tool without installing it permanently:

npx create-react-app my-app

npx will download the tool, run it one time, and then not keep it installed globally. This keeps your computer clean and makes sure you are always using the latest version of the tool.

When should you use npx?

Here are common situations where npx is useful:

Running a tool once, like generating a new project (npx create-react-app, npx create-next-app).
Testing a package before deciding to install it in your project.
Running a script that is already inside your project's node_modules, without typing the full path.

For example, if your project has a package called jest (used for testing) installed locally, you can run it with:

npx jest

This works even though jest was not installed globally on your system. npx knows to look inside your project's node_modules folder first.

npm vs npx: The Simple Difference

npm	npx
Installs and manages packages	Runs packages without installing them permanently
Used to build and set up your project	Used to quickly execute a tool or command
Packages stay on your computer after install	Packages are often removed after use
Example: `npm install axios`	Example: `npx create-react-app my-app`

A simple way to remember it: npm is for keeping, npx is for running.

A Quick Example to Tie It Together

Imagine you want to start a new Node.js project and add a package for handling dates.

# Step 1: Create a new project folder and move into it
mkdir my-project
cd my-project

# Step 2: Set up the project with a package.json file
npm init -y

# Step 3: Install a package to handle dates
npm install dayjs

# Step 4: Use npx to quickly check your project for outdated packages
npx npm-check-updates

In this example, npm was used to set up the project and install a package we plan to use often. npx was used to run a helpful tool one time, without adding it permanently to the project.

Common Mistakes to Avoid

Deleting node_modules without reason. If something goes wrong, you can safely delete this folder and run npm install again to rebuild it.
Ignoring package-lock.json. This file should be kept and shared with your team. It keeps everyone using the same package versions.
Installing everything globally. Try to use npx for one-time tools instead of installing many global packages.
Not checking package trust. Before installing a package, check how popular and well-maintained it is on npmjs.com.

Final Thoughts

npm and npx work together to make JavaScript development easier. npm helps you manage the packages your project depends on, while npx helps you run tools quickly without cluttering your system. Once you understand the difference, you will find it much easier to set up projects, use helpful tools, and keep your work organized.

The Celery Lifecycle: How a Task Gets Registered, Queued, and Run

Olanrewaju Mark — Tue, 11 Aug 2026 21:29:11 +0000

If you have ever needed to send an email, process a payment, or generate a report without making your user wait, you have probably run into Celery. Celery is a tool that lets you run jobs in the background, away from your main app. This article breaks down how it works, step by step, in plain language.

What Is Celery, In Simple Terms

Think of Celery like a restaurant kitchen.

Your app (the waiter) takes an order from a customer.
Instead of cooking the food itself, the waiter drops the order into a queue (the kitchen order rail).
A cook (the worker) picks up the order from the rail and prepares it.
When the food is ready, it goes to a pickup counter (the result backend) where anyone can come check if it's done.

Celery has four main players:

The Producer - your app, the one that creates tasks.
The Broker - the message queue that holds tasks until a worker is free.
The Worker - the process that picks up and runs the tasks.
The Result Backend - where results are stored, if you need them later.

In short: your app sends a task message to the broker. The broker holds it until a worker is free. The worker picks it up, runs the actual function, and (if you set one up) writes the result to the result backend. Your app can then go back and check that result backend to see what happened.

Now let's go through each part.

1. How Tasks Get Registered

Before Celery can run a task, it needs to know the task exists. This is called registration, and it happens the moment your Python code is imported - not when the task runs.

The `@app.task` decorator

You create a Celery app instance, then decorate any function with @app.task. That decorator does not run the function immediately. Instead, it wraps the function and adds it to a task registry - basically a dictionary that Celery keeps internally, mapping a task name to the actual function.

from celery import Celery

app = Celery("myproject")

@app.task
def send_welcome_email(user_id):
    # logic to send an email
    print(f"Sending welcome email to user {user_id}")

The moment Python imports this file, send_welcome_email gets registered under the name myproject.tasks.send_welcome_email (module path + function name, by default).

Why registration matters

Here is the key thing people miss: the producer and the worker are often two separate processes, sometimes on two separate machines. When your app calls send_welcome_email.delay(5), it does NOT run the function. It just creates a message like this:

{
  "task": "myproject.tasks.send_welcome_email",
  "args": [5],
  "kwargs": {},
  "id": "a1b2c3d4-uuid"
}

That message is just a name and some arguments - a string, basically. The actual Python function only needs to exist on the worker's side, because the worker is the one that looks up the name in its own registry and calls the real function.

This is why, if a task is not registered on the worker (say, you forgot to import that module in the worker's app), you will get a KeyError or NotRegistered error even if the producer sent the message fine. The message got sent, but nobody on the worker side knew what myproject.tasks.send_welcome_email meant.

Autodiscovery

In bigger apps (like Django), you don't manually import every task file. Celery gives you app.autodiscover_tasks(), which scans your installed apps for a tasks.py file and imports them automatically, registering every @app.task function it finds inside.

So the registration flow, in words, goes like this: Python imports the file that contains your task → the @app.task decorator fires → the function's name and a reference to it get added to the Celery app's internal registry → from that point on, the app knows this task exists and can hand it work whenever it's called.

2. The Broker URL - The Middleman

The broker is a message queue. Its only job is to hold task messages until a worker is ready to grab one. Celery does not process anything itself - it just passes messages through the broker.

You configure it with a single connection string, the broker URL:

app = Celery(
    "myproject",
    broker="redis://localhost:6379/0"
)

Common broker choices:

Broker	Example URL	Notes
Redis	`redis://localhost:6379/0`	Fast, simple, good for most small-to-medium apps
RabbitMQ	`amqp://guest:guest@localhost:5672//`	More robust, built for messaging, heavier to run
Amazon SQS	`sqs://`	Managed, good if you're already on AWS

What the broker URL actually controls

The broker URL tells Celery:

Which service to talk to (Redis, RabbitMQ, etc.)
Where it lives (host and port)
How to authenticate (username/password, if needed)
Which database/vhost to use (the /0 at the end of a Redis URL picks database 0, for example)

Nothing more. The broker does not know what a task "means." It just stores and forwards messages, like a mailbox. This is why Celery can swap Redis for RabbitMQ without changing a single line of your task code - only the broker URL changes.

Picture it this way: your app pushes a task message into the broker's queue. The broker just sits there holding it. On the other side, one or more workers are constantly listening to that same queue. Whichever worker is free at that moment grabs the next message in line. If you have five workers all pointed at the same broker URL, they are all pulling from the same line - so Celery scales horizontally just by adding more workers behind the same broker.

3. Execution - What Happens When a Task Runs

Once a worker pulls a task message off the broker, here is what happens, step by step:

Deserialize - the worker reads the JSON (or whatever serializer you're using) message and pulls out the task name, args, and kwargs.
Lookup - it checks its local task registry for a function matching that name.
Acknowledge (ack) - by default, Celery acknowledges the message either right before or right after running it, telling the broker "I've got this, you can remove it from the queue." This matters for reliability - if a worker crashes before acking, the broker can redeliver the task to another worker.
Run - the actual Python function executes, using the args/kwargs from the message.
Handle outcome - if it succeeds, the return value is captured. If it raises an exception, Celery marks the task as FAILURE and can optionally retry it, depending on your task's retry settings.
Store result - if a result backend is configured, the worker writes the outcome there.

Put in a single sentence: the broker hands a message to the worker, the worker matches that message to a real function in its registry, runs the function, tells the broker the message is done (so it disappears from the queue), and then, separately, writes whatever the function returned into the result backend.

Concurrency - how one worker runs many tasks

A single worker process does not run tasks one at a time by default. It has a pool of child processes or threads (the default is prefork, which uses multiple OS processes). This is controlled by the --concurrency flag when you start a worker:

celery -A myproject worker --loglevel=info --concurrency=4

This starts 4 child processes, each capable of running a task independently. So if 4 tasks arrive at once, all 4 can run in parallel, not one after another.

4. The Result Backend - Where Answers Go

Not every task needs a result backend. If you just want to fire off an email and forget about it, you don't need one. But if you need to know "did the task succeed?" or "what did it return?", you need a result backend.

app = Celery(
    "myproject",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1"
)

Notice the backend can be a different database index than the broker - /1 instead of /0. You can even use a completely different technology, like storing results in Postgres while your broker stays on Redis.

How you actually use it

result = send_welcome_email.delay(5)

result.status      # 'PENDING', 'STARTED', 'SUCCESS', 'FAILURE'
result.result      # the return value, once it's done
result.get(timeout=10)  # blocks and waits for the result (use carefully)

Behind the scenes, result.status is just Celery checking a key in the result backend (e.g., a Redis key like celery-task-meta-a1b2c3d4) to see what the worker wrote there.

Common mistake: confusing broker and backend

A lot of beginners think the broker stores results. It does not. Once a worker acks a message, the broker deletes it - it has no memory of what happened to that task. The result backend is a completely separate storage system whose only job is to hold outcomes, so your app can look them up later.

	Broker	Result Backend
Purpose	Deliver task messages to workers	Store task outcomes
Lifespan of data	Deleted once task is picked up (acked)	Kept until it expires (`result_expires`) or you check it
Required?	Yes, always	Only if you need to check status/results
Common choice	Redis, RabbitMQ	Redis, Postgres, MongoDB

Putting It All Together

Here is the complete lifecycle of a single task, in order:

You define a task with @app.task. As soon as that file is imported, the task gets added to Celery's internal registry.
Your app calls task.delay(...). Celery serializes the task name plus its arguments into a message - it does not run the function yet.
That message is pushed into the broker's queue, using whatever broker URL you configured.
A free worker, listening on that same broker, pulls the message off the queue.
The worker looks up the task name in its own registry to find the matching function.
The worker runs the real function, using the arguments from the message.
Once the function finishes (or fails), the worker acknowledges the message, so the broker removes it from the queue.
If a result backend is configured, the worker writes the outcome (success, failure, return value) there.
Your app, at any point, can check the result backend by task ID to see the status or grab the return value.

A Quick Practical Example

Since you already work with FastAPI, here's a minimal setup showing all four pieces together:

# celery_app.py
from celery import Celery

app = Celery(
    "reddo_tasks",
    broker="redis://localhost:6379/0",   # where tasks are queued
    backend="redis://localhost:6379/1",  # where results are stored
)

@app.task(bind=True, max_retries=3)
def process_payment_webhook(self, payload: dict):
    try:
        # heavy work: verify signature, update DB, notify user
        return {"status": "processed", "reference": payload.get("reference")}
    except Exception as exc:
        raise self.retry(exc=exc, countdown=10)

# main.py (FastAPI)
from celery_app import process_payment_webhook

@app.post("/webhook")
def handle_webhook(payload: dict):
    task = process_payment_webhook.delay(payload)
    return {"task_id": task.id}

# check status later
from celery.result import AsyncResult
from celery_app import app

def check_task(task_id: str):
    result = AsyncResult(task_id, app=app)
    return {"status": result.status, "result": result.result}

Run the worker separately:

celery -A celery_app worker --loglevel=info --concurrency=4

Summary

Registration happens at import time - @app.task adds the function to an internal registry, and the worker needs that same code imported to run it.
The broker URL tells Celery where the message queue lives. It only moves messages; it does not know what tasks mean or remember their results.
Execution happens inside worker processes, which pull messages, look up the function, run it, and acknowledge the message so it's removed from the queue.
The result backend is a separate store for outcomes. It's optional, and it's how your app checks status or gets return values after the fact.

Once these four pieces click, the rest of Celery - retries, chains, periodic tasks (Celery Beat), routing - is just extra features built on top of this same basic flow.

Offset vs Cursor Pagination: The Mistake Most Backend Engineers Make

Olanrewaju Mark — Sun, 02 Aug 2026 18:44:46 +0000

Your API works perfectly today.

Then six months later, your transaction table grows from 10,000 records to 10million records.

Suddenly, requests that once took 80ms now takes 4seconds.

The question is:

Did you choose the right pagination strategy?

Most backend developers are familiar with pagination, but many don't realize that there are different pagination strategies, and choosing the wrong one can seriously affects your application's performance as data grows.

In this articles, we'll compare the two most common approaches:

Offset Pagination
Cursor Pagination

We'll look at how they work, their advantages and disadvantages, when to use each one, and how companies handling millions of records approach this problem.

What is Pagination?

Pagination is simply the process of breaking a large dataset into smaller chunks.

Imagine your database contains 5 million transactions.

Returning all 5 million records in a single API response would:

Consume excessive memory
Slow down the database
Increase API response time
Create a poor user experience

Instead, we return only a small subset.

For example:

10 records per request

The client then requests additional records when needed.

The question is...

How should the next set of records be fetched?

There are two common answers.

1. Offset Pagination

Offset pagination is probably the first pagination technique every backend developer learns.
A request usually looks like this:

GET /transactions?page=3&limit=10

GET /transactions?offset=20&limit=10

The database query looks something like:

SELECT *
FROM transactions
ORDER BY created_at DESC
LIMIT 10
OFFSET 20;

Here, the database skips the first 20 rows before returning the next 10.

Visual Representation

Suppose we have these transaction IDs:

with

Page = 1
Limit = 5

You get:

Page 2 skips those records and returns:

Simple, Easy and Readable.

Advantage of Offset Pagination

→ Very easy to implement
Almost every framework supports it.

→ Easy to jump to any page
Want page 50?
No problem

?page=50

→ Great for admin dashboards

This is why most admin panels use offset pagination.

Users often want to jump directly to page 8 or page 30

The problem with Offset Pagination

Imagine this query:
OFFSET 5000000

The database still has to scan and skip five million rows before returning the next 10.

That means:

 Skip
 Skip
 Skip
 Skip
 Return 10 rows.

As the table grows larger, this becomes increasingly expensive.

Large offsets lead to slower queries.

Another issue appears when new records are inserted while users are browsing.

Imagine:

Page 1

A new transaction arrives:

101

Now Page 2 may suddenly contain duplicate or missing records because the dataset shifted between requests.

2. Cursor Pagination

Cursor pagination solves this differently.
Instead of saying:

"Skip 500 rows."

It says:

"Continue from the last record I already received."

A request looks like:

GET /transactions?limit=10

The API returns:

{
    "data": [....],
    "next_cursor": "2026-07-31T16:06:30"
}

the client now sends:

GET /transactions?cursor=2026-07-31T16:06:30

The SQL becomes:

SELECT *
FROM transactions
WHERE created_at < '2026-07-31T16:06:30'
ORDER BY created_at DESC
LIMIT 10;

Notice something.

The database isn't skipping millions of rows.

It's jumping directly to the next location using an indexed column.

That's significantly faster.

Production APIs usually Fetch One Extra Record

One interesting trick used in production systems is:

Instead of fetching:

LIMIT 10

The backend fetches:

LIMIT 11

Why?

To determine whether another page exists.

If 11 records are returned:

Return only the first 10
Use the 10th record as the cursor
Set has_next = true

If only 10 records are returned:

has_next = false
next_cursor = null

The frontend now knows there are no more records to load.

Advantage of Cursor Pagination

→ Extremely fast on very large datasets

→ Doesn't become slower over time
Whether your table has:

10,000 rows
10 million rows
500 million rows Performance remains consistent.

→ Prevent duplicate and skipped records

Since the cursor always continues from the last item returned, new inserts don't shift pages in the same way they do with offsets.

FastApi code example

@transaction_router.get(
    "/get_transactions", status_code=status.HTTP_200_OK
)
async def get_transactions(
        limit: Optional[int] = Query(10, ge=1, le=100),
        cursor: Optional[datetime] = Query(None),
        db: Session = Depends(get_db)
):
    limit_offset = limit + 1
    transactions = db.query(TransactionHistory)
    if cursor:
        transactions = transactions.filter(TransactionHistory.created_at < cursor)

    transactions = transactions.order_by(TransactionHistory.created_at.desc()).limit(limit_offset).all()

    if len(transactions) > limit:
        has_next = True
        transactions = transactions[:-1]
        next_cursor = transactions[-1].created_at
    else:
        has_next = False
        next_cursor = None

    return return_response(
        StatRes.SUCCESS,
        "Transactions fetched successfully",
        {
            "transactions": [TransactionResponse.model_validate(t) for t in transactions],
            "next_cursor": next_cursor,
            "has_next":has_next
        }
    )

The Downside

Cursor pagination has one major limitation.

You cannot jump directly to page 75.

The client only knows:

"Give me the next records."

This makes cursor pagination perfect for:

Infinite scrolling
"Load More" buttons
Mobile applications
Social media feeds
Banking transaction history
Notifications
Activity logs

So Which One Should You Use?
The answer is...

It depends on how your users consume the data.

Use Offset Pagination when:

Building admin dashboards
Users need page numbers
Users jump between pages
Total pages and total records are required

Examples:

User management
Product inventory
Employee records
Reports
Admin portals

Use Cursor Pagination when:

Building APIs with huge datasets
Users simply scroll
Performance is critical
Consistency is important
New records are constantly being inserted

Examples:

Banking transaction history
Noitifications
Chat messages
Social media feeds
Activity logs
Mobile applications

Can Both Exists in the Same System?

Absolutely.

This is something many engineers overlook.
Imagine you're building a fintech platform.

Admin Dashboard

The operations team wants:

Page numbers
Total records
Jump to page 18 Offset pagination is the better choice.

Mobile App
Customers simply scroll through transaction history.
They don't care about page 23.
They only want:

"Show me more"

Cursor pagination is the better choice.

The same backend can expose different endpoints or different pagination modes - for different clients.

Final Thoughts

One of the biggest lessons I've learned as a backend engineer is that good API design isn't just about making things work, it's about making them scale.

Offset pagination isn't bad.

Cursor pagination isn't automatically better.
Each solves a different problem.

The best engineers don't ask:

"Which pagination strategy is the best?"

They ask:

"Which pagination strategy best fits how my users interact with this data?"

That shift in thinking is what separates writing APIs that work today from designing APIs that continue to perform as your product grows.

How do you handle pagination in your projects?

Do you stick with offset pagination, prefer cursor pagination, or support both depending on the use case?

I'd love to hear your experience in the comments.

Building a Secure Django REST API from Scratch - DjangGuard

Olanrewaju Mark — Fri, 06 Mar 2026 15:43:52 +0000

Most Django tutorials teach you how to build an API. Very few teach you how to build a secure one.

In this article, I'll walk you through DjangGuard - a Django REST API boilerplate I built that comes with security built in from the start. Not as an afterthought.

By the end, you'll understand how to implement:

JWT authentication with device binding
Redis-backed token blacklisting
Brute-force login protection
Global and endpoint-level rate limiting
Argon2 password hashing
User agent validation middleware

Let's get into it.

Why Security Needs to Come First

When you build an API, it's easy to focus on features. But the moment your API goes live, it becomes a target.

People will try to:

Guess passwords over and over
Steal tokens and use them from different devices
Flood your endpoints with requests
Reuse tokens after a user has logged out

DjangGuard addresses all of these - and it's designed to be a starting point you can build any project on top of.

The Tech Stack

Before we dive in, here's what we're working with:

What	Technology
Framework	Django 5.2 + Django REST Framework
Auth	SimpleJWT + PyJWT
Brute-force protection	django-axes
Rate limiting	django-ratelimit
Caching & token storage	Redis
Password hashing	Argon2
Config management	python-dotenv

Project Structure

backend/
├── api_services/         # Shared utilities (response helper, Redis, logger, etc.)
├── authentication/       # Login, register, logout
├── exception_handler/    # Custom DRF error responses
├── guard/                # Django settings and root URLs
├── middleWares/
│   ├── authenticate/     # User agent validation
│   └── rateLimit/        # Global rate limiting
├── users/                # User profile
└── requirements.txt

Clean and modular. Each responsibility lives in its own place.

1. Custom User Model

The first thing we do is replace Django's default user model. We don't need a username - we use email instead.

class User(AbstractUser, BaseModel):
    username = None
    email = models.EmailField(unique=True)
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    password = models.TextField()
    is_deleted = models.BooleanField(default=False)

    USERNAME_FIELD = "email"
    REQUIRED_FIELDS = []

    objects = CustomUserManager()

We also have a BaseModel that gives every model a hex UUID primary key and created_at / updated_at timestamps:

class BaseModel(models.Model):
    id = models.CharField(primary_key=True, max_length=32, default=hex_uuid, editable=False)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

Using hex UUIDs instead of auto-incrementing integers means your IDs don't leak information about how many users you have.

2. JWT with User Agent Binding

Standard JWT authentication has a problem: if someone steals a token, they can use it from anywhere.

DjangGuard solves this by embedding the user's User-Agent header as a custom claim inside the token at login time.

def get_tokens_for_user(user, extra_claims=None):
    token = AccessToken.for_user(user)
    if extra_claims:
        for key, value in extra_claims.items():
            token[key] = value
    ...

When the user logs in, we capture their User-Agent and bake it into the token:

user_agent = serializer.validated_data.get("user_agent")
token_dict = get_tokens_for_user(user, {"user_agent": user_agent})

Now every token is tied to the device that created it.

3. User Agent Validation Middleware

On every request, our middleware checks if the token's user_agent claim matches the current request's User-Agent.

class UserAgentValidationMiddleware:
    def __call__(self, request):
        auth_header = request.META.get("HTTP_AUTHORIZATION")

        if auth_header:
            token = self.extract_token(auth_header)
            validation_result = self.validate_user_agent(token, request)
            if validation_result is not True:
                return validation_result

        return self.get_response(request)

If there's a mismatch, the token is immediately blacklisted and the request is rejected:

if token_user_agent != current_user_agent:
    redis_service.set(unverified_jti, "blacklisted", expire=timedelta(days=1))
    return JsonResponse(
        {"detail": "User agent mismatch", "message": "Token was issued for a different device/browser"},
        status=401,
    )

This means even if a token gets stolen, it's useless on any device other than the original one.

4. Redis Token Blacklisting

When a user logs out, we blacklist their token's jti (JWT ID) in Redis:

class LogoutView(APIView):
    def post(self, request):
        ...
        payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
        jti = payload.get("jti")
        redis_service.set(jti, "blacklisted", expire=timedelta(days=1))
        return returned_response("success", "Logout successful", status.HTTP_200_OK)

Before any authenticated request is processed, the middleware checks Redis for that jti:

has_been_blacklisted = redis_service.get(unverified_jti)
if has_been_blacklisted and has_been_blacklisted.decode("utf-8") == "blacklisted":
    return JsonResponse({"message": "Token has been blacklisted"}, status=401)

The token expires from Redis automatically after 1 day - matching the token's own lifetime. No cleanup needed.

5. Single Active Session

When a user logs in, we store a mapping of user_id → jti in Redis:

redis_service.set(user.id, token_dict[1], expire=timedelta(days=1))

On each request, the middleware checks if the jti in the token matches what Redis has stored for that user:

is_accessible = redis_service.get(user_id)
if is_accessible and is_accessible.decode("utf-8") != jti:
    return JsonResponse({"message": "Invalid/Revoked Token"}, status=401)

If the user logs in from a new device, the old token becomes invalid. This gives you single active session behavior out of the box.

6. Brute-Force Protection with django-axes

Django-axes tracks failed login attempts and locks accounts automatically.

In settings.py:

AXES_FAILURE_LIMIT = 5       # Lock after 5 failed attempts
AXES_COOLOFF_TIME = 1        # Lockout lasts 1 hour
AXES_RESET_ON_SUCCESS = True # Reset counter on successful login

And we add the axes middleware and backend:

MIDDLEWARE = [
    ...
    "axes.middleware.AxesMiddleware",
    ...
]

AUTHENTICATION_BACKENDS = [
    "axes.backends.AxesBackend",
    "django.contrib.auth.backends.ModelBackend",
]

Axes handles the rest. If someone tries the wrong password 5 times, they're locked out for an hour.

7. Rate Limiting

We have two layers of rate limiting.

Global middleware limits all traffic - 100 requests/hour per IP (unauthenticated) or per user ID (authenticated):

class GlobalRateLimitMiddleware:
    def __call__(self, request):
        if request.user.is_authenticated:
            rate = "100/h"
            key_func = self.get_user_id_key
        else:
            rate = "100/h"
            key_func = self.get_ip_key

        limited = is_ratelimited(request=request, group="global_rate_limit",
                                  rate=rate, key=key_func, method="ALL", increment=True)
        if limited:
            raise Ratelimited()
        ...

Endpoint-level limits on login and register to stop credential stuffing:

@method_decorator(
    [
        ratelimit(key="post:email", rate="5/m", method="POST", block=True),
        ratelimit(key="ip", rate="10/m", method="POST", block=True),
    ],
    name="post",
)
class LoginView(APIView):
    ...

When a limit is hit, the custom exception handler returns a clean 429 response:

if isinstance(exc, Ratelimited):
    return returned_response(
        "failed",
        status=status.HTTP_429_TOO_MANY_REQUESTS,
        message="Slow down a bit!, You are making too many requests.",
    )

8. Argon2 Password Hashing

Argon2 is the strongest password hashing algorithm available today. It won the Password Hashing Competition and is recommended by OWASP.

In settings.py, we set it as the default:

PASSWORD_HASHERS = [
    "django.contrib.auth.hashers.Argon2PasswordHasher",
    "django.contrib.auth.hashers.PBKDF2PasswordHasher",
    "django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher",
]

Django tries each hasher in order. All new passwords use Argon2.

We also enforce strong passwords:

AUTH_PASSWORD_VALIDATORS = [
    {"NAME": "...UserAttributeSimilarityValidator"},
    {"NAME": "...MinimumLengthValidator", "OPTIONS": {"min_length": 12}},
    {"NAME": "...CommonPasswordValidator"},
    {"NAME": "...NumericPasswordValidator"},
]

Minimum 12 characters. No common passwords. No all-numeric passwords.

9. Consistent API Responses

Every response in DjangGuard follows the same structure:

{
  "status": "success",
  "message": "Login successful",
  "data": {
    "access": "<token>"
  }
}

This is handled by a single returned_response helper used everywhere. No more inconsistent error formats across different views. Your frontend always knows what shape to expect.

10. Redis Caching for Profile Data

The user profile endpoint caches the response in Redis for 1 hour to avoid hitting the database on every request:

def get(self, request):
    user = request.user
    key = f"user_profile:{user.id}"
    user_dict = cache.get(key)

    if user_dict:
        return returned_response("success", "User profile retrieved successfully",
                                  status.HTTP_200_OK, user_dict)

    user_dict = {
        "first_name": user.first_name,
        "last_name": user.last_name,
        "email": user.email,
        "date_joined": user.created_at,
    }
    cache.set(key, user_dict, timeout=60 * 60)
    return returned_response("success", "User profile retrieved successfully",
                              status.HTTP_200_OK, user_dict)

Simple but effective. As your user base grows, this matters a lot.

The API Endpoints

Here's a summary of all available endpoints:

Method	Endpoint	Description	Auth Required
`POST`	`/api/auth/register`	Create a new account	No
`POST`	`/api/auth/login`	Login and get a JWT token	No
`POST`	`/api/auth/logout`	Invalidate your token	Yes
`GET`	`/api/users/me`	Get your profile	Yes

Getting Started

1. Clone and install

git clone https://github.com/your-username/DjangGuard.git
cd DjangGuard/backend
pip install -r requirements.txt

2. Create your .env file

SECRET_KEY=your-secret-key-here
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_DB=0
CACHE_REDIS_URL=redis://127.0.0.1:6379/0

3. Run migrations and start the server

python manage.py migrate
python manage.py runserver

That's it. Your hardened API is running.

What You Can Add Next

DjangGuard is a starting point, not a finished product. Here are some things you can layer on top:

Email verification on registration
Password reset via email
Refresh token rotation
Swap SQLite for PostgreSQL before going to production
Docker + docker-compose for easier deployment
Two-factor authentication (2FA)

Final Thoughts

Security isn't something you add at the end. It's something you design from day one.

DjangGuard brings together JWT binding, Redis blacklisting, brute-force protection, rate limiting, and strong hashing - all in one clean codebase you can start a real project with.

If this was helpful, give the repo a ⭐ on GitHub.

engrmarkk / DjangGuard

DjangGuard

A security-focused Django REST API boilerplate with built-in authentication, rate limiting, JWT token management, and Redis-backed caching. Designed to serve as a hardened starting point for building production-ready APIs.

Overview

DjangGuard is a batteries-included Django REST Framework (DRF) backend template built with security at its core. It combines JWT-based authentication, brute-force protection, user-agent binding, Redis-backed token blacklisting, and global rate limiting — all wired together and ready to extend.

Features

JWT Authentication via djangorestframework-simplejwt with custom claims (user agent binding)
Token Blacklisting using Redis — tokens are invalidated on logout and revoked on device mismatch
Brute-Force Protection via django-axes — accounts are locked after 5 failed login attempts (1-hour cooldown)
Global Rate Limiting — 100 requests/hour per IP (unauthenticated) or…

View on GitHub

and share it with someone learning Django.

Have questions or suggestions? Drop them in the comments - I read every one.

Building MatchMetric-AI: A Real-Time Resume Tailoring Engine with Django, WebSockets, and LLMs

Olanrewaju Mark — Tue, 17 Feb 2026 14:45:07 +0000

Applying for jobs is hard. Every resume needs to match the job description perfectly. To solve this, I built MatchMetric-AI, a tool that uses Artificial Intelligence to analyze resumes and give instant feedback.

What does this project do?

Reads PDFs: It extracts text from your resume using the PyPDF library.
AI Analysis: it sends that text to Google Gemini AI to see how well you match a job.
Live Updates: It uses WebSockets so you don't have to refresh the page to see results.
History: It saves your past scans so you can see your progress.

The Tech Stack

Django: The main framework.
Channels & Redis: To make the "Live" WebSocket connection work.
Google Gemini API: The "brain" that reads the resume.
PyPDF: To turn PDF files into plain text.

Here is a deep dive into how I built it.

The Core Vision MatchMetric-AI isn’t just a simple wrapper for an LLM. It’s a full-cycle service that:

Extracts raw text from uploaded PDF resumes.
Analyzes the resume against a specific Job Description.
Streams feedback back to the user instantly using WebSockets.
Persists the history so users can track their application improvements.

Technical Deep Dive

A. Modular Project Structure
I followed a "Service-Oriented" architecture to keep the project clean. Instead of bloating the Django views.py, I separated the logic into distinct modules:

ai/google_genai: Handles all communication with Gemini.
pdf_extract/pypdf_extractor: Contains the logic for parsing PDFs.
apis/resumehistory: Manages the WebSocket consumers and database history.

B. PDF Extraction with PyPDF
One of the first hurdles was converting complex PDF layouts into clean text that an AI can understand. I utilized PyPDF to iterate through document pages and clean the whitespace to ensure the AI prompt doesn't get cluttered with "junk" characters.

from pypdf import PdfReader


class PyPDFExtractor:
    def __init__(self, file_obj):
        self.reader = PdfReader(file_obj)

    def extract_text(self) -> str:
        text = ""
        for page in self.reader.pages:
            page_text = page.extract_text()
            if page_text:
                text += page_text + "\n"
        return text

C. Real-time Communication with Django Channels
Unlike standard HTTP where the user has to wait for a long AI processing time, I implemented WebSockets. This allows the server to maintain an open connection with the client.

The ResumeConsumer handles the connection. Because Gemini’s processing is synchronous, I used sync_to_async to ensure the server remains responsive while the AI "thinks."

D. Solving the ASGI "Handshake" Challenge
The most difficult part of the project was configuring the asgi.py to handle both standard HTTP (for logins) and WebSockets (for AI analysis) simultaneously. I had to ensure that the AuthMiddlewareStack was correctly positioned to allow my WebSocket to "see" the logged-in

application = ProtocolTypeRouter({
    "http": get_asgi_application(),
    "websocket": AuthMiddlewareStack(
        URLRouter(websocket_urlpatterns)
    ),
})

How to Run the Project

Prerequisites
Redis (for the message broker)

Google Gemini API Key (you can get it here: https://aistudio.google.com/)

Installation

Clone & Install, the repo url: https://github.com/engrmarkk/MatchMetric-AI:

pip install -r requirements.txt

Environment Variables: Configure your .env with GOOGLE_API_KEY and REDIS_URL and other variables in the env.example file.
Migrations:

python manage.py migrate

Launch:

chmod +x runapp.sh
./runapp.sh

Testing with Postman

Since this uses WebSockets, you can't test it like a normal website with a frontend library consuming the api.

Login to Django admin with a superuser credential, inspect the browser page and get the sessionid from cookies in the application tab.
Open Postman and start a WebSocket Request.
Go to the headers and set Cookie as the key and sessionid=
Connect to ws://127.0.0.1:8000/ws/resume-tailor/.
Send a JSON message with your resume text and the job description. You need to access the upload endpoint to extract the text from uploaded resume(.pdf).
Watch the AI response appear in real-time!

What I Learned

The biggest challenge was Authentication. I had to make sure the WebSocket knew which user was logged in. I learned how to use Django's sessionid cookies to securely connect the user to their history.

Django 6 Released: Here's How to Use Its Game-Changing Task Feature

Olanrewaju Mark — Mon, 08 Dec 2025 09:21:24 +0000

Attention Django Developers: Exciting developments have just been announced: Django 6.0 has been released, introducing a highly anticipated feature: built-in task queues.

Eliminate the need for third-party Celery setups and complex Redis configurations. Django now seamlessly integrates task management directly within the framework.
Let me show you what this looks like and how you can use it today.

What Are Django Tasks?

Think of Django Tasks as Django's answer to Celery, but built-in. You can:

Run background jobs
Process data asynchronously
Send emails without blocking requests
Handle long-running operations

And the best part? It's all native Django code.

Setting Up Tasks in Django 6

First, make sure you're on Django 6.0+:

pip install Django==6.0

Important: Django 6 requires Python 3.12 or higher! If you're on an older Python version, you'll need to upgrade first.

Your First Django Task

Let's create a simple task that logs a message:

# tasks.py in your app
from django.tasks import task

@task
def log_message(message):
    print(f"Task received: {message}")
    return f"Processed: {message}"

That's it! You've created your first task. The @task decorator tells Django this function should run as a background task.

Running Tasks from Views

Here's how you enqueue tasks from a Django view:

# views.py
from django.http import HttpResponse
from .tasks import log_message

def trigger_task(request):
    # This queues the task to run in background
    log_message.enqueue(message="Hello from Django 6!")

    return HttpResponse("Task enqueued!")

The .enqueue() method adds your task to the queue. Your view returns immediately while the task runs in the background.

Configuring Task Backends

Django 6 comes with two built-in backends:

Immediate Backend (Default) Tasks run right away in the same thread:

# settings.py
TASKS = {
    "default": {
        "BACKEND": "django.tasks.backends.immediate.ImmediateBackend"
    }
}

Good for: Testing and development.

Dummy Backend Tasks don't run at all:

# settings.py
TASKS = {
    "default": {
        "BACKEND": "django.tasks.backends.dummy.DummyBackend"
    }
}

Good for: When you want to test task queuing without execution.

Real-World Example: Email Sender

Let's build something useful - an email sender task:

# tasks.py
from django.tasks import task
from django.core.mail import send_mail

@task
def send_welcome_email(user_email, username):
    subject = f"Welcome, {username}!"
    message = "Thanks for joining our platform."

    send_mail(
        subject=subject,
        message=message,
        from_email="noreply@example.com",
        recipient_list=[user_email]
    )

    return f"Email sent to {user_email}"

Use it in a view:

# views.py
from django.http import JsonResponse
from .tasks import send_welcome_email

def register_user(request):
    # Get user data from request...
    email = request.POST.get('email')
    username = request.POST.get('username')

    # Queue email task (won't block the response)
    send_welcome_email.enqueue(
        user_email=email,
        username=username
    )

    return JsonResponse({"status": "Registration complete!"})

The user gets an immediate response while the email sends in the background.

Need Production-Ready? Use django-tasks

For production use, you'll want a proper backend. The Django community is already building solutions. Here's one option:

pip install django-tasks

Configure it in settings:

# settings.py
INSTALLED_APPS = [
    'django_tasks',
    'django_tasks.backends.database',
    # ... your other apps
]

TASKS = {
    "default": {
        "BACKEND": "django_tasks.backends.database.DatabaseBackend"
    }
}

Run migrations and start the worker:

python manage.py migrate
python manage.py runserver
python manage.py db_worker  # In a separate terminal

Current Limitations

As Django 6 just released, keep in mind:

Built-in backends are basic – Immediate and Dummy only
Community backends are emerging – Watch for django-tasks-redis, django-tasks-rabbitmq
Python 3.12+ required – Time to upgrade!

Wrapping Up

Django 6's built-in tasks are exciting! They won't replace Celery overnight, but they offer a simpler alternative for many use cases.

When to use Django Tasks:

Simple background jobs
Email sending
Data processing
When you want minimal setup

Note: As Django 6 is brand new, expect rapid improvements to the task system. Check the official Django documentation for the latest updates.

Beyond Database Locks: Managing Concurrent User Transactions

Olanrewaju Mark — Thu, 23 Oct 2025 09:53:22 +0000

The Simultaneous Update Anomaly

Imagine this nightmare scenario:

Your user, Sarah, sells a product for $100.
Two buyers pay her at the exact same moment.
Two "credit alert" webhooks hit your server simultaneously.
Both payments are for the same amount, to the same user.

What happens?

Worker 1 checks Sarah's balance: $0.
Worker 2 also checks Sarah's balance: $0.
Worker 1 adds $100. New balance: $100.
Worker 2 adds $100. New balance: $100.

Result: Sarah only received $100 instead of $200! Money has vanished.

This isn't a multiple-user problem. This is a single-user concurrency problem, and it's every payment processor's worst nightmare. Your database's with_for_update() lock is useless here because the two transactions are operating on separate connections from different workers.

Why Your Database Can't Save You

Think of your database row lock as a lock on Sarah's account file in a filing cabinet.

Worker 1 walks up, opens the cabinet, and takes out Sarah's file.
Worker 2 walks up at the same time, opens the same cabinet, and takes out the same file.
Both make copies, update them, and put them back.

They never knew the other was there! This is the chaos of concurrent systems.

The Solution: The "One-Operation-Per-User" Lock

We need to put a "BUSY" sign on Sarah herself. Not just her file, but on the very idea of processing her transaction. We need to ensure that for any critical operation on a user, only one request is processed at a time.

Let me show you the exact weapon we need, a distributed user lock.

Code Deep Dive: Building the User Traffic Cop

Here's how we implement a lock that works across all your servers:

from threading import Lock
from .responses import http_response_400
from functools import wraps
from apis.utils import get_redis_counter

lock = Lock()
top_up_locks = {}

def one_deposit_at_a_time(f):
    @wraps(f)
    def decorated_function(*args, **kwargs):
        from apis.crud import get_user_by_account_number
        # get account number from request
        payload = request.get_json()
        user = get_user_by_account_number(payload["account_number"])
        user_id = user.id
        key = f"one_top_up:{user_id}"
        with lock:
            if key not in top_up_locks:
                top_up_locks[key] = Lock()
            top_up_lock = top_up_locks[key]
        with top_up_lock:
            # This is to get the counter to check if it exist (increment = False)
            one_top_up = get_redis_counter(key, ex_time=5, increment=False)
            if one_top_up:
                return http_response_400(
                    {
                        'status': 'failed',
                        'msg': 'One transaction at a time pls'
                    }
                )
            # This is to set the counter and set it to be released in 5 secs
            get_redis_counter(key, ex_time=5, increment=True)
            return f(*args, **kwargs)
    return decorated_function

Let's break down the example provided, step-by-step.

Step 1: The Tools We Need

We use two main tools to build our distributed lock:

A Global Dictionary (top_up_locks): This stores our lock objects for each user.
A Master Lock (lock): This is a safety lock to make sure we don't create two locks for the same user at the same time, a lock for our locks!

from threading import Lock
lock = Lock() # The master lock
top_up_locks = {} # The locker where we store all user locks

Step 2: The "Decorator" - A Reusable Shield

The code uses a decorator called @one_deposit_at_a_time. Think of a decorator as a wrapper. It lets you add extra logic (like a lock) to any function without changing the function's own code.

def one_deposit_at_a_time(f): # 'f' is the original function
    @wraps(f)
    def decorated_function(*args, **kwargs):
        # ... OUR LOCKING LOGIC GOES HERE ...
        # Once the lock is acquired, we run the original function
        return f(*args, **kwargs)
    return decorated_function

Step 3: Creating a Unique Key for Each User

Every user needs their own personal lock. We don't want to lock everyone because one user is making a deposit! We create a unique key based on the user's ID.

payload = request.get_json()
user = get_user_by_account_number(payload["account_number"])
user_id = user.id
key = f"one_top_up:{user_id}" # e.g., "one_top_up:user_789"

Step 4: The Two-Step Locking Dance

This is the genius part. We perform two locking steps to be perfectly safe.

Step 4a: Get (or Create) the User's Personal Lock

We use the master lock to safely look inside our global dictionary.

With the master lock held, we check: "Does a lock for user_789 already exist?"
If not, we create a new Lock() object and store it in the dictionary.
We then immediately release the master lock. We are now only dealing with the one user's lock.

with lock: # Hold the master lock very briefly
    if key not in top_up_locks:
        top_up_locks[key] = Lock() # Create a new lock for this user
    top_up_lock = top_up_locks[key] # Get a reference to it

Step 4b: Acquire the User's Personal Lock

Now we try to acquire the lock for user_789.

with top_up_lock: will wait here if another request for the same user is already holding the lock.
Once the first request is done, this one will acquire the lock and proceed.
This ensures that for this user, requests are processed one after the other, even across multiple workers.

with top_up_lock: # This waits here if the lock is already held
    # Now we are safe to check the database and perform the operation

What Happens Now When Two Payments Arrive:

TIMELINE:

T=0s: Payment A and Payment B arrive simultaneously for Sarah
T=0.1s: Payment A acquires the lock for user_sarah_123
T=0.1s: Payment B tries to acquire the lock → FAILS! Gets "please wait" message
T=1.1s: Payment A finishes processing, releases the lock
T=1.2s: Payment B automatically retries, acquires the lock, processes successfully
T=2.3s: Payment B finishes

Result: Sarah gets $200 correctly! The second payment was simply asked to wait its turn.

Conclusion: From Chaos to Control

The double deposit scenario reveals a fundamental truth: database transactions alone cannot protect you from distributed system problems. You need application-level coordination.

By implementing a simple user-based distributed lock, you transform your system from one that loses money under pressure to one that calmly queues transactions and processes them with perfect accuracy.

Next time two credit alerts arrive at once, your system will handle them with the grace of a seasoned traffic director: "You first, then you." No panic, no lost money, just rock-solid reliability.

Handling 404 and 500 Errors Gracefully in Django with Custom JSON Responses

Olanrewaju Mark — Sat, 30 Aug 2025 11:12:33 +0000

When building RESTful APIs with Django, the default error pages 404 Not Found (when you try to access an unavailable url), 500 Internal Server Error, return HTML responses. While this is fine for traditional web apps, API clients expect JSON responses instead.

In this article, I’ll walk through how to override Django’s error handlers using handler404 and handler500, and return consistent, structured JSON responses for your API.

Why Customize Error Handlers?

By default, Django shows pages like this when an error occurs:

404 → “Page not found” (HTML template)
500 → “Server error” (HTML template)

For API consumers, this is not helpful. Instead, something like this should be returned:

{
  "status": "Failed",
  "message": "he requested resource was not found"
}

{
  "status": "Failed",
  "message": "Internal Server Error"
}

Setting Up Custom Exception Handlers

First, create a file called custom_exceptions.py inside your API service folder (for example api_services/custom_exceptions.py).

# api_services/custom_exceptions.py
from django.http import JsonResponse

class CustomException:
    @staticmethod
    def custom_404_view(request, exception=None):
        return JsonResponse(
            {"status": "Failed", "message": "The requested resource was not found"},
            status=404
        )

    @staticmethod
    def custom_500_view(request):
        return JsonResponse(
            {"status": "Failed", "message": "Internal Server Error"},
            status=500
        )

custom_404_view → handles invalid routes.
custom_500_view → handles unexpected server errors.

We’re using JsonResponse here because it works outside of Django REST Framework (DRF) and guarantees valid JSON output.

Registering Handlers in `urls.py`

Now tell Django to use these handlers globally. In your project’s main urls.py file:

from django.contrib import admin
from django.urls import path, include
from django.conf.urls import handler404, handler500
from api_services.custom_exceptions import CustomException

BASE_PREFIX = "api/v1"

urlpatterns = [
    path("admin/", admin.site.urls),
    path(f"{BASE_PREFIX}/users/", include("apis.users.urls")),
]

# Attach custom error handlers
handler404 = CustomException.custom_404_view
handler500 = CustomException.custom_500_view

Important: Turn Off Debug Mode

By default, Django shows detailed debug pages when DEBUG = True.
This is useful for developers, but it overrides your custom error handlers.
To make your JSON handlers work, set this in settings.py:

DEBUG = False
ALLOWED_HOSTS = ["*"]  # Or set to your domain(s)

Now Django will call your custom_404_view and custom_500_view instead of showing HTML pages.

Testing the Setup

Run your server

python manage.py runserver

Try hitting an invalid endpoint:

GET http://127.0.0.1:8000/api/v1/use

You’ll now get a clean JSON response:

{
  "status": "Failed",
  "message": "The requested resource was not found"
}

For 500 errors, you can temporarily raise an exception in a view and see the JSON response.

Why This Matters

✅ Cleaner API: Clients always get JSON, no ugly HTML errors.
✅ Consistency: Same error format as your success responses.
✅ Better UX: Frontend apps can handle errors gracefully.

When to Use LabelEncoder and OneHotEncoder in Machine Learning

Olanrewaju Mark — Mon, 28 Jul 2025 13:32:18 +0000

Imagine trying to teach a computer what Small, Medium, and Large mean.
To you, these words are simple. But for a machine, they’re just strange strings of letters.
Machines don’t understand text, they only understand numbers.
So before we train a machine learning model, we need to translate these text categories into numbers that the model can actually process.
Two popular translators are:

LabelEncoder
OneHotEncoder But when do you use each one? Let’s make this super clear.

1. What Is Categorical Data?

Categorical data is data made up of labels or names, not numbers.
Think about these examples:

Size: Small, Medium, Large
Weather: Sunny, Rainy, Cloudy
Cities: Lagos, Abuja, Kano We can’t just throw these words at a model, we need to convert them into numbers first.

2. Label Encoding

Label Encoding simply assigns a number to each category, starting from 0.

Example with sizes:

Small  → 0  
Medium → 1  
Large  → 2

When Should You Use Label Encoding?

Use it when the categories have a natural order or ranking.
Examples:

Low < Medium < High
Cold < Warm < Hot
Small < Medium < Large

Never use LabelEncoder for categories like colors or city names — because the model will think Green (2) > Red (0), which is meaningless!

from sklearn.preprocessing import LabelEncoder

sizes = ["Small", "Large", "Medium", "Small", "Large"]

label_encoder = LabelEncoder()
encoded_sizes = label_encoder.fit_transform(sizes)

print("Original:", sizes)
print("Encoded:", encoded_sizes)
print("Classes:", label_encoder.classes_)

Output:

Original: ['Small', 'Large', 'Medium', 'Small', 'Large']
Encoded: [2 0 1 2 0]
Classes: ['Large' 'Medium' 'Small']

Notice how it encoded alphabetically (Large=0, Medium=1, Small=2).
If you want Small=0, Medium=1, Large=2, you can map it manually:

size_order = {'Small': 0, 'Medium': 1, 'Large': 2}

3. One-Hot Encoding

One-Hot Encoding creates separate columns for each category and marks them with 0 or 1.
One-Hot Encoding creates separate columns for each category and marks them with 0 or 1.

For example:

Color: Red   → [1, 0, 0]
       Blue  → [0, 1, 0]
       Green → [0, 0, 1]

This way, no category is greater or less than another.

When Should You Use One-Hot Encoding?

Use it when the categories have no order, like colors, cities, or animal names.
It avoids the fake ranking problem that LabelEncoder might create.

from sklearn.preprocessing import OneHotEncoder
import numpy as np

colors = np.array(["Red", "Blue", "Green", "Blue", "Red"]).reshape(-1, 1)

onehot_encoder = OneHotEncoder(sparse_output=False)
encoded_colors = onehot_encoder.fit_transform(colors)

print("Original:", colors.flatten())
print("Encoded:\n", encoded_colors)
print("Categories:", onehot_encoder.categories_)

Output:

Original: ['Red' 'Blue' 'Green' 'Blue' 'Red']
Encoded:
 [[0. 0. 1.]
  [1. 0. 0.]
  [0. 1. 0.]
  [1. 0. 0.]
  [0. 0. 1.]]
Categories: [array(['Blue', 'Green', 'Red'], dtype=object)]

4. LabelEncoder vs OneHotEncoder

Think of it like this:

LabelEncoder says: I’ll give each category a number. You figure out what it means.

OneHotEncoder says: I’ll give each category its own column so no one feels more important than the other.

Aspect	LabelEncoder	OneHotEncoder
Best for	Ordered categories	Unordered categories
Output	Single column (0, 1, 2...)	Multiple columns (0/1)
Risk	Fake order for labels	No fake order
Example	Small < Medium < Large	Red, Blue, Green

5. What About 2 Categories?

If you only have 2 categories, LabelEncoder is fine because it will just give 0 and 1.

Example:

from sklearn.preprocessing import LabelEncoder

binary = ["Yes", "No", "Yes", "No"]

encoder = LabelEncoder()
encoded = encoder.fit_transform(binary)

print("Encoded:", encoded)  # [1 0 1 0]
print("Classes:", encoder.classes_)  # ['No' 'Yes']

Here:

No  → 0
Yes → 1

6. Things to Keep in Mind

Use LabelEncoder when your data has a natural order (e.g., Small < Medium < Large).
Use OneHotEncoder when your data has no order (e.g., colors, cities).
For 2 categories, LabelEncoder automatically uses 0 and 1.
The order of rows in your CSV does not matter.

Think of LabelEncoder as the tool for ranking categories that have order.
Think of OneHotEncoder as the tool for naming categories when order doesn’t exist.

If you mix them up (like using LabelEncoder on colors), your model might get confused and make bad predictions.

ACID Operation in Backend Engineering: Building Data Integrity Like a Pro

Olanrewaju Mark — Thu, 24 Jul 2025 10:02:37 +0000

When you build a banking app, an e-commerce checkout, or any critical backend system, data integrity is everything. A single failed transaction could cause duplicate charges, missing payments, or worse.

To prevent such disasters, databases use ACID properties:

Atomicity
Consistency
Isolation
Durability

In this article, we’ll break each property into plain English explanations with Python code examples using SQLAlchemy.

1. Atomicity – All or Nothing

Atomicity means that a transaction is indivisible.

If part of the transaction fails, the entire operation is rolled back.

Example Scenario

Alice is transferring $300 to Bob:

Subtract $300 from Alice’s balance.
Add $300 to Bob’s balance.

If step 2 fails, step 1 must be undone.

Code Example

from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.orm import sessionmaker, declarative_base

Base = declarative_base()

class Account(Base):
    __tablename__ = 'accounts'
    id = Column(Integer, primary_key=True)
    name = Column(String, unique=True)
    balance = Column(Integer, default=0)

engine = create_engine("sqlite:///bank.db", echo=False)
Session = sessionmaker(bind=engine)
Base.metadata.create_all(engine)

def atomic_transfer(session, from_id, to_id, amount):
    try:
        from_acc = session.query(Account).filter(Account.id == from_id).with_for_update().first()
        to_acc = session.query(Account).filter(Account.id == to_id).with_for_update().first()

        if from_acc.balance < amount:
            raise ValueError("Insufficient funds")

        from_acc.balance -= amount
        to_acc.balance += amount

        session.commit()  # Only commit if all steps succeed
        print("Transfer completed.")
    except Exception as e:
        session.rollback()  # Undo everything if something fails
        print("Transfer failed:", e)

# Run a test
session = Session()
atomic_transfer(session, from_id=1, to_id=2, amount=300)
session.close()

2. Consistency – Valid State Before and After

Consistency ensures that data always follows rules.
For example:

Balances cannot go negative.
Total money in the system should remain correct.

Example

# Consistency check
if from_acc.balance < amount:
    raise ValueError("Insufficient funds: cannot go negative")

Here, we ensure no transaction breaks the system’s rules.

3. Isolation – Transactions Don’t Interfere

Isolation means that concurrent transactions behave as if they’re running one after another, even when they’re processed at the same time.

Example

Two transfers happening at once:

Alice transfers $200 to Bob.
Bob transfers $100 to Charlie.

Without isolation, their operations might overlap and cause wrong balances.
Code Example

Using with_for_update() to lock rows during the transfer:

from_acc = (
            session.query(Account)
            .filter(Account.id == from_id)
            .with_for_update()
            .first()
        )
to_acc = (
            session.query(Account)
            .filter(Account.id == to_id)
            .with_for_update()
            .first()
        )

This works for a single worker, but when you have multiple workers or parallel requests, isolation gets tricky.

4. Durability – Changes Persist

Durability guarantees that once a transaction is committed, the changes survive crashes.

Example

def durable_deposit(session, account_id, amount):
    try:
        acc = session.query(Account).filter(Account.id == account_id).with_for_update().first()
        acc.balance += amount
        session.commit()
        print(f"Deposited {amount} to {acc.name}. This change is permanent.")
    except Exception as e:
        session.rollback()
        print("Deposit failed:", e)

Even if your server restarts after session.commit(), the deposit remains in the database.

5. Full Example: Deposit + Transfer

def deposit_and_transfer(session, deposit_to, deposit_amt, transfer_from, transfer_to, transfer_amt):
    try:
        # Deposit
        acc = session.query(Account).filter(Account.id == deposit_to).with_for_update().first()
        acc.balance += deposit_amt
        print(f"Deposited {deposit_amt} to {acc.name}. New balance: {acc.balance}")

        # Transfer
        from_acc = session.query(Account).filter(Account.id == transfer_from).with_for_update().first()
        to_acc = session.query(Account).filter(Account.id == transfer_to).first()

        if from_acc.balance < transfer_amt:
            raise ValueError("Insufficient funds for transfer")

        from_acc.balance -= transfer_amt
        to_acc.balance += transfer_amt

        session.commit()
        print("Deposit and transfer successful.")
    except Exception as e:
        session.rollback()
        print("Operation failed:", e)

This ensures all steps succeed or all fail.

Why ACID Matters

Atomicity: No partial transfers.
Consistency: No invalid states.
Isolation: No clashing transactions.
Durability: Data persists after commit.

Coming Next: Isolation Under Pressure

Our examples work well when there’s one worker.
But with 4 workers handling two webhooks at the same time, things can break.
We’ll uncover why with_for_update() is not enough and how to use distributed locks or queues to fix it.

Poetry Explained: A Better Tool for Managing Python Projects

Olanrewaju Mark — Wed, 23 Jul 2025 09:44:56 +0000

If you’ve worked with Python, you’ve probably used pip to install packages. It’s the default tool in Python to add libraries like Flask, Requests, or Pandas.

But there's another tool called Poetry that does more than just install packages. Poetry helps manage your whole Python project: dependencies, virtual environments, versioning, and publishing, all in one place.

In this article, we’ll explain:

What Poetry is
How to install and use it
How it compares with pip
When to choose Poetry over pip

Let’s begin!

📦 What Is Poetry?

Poetry is a Python dependency and package manager. It was created to solve the common problems Python developers face, such as:

Messy requirements.txt files
Confusing virtual environments
Publishing packages to PyPI manually

Poetry uses a single file — pyproject.toml — to manage your dependencies and project configuration.

It also creates and manages a virtual environment automatically, so you don't have to use tools like venv or virtualenv.

🔧 Installing Poetry

To install Poetry, you can run this one-line command in your terminal:

curl -sSL https://install.python-poetry.org | python3 -

After installation, make sure it works:

poetry --version

If that prints something like Poetry (version 2.1.3), you’re good to go!

🚀 How to Use Poetry

1. Create a New Project

poetry new my_project

This creates a folder my_project with this structure:

my_project/
├── pyproject.toml
├── README.rst
├── my_project/
│   └── __init__.py
└── tests/
    └── test_my_project.py

2. Add a Dependency

poetry add requests

Poetry:

Adds requests to the pyproject.toml
Installs it in a virtual environment
Creates or updates the poetry.lock file

3. Activate the Virtual Environment

Poetry creates a virtual environment automatically.
To activate it, run:

eval $(poetry env activate)

After this, you are inside the Poetry-managed environment.
You can now run Python directly:

python main.py

4. Run Without Activating

You can also run commands without manually activating the environment:

poetry run python main.py

5. Install All Dependencies

When you clone a project, simply run:

poetry install

It installs all the packages listed in pyproject.toml and poetry.lock.

📁 Poetry vs pip: Key Differences

Feature	`pip`	`poetry`
Dependency management	Manual via `requirements.txt`	Automatic via `pyproject.toml` and `poetry.lock`
Virtual environments	You set it up yourself	Automatically created and managed
Adding packages	`pip install`	`poetry add`
Removing packages	`pip uninstall`	`poetry remove`
Dependency resolution	Basic	Advanced (checks compatibility)
Lock file	No built-in support	Yes (`poetry.lock`)
Project creation	Not available	`poetry new` or `poetry init`
Publishing to PyPI	Manual (twine etc.)	Built-in (`poetry publish`)

In short, pip only installs packages. Poetry manages the whole project.

✅ Why Use Poetry?

Use Poetry if:

You want a clean and consistent environment
You are building a Python library or application
You want to avoid manually editing requirements files
You want built-in tools for versioning and publishing

Use pip if:

You're working on simple scripts
You don’t want extra tools
You prefer traditional virtualenv setups

pyproject.toml Example

[tool.poetry]
name = "my_project"
version = "0.1.0"
description = "My awesome Python project"
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.10"
requests = "^2.31.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

This is what controls your project. It replaces setup.py, requirements.txt, and MANIFEST.in — all in one file.

🏁 Conclusion

Poetry upgrades your Python workflow. It takes care of dependencies, virtual environments, and even publishing. With just one command: eval $(poetry env activate), you can start coding without worrying about setup. While pip is great for simple tasks, Poetry is better for serious projects.

If you are tired of managing requirements.txt and venv by hand, try Poetry. It makes Python development faster, cleaner, and less frustrating.

Introducing Quart: A Modern Alternative to Flask (with Async Support)

Olanrewaju Mark — Mon, 14 Jul 2025 13:17:21 +0000

If you've used Flask before and loved how simple it is to build APIs, you’ll probably enjoy Quart too. Quart is like Flask, but with built-in async support, making it better for handling modern, high-speed web applications.

💡 Why Quart?

Flask is great, but it wasn’t built with async/await in mind. In today’s world where performance and speed are critical, especially for APIs having support for asynchronous code (without workarounds) is a big win.

Quart gives you:

The same API and structure as Flask
Support for async views, database calls, etc.
Works well with tools like SQLAlchemy, JWT, and more

🔁 Flask vs. Quart (Basic Example)

Flask:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello from Flask!"

Quart:

from quart import Quart

app = Quart(__name__)

@app.route("/")
async def hello():
    return "Hello from Quart!"

You’ll notice the main difference is async def instead of def. That allows you to use await inside the route, like calling an async database or HTTP request.

🔐 JWT Authentication Example in Quart

from quart import Quart
from quart_jwt_extended import JWTManager, jwt_required, create_access_token

app = Quart(__name__)
app.config["JWT_SECRET_KEY"] = "your-secret-key"

jwt = JWTManager(app)

@app.route("/login", methods=["POST"])
async def login():
    # Dummy example
    access_token = create_access_token(identity="user123")
    return {"access_token": access_token}

@app.route("/protected")
@jwt_required
async def protected():
    return {"message": "This is a protected route"}

📁 Example Project

I’ve created a small Quart project that shows how to:

Set up Quart with async SQLAlchemy
Use JWT for authentication
Organize routes and config

👉 Check it out on GitHub:

engrmarkk / quart-api

Quart API

A Python-based asynchronous REST API built with Quart, SQLAlchemy (async), and [PostgreSQL], using Poetry for dependency management.

🚀 Features

Fast async API with Quart
PostgreSQL + SQLAlchemy (async ORM)
JWT authentication with quart-jwt-extended
Fully typed models
Alembic for migrations

🛠️ Requirements

Python 3.10+
Poetry
PostgreSQL database

📦 Installation

1. Clone the repository

git clone https://github.com/engrmarkk/quart-api.git
cd quart-api

2. Install dependencies

poetry install

3. Create `.env`

touch .env

Edit .env and set your database credentials or JWT secret, e.g.:

DB_URI=postgresql+asyncpg://username:password@localhost:5432/quart-api # or your external db (but +asyncpg: should be present in the uri)
SECRET_KEY=supersecretkey

🗃️ Database Setup

Create the PostgreSQL database

CREATE DATABASE quart_api;

Run migrations

poetry run alembic upgrade head

🔐 JWT Auth (Optional Setup)

Ensure you're using quart-jwt-extended. Poetry handles this if you ran poetry install.

▶️ Running the Server

chmod u+x run_it.sh # to give the file permission the first time

…

View on GitHub

🧠 Final Thoughts

Quart is a great choice if you're already familiar with Flask and want to step into the world of asynchronous Python. It lets you build scalable, modern APIs without changing much in your existing workflow.

So if you're starting a new project, give Quart a try!

DEV Community: Olanrewaju Mark

Stop Confusing npm and npx: Here's What Every Developer Should Know

npm and npx: A Simple Guide for Everyone

What is npm?

Why do developers use npm?

Getting started with npm

Starting a new project

Installing a package

Installing all packages for a project

Removing a package

Updating packages

What is npx?

When should you use npx?

npm vs npx: The Simple Difference

A Quick Example to Tie It Together

Common Mistakes to Avoid

Final Thoughts

The Celery Lifecycle: How a Task Gets Registered, Queued, and Run

What Is Celery, In Simple Terms

1. How Tasks Get Registered

The @app.task decorator

Why registration matters

Autodiscovery

2. The Broker URL - The Middleman

What the broker URL actually controls

3. Execution - What Happens When a Task Runs

Concurrency - how one worker runs many tasks

4. The Result Backend - Where Answers Go

How you actually use it

Common mistake: confusing broker and backend

Putting It All Together

A Quick Practical Example

Summary

Offset vs Cursor Pagination: The Mistake Most Backend Engineers Make

What is Pagination?

1. Offset Pagination

Visual Representation

Advantage of Offset Pagination

The problem with Offset Pagination

2. Cursor Pagination

Production APIs usually Fetch One Extra Record

Advantage of Cursor Pagination

FastApi code example

The Downside

Can Both Exists in the Same System?

Final Thoughts

Building a Secure Django REST API from Scratch - DjangGuard

Why Security Needs to Come First

The Tech Stack

Project Structure

1. Custom User Model

2. JWT with User Agent Binding

3. User Agent Validation Middleware

4. Redis Token Blacklisting

5. Single Active Session

6. Brute-Force Protection with django-axes

7. Rate Limiting

8. Argon2 Password Hashing

9. Consistent API Responses

10. Redis Caching for Profile Data

The API Endpoints

Getting Started

What You Can Add Next

Final Thoughts

engrmarkk / DjangGuard

DjangGuard

Table of Contents

Overview

Features

Building MatchMetric-AI: A Real-Time Resume Tailoring Engine with Django, WebSockets, and LLMs

What does this project do?

The Tech Stack

Technical Deep Dive

How to Run the Project

Testing with Postman

What I Learned

Django 6 Released: Here's How to Use Its Game-Changing Task Feature

What Are Django Tasks?

Setting Up Tasks in Django 6

Your First Django Task

The `@app.task` decorator

Registering Handlers in `urls.py`