DEV Community: Billy Okeyo

Distributed Locks Explained: Technologies That Implement Distributed Locks

Billy Okeyo — Fri, 17 Jul 2026 00:00:00 +0000

Now that we understand how distributed locks work conceptually, the next question becomes:

Where do these locks actually live?

Unlike database transactions, distributed locks cannot rely on the memory of a single application server. Remember, our application might be running on five, ten, or even hundreds of machines, and every server must consult the same source of truth before deciding whether it may perform a particular operation.

Over the years, several technologies have emerged to solve this coordination problem. Although they all provide distributed locking capabilities, they were designed with slightly different goals in mind. Let’s look at the most common ones.

Redis

For most web applications, Redis is by far the most popular choice. Originally designed as an in-memory data store, Redis is incredibly fast, making it an excellent candidate for lightweight coordination tasks.

Acquiring a lock in Redis is surprisingly straightforward. A server attempts to create a key using an atomic command that says, in effect:

“Create this key only if it doesn’t already exist.”

If Redis successfully creates the key, the server owns the lock; if the key already exists, another server is already performing the work. Because Redis executes this operation atomically, two servers can never successfully create the same lock at the same time.

A simplified example looks like this:

SET invoice-generation

Server-A

NX

EX 30

The command says:

Create the key only if it doesn’t already exist (NX).
Automatically expire it after thirty seconds (EX 30).

In one atomic operation, Redis both acquires the lock and ensures it won’t remain forever if the server crashes. For many applications, this is all that’s needed.

Why Redis Is So Popular

Redis has become the default choice for distributed locks because it satisfies three important requirements. First, it’s extremely fast: since Redis stores data in memory rather than on disk, lock acquisition usually takes only a few milliseconds. Second, many applications already use Redis for caching, sessions, queues, or rate limiting, so adding distributed locking often requires little additional infrastructure. Finally, Redis has mature client libraries for virtually every programming language, with Laravel, Django, Spring Boot, .NET, Node.js, and Go all providing excellent Redis support.

For the vast majority of business applications, Redis offers an excellent balance between simplicity, performance, and reliability.

The Challenge with a Single Redis Instance

Suppose your entire application depends on one Redis server. Everything works perfectly, then Redis crashes. Suddenly, no application server can acquire new locks, and even worse, if Redis loses its in-memory state during a restart, locks may disappear unexpectedly.

This introduces a new challenge: the coordinator itself has become a single point of failure. For many applications, this risk is acceptable. For others, particularly financial systems or globally distributed services, it isn’t. This challenge led to one of the most discussed topics in distributed systems: the Redlock algorithm.

The Redlock Algorithm

Redlock was proposed by Redis creator Salvatore Sanfilippo as a way to make Redis-based distributed locks more resilient. Instead of relying on one Redis server, Redlock uses multiple independent Redis instances.

Imagine five Redis servers.

Redis 1

Redis 2

Redis 3

Redis 4

Redis 5

When acquiring a lock, the application attempts to obtain it from all five servers, and the lock is considered successful only if a majority agree.

For example:

Acquire Lock

↓

Redis 1 ✅

Redis 2 ✅

Redis 3 ✅

Redis 4 ❌

Redis 5 ❌

Three out of five succeeded, so the application proceeds. If only two servers grant the lock, the operation fails because a majority wasn’t reached. This approach significantly reduces the impact of individual Redis failures.

However, Redlock is also one of the most debated algorithms in distributed systems. Some engineers argue that it’s sufficient for many practical systems, while others, including Martin Kleppmann, have published detailed critiques explaining situations where Redlock may not provide the guarantees developers expect. The important lesson isn’t that Redlock is good or bad; it’s that distributed systems involve trade-offs, and understanding those trade-offs matters more than memorizing algorithms.

ZooKeeper

Long before Redis became popular for distributed locking, many large distributed systems relied on Apache ZooKeeper. ZooKeeper was designed specifically for coordination. Rather than functioning as a cache, it provides services such as:

Distributed locks
Leader election
Configuration management
Service discovery

Think of ZooKeeper as a highly reliable coordinator for distributed applications. Its primary goal isn’t speed, it’s correctness. Large systems such as Apache Kafka, Hadoop, and HBase have historically relied on ZooKeeper to coordinate clusters of machines. If your application requires complex distributed coordination rather than simple locking, ZooKeeper remains an excellent choice.

etcd

If you’ve worked with Kubernetes, you’ve already encountered etcd, even if you didn’t realize it. Every Kubernetes cluster stores its configuration inside etcd. Like ZooKeeper, etcd is a distributed key-value store designed for coordination rather than caching. It provides:

Distributed locks
Leader election
Configuration storage
Consensus
Service coordination

Unlike Redis, etcd prioritizes consistency over raw performance. Its API is also designed around long-lived leases, making lock management particularly elegant. Modern cloud-native applications frequently choose etcd when they already operate within Kubernetes ecosystems.

Consul

HashiCorp Consul occupies a similar space. Although many developers know Consul for service discovery, it also provides distributed locking capabilities through sessions. Organizations using Consul often rely on it for:

Service registration
Health checks
Distributed configuration
Leader election
Distributed locks

Like ZooKeeper and etcd, Consul focuses on reliable coordination across distributed infrastructure.

Which Technology Should You Choose?

There isn’t a universal answer. Instead, the right choice depends on your application’s requirements. If you’re building a typical web application that already uses Redis, implementing distributed locks with Redis is usually the simplest and most practical solution. If you’re coordinating hundreds of services across a Kubernetes cluster, etcd may integrate more naturally with your infrastructure. If your organization already uses Consul or ZooKeeper for service coordination, leveraging those existing systems often makes more sense than introducing Redis solely for locking.

Choosing a technology isn’t just about features. It’s also about operational complexity, existing infrastructure, and the guarantees your business requires. The important thing to remember is this: distributed locks are a concept, while Redis, ZooKeeper, etcd, and Consul are simply different tools for implementing that concept. Understanding the underlying idea is far more valuable than becoming attached to a specific technology.

Do You Always Need a Distributed Lock?

Reading this article, it might be tempting to conclude that distributed locks are the solution to every concurrency problem. They’re not. In fact, many applications never need them. If your application runs on a single server, ordinary in-memory locks are often sufficient, and if your database transaction already guarantees correctness, introducing a distributed lock may simply add unnecessary complexity.

Distributed locks are powerful, but they should be introduced only when multiple independent application instances genuinely need to coordinate shared work. Like every distributed systems technique, they solve a very specific class of problems, and the best engineering decision is often knowing when not to use them.

Common Mistakes When Using Distributed Locks

Like many distributed systems concepts, distributed locks appear deceptively simple: acquire a lock, perform some work, release the lock. In practice, however, there are several subtle mistakes that can introduce bugs that are even harder to diagnose than the problem the lock was intended to solve. Understanding these pitfalls is just as important as understanding distributed locks themselves.

Assuming a Lock Lasts Forever

One of the most common mistakes is forgetting that distributed locks usually have an expiration time. Suppose a server acquires a lock with a TTL of thirty seconds, and the developer assumes the operation will always finish within that window. Months later, a new feature makes the operation take forty-five seconds. The lock expires while the first server is still working, another server acquires the same lock and begins executing the exact same task, and suddenly, duplicate work appears again.

Choosing an appropriate TTL, and renewing it for long-running tasks when necessary, is essential.

Forgetting to Release the Lock

Although expiration protects against permanent deadlocks, applications should still release locks as soon as the protected work finishes. Holding a lock longer than necessary reduces concurrency and delays other servers waiting to perform legitimate work.

A good rule is simple:

Hold the lock only for the work that genuinely requires exclusive access.

Everything else should happen outside the lock whenever possible.

Protecting Too Much Code

Developers sometimes wrap entire workflows inside a distributed lock. For example:

Acquire Lock

↓

Call External Payment API

↓

Generate PDF

↓

Upload File

↓

Send Email

↓

Release Lock

This means every other server waits while network requests, file generation, and email delivery are taking place. Often, only a small portion of the workflow actually requires exclusive access. A better approach is to keep the critical section as short as possible.

Acquire Lock

↓

Update Shared Resource

↓

Release Lock

↓

Generate PDF

↓

Send Email

The shorter the lock, the better the system scales.

Distributed Locks vs Database Locks

At first glance, distributed locks and database locks appear very similar. Both prevent concurrent operations and both coordinate access to shared resources, but they operate at completely different levels.

A database lock protects data inside the database. For example, when two transactions attempt to update the same customer record, the database can lock that row until one transaction completes. Everything happens within the database engine itself.

A distributed lock protects work performed by application servers. Instead of preventing two transactions from updating the same row, it prevents two application instances from starting the same business process. Consider generating monthly invoices: before any invoice rows even exist in the database, every server must first decide whether it should begin the job. That decision happens outside the database, and a distributed lock coordinates that decision.

An easy way to remember the difference is:

Database locks protect data. Distributed locks protect business operations.

In many systems, you’ll use both together. A distributed lock ensures only one server begins generating invoices, and database transactions then ensure every invoice is written consistently.

Distributed Locks vs Optimistic Concurrency

Another concept frequently confused with distributed locks is optimistic concurrency. Optimistic concurrency assumes conflicts are relatively rare. Instead of preventing multiple users from editing the same record, it detects whether someone else changed the data before saving.

Suppose two employees open the same customer profile, and each begins editing. The system stores a version number alongside the record. When Employee A saves, the version changes from 5 to 6. When Employee B later attempts to save, the application notices that the version has already changed. Rather than silently overwriting Employee A’s work, it rejects the update and asks Employee B to refresh the page. No locking was required; the conflict was simply detected before committing.

Distributed locks take a different approach. Instead of detecting conflicts afterward, they prevent conflicting work from starting in the first place. Neither technique is universally better: optimistic concurrency works well when conflicts are uncommon, while distributed locks work best when duplicate execution would be expensive or dangerous.

Best Practices

As with most distributed systems techniques, simplicity is your friend. If you’re considering introducing distributed locks into your application, the following guidelines will help you avoid many common problems.

Keep Critical Sections Small

Acquire the lock immediately before modifying shared resources, and release it immediately afterward. The less work performed while holding the lock, the better your system scales.

Always Use Lock Expiration

Servers crash, containers restart, and networks fail. Never assume your application will always release its lock correctly. Expiration protects the rest of the system from waiting forever.

Verify Lock Ownership

Before releasing a lock, ensure your application still owns it. Ownership checks prevent one server from accidentally deleting another server’s lock after an expiration or retry.

Design for Failure

Distributed systems should always assume that something will eventually fail. Ask yourself:

What happens if Redis becomes unavailable?
What happens if the server crashes?
What happens if the network partitions?
What happens if the lock expires unexpectedly?

Thinking through failure scenarios early often prevents painful production incidents later.

Combine Reliability Patterns

Distributed locks are rarely used in isolation. Production systems often combine multiple reliability techniques. For example:

Idempotency prevents duplicate requests.
Transactions guarantee atomic database updates.
Isolation Levels provide predictable views of data.
Distributed Locks coordinate multiple application servers.

Each technique solves a different problem. Together, they create systems that continue behaving correctly even under heavy load and unexpected failures.

Bringing It All Together

At this point in the series, we’ve explored several concepts that all contribute to building reliable backend systems. Each one addresses a different question.

Concept	Question It Answers
Idempotency	What if the same request is sent twice?
Race Conditions	What if multiple requests modify the same data simultaneously?
Database Transactions	What if part of my operation fails?
Isolation Levels	What should transactions be allowed to see while others are running?
Distributed Locks	How do multiple servers agree who should perform a task?

Notice how these concepts complement one another. None replaces the others; reliable systems are built by combining the right tools for the right problems.

Final Thoughts

As applications grow, the biggest challenges often aren’t writing business logic. They’re coordinating work across multiple users, multiple requests, multiple transactions, and eventually multiple servers.

Distributed locks exist because modern applications are no longer confined to a single machine. They’re deployed across clusters, containers, cloud regions, and worker nodes that all need to cooperate without constantly stepping on each other’s toes.

Like transactions and isolation levels, distributed locks aren’t something you’ll use for every feature. But when you do need them, they can be the difference between a system that behaves predictably and one that quietly creates duplicate invoices, repeated payments, or inconsistent business data.

The next time you design a background job, scheduled task, or critical workflow, ask yourself one simple question:

“What happens if two servers try to do this at exactly the same time?”

If the answer is “something bad,” you’ve probably found a place where a distributed lock belongs.

What’s Next?

We’ve now covered:

Contract Testing
Idempotency
Race Conditions
Database Transactions
Database Concurrency
Database Isolation Levels
Distributed Locks

So far, every concept has focused on keeping operations consistent while they’re happening. But there’s another challenge waiting. Imagine you’ve successfully updated your database inside a transaction and now need to publish an event to Kafka, RabbitMQ, or another message broker. What happens if the database commit succeeds, but publishing the event fails? Or worse, what if the event is published but the transaction rolls back?

This problem has caused countless production incidents in distributed systems. In the next article, we’ll explore The Outbox Pattern Explained: Publishing Events Without Losing Data, one of the most widely used patterns for ensuring your database and message broker stay in sync.

Distributed Locks Explained: Coordinating Work Across Multiple Servers

Billy Okeyo — Mon, 13 Jul 2026 00:00:00 +0000

“Database locks protect rows. Distributed locks protect systems.”

Imagine your application has been wildly successful. What started as a simple web application running on a single server now runs across multiple machines behind a load balancer. Requests are shared between application instances, background workers process jobs independently, and scheduled tasks run on every server.

From the outside, everything looks better than ever: pages load faster, traffic scales effortlessly, and users are happy. Then one morning, your finance department calls. Every customer has received four monthly invoices.

Nothing appears wrong with the code. The invoice generation job is scheduled to run once every month, so why did it execute four times? The answer is surprisingly simple: you now have four application servers. At midnight, every server woke up, checked the scheduler, and independently decided that it was responsible for generating invoices. From each server’s perspective, everything was perfectly correct; collectively, however, they created a costly mistake.

Now imagine a different scenario. Your payment gateway sends a webhook confirming a successful payment. The webhook is delivered to one of your load-balanced servers, but a retry occurs because the payment provider doesn’t receive a response quickly enough, so another server processes the same webhook. Both servers begin updating balances, both create accounting entries, and both generate receipts.

You’ve already learned how idempotency protects against duplicate requests and how transactions ensure database operations succeed together.

But neither of those concepts answers a new question:

How do multiple application servers agree that only one of them should perform a particular piece of work?

That problem is solved by distributed locks.

As applications grow beyond a single server, distributed locks become one of the most important coordination mechanisms in modern software engineering.

Why Database Locks Are No Longer Enough

Earlier in this series, we explored database transactions and row-level locking.

Suppose two transactions attempt to update the same bank account.

Using a statement such as:

SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;

the database ensures only one transaction can modify that row at a time.

This works beautifully because both transactions are coordinating through the same database.

Now consider a different problem.

Suppose you have four application servers running the exact same code, and each server has a scheduler responsible for calculating monthly loan interest. Midnight arrives, and every server starts the same scheduled job. None of them are updating the same database row immediately. Instead, they’re deciding whether to begin an entire business process. The database has nothing to lock yet, and by the time one server begins writing data, the others have already started processing. The result is duplicated work.

Database locks are excellent at protecting individual records, but they are not designed to coordinate entire applications spread across multiple machines. This is the gap distributed locks were created to fill.

What Is a Distributed Lock?

A distributed lock is a coordination mechanism that allows multiple independent servers to agree that only one of them may perform a particular operation at a given time. Instead of protecting a single database row, a distributed lock protects an entire business activity.

Imagine a conference room with a single key. Anyone can use the room, but only the person holding the key may enter, and everyone else must wait until the key is returned. A distributed lock works in much the same way: before performing an operation, a server first attempts to acquire the lock. If the lock is available, the server proceeds; if another server already owns the lock, the operation waits, retries, or exits.

The important point is that every server asks the same central authority for permission before beginning work. That authority might be Redis, ZooKeeper, etcd, or another distributed coordination system.

A Simple Example

Imagine four application servers.

        Load Balancer
              │
   ┌──────────┼──────────┐
   │          │          │
Server A   Server B   Server C
   │          │          │
        Server D

At midnight, each server checks whether it’s time to generate invoices.

Without a distributed lock:

Server A → Generate invoices ✅

Server B → Generate invoices ✅

Server C → Generate invoices ✅

Server D → Generate invoices ✅

Four executions, four invoices, and one very unhappy finance department.

With a distributed lock:

Server A → Acquire Lock ✅

Server B → Lock Exists

Server C → Lock Exists

Server D → Lock Exists

Only Server A proceeds, and everyone else exits. The invoices are generated exactly once.

Real-World Examples

Distributed locks appear in far more places than most developers realize. Whenever multiple application instances could accidentally perform the same work, a distributed lock becomes a potential solution.

Scheduled Jobs

Suppose your loan management platform calculates accrued interest every night at midnight. Without coordination, every application server performs the calculation independently, interest may be applied multiple times, and a distributed lock ensures exactly one server performs the calculation.

Payment Processing

Imagine receiving a payment webhook from M-Pesa. Network retries cause multiple servers to receive the same notification, and without coordination, several servers may attempt to update balances simultaneously. A distributed lock allows only one server to process the payment while the others simply exit.

Inventory Management

Only one laptop remains in stock, and two servers receive purchase requests simultaneously. Each attempts to reserve the item. Although transactions protect database consistency, a distributed lock can coordinate reservation workflows across multiple application instances before they even begin modifying inventory.

Sending Emails

Marketing decides to send a promotional email to one million subscribers, and your email scheduler is deployed across five worker nodes. Without coordination, every worker starts sending the campaign and customers receive the same email five times. With a distributed lock, only one worker initiates the campaign while the others remain idle.

Report Generation

Generating annual financial reports may take several minutes. Without coordination, multiple servers might begin generating the exact same report simultaneously, wasting CPU time and increasing database load. A distributed lock ensures only one report generation process is active.

When Should You Consider a Distributed Lock?

A useful rule of thumb is to ask yourself a simple question:

What would happen if two servers performed this operation at exactly the same time?

If the answer is:

Duplicate invoices
Duplicate payments
Duplicate notifications
Duplicate accounting entries
Duplicate interest calculations

then the operation is probably a candidate for a distributed lock.

Not every feature needs one. Most HTTP requests don’t, reading data doesn’t, and serving web pages doesn’t. Distributed locks are primarily useful for shared background work and critical business processes where duplicate execution would produce incorrect results.

Distributed Locks Are About Coordination

One misconception worth addressing early is that distributed locks replace database transactions.

They don’t. Transactions guarantee consistency inside the database; distributed locks coordinate between application servers. The two solve different problems.

In practice, many enterprise systems use both together. A server first acquires a distributed lock, then begins a database transaction. When the transaction completes successfully, the server releases the distributed lock. The lock ensures only one server performs the work, and the transaction ensures the database remains consistent while that work is being performed. Together, they provide a powerful foundation for building reliable distributed systems.

At this point, we’ve answered why distributed locks exist.

The next question is equally important:

How do multiple servers actually agree on who owns the lock?

How Distributed Locks Work

At a high level, every distributed lock follows the same basic workflow.

Before performing a critical operation, an application asks a shared coordination service for permission to proceed. If the lock is available, the application acquires it and begins its work; if another server already owns the lock, the application waits, retries later, or simply exits. Once the work has been completed, the lock is released, allowing another server to acquire it.

Although different technologies implement this process differently, the underlying idea remains remarkably simple.

Application Server

        │

Request Lock

        │

───────────────
 Lock Service
───────────────

Lock Available?

   │          │

  Yes         No

   │          │

Acquire      Wait / Retry / Exit

   │

Perform Work

   │

Release Lock

The important detail is that every application instance talks to the same lock service. Without a shared source of truth, each server would simply believe it owned the lock.

Acquiring a Lock

Imagine four servers attempting to generate monthly invoices.

Each server sends a request to Redis asking for a lock called:

invoice-generation

Redis receives the requests almost simultaneously. The first request succeeds, and Redis stores something similar to:

invoice-generation

Owner: Server A

Expires: 30 seconds

When the remaining servers ask for the same lock, Redis responds that the lock already exists. Only Server A continues. Servers B, C, and D either wait, retry after a short delay, or abandon the operation altogether.

The beauty of distributed locks lies in their simplicity: instead of every server making its own decision, they all trust a single coordinator.

Holding the Lock

Once a server has successfully acquired the lock, it proceeds with the protected operation.

This might involve:

Generating invoices.
Processing a payment.
Calculating loan interest.
Sending reminder emails.
Synchronizing inventory.

During this period, every other server attempting the same operation sees that the lock is already owned. Rather than performing duplicate work, those servers simply back off. The lock effectively becomes a reservation saying that someone is already doing this work and others should wait.

Releasing the Lock

When the protected work completes successfully, the server releases the lock.

Acquire Lock

↓

Process Work

↓

Release Lock

Once Redis removes the lock, another server is free to acquire it if necessary. Releasing the lock is just as important as acquiring it. A lock that is never released eventually blocks every future attempt to perform that operation.

The Problem with Permanent Locks

Now consider something less pleasant.

Server A acquires the invoice-generation lock, but halfway through generating invoices, the server crashes. Perhaps the machine loses power, perhaps Kubernetes terminates the container, or perhaps someone accidentally restarts the application. The important point is that Server A never gets the opportunity to release its lock.

If the lock remained permanent, every future invoice generation attempt would fail because the system would forever believe Server A still owned the lock. This is one of the biggest differences between traditional application locks and distributed locks: distributed systems must always assume that servers can disappear without warning.

Lock Expiration (TTL)

To solve this problem, distributed locks almost always include an expiration time, often called a Time-To-Live (TTL).

Instead of storing only the lock name, the lock service stores something like:

Lock:

invoice-generation

Owner:

Server A

Expires:

30 seconds

If Server A completes successfully, it releases the lock before those thirty seconds expire. If Server A crashes, Redis automatically deletes the lock after the TTL expires, allowing another server to continue the work instead of waiting forever.

Think of it like borrowing a meeting room: rather than reserving it indefinitely, your booking automatically expires after one hour. If you forget to leave, the reservation eventually disappears and someone else can use the room. TTL prevents abandoned locks from permanently blocking the system.

Choosing the Right TTL

Choosing a lock duration isn’t as straightforward as it might seem.

Suppose generating invoices normally takes ten seconds, and a thirty-second TTL provides plenty of room for occasional delays. But what if one month the process unexpectedly takes forty-five seconds? The lock expires after thirty seconds, another server acquires it, and now both servers are generating invoices simultaneously. You’ve accidentally recreated the very problem the lock was supposed to prevent.

On the other hand, choosing an extremely long TTL isn’t ideal either. If a server crashes while holding a lock that expires after thirty minutes, every other server must wait half an hour before continuing. Finding the right TTL therefore requires understanding how long your operation normally takes, while leaving enough room for occasional delays.

Some distributed lock implementations even allow servers to periodically renew the TTL while they’re still actively working. This approach, often called a heartbeat, keeps long-running operations alive without requiring excessively long expiration times.

What Happens If Two Servers Ask at the Same Time?

One question naturally arises: what happens if two servers request the lock at exactly the same millisecond? The answer depends on the lock service.

Redis, ZooKeeper, etcd, and similar systems perform lock acquisition atomically. That means checking whether the lock exists and creating it happen as a single indivisible operation. There is never a moment when both servers successfully acquire the same lock: one request succeeds and the other fails. This atomicity is exactly what makes distributed locks reliable; without it, two servers could both believe they owned the lock, defeating the entire purpose.

Lock Ownership Matters

Imagine Server A acquires a lock. Before finishing its work, the lock expires because the TTL was too short, and Server B now acquires the same lock. Moments later, Server A finally finishes and attempts to release it. If the lock service simply deleted the lock without checking ownership, Server A would accidentally remove Server B’s lock, Server C could now acquire it, and suddenly two servers are working simultaneously again.

To prevent this, distributed lock implementations associate every lock with a unique owner identifier. When releasing a lock, the application must prove that it is still the owner; if ownership has already changed, the release request is ignored. This simple verification prevents one server from accidentally deleting another server’s lock.

Distributed Locks Aren’t Magic

It’s important to understand that distributed locks don’t eliminate failures. Servers can still crash, networks can still become partitioned, and Redis instances can still fail. Distributed locks simply provide a coordinated way for multiple application instances to make decisions despite those realities. They reduce duplicate work, improve consistency, and coordinate critical business operations, but like every distributed systems technique, they must be implemented carefully and combined with other reliability mechanisms such as transactions, retries, idempotency, and monitoring.

In the next section, we’ll explore the most common technologies used to implement distributed locks, including Redis, Redlock, ZooKeeper, etcd, and Consul, along with the strengths and weaknesses of each approach.

Database Isolation Levels Explained: Choosing the Right Consistency Guarantees

Billy Okeyo — Fri, 10 Jul 2026 00:00:00 +0000

Understanding the Four SQL Isolation Levels

Now that we’ve seen the kinds of problems concurrent transactions can create, the next question is obvious:

How does a database prevent them?

The answer lies in isolation levels.

Rather than enforcing a single set of rules for every application, relational databases allow developers to choose how isolated transactions should be from one another.

This flexibility exists because different applications have different priorities.

A banking application transferring millions of shillings every day values consistency above almost everything else.

A reporting dashboard displaying website traffic may prefer speed over perfect accuracy.

Isolation levels allow the database to balance these competing requirements.

As isolation becomes stronger, transactions observe more consistent data, but the database also has to coordinate more aggressively, often reducing concurrency.

As isolation becomes weaker, transactions execute more freely, improving performance but increasing the likelihood of observing changing data.

The SQL standard defines four isolation levels.

Each one builds upon the guarantees of the previous level.

Read Uncommitted

Read Uncommitted is the weakest isolation level defined by the SQL standard.

At this level, transactions are allowed to read changes made by other transactions even if those changes haven’t yet been committed.

Returning to our banking example, imagine Alice begins transferring KES 20,000 to another account.

The database deducts the money from her balance but hasn’t yet committed the transaction.

Another transaction immediately reads Alice’s account.

Instead of seeing KES 50,000, it now sees KES 30,000, even though the transfer could still fail and be rolled back.

That second transaction has just performed a dirty read.

The advantage of Read Uncommitted is that transactions almost never wait for one another.

Because the database performs very little coordination, throughput can be extremely high.

The downside is that applications may make decisions using data that never officially existed.

For most business applications, this is unacceptable.

Imagine calculating payroll using salaries that are eventually rolled back or approving a loan based on a balance that disappears moments later.

Fortunately, very few modern relational databases actually encourage Read Uncommitted.

Many databases either discourage it entirely or internally behave more conservatively even when it’s requested.

In practice, you’ll rarely choose this isolation level for production systems.

Read Committed

Read Committed is the default isolation level in databases such as PostgreSQL, Oracle, and SQL Server.

Instead of allowing transactions to read uncommitted changes, the database only exposes data that has already been committed.

This immediately eliminates dirty reads.

Returning to Alice’s transfer, suppose another transaction checks her balance while the transfer is still running.

Instead of seeing the temporary balance of KES 30,000, it continues seeing the previously committed balance of KES 50,000 until the transfer completes.

Only after the transaction commits does the new balance become visible.

This makes Read Committed an excellent general-purpose isolation level.

Applications never observe incomplete work, while the database still allows a high degree of concurrency.

However, Read Committed doesn’t solve every problem.

Suppose your transaction reads Alice’s balance at the beginning of a report.

A few seconds later, another transaction deposits KES 100,000 into the account and commits.

If your report queries the balance again before finishing, you’ll now see a different value.

The same row has changed during your transaction.

Read Committed prevents dirty reads, but it still allows non-repeatable reads and phantom reads.

For many applications, that’s a perfectly acceptable trade-off.

Read Committed Timeline

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 150,000

COMMIT

Both values are valid.

The important difference is that Transaction A never sees incomplete or rolled-back data.

It only observes committed changes.

Repeatable Read

Suppose you’re generating an end-of-day financial report.

Your transaction calculates the total account balance across thousands of customers.

Halfway through generating the report, another transaction updates several account balances.

If your report re-reads those accounts later, the totals may no longer match the values used earlier in the report.

This is exactly the situation Repeatable Read was designed to solve.

At this isolation level, once a transaction reads a row, subsequent reads of that same row always return the same version for the lifetime of the transaction.

Even if another transaction updates the row and commits, your transaction continues working with the original version.

It’s as though your transaction receives its own private snapshot of the database.

This provides a much more consistent view of the data, making it particularly useful for reporting systems and financial calculations.

However, Repeatable Read doesn’t necessarily prevent new rows from appearing that satisfy your query conditions.

Depending on the database implementation, phantom reads may still occur, although databases like PostgreSQL use Multi-Version Concurrency Control (MVCC) to eliminate many of these anomalies without locking every row.

This is one of the reasons database behavior differs slightly across vendors.

We’ll return to that shortly.

Repeatable Read Timeline

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 50,000 ✅

COMMIT

Although the database now contains KES 150,000, Transaction A continues seeing KES 50,000 because it is working from a consistent snapshot.

Serializable

Serializable is the strongest isolation level defined by the SQL standard.

The easiest way to understand it is to imagine that every transaction runs one after another instead of simultaneously.

Internally, the database may still execute many transactions concurrently, but it guarantees that the final result is identical to some serial execution order.

Returning to our concert ticket example, suppose only one seat remains.

Customer A begins purchasing the ticket.

Customer B attempts to purchase the same seat at exactly the same time.

Under Serializable isolation, the database ensures that only one transaction succeeds.

The other transaction must either wait, retry, or fail.

The database refuses to produce a result that couldn’t happen if the transactions had executed one after another.

This provides the strongest possible consistency guarantees.

It also comes at the highest performance cost.

Serializable transactions often require additional locking, conflict detection, or transaction retries.

For systems processing large volumes of concurrent requests, this can reduce throughput significantly.

Because of this, Serializable is typically reserved for situations where correctness is absolutely critical.

Financial ledgers, securities trading systems, and certain accounting operations are common examples.

Isolation Levels at a Glance

Isolation Level	Dirty Reads	Non-Repeatable Reads	Phantom Reads	Performance
Read Uncommitted	❌ Possible	❌ Possible	❌ Possible	⭐⭐⭐⭐⭐
Read Committed	✅ Prevented	❌ Possible	❌ Possible	⭐⭐⭐⭐
Repeatable Read	✅ Prevented	✅ Prevented	⚠ Depends on database	⭐⭐⭐
Serializable	✅ Prevented	✅ Prevented	✅ Prevented	⭐⭐

The table makes an important pattern clear.

As isolation increases, the number of concurrency anomalies decreases.

At the same time, the amount of coordination required by the database increases.

This is why there is no universally “best” isolation level.

The appropriate choice always depends on the requirements of your application.

Database Isolation Levels in Popular Databases

One detail that often surprises developers is that not every relational database implements isolation levels in exactly the same way.

The SQL standard defines the four isolation levels, but database vendors have some flexibility in how they achieve those guarantees.

For example, PostgreSQL relies heavily on Multi-Version Concurrency Control (MVCC). Instead of locking rows aggressively, PostgreSQL keeps multiple versions of a row and allows transactions to read a consistent snapshot of the data. This approach provides excellent concurrency while maintaining strong consistency.

MySQL’s InnoDB storage engine also supports MVCC but implements certain isolation behaviors differently. In particular, its default Repeatable Read isolation level prevents many phantom reads by using a combination of snapshot reads and gap locks.

SQL Server, on the other hand, traditionally relies more heavily on locking, although it also offers snapshot-based isolation levels that can be enabled when appropriate.

As a developer, you don’t need to memorize every implementation detail.

The important lesson is this:

Always understand how your specific database implements isolation before assuming its behavior.

The SQL standard provides the vocabulary, but your database documentation explains the exact behavior.

Choosing the Right Isolation Level

After learning about all four isolation levels, it’s natural to ask:

“Which one should I actually use?”

The honest answer is:

It depends on what you’re building.

Suppose you’re developing a dashboard that displays the number of users currently online.

If the number changes while someone refreshes the page, that’s perfectly acceptable.

There’s little value in sacrificing performance just to ensure every count remains identical throughout a transaction.

Read Committed is usually more than sufficient.

Now consider a payroll system.

Calculating employee salaries requires reading thousands of records while ensuring the figures don’t change halfway through the calculation.

If one employee’s salary is updated while payroll is being processed, the final report could contain inconsistent totals.

Repeatable Read becomes a much better fit because it provides a stable snapshot throughout the transaction.

Finally, imagine a securities trading platform or a banking ledger where even a single inconsistency could have significant financial consequences.

Here, correctness is more important than throughput.

Serializable isolation is often the safest choice, even if it means transactions occasionally wait or retry.

The goal isn’t to choose the strongest isolation level.

The goal is to choose the weakest isolation level that still guarantees the correctness your application requires.

Doing so allows the database to maximize concurrency without sacrificing business integrity.

Isolation Levels and Performance

One mistake developers sometimes make is assuming higher isolation is always better.

In reality, every additional guarantee comes at a cost.

Higher isolation levels typically require the database to:

Coordinate more transactions.
Acquire additional locks or maintain more snapshots.
Detect conflicts more aggressively.
Delay or retry conflicting transactions.

As concurrency increases, these costs become more noticeable.

A high-traffic e-commerce platform processing thousands of orders every minute cannot afford unnecessary waiting if a lower isolation level already satisfies its business rules.

Likewise, a financial institution cannot sacrifice correctness simply to process a few extra transactions per second.

Finding the right balance is part of designing reliable software.

Isolation Levels vs Transactions vs Race Conditions

At this point in the series, we’ve covered three concepts that are closely related but often confused.

Let’s put them side by side.

Transactions

Transactions answer the question:

What happens if my operation fails halfway through?

They ensure a group of related database operations either all succeed together or all fail together.

Without transactions, partial updates can leave your data inconsistent.

Race Conditions

Race conditions answer a different question:

What happens if two requests modify the same data at the same time?

These problems arise because multiple users or systems interact with shared data concurrently.

The outcome often depends entirely on timing.

Transactions alone don’t eliminate race conditions.

Additional mechanisms such as locking, optimistic concurrency, or stronger isolation levels are often required.

Isolation Levels

Isolation levels answer yet another question:

While another transaction is running, what am I allowed to see?

Should your transaction observe unfinished work?

Should it continue seeing the same data even after another transaction commits?

Should it behave as though it’s the only transaction running?

Isolation levels define these rules.

Together, these three concepts form the foundation of reliable database applications.

They complement one another rather than compete.

A payment system, for example, might use:

Idempotency to prevent duplicate payment requests.
Transactions to ensure payment records and account balances remain synchronized.
Read Committed or Serializable isolation to guarantee consistent reads.
Locking to prevent concurrent modifications of the same account.

No single technique solves every reliability problem.

Reliable systems combine several techniques, each addressing a different class of failure.

Practical Advice for Backend Developers

If you’re just beginning your backend engineering journey, don’t feel pressured to master every isolation level immediately.

Instead, focus on developing the habit of asking the right questions whenever you design a feature.

For example:

Could another user modify this data while I’m reading it?
If the same query runs twice, should it return the same result?
What happens if another transaction inserts new rows before mine finishes?
Is perfect consistency necessary, or is slightly stale data acceptable?
Would optimistic concurrency or explicit locking be a better solution?

Thinking through these questions early in the design process often prevents bugs that are incredibly difficult to diagnose later in production.

Many concurrency issues aren’t caused by writing incorrect code.

They’re caused by making incorrect assumptions about how multiple users interact with the same data simultaneously.

Final Thoughts

Database isolation levels are often presented as a collection of definitions that developers are expected to memorize.

In reality, they’re much simpler than they first appear.

They’re simply different answers to one question:

How much should one transaction be allowed to observe while another transaction is still working?

Lower isolation levels prioritize concurrency, allowing more users to interact with the database simultaneously.

Higher isolation levels prioritize consistency, ensuring every transaction sees a predictable view of the data.

Neither approach is universally correct.

The right choice depends entirely on the problem you’re solving.

As your applications grow, understanding isolation levels becomes increasingly important because concurrency is no longer the exception—it’s the norm.

Every online store, banking application, inventory system, booking platform, and loan management system eventually reaches a point where multiple transactions compete for the same data.

The developers who understand isolation levels don’t simply build applications that work.

They build applications that continue working correctly under real-world load.

What’s Next?

In this series we’ve explored:

Idempotency
Race Conditions
Database Transactions
Database Isolation Levels

We’ve learned how to protect our systems from duplicate requests, concurrent updates, partial failures, and inconsistent reads.

But one challenge still remains.

Everything we’ve discussed assumes our application is running on a single database.

What happens when your application is running on ten servers, each processing requests simultaneously?

A normal database lock isn’t always enough.

In the next article, we’ll explore Distributed Locks Explained: Coordinating Work Across Multiple Servers, where we’ll see how systems like Redis, ZooKeeper, and etcd help ensure that only one application instance performs a critical operation at a time.

Database Isolation Levels Explained: Why Two Transactions Can See Different Data

Billy Okeyo — Mon, 06 Jul 2026 00:00:00 +0000

“Transactions guarantee that your work completes correctly. Isolation levels determine what everyone else is allowed to see while that work is happening.”

In the previous article, we explored database transactions and learned how they ensure multiple database operations either succeed together or fail together. Transactions protect applications from partial execution, preventing situations where money is deducted from one account without being deposited into another or where inventory is reduced without successfully creating an order.

But transactions solve only part of the problem.

Modern applications rarely have just one user interacting with the database at a time. Thousands of customers may be placing orders, updating records, making payments, or querying reports simultaneously. Each of these actions runs inside its own transaction, and more often than not, several transactions are accessing the same data at exactly the same time.

This raises an important question:

What should one transaction be allowed to see while another transaction is still running?

Imagine opening your banking application to check your account balance.

At the exact same moment, your employer’s payroll system is depositing your monthly salary into the same account.

Should your transaction see the new balance immediately?

Should it continue seeing the old balance until the salary transaction finishes?

Should it wait until the payroll transaction completes before showing you anything at all?

Each answer is technically valid depending on how the database is configured.

Now imagine an online store where only one laptop remains in stock.

Customer A begins placing an order.

Before their transaction finishes, Customer B checks the product page.

Should Customer B still see one laptop available?

Should they see zero?

Should they wait until Customer A’s purchase either succeeds or fails?

Again, the answer depends on the database’s isolation level.

Isolation levels define the rules governing how concurrent transactions interact with one another. They determine whether one transaction can observe another transaction’s work before it has been completed, whether repeated reads always return the same result, and whether new rows appearing during a transaction should be visible immediately.

Although isolation levels are often introduced as an advanced database topic, they’re really about one simple idea:

How much of another transaction’s work should your transaction be allowed to see?

The answer has significant consequences for both correctness and performance.

In this article, we’ll explore why isolation levels exist, the concurrency problems they solve, the four SQL standard isolation levels, and how to choose the right one for your application.

Why Isolation Exists

To understand isolation, imagine you’re reading a book in a library.

Halfway through chapter three, someone walks over, quietly replaces several pages with new ones, and walks away.

You continue reading without realizing anything changed.

The beginning of the chapter describes one story.

The ending describes another.

Nothing makes sense.

Databases can experience a remarkably similar problem.

When multiple transactions execute simultaneously, each transaction may be reading data while another transaction is actively changing it.

Without rules governing these interactions, applications could make decisions based on incomplete information, outdated values, or data that is eventually discarded.

Isolation exists to prevent these situations.

Rather than allowing every transaction unrestricted access to every change happening in the database, the database controls what each transaction can observe and when it can observe it.

Think of it as putting walls between transactions.

Some walls are very thin.

Transactions can see almost everything happening around them.

Other walls are much thicker.

Transactions operate almost as though they’re the only users of the database.

The thicker the wall, the more isolated the transaction becomes.

The Trade-Off Between Consistency and Performance

At first glance, it might seem obvious that every database should simply use the highest possible isolation level.

After all, if stronger isolation produces more consistent data, why wouldn’t every system choose it?

The answer lies in performance.

Imagine a supermarket with only one checkout counter.

Every customer waits patiently in line.

Because only one cashier is serving customers, inventory updates happen one at a time.

Mistakes are rare.

Unfortunately, the queue becomes enormous.

Now imagine opening ten checkout counters.

Customers move much faster.

However, all ten cashiers are now updating the same inventory system simultaneously.

Keeping everything synchronized becomes much more difficult.

Databases face exactly the same challenge.

Higher isolation levels provide stronger guarantees about data consistency, but they often require additional locking, coordination, and waiting.

Lower isolation levels allow more transactions to execute concurrently, increasing throughput and reducing latency, but they also increase the likelihood that transactions observe changing data.

Isolation levels are therefore a balancing act between two competing goals:

Consistency, ensuring every transaction sees predictable and reliable data.
Concurrency, allowing as many users as possible to interact with the system simultaneously.

Different applications make different choices.

A banking application processing financial transfers typically prioritizes correctness over raw performance.

An analytics dashboard generating sales reports might tolerate slightly older data if it means thousands of users can run reports simultaneously without slowing the system.

Neither approach is universally correct.

The appropriate isolation level depends entirely on your business requirements.

Concurrency Anomalies: The Problems Isolation Levels Exist to Solve

Isolation levels were not invented simply to make databases more complicated.

They exist because concurrent transactions can produce behaviors that most developers would consider surprising—or even dangerous.

These unexpected behaviors are collectively known as concurrency anomalies.

Every isolation level is essentially a trade-off between preventing these anomalies and maintaining good performance.

Before discussing the isolation levels themselves, it’s important to understand the problems they are designed to solve.

The four anomalies you’ll encounter most often are:

Dirty Reads
Non-Repeatable Reads
Phantom Reads
Lost Updates

Each represents a different way concurrent transactions can interfere with one another.

Let’s begin with the simplest.

Dirty Reads

Imagine Alice has KES 50,000 in her account.

She initiates a transfer of KES 20,000 to another account.

The banking system begins processing the transaction.

The first step deducts the money from Alice’s balance.

Before the transaction finishes, another process—perhaps an ATM balance inquiry or an online banking session—checks Alice’s account.

At that moment, it sees a balance of KES 30,000.

Everything seems perfectly normal.

Then something unexpected happens.

The transfer fails because the destination account no longer exists.

The database rolls back the transaction.

Alice’s balance immediately returns to KES 50,000.

The second transaction has now made a decision based on information that never officially existed.

It observed data that was eventually discarded.

This is known as a Dirty Read.

A dirty read occurs when one transaction reads data written by another transaction before that transaction has been committed.

The easiest way to understand it is to imagine reading someone’s unfinished draft before they’ve decided whether to keep or delete it.

The version you read may never become the final version.

Making business decisions based on that draft could lead to incorrect outcomes.

Fortunately, most modern relational databases prevent dirty reads by default because they are rarely desirable in business applications.

The SQL standard still defines them because they help explain the spectrum of isolation levels.

Timeline of a Dirty Read

Transaction A                     Transaction B

BEGIN

Balance = 50,000

↓

Update Balance = 30,000

                               Read Balance = 30,000 ❌

↓

Transfer Fails

↓

ROLLBACK

Balance returns to 50,000

Transaction B has observed a value that disappeared moments later.

From the perspective of the database, that balance never officially existed.

Yet another transaction already acted as though it did.

This is precisely the type of inconsistency isolation levels are designed to prevent.

Non-Repeatable Reads

Suppose you’re building an online banking application.

A customer opens the app and views their account balance. At that moment, the database reports a balance of KES 50,000.

The customer decides to transfer KES 40,000 to another account, but before confirming the transfer, the application performs one final balance check to ensure sufficient funds are still available.

This seems like a perfectly reasonable workflow.

However, between the first balance check and the second, another transaction deposits KES 100,000 into the same account.

When the application performs the second query, the balance is no longer KES 50,000.

It’s now KES 150,000.

Nothing is technically wrong.

The second transaction committed successfully.

The balance genuinely changed.

The surprising part is that the same transaction read the same row twice and received two different answers.

This phenomenon is known as a Non-Repeatable Read.

Unlike a dirty read, the second transaction isn’t reading uncommitted data. Every value it sees has been permanently committed to the database.

The inconsistency comes from the fact that another transaction modified the row while the first transaction was still running.

Imagine reading yesterday’s newspaper while someone keeps replacing pages with today’s edition.

The information isn’t incorrect.

It’s simply inconsistent because the document changed while you were reading it.

For many applications, this isn’t a problem.

If you’re refreshing a weather dashboard or checking the number of users currently online, it’s perfectly acceptable for values to change between two queries.

However, systems that rely on a stable snapshot of data—such as financial reporting, payroll processing, or end-of-day reconciliation—often require the same query to return the same result throughout the entire transaction.

Timeline of a Non-Repeatable Read

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 150,000 ❌

COMMIT

Transaction A never modified the balance itself.

It simply asked the same question twice and received two different answers because another committed transaction changed the underlying data in the meantime.

Real-World Example: Updating a Customer Profile

Consider an insurance application where a customer service representative opens a customer’s profile.

The representative spends several minutes reviewing the information before approving a policy update.

Meanwhile, another employee updates the customer’s phone number and address.

When the representative finally clicks Save, the application may now be working with information that is different from what was originally displayed.

Depending on how the application handles these changes, it could accidentally overwrite newer data or make decisions using outdated information.

This isn’t a database bug.

It’s simply the natural consequence of multiple users interacting with the same record at the same time.

Applications that require users to work with a consistent view of the data often use higher isolation levels or optimistic concurrency controls to detect these situations before committing changes.

Phantom Reads

Now let’s consider a different scenario.

Instead of reading a single row twice, imagine you’re querying an entire collection of rows.

Suppose you’re generating a report showing all loan applications submitted today.

Your first query returns:

Loan Applications Submitted Today

-------------------------------

Loan #101

Loan #102

Loan #103

Total: 3

While your report is still running, another loan application is submitted and committed to the database.

A few moments later, your transaction performs the exact same query again.

This time the results look different.

Loan Applications Submitted Today

-------------------------------

Loan #101

Loan #102

Loan #103

Loan #104

Total: 4

Notice what changed.

None of the existing rows were modified.

Instead, an entirely new row appeared.

This is called a Phantom Read.

A phantom read occurs when the same query returns a different set of rows because another transaction inserted, updated, or deleted records that match the query’s search criteria.

Think of it like counting the number of people in a room.

You count 20 people.

While you’re writing the number down, someone walks into the room.

You count again.

Now there are 21 people.

Nothing about the original twenty people changed.

The difference is that a new “phantom” appeared between your two observations.

Timeline of a Phantom Read

Transaction A                     Transaction B

BEGIN

SELECT *

WHERE loan_date = TODAY

Returns 3 rows

                               BEGIN

                               INSERT Loan #104

                               COMMIT

SELECT *

WHERE loan_date = TODAY

Returns 4 rows ❌

COMMIT

Unlike a non-repeatable read, where an existing row changes, phantom reads involve the appearance or disappearance of entire rows.

Why Phantom Reads Matter

Imagine you’re calculating today’s total revenue for financial reporting.

Your reporting transaction begins at 5:00 PM and starts aggregating sales.

While it’s still processing, new sales continue being recorded.

Different parts of the report may now be working with different datasets.

The total revenue calculated on page one might not match the detailed transaction list generated on page five because new rows appeared while the report was still executing.

In reporting systems, this can produce confusing and inconsistent results.

Higher isolation levels solve this problem by ensuring the transaction sees a consistent snapshot of the data throughout its lifetime, even if other transactions continue inserting new rows.

Lost Updates

The final concurrency anomaly is perhaps the most dangerous because it silently discards valid work.

Imagine two warehouse employees looking at the same inventory record.

The system currently shows:

Laptop Stock = 10

Employee A sells one laptop.

Employee B also sells one laptop at almost exactly the same time.

Both employees read the current stock before making their update.

Each calculates the new quantity as:

10 - 1 = 9

Employee A saves.

The inventory becomes:

A fraction of a second later, Employee B saves.

The inventory is still:

One of the updates has effectively disappeared.

The correct inventory should now be 8, but because both transactions started from the same original value, one update overwrote the other.

This is known as a Lost Update.

Unlike the previous anomalies, nothing appears obviously wrong.

No errors occur.

No constraints are violated.

The database happily accepts both updates.

The problem is that one user’s work has unintentionally replaced another’s.

Lost updates are one of the primary reasons databases provide row locking, optimistic concurrency control, and stronger isolation levels.

Without these protections, applications that receive many simultaneous updates—such as inventory systems, banking platforms, or booking applications—can slowly drift away from reality without anyone noticing.

Now that we understand the problems, the next question is obvious: How do databases prevent them? That’s exactly what we’ll cover in Part 2.”

Database Transactions Explained: Keeping Data Correct When Things Go Wrong

Billy Okeyo — Fri, 03 Jul 2026 00:00:00 +0000

“Transactions are not just a database feature—they’re one of the fundamental building blocks of reliable software.”

Imagine you’re transferring KES 10,000 from your savings account to a friend.

From your perspective, it’s a single action. You tap “Send Money”, authenticate the transaction, and wait for confirmation. Behind the scenes, however, the banking system performs several independent operations. It verifies that you have sufficient funds, deducts the amount from your account, credits your friend’s account, records the transaction in a ledger, updates account balances, and generates a receipt.

Each of these operations is important, but together they represent a single business action: a money transfer.

Now imagine the server crashes immediately after deducting the money from your account but before crediting your friend.

The result is disastrous. Your balance has decreased, your friend never receives the money, and unless additional recovery mechanisms exist, the system is left in an inconsistent state. From a customer’s perspective, the money has simply disappeared.

The same problem appears outside banking.

An e-commerce application might create an order, reduce inventory, process a payment, generate an invoice, and send a confirmation email. If the payment succeeds but the order creation fails, the customer has paid for a product that the system doesn’t believe exists. Likewise, in a loan management system, a repayment may update the outstanding balance, post accounting entries, and generate a receipt. If only some of those updates complete, financial records quickly become unreliable.

These problems aren’t caused by bad algorithms or poor business logic. They’re caused by partial execution when only part of a larger operation succeeds.

This is precisely the problem database transactions were designed to solve.

A transaction ensures that multiple database operations behave as a single unit of work. Either every operation succeeds together, or every operation is rolled back as though nothing ever happened. There is no halfway point where your system is left in an inconsistent state.

For backend developers, understanding transactions is just as important as understanding APIs or databases themselves. They are the foundation upon which reliable financial systems, booking platforms, inventory systems, healthcare applications, and countless other business-critical systems are built.

What Is a Database Transaction?

A database transaction is a collection of one or more database operations that the database treats as a single logical operation.

Instead of thinking about individual SQL statements, think about the business process they represent.

Suppose a customer purchases the last laptop in your online store. That single purchase might require your application to:

Create an order.
Deduct one item from inventory.
Reserve the shipment.
Record the payment.
Create an invoice.

Although these are separate SQL statements, they represent one business event. Either all of them should succeed, or none of them should.

That’s exactly what a transaction guarantees.

Without transactions, every statement executes independently. If statement number four fails, the previous three remain committed, leaving your data inconsistent.

With transactions, the database waits until you’re satisfied that every operation has completed successfully. Only then are the changes permanently saved.

Understanding Transactions Through a Simple Example

Let’s return to the banking example.

Alice wants to transfer KES 10,000 to Bob.

A simplified version of the SQL might look like this:

UPDATE accounts
SET balance = balance - 10000
WHERE account_id = 1;

UPDATE accounts
SET balance = balance + 10000
WHERE account_id = 2;

At first glance, this seems perfectly reasonable.

But imagine the database crashes immediately after the first statement executes.

Alice’s balance has already been reduced.

Bob’s balance has not increased.

The system now contains incorrect financial data.

This is why production systems rarely execute related operations independently.

Instead, they wrap them inside a transaction.

BEGIN;

UPDATE accounts
SET balance = balance - 10000
WHERE account_id = 1;

UPDATE accounts
SET balance = balance + 10000
WHERE account_id = 2;

COMMIT;

If every statement succeeds, the database executes the COMMIT, making the changes permanent.

If any statement fails before that point, the application issues a ROLLBACK, and the database restores itself to exactly the state it was in before the transaction began.

To the outside world, it appears as though the failed transfer never happened.

This “all-or-nothing” behavior is what makes transactions so valuable.

The Lifecycle of a Transaction

Every transaction follows a predictable lifecycle.

BEGIN
   │
Execute SQL Operations
   │
Everything Successful?
   │
 ┌─ Yes ─────────────┐
 │                   │
COMMIT          Changes Saved
 │
 └─ No ──────────────┐
                     │
                 ROLLBACK
                     │
          Database Restored

The process begins with BEGIN, which tells the database to temporarily hold all modifications rather than immediately committing them.

The application then performs one or more operations. These could involve inserting records, updating balances, deleting data, or modifying relationships between tables.

If every operation succeeds, the application calls COMMIT. At that point, the database permanently saves all changes.

If anything goes wrong along the way, perhaps a validation error, a database constraint violation, or even an unexpected server failure, the transaction is rolled back, discarding every change made since BEGIN.

The beauty of this model is that the database itself guarantees consistency. Developers don’t need to manually undo every failed operation because the database handles that responsibility.

The Four ACID Properties

When developers discuss transactions, you’ll almost always hear the term ACID.

Despite sounding intimidating, ACID simply describes the guarantees that modern relational databases provide when executing transactions.

Atomicity

Atomicity means that a transaction is indivisible.

Either every operation succeeds, or none of them do.

Returning to our banking example, it makes no sense for money to be deducted from one account without being added to another. The transaction must succeed completely or fail completely.

Think of flipping a light switch.

The light cannot be half on.

Similarly, a transaction cannot be half completed.

Consistency

Consistency ensures that every transaction leaves the database in a valid state.

Business rules should always remain true.

If your application enforces that inventory can never become negative, then a successful transaction should never violate that rule.

Likewise, if your accounting system requires every journal entry to balance, no committed transaction should ever leave the ledger unbalanced.

Consistency isn’t about preventing bugs in your application logic; it’s about ensuring that completed transactions respect the rules defined by your database and your business.

Isolation

Isolation becomes important when multiple users interact with the system simultaneously.

Imagine two customers attempting to purchase the last available ticket for a concert.

Without proper isolation, both requests may read the inventory before either updates it. Both believe the ticket is available, and both complete the purchase.

You’ve now sold the same seat twice.

Isolation ensures that concurrent transactions don’t interfere with one another in ways that produce inconsistent results.

In our previous article, we discussed race conditions situations where multiple requests compete to modify the same data. Isolation is one of the database mechanisms used to prevent those concurrency problems.

We’ll explore isolation levels in greater depth in the next article because they deserve an entire discussion of their own.

Durability

Durability guarantees that once a transaction has been committed, the changes are permanent.

Even if the server loses power immediately after the commit, the database ensures that committed data survives.

Modern databases achieve this through techniques such as write-ahead logging, transaction logs, and crash recovery.

For developers, the important takeaway is simple: once the database confirms a successful commit, you can trust that the data has been safely stored.

Transactions Solve Partial Failures: Not Every Problem

One misconception among newer developers is that transactions magically solve every data consistency problem.

They don’t.

Transactions protect against partial execution.

Suppose your application updates three tables and crashes after updating the second one. A transaction ensures that the database rolls everything back, preventing inconsistent data. However, transactions don’t automatically solve concurrency problems.

Imagine two users attempting to withdraw money from the same account simultaneously. Each transaction independently checks the balance before either completes. If both see the same balance and both proceed, the final result may still be incorrect depending on your isolation level. This isn’t a transaction problem. It’s a concurrency problem.

That’s why understanding race conditions and transactions together is so important. They solve different classes of reliability issues.

Common Places You’ll Use Transactions

Transactions appear almost everywhere in modern backend systems.

Payment processing is perhaps the most obvious example. Charging a customer’s card, recording the payment, updating invoices, and generating accounting entries should either all succeed or all fail together.

Inventory management systems use transactions to ensure stock counts remain accurate even when multiple customers are purchasing products simultaneously.

Booking platforms rely on transactions to reserve hotel rooms, airline seats, or event tickets without creating conflicting reservations.

Loan management systems use transactions when posting repayments, updating outstanding balances, calculating accrued interest, and recording accounting entries.

Healthcare systems use them to ensure patient records, prescriptions, billing information, and appointment schedules remain synchronized.

Any time a business operation spans multiple database changes, a transaction is usually involved.

Common Mistakes Developers Make

One of the most common mistakes is keeping transactions open for too long.

Imagine starting a transaction, calling a third-party payment API, waiting several seconds for a response, and only then committing the transaction.

During that entire period, database resources may remain locked, reducing performance for other users.

A better approach is to perform external API calls before starting the transaction whenever possible, keeping the transaction focused solely on database operations.

Another common mistake is assuming transactions automatically protect against concurrent updates. As we’ve already seen, concurrency introduces an entirely different set of challenges that require locking strategies or appropriate isolation levels.

Finally, developers sometimes forget that transactions should represent business operations not individual SQL statements. Wrapping every single query in its own transaction rarely provides meaningful benefits.

Transactions in Modern Frameworks

Fortunately, most frameworks make transactions straightforward to use.

Laravel offers the DB::transaction() helper.

Django provides transaction.atomic().

Entity Framework supports BeginTransactionAsync().

Spring Boot uses the @Transactional annotation.

Although the syntax differs, the underlying principle never changes. The framework simply tells the database when to begin the transaction, when to commit it, and when to roll it back if something goes wrong.

Understanding the concept matters far more than memorizing framework-specific syntax.

Transactions, Idempotency, and Race Conditions

If you’ve been following this series, you may have noticed that each concept addresses a different reliability challenge.

Idempotency protects against duplicate requests by ensuring that repeating the same request doesn’t produce duplicate side effects.

Race conditions occur when multiple requests compete to modify shared data simultaneously, leading to unpredictable outcomes.

Transactions ensure that a group of related database operations either all succeed together or all fail together.

Reliable backend systems typically rely on all three.

Imagine a payment API.

Idempotency prevents customers from being charged twice if they retry a request.

Transactions ensure that charging the customer, recording the payment, and updating account balances either all succeed or all fail.

Proper concurrency control ensures that two simultaneous payment requests don’t corrupt shared data.

Each concept complements the others rather than replacing them.

Final Thoughts

Transactions are one of the reasons relational databases remain so powerful. They provide developers with a reliable mechanism for preserving data integrity even when failures occur.

As systems become larger and more distributed, failures become inevitable. Servers crash, networks fail, APIs time out, and users submit requests simultaneously. Transactions don’t eliminate those realities, but they ensure your database remains consistent when they happen.

Whenever you’re implementing a feature that modifies multiple pieces of related data, pause for a moment and ask yourself:

What happens if this operation fails halfway through?

If the answer is “my system ends up in an inconsistent state,” then you’ve almost certainly found a place where a database transaction belongs.

In the next article, we’ll build on this foundation by exploring database isolation levels and why two perfectly valid transactions can still interfere with one another when they run at the same time.

Race Conditions Explained: The Concurrency Bug Every Backend Developer Should Understand

Billy Okeyo — Sun, 28 Jun 2026 00:00:00 +0000

Imagine you’re trying to buy the last ticket for your favorite concert.

The website shows:

Only 1 ticket remaining.

At exactly the same moment, someone else clicks Buy.

Both of you complete payment.

Both of you receive confirmation emails.

But there was only one ticket.

How did two people successfully purchase the same seat?

Now imagine the same thing happening in a banking application.

Two ATM withdrawals happen at almost the same time.

Both check the balance before either transaction finishes.

Both think there’s enough money.

Both approve the withdrawal.

The account ends up with a negative balance.

Neither application is necessarily “broken.”

Instead, they suffer from one of the most common problems in software engineering:

Race conditions.

Race conditions are among the hardest bugs to reproduce because they don’t happen every time. They may only appear under heavy traffic, high concurrency, or perfect timing. Everything works during testing until your application reaches production.

In this article, we’ll explore what race conditions are, why they happen, how they relate to idempotency, and the techniques developers use to prevent them.

What Is a Race Condition?

A race condition occurs when two or more operations access and modify the same piece of data at the same time, and the final result depends on the order in which they execute.

The important phrase is:

The outcome depends on timing.

That’s what makes race conditions so dangerous.

Sometimes everything works.

Sometimes everything breaks.

The exact same code can produce different results simply because two requests happened a few milliseconds apart.

A Simple Analogy

Imagine two people standing in front of a cookie jar.

The jar contains exactly one cookie.

Both people look inside.

Both see one cookie.

Both reach in.

Both believe they’ll get the cookie.

Reality says otherwise.

Only one cookie exists.

The mistake happened because both people checked the state before either updated it.

Software behaves exactly the same way.

A Real Banking Example

Suppose an account has:

Balance = $100

Two withdrawal requests arrive simultaneously.

Request A → Withdraw $80

Request B → Withdraw $50

Without synchronization:

Request A
↓

Read Balance ($100)

------------

Request B

↓

Read Balance ($100)

------------

Request A

Balance = $20

------------

Request B

Balance = $50

Depending on timing, the final balance might be:

$20

or

$50

or

-$30

None of these outcomes are guaranteed.

This is a race condition.

Why It Works During Development

Most developers test applications alone.

One request.

One browser.

One user.

Everything works perfectly.

Production is different.

Imagine:

5,000 users
Hundreds of requests per second
Multiple application servers
Database replication
Network latency

The probability of two requests colliding becomes much higher.

Race conditions often appear only after an application becomes successful.

Ironically, scaling your application can reveal bugs that never existed during development.

Race Conditions vs Idempotency

If you’ve read my previous article on idempotency, if not, read it first here. You might wonder whether they’re the same thing.

They’re related, but they solve different problems.

Idempotency answers:

What if the same request is sent twice?

Example:

POST /payments

↓

Retry

↓

POST /payments

The solution is an Idempotency-Key.

The same request produces the same result.

Race conditions answer:

What if different requests happen at the same time?

Example:

User A buys last ticket

↓

User B buys last ticket

These are two legitimate requests from different users.

An idempotency key won’t help because the requests are not duplicates.

Race conditions require synchronization, not deduplication.

Real-World Examples

Airline Seat Booking

Only one seat remains.

Two customers purchase it simultaneously.

Without proper locking:

Seat sold twice
Refund required
Customer frustration

E-Commerce Inventory

Stock:

Laptop

Quantity = 1

Two customers purchase simultaneously.

Without protection:

Inventory becomes:

-1

Now you’ve sold a product you don’t have.

Loan Approval Systems

Imagine two loan officers reviewing the same application.

Officer A:

Approve.

Officer B:

Reject.

If both updates happen simultaneously without coordination, the final loan status depends entirely on timing.

Coupon Redemption

Promotion:

First 100 customers only

If multiple requests update the redemption count simultaneously, you might accidentally issue:

103

105

110

coupons.

How Race Conditions Happen

Most race conditions follow this pattern:

Read

↓

Modify

↓

Write

The danger lies between Read and Write.

Example:

Read Balance

↓

Calculate New Balance

↓

Update Balance

If another request changes the balance between those steps, your calculation becomes outdated.

This is known as a Lost Update problem.

Solution 1: Database Transactions

Transactions ensure multiple operations succeed or fail together.

Example:

BEGIN;

SELECT balance
FROM accounts
WHERE id = 1;

UPDATE accounts
SET balance = balance - 80
WHERE id = 1;

COMMIT;

Transactions protect data consistency.

However, they don’t automatically eliminate every race condition.

Isolation levels matter too.

Solution 2: Row-Level Locks

Many relational databases allow locking a row while it’s being updated.

Example:

SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;

Now:

Request A locks the row.

Request B must wait.

Only after Request A finishes can Request B continue.

This guarantees consistent updates.

Solution 3: Optimistic Locking

Instead of preventing conflicts, optimistic locking detects them.

Imagine a version number.

Account

Balance = 100

Version = 5

Update:

WHERE Version = 5

If another request updates the row first:

Version = 6

Your update fails.

The client retries with fresh data.

Optimistic locking works well when collisions are relatively rare.

Solution 4: Distributed Locks

What if your application runs on multiple servers?

Server A

↓

Database

↑

Server B

A normal in-memory lock won’t work because each server has its own memory.

Instead, developers use distributed locking systems like:

Redis (Redlock)
ZooKeeper
etcd
Consul

These coordinate access across multiple application instances.

Solution 5: Atomic Database Operations

Sometimes you don’t need to:

Read

↓

Calculate

↓

Write

Instead, let the database perform the update atomically.

Bad:

SELECT quantity;

quantity--;

UPDATE products;

Better:

UPDATE products

SET quantity = quantity - 1

WHERE quantity > 0;

Now the database guarantees consistency.

Detecting Race Conditions

One reason race conditions are difficult is that they rarely appear during manual testing.

Ways to uncover them include:

Load testing with concurrent users
Stress testing
Running parallel integration tests
Simulating delayed responses
Chaos engineering
Monitoring production logs

If a bug only appears “sometimes,” concurrency should be one of your first suspects.

Best Practices

When designing systems that handle shared data:

Keep transactions short.
Avoid long-running locks.
Prefer atomic database operations where possible.
Use optimistic locking when contention is low.
Use pessimistic locking for critical resources.
Test with concurrent requests, not just sequential ones.
Understand your database’s transaction isolation levels.

Most importantly, always ask:

“What happens if two users do this at exactly the same time?”

Final Thoughts

Race conditions aren’t caused by bad developers, they’re caused by systems becoming concurrent.

As applications grow, users interact simultaneously, background jobs overlap, and services communicate in parallel. Timing becomes unpredictable.

That’s why building reliable software isn’t just about writing correct logic for one request. It’s about ensuring your logic remains correct when hundreds or thousands of requests happen together.

If idempotency protects you from duplicate requests, race condition handling protects you from competing requests.

Together, they form two of the most important building blocks for designing resilient APIs and distributed systems.

The next time you write code that reads, modifies, and writes shared data, pause for a moment and ask:

“What happens if someone else does this at the exact same time?”

If you don’t know the answer, you’ve just found your next engineering problem to solve.

Idempotency Explained: Building APIs That Survive Retries

Billy Okeyo — Thu, 25 Jun 2026 00:00:00 +0000

Imagine you're purchasing a product online.

You click the "Pay Now" button.

Nothing happens.

After a few seconds, you assume the request failed, so you click the button again.

And again.

A few minutes later, you discover you've been charged three times.

What happened?

From the user's perspective, the payment seemed to fail. From the server's perspective, however, it successfully processed every request it received.

This is one of the most common problems in distributed systems, and it's exactly why idempotency exists.

Whether you're building payment systems, booking platforms, inventory management software, or any API that changes data, retries are inevitable. Networks fail, clients time out, mobile connections drop, and users double-click buttons.

A well-designed API should survive these retries without creating duplicate side effects.

In this article, we'll explore what idempotency is, why it matters, and how to implement it in your own APIs.

What Is Idempotency?

In simple terms, an idempotent operation can be performed multiple times without changing the final result beyond the first successful execution.

For example:

Turn on the light.
Turn on the light again.
Turn on the light again.

The light is still ON.

Nothing new happened after the first request.

The final state remains the same.

That's idempotency.

Now compare it with this:

Deposit $100
Deposit $100
Deposit $100

Your account balance increases by $300.

This operation is not idempotent because every request changes the system.

Why Retries Happen

Many developers assume users only send one request.

Reality is different.

Requests are retried because of:

Slow internet connections
Gateway timeouts
Reverse proxies
Mobile network interruptions
Browser refreshes
Double-clicking buttons
Client retry mechanisms
Load balancers
Microservice communication failures

Imagine this timeline:

Client -------- POST /payments --------> API

             Payment succeeds

API -------- 200 OK --------X

(Response never reaches client)

Client waits...

Client retries.

POST /payments again.

The client believes the payment failed.

The server already completed it.

Without idempotency...

The payment happens twice.

HTTP Methods and Idempotency

HTTP itself distinguishes between idempotent and non-idempotent methods.

GET

GET /users/10

Read the user.

Call it once.

Call it 100 times.

Nothing changes.

Idempotent

PUT

PUT /users/10
{
   "name": "Billy"
}

Replacing the same resource repeatedly produces the same result.

Idempotent

DELETE

DELETE /users/10

Delete the user.

Deleting an already deleted user doesn't delete them twice.

The final state is still:

User does not exist.

Idempotent

POST

POST /orders

Create a new order.

Call it twice.

You now have two orders.

Not idempotent

This is why POST requests often require additional protection.

Why Payment APIs Use Idempotency Keys

Payment providers like Stripe popularized the use of Idempotency Keys.

The idea is simple.

The client generates a unique identifier.

Example:

Idempotency-Key:
6ab89d3b-acde-4d71-b20d-483d8d0ef091

Every retry sends the same key.

POST /payments

Idempotency-Key:
6ab89d3b-acde-4d71-b20d-483d8d0ef091

When the server receives the request:

Check if this key already exists.
If not, process the payment.
Save both the key and the response.
Return the response.

If the same request arrives again with the same key:

Instead of charging the customer again...

Return the previously stored response.

Client

POST /payments
Key: ABC123

↓

Server

Charge customer

↓

Store

ABC123 → Payment #456

↓

Return success

---

Retry

POST /payments
Key: ABC123

↓

Lookup

ABC123 exists

↓

Return Payment #456

No second charge.

Implementing Idempotency

A common workflow looks like this.

Step 1

Receive request.

POST /orders

Headers

Idempotency-Key:
XYZ987

Step 2

Search database.

SELECT *
FROM idempotency_keys
WHERE key = 'XYZ987'

Found?

Yes.

Return stored response.

Done.

Step 3

Not found?

Create the resource.

Create Order

Step 4

Store:

Key

Response

Status Code

Timestamp

Now every retry returns the same response.

Example in Node.js (Express)

app.post("/payments", async (req, res) => {
    const key = req.header("Idempotency-Key");

    const existing = await Idempotency.findOne({ key });

    if (existing) {
        return res.status(existing.status).json(existing.response);
    }

    const payment = await processPayment(req.body);

    await Idempotency.create({
        key,
        status: 201,
        response: payment,
    });

    return res.status(201).json(payment);
});

Python (FastAPI)

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/payments")
async def create_payment(request: Request):
    key = request.headers.get("Idempotency-Key")

    existing = await Idempotency.find_one(key=key)

    if existing:
        return JSONResponse(
            content=existing.response,
            status_code=existing.status,
        )

    body = await request.json()
    payment = await process_payment(body)

    await Idempotency.create(
        key=key,
        status=201,
        response=payment,
    )

    return JSONResponse(content=payment, status_code=201)

C# (ASP.NET Core)

app.MapPost("/payments", async (
    HttpRequest request,
    IdempotencyStore store,
    PaymentService payments) =>
{
    var key = request.Headers["Idempotency-Key"].ToString();

    var existing = await store.FindAsync(key);
    if (existing is not null)
    {
        return Results.Json(existing.Response, statusCode: existing.Status);
    }

    var body = await request.ReadFromJsonAsync<PaymentRequest>();
    var payment = await payments.ProcessAsync(body!);

    await store.CreateAsync(new IdempotencyRecord(key, 201, payment));

    return Results.Json(payment, statusCode: StatusCodes.Status201Created);
});

Go

func createPayment(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")

    existing, err := idempotency.FindOne(r.Context(), key)
    if err == nil && existing != nil {
        w.WriteHeader(existing.Status)
        json.NewEncoder(w).Encode(existing.Response)
        return
    }

    var body PaymentRequest
    if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    payment, err := processPayment(r.Context(), body)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    if err := idempotency.Create(r.Context(), IdempotencyRecord{
        Key:      key,
        Status:   http.StatusCreated,
        Response: payment,
    }); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(payment)
}

Laravel

Route::post('/payments', function (Request $request) {
    $key = $request->header('Idempotency-Key');

    $existing = Idempotency::where('key', $key)->first();

    if ($existing) {
        return response()->json($existing->response, $existing->status);
    }

    $payment = processPayment($request->all());

    Idempotency::create([
        'key' => $key,
        'status' => 201,
        'response' => $payment,
    ]);

    return response()->json($payment, 201);
});

The logic is surprisingly simple.

The complexity comes from storing and managing the keys correctly.

Where Should Idempotency Keys Be Stored?

Options include:

Database

Best for most applications.

Pros:

Persistent
Reliable
Easy to query

Cons:

Slightly slower

Redis

Excellent for high-volume APIs.

Pros:

Extremely fast
TTL support
Easy expiration

Many APIs automatically expire keys after 24 hours.

In-Memory

Useful only during development.

Not recommended for production.

Restarting the server loses everything.

Common Mistakes

Reusing Keys

Every logical operation should have its own unique key.

Bad:

ABC123

used today

used tomorrow

Good:

New checkout

↓

Generate new UUID

Ignoring Request Differences

Suppose the first request is:

$50

The retry is:

$500

Same key.

Different body.

The server should reject this request because the key is being reused for a different operation.

Never Expiring Keys

Keeping millions of old keys forever wastes storage.

Most APIs expire them after:

24 hours
48 hours
7 days

depending on business requirements.

Real-World Use Cases

Idempotency is valuable anywhere duplicate requests could have costly consequences.

Examples include:

Payment processing
Bank transfers
Order creation
Hotel reservations
Flight bookings
Ticket purchases
Subscription billing
Inventory updates
Webhook processing
Email sending
Message queues

If performing the same action twice could create an incorrect outcome, idempotency is worth considering.

When You Don't Need Idempotency

Not every endpoint needs an idempotency key.

For example:

GET /posts

No state changes.

No duplicates.

No problem.

Likewise, endpoints such as:

Search
Filtering
Reading reports
Viewing profiles

are already naturally idempotent.

Reserve idempotency mechanisms for operations where retries could create unintended side effects.

Final Thoughts

Idempotency isn't just an implementation detail—it's a reliability feature.

In distributed systems, retries are normal. Networks are unreliable, users click buttons more than once, and clients retry requests automatically. Instead of hoping those situations never happen, design your APIs to handle them gracefully.

By using idempotency keys, storing responses, validating retries, and choosing the right storage strategy, you can prevent duplicate orders, repeated payments, and other costly errors.

A resilient API isn't one that never receives duplicate requests.

It's one that produces the correct outcome even when duplicate requests inevitably arrive.

The next time you design a POST endpoint, ask yourself:

"What happens if this request is sent twice?"

If the answer is "something bad," it's probably time to add idempotency.

Debugging Is a Skill Nobody Teaches You

Billy Okeyo — Mon, 04 May 2026 00:00:00 +0000

You’ve been staring at the same bug for 2 hours.
You’ve restarted the server. Cleared cache. Added random console.logs.
Somehow… it still doesn’t work.

At some point, you stop coding and start guessing.

And that’s the real problem.

The issue isn’t the bug.
It’s that nobody actually teaches debugging as a skill.

The Way Most Developers Debug

Let’s be honest. Most of us learned debugging like this:

Sprinkle console.log everywhere
Change random lines and hope something works
Copy-paste error messages into Google
Restart everything “just in case”

It sometimes works.

But it’s slow, frustrating, and unreliable.

It’s not debugging.

It’s trial and error disguised as progress.

What Debugging Actually Is

Here’s the mindset shift that changes everything:

Debugging is not about fixing code.
It’s about finding where your mental model diverges from reality.

You think the system works one way.

Reality says otherwise.

Your job is to close that gap.

The Debugging Mindset

Before tools, before techniques, this is what matters most.

1. Assume Your Assumptions Are Wrong

If something doesn’t work, at least one thing you believe is false.

Your job is to find it.

2. Narrow the Problem Space

Bad debugging:

“Something is wrong with the app”

Good debugging:

“The issue happens only when this function runs after login”

3. Reproduce Before Fixing

If you can’t reliably reproduce the bug, you don’t understand it.

And if you don’t understand it, your fix is luck—not skill.

4. One Change at a Time

If you change 5 things and it works…
which one fixed it?

You don’t know.

That’s how bugs come back later.

5. Understand Before You Patch

Quick fixes feel good.

Understanding the root cause makes you dangerous (in a good way).

A Repeatable Debugging Process

This is where things become practical.

Step 1: Reproduce the Bug

Make it happen consistently.

Click button → error appears  
Refresh → still happens  
Different browser → still happens

If it’s inconsistent, your first task is to find the pattern.

Step 2: Define Expected vs Actual

Write it down clearly.

Expected: API returns user data  
Actual: API returns empty array

This step alone eliminates confusion.

Step 3: Isolate the Problem

Shrink the scope.

Comment out unrelated code
Remove layers (UI → API → DB)
Test pieces independently

Think of it like this:

[ UI ] → [ API ] → [ Database ]

Which layer is lying?

Step 4: Form a Hypothesis

Be explicit:

“I think the API is returning empty data because the query filter is wrong.”

Now you’re not guessing—you’re testing a theory.

Step 5: Test the Hypothesis

Use targeted tools:

Logs
Breakpoints
Network inspector

Example:

console.log("User ID:", userId)

But intentional—not random.

Step 6: Fix and Verify

Fix it.

Then confirm:

Does it work in all cases?
Did you break something else?

Step 7: Understand the Root Cause

This is where most devs stop too early.

Don’t just fix it—explain it:

“The bug happened because the state updated asynchronously, and we read it too early.”

Now you’ve learned something reusable.

Real Example: “The API Is Broken” (But It’s Not)

Let’s walk through a real scenario.

The Bug

Frontend shows: No data available

Initial Assumption

“The API is broken.”

Step 1: Check Network Tab

You open DevTools → Network:

API returns correct data

So… not the API.

Step 2: Check State

console.log(data)

It logs:

[]

Empty array.

Step 3: Trace the Flow

useEffect(() => {
  fetchData()
}, [])

Inside fetchData:

setData(response.data)
console.log(data) // still empty

The Problem

React state updates are asynchronous.

You’re logging before state updates.

The Fix

useEffect(() => {
  fetchData()
}, [])

useEffect(() => {
  console.log(data)
}, [data])

The Lesson

The bug wasn’t in the API.
It was in your mental model of how state updates work.

Tools That Actually Help

Not everything is about tools—but the right ones matter.

1. Browser DevTools (Underrated Powerhouse)

Network tab → verify API calls
Console → inspect runtime values
Application tab → check storage

2. Breakpoints (Game Changer)

Instead of spamming logs:

Pause execution and inspect state live

3. Intentional Logging

Bad:

console.log("here")

Good:

console.log("User after login:", user)

4. Stack Traces

Read them.

They literally tell you:

Where the error happened
What triggered it

5. Rubber Duck Debugging

Explain the bug out loud.

Yes, seriously.

You’ll often solve it mid-explanation.

Problem → Hypothesis → Test → Learn → Repeat

Common Debugging Traps

Avoid these and you’ll already be ahead of most devs:

Fixing Symptoms Instead of Causes

You silence the error… but the bug is still there.

Changing Too Many Things at Once

Now you don’t know what worked.

Ignoring Error Messages

The error is literally telling you what’s wrong.

Read it.

Assuming the Bug Is “Weird”

It’s almost never weird.

It’s misunderstood.

“It Works on My Machine”

This is not a flex.

It’s a clue.

A Better Mental Model

Expectation ≠ Reality  
        ↓  
Investigate the gap

That’s debugging.

Final Thought

The best developers aren’t the ones who write perfect code.
They’re the ones who can quickly understand why things break.

Debugging isn’t a side skill.

It is the job.

Stripe Connect on Accounts v2 — Standard, Express, and Custom, Rebuilt Around Configurations (with HTTP, Python, C#, and PHP)

Billy Okeyo — Mon, 06 Apr 2026 00:00:00 +0000

This is the Accounts v2 companion to the original Connect guide (Accounts v1). Same platform concepts—Standard, Express, Custom, money flow, compliance—but the API shape is the one Stripe describes in Connect and the Accounts v2 API. Use the older post when you must stay on v1 (Account.create with type=express, OAuth-only Standard flows, etc.). Use this post when you are designing around one Account, configurations, and customer_account.

Heads-up: v2 uses different endpoints and JSON than classic Accounts CRUD. Always confirm API version, preview flags, and SDK support in Stripe’s documentation before shipping production code.

What is Stripe Connect?

Stripe Connect is for platforms that move money between buyers, sellers, and your platform. You connect connected accounts to your platform account so you can:

Collect payments from customers
Pay out to sellers or service providers
Take application fees where appropriate

Stripe holds the payments rail; you own product and UX choices.

Two ideas at once: “style” (Standard / Express / Custom) and “shape” (v2 configurations)

The original guide compares Standard, Express, and Custom as who owns onboarding, dashboards, and compliance.

Accounts v2 adds a separate axis: what capabilities a single Account has, expressed as configurations:

Configuration (v2)	Role in plain language
`merchant`	Accept card payments, get paid out—what you usually wanted from aconnected account selling on your platform.
`customer`	Bebilled like a customer (subscriptions, invoices to your platform) using the same Account identity—often replacing a separate Customer object for that business.
`recipient`	Receivetransfers (e.g. indirect charges), using the v2 transfer capabilities Stripe documents for recipients.

You can assign one or more configurations to the same Account. That is the core promise of v2: one identity, multiple roles, less manual ID mapping.

The Standard / Express / Custom choice is still real: it describes OAuth vs Stripe-hosted onboarding vs fully custom UI. On v2 you implement those experiences while creating and updating Accounts through the v2 surface (where supported).

Standard vs Express vs Custom (unchanged product trade-offs)

Feature / aspect	Standard	Express	Custom
Who owns the Stripe relationship	The user’sexisting Stripe user; you connect via OAuth.	You createconnected accounts; Stripe hosts onboarding and a light** dashboard.	You ownall UX; Stripe is invisible to end users.
KYC / onboarding	User usesStripe Dashboard.	Stripe-hosted onboarding (Account Links, etc., per current docs).	You collect data and satisfy Stripe requirements via API.
Dashboard	FullStripe dashboard for the user.	Express dashboard.	None unless you build it.
Best when	Sellersalready have Stripe.	You needspeed and shared compliance.	You needfull branding and control.

On v2, you still make these product choices—but you attach merchant / customer / recipient configurations to match what each connected business must do.

Architectural overview

Fund flow is unchanged at a high level:

flowchart LR
  C[Customer] --> P[Your platform]
  P --> A[Connected Account]
  A --> B[Bank payout]

What changes in v2 is object modeling:

One Account can represent both “seller” and “buyer of your SaaS” if you add merchant and customer configurations.
APIs that took customer may accept customer_account with an Account ID—see using Accounts as customers.

Accounts v2 primitives (before code)

Stripe’s v2 Account objects differ from v1’s flat Account create. You typically send:

identity — country, entity type, business details.
configuration — nested blocks such as merchant, customer, recipient, each with capabilities you request (card_payments, balance / payout capabilities as named in current docs).
defaults — currency, locales, responsibilities (fees / losses collector), etc.
include — ask the API to return nested sections (e.g. configuration.merchant, requirements).

Responses may omit fields unless you include them—see Stripe’s note on includable response values.

API version: v2 calls often require a specific Stripe-Version header (including preview versions while the API evolves). Set this from the exact value in Stripe’s Accounts v2 docs for your integration window.

Implementation: creating an Account on v2 (HTTP-first)

Official examples are REST + JSON. Below is illustrative—copy field names, capability keys, and version from the current docs, not from this blog alone.

cURL (canonical pattern from Stripe)

Stripe documents POST /v2/core/accounts with a JSON body. Conceptually:

curl -X POST https://api.stripe.com/v2/core/accounts \
  -H "Authorization: Bearer sk_test_..." \
  -H "Stripe-Version: <version-from-stripe-docs>" \
  -H "Content-Type: application/json" \
  --data '{
    "contact_email": "seller@example.com",
    "display_name": "Example Seller",
    "identity": {
      "country": "us",
      "entity_type": "company",
      "business_details": { "registered_name": "Example Co" }
    },
    "configuration": {
      "merchant": {
        "capabilities": {
          "card_payments": { "requested": true }
        }
      },
      "customer": {
        "capabilities": {
          "automatic_indirect_tax": { "requested": true }
        }
      }
    },
    "defaults": {
      "currency": "usd",
      "responsibilities": {
        "fees_collector": "stripe",
        "losses_collector": "stripe"
      },
      "locales": ["en-US"]
    },
    "include": [
      "configuration.customer",
      "configuration.merchant",
      "identity",
      "requirements"
    ]
  }'

This shows the mental model: one create, multiple configurations, explicit includes.

Python (generic HTTP client)

Until your stripe SDK exposes stable helpers for every v2 path, httpx or requests keeps you aligned with the docs:

import os
import requests

def create_account_v2():
    r = requests.post(
        "https://api.stripe.com/v2/core/accounts",
        headers={
            "Authorization": f"Bearer {os.environ['STRIPE_SECRET_KEY']}",
            "Stripe-Version": "<version-from-stripe-docs>",
            "Content-Type": "application/json",
        },
        json={
            # ...same structure as the cURL example...
        },
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

PHP (Laravel-style)

Use Guzzle or Laravel Http::withHeaders([...])->post(...) with the same URL, Stripe-Version, and JSON body. Keep secrets in config, not in source control.

C# (.NET)

Use HttpClient with StringContent(json, Encoding.UTF8, "application/json") and the same headers. Deserialize the JSON response into your own DTOs that track id, requirements, and configuration.* state.

OAuth / Standard accounts: Stripe currently directs platforms that authenticate with OAuth to connected accounts to continue using v1 for that path. Treat Standard as documented in the v1 guide until your OAuth + v2 story is explicitly supported for your use case—see Accounts v2 and OAuth.
{: .prompt-tip }

Onboarding links (Express-style flows)

For Express-like experiences you still send users through Stripe-hosted onboarding where the product allows it. With a connected account ID returned from v2 (acct_...), Account Links (v1 resource) remain the usual tool for account_onboarding—the same pattern as the v1 article, but the account value may come from a v2 create. Verify compatibility for your API version in Stripe’s docs.

Python (v1 Account Links API, account id from v2 create):

import stripe
stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

link = stripe.AccountLink.create(
    account=connected_account_id,
    refresh_url="https://yourapp.com/reauth",
    return_url="https://yourapp.com/complete",
    type="account_onboarding",
)

Custom-style integrations on v2

Custom still means: you own KYC collection, ToS acceptance, and ongoing verification. On v2 you express that by:

Sending complete identity data the API requires.
Requesting merchant / recipient capabilities your product needs.
Handling requirements and webhooks the same way you would for Custom on v1—only the payload shape differs.

You may still use tos_acceptance-style fields where the v2 schema maps them; follow Stripe’s v2 reference for exact property names.

Using Accounts as customers (`customer_account`)

Where you used customer=cus_..., many flows accept customer_account=acct_... for an Account that has the customer configuration. Example pattern from Stripe (conceptual):

curl https://api.stripe.com/v1/setup_intents \
  -u "sk_test_...:" \
  -H "Stripe-Version: <version-from-stripe-docs>" \
  -d customer_account=acct_123 \
  -d "payment_method_types[]=card" \
  -d confirm=true \
  -d usage=off_session

Details and supported objects live under using Accounts as customers.

Checking balances

For many Connect operations, connected account scoping is unchanged. Python:

balance = stripe.Balance.retrieve(stripe_account=account_id)

PHP:

$balance = \Stripe\Balance::retrieve([], ['stripe_account' => $accountId]);

C#:

var balance = await stripe.Balance.GetAsync(
    new BalanceGetOptions(),
    new RequestOptions { StripeAccount = accountId }
);

Confirm in Stripe’s docs whether your v2 account IDs behave identically for every Balance and v1 helper you rely on during migration.

Compliance responsibilities

The Standard / Express / Custom compliance split from the original article still applies who collects KYC and who owns disputes. v2 can reduce duplicate identity collection when you add configurations to an existing Account instead of opening a second Customer record.

Responsibility	Standard	Express	Custom
KYC	Stripe	Mostly Stripe	You
Tax reporting	Stripe-heavy	Shared	Often you
PCI	Stripe-hosted elements	Shared	Mostly you
Disputes	Stripe-heavy	Shared	Often you

Choosing a path

Scenario	Style to favor	v2 angle
Sellers already on Stripe; OAuth	Standard	Oftenv1 OAuth until Stripe supports your OAuth + v2 plan
Fast marketplace onboarding	Express	v2 `Account` + `merchant` + Account Links
White-label, embedded finance	Custom	v2 full `identity` + capabilities + your UI
Same business pays youand sells on your platform	Express or Custom	Same*`Account`, `merchant`* + `customer` configurations

Strategic considerations

Time-to-market: Standard (when OAuth fits) < Express < Custom.
API surface: v2 adds configuration discipline—plan for migration from v1, not an eternal split (Stripe discourages maintaining both versions simultaneously). - Reference
SDKs: Expect to use HTTP for some v2 paths until your language SDK is fully aligned.

Final thoughts

Accounts v2 does not erase Standard / Express / Custom—it repackages how you represent connected users in the API. Start from Connect and the Accounts v2 API, add using Accounts as customers when the same legal entity both sells and buys from your platform, and keep the v1 Connect guide handy for OAuth flows and legacy snippets until you have fully moved.

Either way, you still trade off control, compliance, and complexity—only the object model got a long-overdue upgrade.

Contract Testing: Prevent Breaking Changes Before Production

Billy Okeyo — Thu, 19 Mar 2026 00:00:00 +0000

“It worked locally. Tests passed. But production still broke.”

If you’re building distributed systems with multiple services and frontends, you’ve likely encountered this (whether using .NET + Angular , Node.js + React , Python + Vue , or any combination):

A backend change gets deployed
The frontend suddenly breaks
No tests warned you

The issue isn’t always logic.

It’s often a broken contract between systems.

The Problem: Silent API Breakages

In a typical setup:

Service Provider: An API or microservice
Service Consumer: A frontend, app, or another service
Communication: JSON over HTTP (or any protocol)

Everything depends on one thing:

The consumer and provider agreeing on what data looks like

A Real Example

Let’s use a .NET API and Angular frontend as an example (though this applies to any tech stack).

API Provider (Initial Version)


public class UserDto
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
}




[HttpGet("{id}")]
public IActionResult GetUser(int id)
{
    return Ok(new UserDto
    {
        Id = 1,
        Name = "Billy",
        Email = "billy@example.com"
    });
}

Consumer Interface (Angular Example)

export interface User {
  id: number;
  name: string;
  email: string;
}

Everything works perfectly.

Then a “Small” Change Happens

public class UserDto
{
    public int Id { get; set; }
    public string FullName { get; set; } // renamed
    public string Email { get; set; }
}

Production Result

user.name // undefined

The UI breaks.

Why Didn’t Traditional Tests Catch This?

Unit tests → passed (backend logic is fine)
Integration tests → passed (they used the old model)
The API still returns valid JSON

But the contract between systems changed —and traditional tests don’t verify that.

What is Contract Testing?

Contract testing ensures that your service provider always matches what the consumer expects.

A contract defines:

Request format
Response structure
Required fields
Data types

In Simple Terms

The consumer (frontend, app, or service) defines expectations The provider (API or service) must satisfy them

Consumer vs Provider

Consumer

Calls the service/API
Defines expected structure
Examples: Angular frontend, React app, mobile app, another microservice

Provider

Returns the data
Must not break expectations
Examples: .NET API, Node.js backend, Python service, GraphQL endpoint

How Contract Testing Works

Instead of relying only on integration tests:

Angular defines expectations
A contract file is generated
.NET verifies the contract

Flow

Consumer Test (e.g., Angular)
      ↓
Generates Contract
      ↓
Saved as JSON/YAML
      ↓
Provider verifies against contract (e.g., .NET API)

Practical Example

(.NET backend + Angular frontend as an example)

Step 1: Define Expectations in Consumer (Angular example)

const expectedUser = {
  id: 1,
  name: "Billy",
  email: "billy@example.com"
};

it("should fetch user correctly", async () => {
  const user = await userService.getUser(1);
  expect(user).toEqual(expectedUser);
});

Generated Contract (Simplified)

{
  "request": {
    "method": "GET",
    "path": "/api/users/1"
  },
  "response": {
    "body": {
      "id": 1,
      "name": "Billy",
      "email": "billy@example.com"
    }
  }
}

Step 2: Verify in Provider (.NET API example)

Install Pact:

dotnet add package PactNet

Provider Test

[Fact]
public void VerifyPact()
{
    var pactVerifier = new PactVerifier();

    pactVerifier
        .ServiceProvider("UserApi", "http://localhost:5000")
        .WithFileSource(new FileInfo("pacts/userapi-angular.json"))
        .Verify();
}

If Provider Breaks the Contract


1

public string FullName { get; set; }

The test fails immediately
You catch the issue before deployment

Where Contract Testing Fits

E2E Tests
(user journeys)

Integration Tests
(service interactions)

Contract Tests 
(API agreements)

Unit Tests
(business logic)

Why This Matters in Real Projects

Safer Refactoring

Change DTOs, schema, or API responses without fear. Contract tests verify nothing broke.

Independent Development

Consumer and provider teams move faster. Changes are caught instantly, not in production.

Faster Debugging

Failures clearly show what broke

Stronger CI/CD Pipelines

Consumer Build → Generate Contract
Provider Build → Verify Contract
Deploy → Only if both pass

This works with any tech stack.

Common Mistakes

Over-Specifying Data

Bad:

"name": "Billy Okeyo"

Good:

"name": "string"

Testing Everything

Only validate fields your frontend actually uses

Ignoring Versioning

Breaking contracts without versioning leads to production issues

Best Practices

Keep Contracts Minimal

Focus only on required fields

Version Your API

/api/v1/users
/api/v2/users

Automate in CI/CD

Contracts should be generated and verified automatically

Use Realistic Data

Avoid unrealistic mocks

Final Takeaway

Unit tests verify logic Integration tests verify systems E2E tests verify user flows

Contract tests verify agreements

“Most production bugs aren’t failures… they’re misunderstandings between systems.”

Contract testing eliminates those misunderstandings.

Why Your Django App Needs Redis and Celery in Production

Billy Okeyo — Mon, 16 Mar 2026 00:00:00 +0000

Django is an incredibly powerful framework for building web applications quickly. However, as your application grows, certain tasks begin to slow down request-response cycles.

Examples include:

Sending emails
Generating reports
Processing uploaded files
Running background analytics
Sending notifications

Running these tasks during an HTTP request can make your application slow and unreliable.

This is where Celery and Redis come in.

Together they allow you to run background jobs asynchronously without blocking your main application.

The Problem with Synchronous Tasks

Imagine a user submits a request that triggers an operation that takes 10 seconds.

For example:

Generating a financial report
Parsing a large document
Sending multiple emails

If your application processes this synchronously:


1

User Request → Django → Long Task → Response

The user waits for the entire process to finish.

This leads to:

Slow responses
Poor user experience
Possible request timeouts

Introducing Celery

Celery is a distributed task queue that allows you to run background jobs outside the request-response cycle.

Instead of executing tasks immediately, Django sends the job to a queue.

A worker then processes it asynchronously.

The flow becomes:


User Request
    ↓
Django
    ↓
Queue Task
    ↓
Immediate Response
    ↓
Celery Worker
    ↓
Executes Job

This makes your application fast and scalable.

Why Redis?

Celery requires a message broker to manage task queues.

Redis is commonly used because it is:

Extremely fast
Lightweight
Easy to deploy
Perfect for queues and caching

Redis stores the tasks until workers pick them up.

Example: Sending Email in the Background

Instead of sending email directly in a Django view:

send_mail(
    "Welcome",
    "Thanks for signing up",
    "noreply@example.com",
    [user.email],
)

You create a Celery task:

from celery import shared_task
from django.core.mail import send_mail

@shared_task
def send_welcome_email(email):
    send_mail(
        "Welcome",
        "Thanks for signing up",
        "noreply@example.com",
        [email],
    )

Then call it asynchronously:

send_welcome_email.delay(user.email)

The user gets an immediate response while the email is processed in the background.

Common Use Cases for Celery

Celery is useful for many production tasks:

Email sending

Welcome emails
Notifications
Password resets

Data processing

Financial calculations
AI processing
Data pipelines

Scheduled tasks

Daily reports
Cleaning expired sessions
Updating analytics

Running Celery in Production

A typical Django production stack might look like this:

Users
  ↓
Nginx
  ↓
Gunicorn
  ↓
Django App
  ↓
Redis (Broker)
  ↓
Celery Workers

Each component plays a role:

Nginx handles web traffic
Gunicorn runs Django
Redis manages task queues
Celery workers execute background jobs

Scaling Celery Workers

One of Celery’s biggest advantages is scalability.

If tasks increase, you simply add more workers.

celery -A project worker --loglevel=info --concurrency=4

More workers mean faster task processing.

Final Thoughts

Celery and Redis are essential tools for running Django applications at scale.

They allow you to:

Improve response times
Handle heavy workloads
Build scalable architectures
Run background processing reliably

If your Django application handles tasks that take more than a few seconds, moving them to Celery is one of the best architectural decisions you can make.

A Practical Guide to File Uploads (Images, Excel, CSV) in Angular + Django

Billy Okeyo — Mon, 23 Feb 2026 00:00:00 +0000

File uploads are one of those features that look simple… until they aren’t.

Images need previewing. CSV files need parsing. Excel files need validation. Large files need handling. And suddenly your “simple upload” becomes a full feature.

In this guide, we’ll build a practical and production-ready file upload system using:

Angular (Frontend)
Django + Django REST Framework (Backend)

We’ll cover:

Image uploads with preview
CSV uploads and parsing
Excel uploads and processing
Validation and security best practices

Backend Setup (Django + DRF)

1. Install Dependencies

pip install djangorestframework pillow openpyxl pandas

pillow → Image processing
openpyxl → Excel support
pandas → CSV & Excel parsing

2. Configure Media Files

settings.py

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

urls.py (project level)

from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    # your urls
]

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Part 1: Image Upload

Django Model

from django.db import models

class Profile(models.Model):
    name = models.CharField(max_length=100)
    image = models.ImageField(upload_to='profiles/')

Serializer

from rest_framework import serializers
from .models import Profile

class ProfileSerializer(serializers.ModelSerializer):
    class Meta:
        model = Profile
        fields = ' __all__'

View

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, FormParser
from rest_framework import status

class ProfileUploadView(APIView):
    parser_classes = [MultiPartParser, FormParser]

    def post(self, request):
        serializer = ProfileSerializer(data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Angular Frontend (Image Upload)

HTML

<input type="file" (change)="onFileSelected($event)" accept="image/*" />
<img *ngIf="previewUrl" [src]="previewUrl" width="200" />
<button (click)="upload()">Upload</button>

Component

selectedFile!: File;
previewUrl: string | ArrayBuffer | null = null;

onFileSelected(event: any) {
  this.selectedFile = event.target.files[0];

  const reader = new FileReader();
  reader.onload = () => {
    this.previewUrl = reader.result;
  };
  reader.readAsDataURL(this.selectedFile);
}

upload() {
  const formData = new FormData();
  formData.append('name', 'Billy');
  formData.append('image', this.selectedFile);

  this.http.post('http://localhost:8000/api/upload/', formData)
    .subscribe(res => console.log(res));
}

Part 2: CSV Upload and Parsing

Django View for CSV

import pandas as pd
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser

class CSVUploadView(APIView):
    parser_classes = [MultiPartParser]

    def post(self, request):
        file = request.FILES['file']

        if not file.name.endswith('.csv'):
            return Response({'error': 'Invalid file type'}, status=400)

        df = pd.read_csv(file)

        # Example processing
        total_rows = len(df)
        columns = list(df.columns)

        return Response({
            'rows': total_rows,
            'columns': columns
        })

Angular CSV Upload

uploadCSV(file: File) {
  const formData = new FormData();
  formData.append('file', file);

  this.http.post('http://localhost:8000/api/upload-csv/', formData)
    .subscribe(res => console.log(res));
}

Part 3: Excel Upload (.xlsx)

Django View for Excel

class ExcelUploadView(APIView):
    parser_classes = [MultiPartParser]

    def post(self, request):
        file = request.FILES['file']

        if not file.name.endswith('.xlsx'):
            return Response({'error': 'Invalid file type'}, status=400)

        df = pd.read_excel(file)

        summary = {
            'rows': len(df),
            'columns': list(df.columns)
        }

        return Response(summary)

Validation & Security Best Practices

1. Limit File Size

In settings.py:

DATA_UPLOAD_MAX_MEMORY_SIZE = 5242880 # 5MB

2. Validate File Type Properly

Don’t rely only on extension.

if file.content_type not in ['image/jpeg', 'image/png']:
    return Response({'error': 'Invalid image format'}, status=400)

3. Rename Uploaded Files

import uuid

def upload_to(instance, filename):
    ext = filename.split('.')[-1]
    return f"uploads/{uuid.uuid4()}.{ext}"

4. Handle Large Files with Streaming

For very large CSV files, process in chunks:

for chunk in pd.read_csv(file, chunksize=1000):
    # process chunk
    pass

Bonus: Returning Processed Data

Sometimes you don’t just upload — you process and return a result file.

Example: Generate processed CSV

from django.http import HttpResponse

response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="processed.csv"'
df.to_csv(response, index=False)
return response

Final Thoughts

File uploads are not just about saving files.

They are about:

Validation
Security
User experience
Scalability

When done correctly, they become powerful data pipelines inside your application.

If you’re building admin systems, reporting tools, learning platforms, or fintech dashboards — mastering uploads is essential.