Lock Management Inside a Process: Why Native Locks Alone Are Not Enough

Hello, I'm Maneshwar. I'm working on FreeDevTools online currently building **one place for all dev tools, cheat codes, and TLDRs* — a free, open-source hub where developers can quickly find and use tools without any hassle of searching all

So far, we’ve treated file locking as something that naturally coordinates concurrency across transactions.

That assumption mostly holds across processes. However, today’s learning exposes a subtle but critical truth:

Native file locks are process-wide, not file-descriptor–specific.

This single detail fundamentally changes how SQLite must manage locks internally.

The Problem with Multiple File Descriptors

Linux file locks are applied using fcntl, and they are associated with:

• a process
• an inode
• a byte range

They are not associated with:

• a specific file descriptor
• a thread
• a database connection

This means that if the same process opens the same file multiple times and even under different names, all lock operations still target the same inode.

Consider the following situation:

int fd1 = open("file1", ...);
int fd2 = open("file2", ...);  // hard link or symlink to file1

Even though file1 and file2 appear different, they may resolve to the same inode.

Now imagine:

• Thread T1 acquires a read lock on a region using fd1
• Thread T2 (or even T1 again) requests a write lock on the same region using fd2

The kernel does not treat these as independent locks.

Instead:

• The write lock silently overrides the read lock
• Because both requests come from the same process
• And they target the same inode and byte range

This override happens automatically and unintentionally.

Why This Is Dangerous for SQLite

SQLite allows multiple database connections in the same process and possibly across multiple threads

Each connection may run its own transaction and believe it has its own locking state

But Linux enforces only one lock per process per file region.

As a result:

• Two SQLite connections inside the same process
• Can accidentally interfere with each other
• Even though they appear logically independent

This means native locks cannot be used directly to manage transactional concurrency within a process.

Native locks are sufficient between processes, but insufficient inside a single process.

SQLite’s Solution: Internal Lock Tracking

To solve this, SQLite introduces an abstraction layer above native file locks.

Instead of blindly calling fcntl, SQLite:

1. Tracks all open database files internally
2. Groups file descriptors by inode
3. Manages logical lock state itself
4. Calls fcntl only when a real lock transition is required

The cornerstone of this design is the unixinodeinfo structure.

The unixinodeinfo Object: One per Inode per Process

Whenever SQLite opens a database file, it determines the file’s identity using:

• st_dev → device number
• st_ino → inode number

These two values uniquely identify a file on the system.

For each unique (st_dev, st_ino) pair, SQLite creates exactly one unixinodeinfo object per process.

This object represents the process-wide locking state for that specific database file

A process can never have two unixinodeinfo objects for the same inode.

Global Inode List and Synchronization

All unixinodeinfo objects are stored in a global doubly linked list called inodeList.

Key properties:

• Indexed by (device number, inode number)
• Created lazily on first use
• Destroyed when no longer needed
• Protected by a global mutex

This ensures correctness under multithreading, consistent lock tracking across connections

Reference Counting: Tracking File Opens

Each unixinodeinfo object contains a reference count nRef.

This value represents how many times the process has opened the same database file across all threads and connections

Behavior:

• When a database file is opened → nRef++
• When a connection closes → nRef--
• When nRef reaches zero → the object is removed and destroyed

This ensures internal lock state exists only while the file is in use.

Lock Requests: Internal First, Native Later

When a thread requests a lock:

1. SQLite locates the unixinodeinfo for the file
2. It checks current internal lock state
3. It determines whether a real lock transition is required
4. Only then does it invoke fcntl

For example:

• RESERVED → SHARED → only increments nShared, no system call
• RESERVED → EXCLUSIVE → requires a native write lock via fcntl

This minimizes syscalls and prevents accidental lock overrides.

The unixFile Object: One per Connection

While unixinodeinfo is process-wide, SQLite also maintains a per-connection structure called unixFile.

Each open database connection has its own unixFile object.

This object represents one logical database connection, stores the actual OS file descriptor (h), tracks the lock state for that connection via eFileLock and points to the shared unixinodeinfo

Why This Design Matters

This internal lock-tracking layer allows SQLite to:

• Safely manage concurrency within a single process
• Prevent native lock overrides
• Preserve serializable isolation
• Avoid excessive system calls
• Maintain correctness across threads and connections

Without this layer, SQLite would be fundamentally unsafe in multi-connection, multi-threaded applications.

My experiments and hands-on executions related to SQLite will live here: lovestaco/sqlite

References:

SQLite Database System: Design and Implementation. N.p.: Sibsankar Haldar, (n.d.).

👉 Check out: FreeDevTools

Any feedback or contributors are welcome!

It’s online, open-source, and ready for anyone to use.

⭐ Star it on GitHub: freedevtools