DEV Community: Swaraj Puppalwar

LioranDB V2 Pre-Alpha Benchmark Summary

Swaraj Puppalwar — Wed, 29 Jul 2026 13:26:36 +0000

Over the past few weeks I've been benchmarking LioranDB V2, a document database engine written completely from scratch in Rust.

Instead of publishing screenshots or selected numbers, I've decided to publish the raw benchmark logs so anyone can inspect the results themselves.

This is Pre-Alpha software. The goal isn't to claim perfection, but to be transparent about where the database stands today and where it's heading.

Test Environment

All benchmarks were executed on my personal development machine rather than a dedicated server.

Hardware

Laptop: HP OMEN Gaming Laptop
Processor: Intel Core Ultra 7 (14th Generation)
CPU Cores: 16 Physical Cores
Memory: 24 GB DDR5 @ 5600 MT/s
Storage: NVMe PCIe SSD
Operating System: Windows

LioranDB Configuration

Language: Rust
Nodes: 4
Partitions: 8
Worker Threads: 32
Batch Size: 256

Features Enabled

Secondary Indexes
Full-Text Search
Write-Ahead Log (WAL)
MVCC
Parallel Read & Write Execution

Note: These benchmarks were intentionally performed on a consumer laptop instead of enterprise server hardware. The purpose was to evaluate LioranDB V2's real-world performance on hardware that many developers already own. Dedicated server hardware with higher core counts, faster storage, and larger memory capacity is expected to deliver significantly higher throughput.

Write Performance

The write benchmarks were executed using realistic document workloads while maintaining durability through the WAL.

Results reached approximately

23K to 25K writes/sec during the benchmark
Sustained write throughput across millions of documents
Stable WAL group commit behavior

Benchmark logs:

Read Performance

Read benchmarks include

Primary Key Lookups
Secondary Index Queries
Range Queries
Pagination
Full Text Search

During the benchmarks the engine demonstrated

Low millisecond latency
High parallel throughput
Efficient index lookups
Stable read performance under concurrency

Benchmark logs

Mixed Workload

Real applications rarely perform only reads or only writes.

The soak benchmark continuously executed mixed workloads consisting of writes together with concurrent reads.

Target workload:

~10K Writes/sec
~25K Reads/sec
~35K Total Operations/sec

The goal was to evaluate long-running stability rather than peak numbers.

Benchmark log

https://www.liorandb.com/benchmark_ldb/soak/soak_10m.txt

Crash Recovery Test

Reliability matters more than benchmark numbers.

LioranDB V2 was repeatedly terminated during active writes and restarted to verify WAL recovery.

The crash benchmark performs multiple forced crashes while documents are still being inserted.

Recovery validates:

WAL replay
Metadata consistency
Duplicate prevention
Missing document detection

Benchmark log

https://www.liorandb.com/benchmark_ldb/crash/crash_test_1m_9c.txt

What These Benchmarks Show

Current pre-alpha results demonstrate:

Multi-node architecture
Parallel execution
Durable WAL commits
Secondary indexes
Full-text search
Sustained write throughput
Concurrent reads and writes
Crash recovery support

The project is still under active development, and these benchmarks are intended to provide transparent insight into its current capabilities rather than serve as marketing claims.

Benchmark Repository

All raw benchmark logs are publicly available.

https://www.liorandb.com/benchmark_ldb/

About LioranDB

LioranDB is an open-source distributed document database written from scratch in Rust.

The project focuses on building modern developer infrastructure with performance, simplicity, and transparency as core principles.

Website

https://www.liorandb.com

About Lioran Group

LioranDB is developed by Lioran Developer Solutions (LDS).

Website

https://www.lioransolutions.com

Company

https://www.lioran.group

About the Author

Swaraj Puppalwar

Founder & CTO, Lioran Group

GitHub

https://github.com/UltronTheAI

Feedback, benchmark reviews, and contributions are always welcome. Every benchmark log is public so anyone can inspect the results, reproduce the tests, and help improve the project.

From Embedded Engine to Distributed Database: LioranDB's Cluster Architecture

Swaraj Puppalwar — Fri, 24 Jul 2026 18:59:36 +0000

LioranDB begins with a local transactional storage engine, but its architecture extends beyond one process.

The DBMS backend can operate in several modes:

Local
Multi-core
Multi-node
Remote cluster

One DBMS interface

The document layer talks through a backend abstraction.

A read transaction may therefore target:

Local(ReadTxn)
MultiCore
MultiNode
Cluster

The document query layer does not need a completely different API for each deployment model.

Multi-core routing

In multi-core mode, a table and key are mapped to one engine partition.

table + key
    ↓
partition hash
    ↓
local engine partition

Multi-node routing

In multi-node mode, the operation first selects a node and then a partition inside that node.

table + key
    ↓
node
    ↓
partition
    ↓
engine

Global scans visit several partitions or nodes and merge the returned keys.

WAL-based replication

The cluster crate includes a leader-driven replication abstraction.

A replicator:

Fetches WAL entries from a starting LSN.
Limits the batch by entries and bytes.
Sends the batch to a follower.
Receives the follower's acknowledged LSN.
Continues from the next LSN.

The current abstraction is synchronous, but it cleanly separates WAL production from transport.

That transport can later be backed by RPC.

gRPC server

The server crate opens a node using:

Cluster configuration
Node identity
On-disk engine configuration
TLS certificates
Certificate authority
ACL rules
gRPC listen address
Metrics address

This is the boundary where the database becomes a network service instead of an embedded library.

The larger idea

A distributed database should not be designed by gluing networking onto a single global engine at the end.

It needs internal seams for:

Partition ownership
Request routing
WAL shipping
Follower application
Read preferences
Node configuration
Security
Observability

LioranDB's architecture is building those seams from the storage layer upward.

The result is a path from one Rust process to a multi-core and eventually replicated database system without replacing the entire core halfway through.

That is the architectural long game.

LioranDB is developed by Swaraj Puppalwar under Lioran Group.

Explore the project:

Progressive Readiness: How LioranDB Opens Before Everything Is Finished

Swaraj Puppalwar — Fri, 24 Jul 2026 18:58:29 +0000

Database startup is rarely one binary moment.

After a crash or cold start, different capabilities become safe at different times.

LioranDB models this explicitly through progressive readiness states.

Startup states

The DBMS can move through states such as:

OPENING
RECOVERING
INDEX_REPLAYING
READ_DEGRADED
READ_READY
FINALIZING
WRITE_READY
CORRUPT

This is more informative than a single Boolean called ready.

Different kinds of readiness

LioranDB separately tracks whether it can:

Serve primary-key reads
Serve indexed reads
Accept degraded writes
Accept fully ready writes
Query through an index replay overlay
Declare all background state drained

For example, primary documents may already be recoverable while secondary indexes are still replaying mutations.

In that situation, refusing every read may be unnecessarily conservative.

But allowing an index query would return incomplete results.

Capability gates

The engine checks facts such as:

Primary recovery complete
Catalog loaded
Logical primary view ready
Manifest valid
WAL replay complete
Secondary index ready
Text index ready

If a capability is blocked, the readiness report records the reasons.

That makes startup behaviour observable and testable.

Timelines and metrics

LioranDB records timestamps for events such as:

API handle creation
Primary recovery
Catalog readiness
WAL replay completion
Index overlay availability
Secondary readiness
Text readiness
Full write readiness

It also tracks rejected operations before each capability becomes safe.

Why this architecture matters

Large databases can take meaningful time to rebuild indexes or drain recovery work.

A progressive readiness model allows a service to expose safe functionality earlier without pretending the whole system is ready.

The important word is safe.

Fast startup should not mean returning subtly incomplete data.

It should mean identifying the earliest point at which each operation becomes correct.

LioranDB is created by Swaraj Puppalwar at Lioran Group.

Links:

The I/O Scheduler That Stops Background Work From Killing Latency

Swaraj Puppalwar — Fri, 24 Jul 2026 18:57:28 +0000

A database performs many kinds of disk activity:

WAL fsync
Primary data flushes
Document-store writes
Index flushes
Compaction
Text compaction
Checkpoint sync
Metadata writes
Cleanup

If all of them hit the disk without coordination, foreground latency becomes chaotic.

LioranDB contains a global I/O scheduler to manage this contention.

I/O classes

Operations are assigned to classes such as:

WalFsync
Metadata
PrimaryFlush
DocStore
IndexFlush
IndexMaintenance
Compaction
TextCompaction
CheckpointSync

Each class has a relative priority.

WAL fsync receives the highest weight because committed transactions may be waiting for it.

Background compaction receives less priority because it can usually pause briefly without breaking correctness.

Token-based budgeting

The scheduler maintains a byte budget per second.

Operations consume tokens as they perform I/O.

The budget supports short bursts but prevents background workers from continuously saturating the device.

The default budget can be configured through environment variables.

Foreground awareness

The scheduler records recent foreground activity.

When user-facing writes are active, optional background operations can yield.

Examples include:

Secondary compaction
Text compaction
Cleanup

This reduces the chance that a giant merge causes sudden WAL latency spikes.

Sync coordination

Physical syncs are especially expensive.

LioranDB tracks sync ownership for:

WAL
Checkpoint
Document store
Metadata
Compaction

It also records whether WAL fsync was blocked by another subsystem.

That gives engineers evidence instead of vibes when investigating latency.

Circuit-breaker behaviour

The code also models background-I/O health states such as:

Healthy
Elevated
Critical
Recovery

When WAL latency becomes dangerous, background work can use smaller chunks or pause more aggressively.

Databases are not CPU programs that occasionally touch disk.

They are I/O orchestration systems wearing an API as a hat.

Once several durability and maintenance pipelines exist, a scheduler becomes essential to keep them from wrestling in the storage aisle.

Built by Swaraj Puppalwar under Lioran Group.

Checkpoints and Atomic Manifests in LioranDB

Swaraj Puppalwar — Fri, 24 Jul 2026 18:56:23 +0000

A WAL can recover recent writes, but replaying an ever-growing log on every startup would eventually become unacceptable.

Databases therefore create checkpoints.

What does a checkpoint mean?

A checkpoint advances the durable on-disk state.

Conceptually:

WAL contains operations up to LSN 1,000,000
Data pages are flushed up to LSN 900,000
Checkpoint LSN becomes 900,000

The engine now knows that older WAL records below the safe reclaim boundary may no longer be required.

The manifest

LioranDB stores critical engine metadata in a MANIFEST file.

It includes values such as:

Page size
Next page ID
Next LSN
Checkpoint LSN
Freelist root
Table root pages

Without this metadata, the engine would not know where its tables begin or which log range is already reflected on disk.

Atomic replacement

Updating metadata in place is dangerous.

A crash halfway through the write could leave a partially written manifest.

LioranDB instead follows an atomic replacement flow:

Encode new manifest
    ↓
Write temporary file
    ↓
fsync temporary file
    ↓
Rename temporary file over MANIFEST
    ↓
Sync directory entry

The rename allows the old or new complete manifest to exist, instead of a half-manifest chimera crawling out of a power failure.

Bounded checkpoint slices

Checkpointing can also compete with foreground writes for disk bandwidth.

LioranDB supports limits based on:

Bytes per checkpoint slice
Wall-clock time per slice
Dirty-byte targets
WAL quiet periods
Periodic full checkpoints

The checkpoint can yield and continue later rather than monopolizing storage.

A checkpoint is not simply “save database now.”

It is a coordinated agreement between WAL durability, dirty pages, metadata publication, index state, and log reclamation.

Getting that agreement wrong is how databases lose acknowledged data.

Getting it right is how startup stays fast without compromising recovery.

LioranDB is developed by Swaraj Puppalwar at Lioran Group.

Explore:

Inside LioranDB's Full-Text Search Segments

Swaraj Puppalwar — Fri, 24 Jul 2026 18:54:33 +0000

A normal secondary index can answer:

status = "active"

It cannot efficiently answer:

documents containing "distributed database"

LioranDB therefore has a dedicated text-segment architecture.

Tokenization

Text is split on non-alphanumeric characters.

Depending on index options, tokens can be normalized to lowercase and filtered through stopwords.

"Building Distributed Databases"

becomes

["building", "distributed", "databases"]

Segment contents

A LioranDB text segment can contain several files and structures:

Term dictionary
Posting lists
Document map
Document-length norms
Optional term positions
Bloom filter
Segment metadata

A posting connects a term to the local documents containing it.

"database" → [doc 2, doc 8, doc 19]

Positions can record where the term appears inside each document.

That enables more advanced query behaviour and phrase-aware features.

Global and local document IDs

Each segment assigns compact local IDs to its documents.

A separate document map translates them back to global document IDs.

This keeps postings smaller while preserving the external identity of the record.

Bloom filters

Each segment also maintains a Bloom filter for terms.

Before reading a segment's postings, the query path can test whether the term might exist there.

A negative answer is definitive.

A positive answer means the segment may contain the term and should be checked.

Query modes

The text query layer supports modes such as:

AND
OR

It also emits scored documents and metrics including:

Query time
Postings read
Candidate documents
Segments searched

Full-text search is essentially a specialized database living beside the document database.

Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake.

Built by Swaraj Puppalwar under Lioran Group.

Learn more:

Why LioranDB Uses LSM-Based Secondary Indexes

Swaraj Puppalwar — Fri, 24 Jul 2026 18:53:25 +0000

A document database usually stores a primary record by document ID.

But applications also ask questions such as:

Find users where status = "active"
Find invoices where customerId = 42
Find posts where category = "rust"

Those queries need secondary indexes.

LioranDB implements secondary indexing through an LSM-style index store.

The mutation path

When a document changes, the DBMS creates index mutations.

These mutations are written to an index mutation log and then applied asynchronously.

Document write
    ↓
Index mutation log
    ↓
Pending mutation queue
    ↓
Index memtable
    ↓
Immutable table
    ↓
Segment file
    ↓
Compaction

This separates the primary document write path from heavier index maintenance.

Why LSM?

An LSM design converts many random updates into sequential batches.

Instead of repeatedly rewriting one large index structure, the engine creates immutable sorted segments.

Reads search across those segments and newer in-memory data.

Background compaction later merges segments.

Different workers for different workloads

LioranDB defines separate worker groups for:

Secondary index application
Text index application
Index flushing
Secondary compaction
Text compaction

This matters because secondary and text indexes have very different costs.

A secondary mutation may be a compact key-to-document mapping.

A text mutation may generate many terms and postings.

Treating them as identical would make scheduling less predictable.

Lag and safety limits

Asynchronous indexing introduces index lag.

LioranDB tracks conditions such as:

Pending mutations
Mutation log bytes
Lag in LSNs
Lag in milliseconds
Segment counts
Compaction debt

Soft limits begin rescue work.

Hard limits trigger stronger backpressure.

This prevents the primary writer from outrunning the indexing pipeline forever.

LSM indexes are not free speed.

They exchange immediate random writes for background complexity, read amplification, compaction, and lag management.

Used carefully, that trade is extremely powerful.

Used carelessly, it creates a segment confetti cannon.

LioranDB is built by Swaraj Puppalwar at Lioran Group.

Links:

Transactions in LioranDB: Read Timestamps and Write Sets

Swaraj Puppalwar — Fri, 24 Jul 2026 18:52:18 +0000

A database transaction is not just a function containing several writes.

It defines what data the operation can observe and how its changes become visible.

LioranDB separates transactions into read and write transactions.

Read transactions

A read transaction stores a read timestamp.

Every lookup uses that timestamp:

get_at(read_timestamp, table, key)

A scan follows the same idea.

scan_at(read_timestamp, table, range)

The timestamp allows the engine to define which version of a value is visible to that transaction.

Write transactions

A write transaction contains:

Transaction ID
Read timestamp
Write set

The write set is a list of operations:

Put {
    table,
    key,
    value
}

Delete {
    table,
    key
}

Calling put() does not immediately modify every persistent structure.

Instead, the transaction accumulates operations.

On commit, the complete write set is passed to the engine.

Why collect a write set?

This design enables the engine to:

Validate conflicts
Assign a commit timestamp
Append one logical transaction to the WAL
Apply several operations atomically
Collect commit statistics
Route operations before publishing them

A transaction can therefore fail before partially exposing its updates.

Isolation modes

The engine configuration currently exposes two isolation choices:

SnapshotIsolation
ReadCommitted

Snapshot isolation provides a stable logical view and write-write conflict detection.

Read committed exposes the latest committed state according to the engine's disk-mode semantics.

Isolation is not a checkbox that magically creates correctness.

Every read path, write path, index update, WAL record, and recovery operation must agree on transaction visibility.

The transaction structs in LioranDB look compact because most of the complexity lives underneath them.

That is usually a good sign.

A clean transaction API should hide the machinery, not leak the whole engine through every method signature.

Created by Swaraj Puppalwar under Lioran Group.

How LioranDB Scales Across CPU Cores With Partitions

Swaraj Puppalwar — Fri, 24 Jul 2026 18:51:17 +0000

A single global database lock is an excellent way to turn a 24-core machine into a decorative heater.

LioranDB uses partitions to scale work across CPU cores.

Independent engine partitions

A partitioned LioranDB node runs multiple independent storage-engine instances.

Each partition has its own:

WAL
Checkpoint pipeline
Flush pipeline
On-disk directory
Memtables
B+ tree state

The directory layout resembles:

data/
└── partitions/
    ├── 0000/
    ├── 0001/
    ├── 0002/
    └── 0003/

Routing keys

Keys are assigned to partitions using a deterministic hash.

Conceptually:

partition = hash(key) % partition_count

The same key always reaches the same partition.

A write therefore touches only the engine responsible for that key instead of contending on one global structure.

Parallel startup

Partitions are also opened in parallel.

LioranDB tracks startup metrics such as:

Total wall-clock open time
Sum of partition open times
Maximum partition time
P50 and P95 open time
Files opened
Bytes read
Peak parallel partition workers

This distinction is useful.

If four partitions each take five seconds but open concurrently, the serial sum is twenty seconds while wall-clock startup is closer to five.

Scans are different

Exact-key operations route to one partition.

A full range scan may need to scan every partition and merge the results into sorted order.

That means partitioning improves point-operation scalability but makes global ordered operations more complex.

This is a standard distributed-systems tradeoff appearing inside a single machine.

From multi-core to multi-node

The partition abstraction is also the foundation for future node-level distribution.

A partition can eventually be:

Owned by one node
Replicated to followers
Moved to another node
Routed through a cluster client

Partitioning is therefore more than a performance optimization.

It is the seam where a single-node database begins evolving into a distributed one.

LioranDB is developed by Swaraj Puppalwar at Lioran Group.

Explore:

Memtables: The Fast Write Buffer Inside LioranDB

Swaraj Puppalwar — Fri, 24 Jul 2026 18:49:44 +0000

Disk structures are durable, but updating them for every write is expensive.

LioranDB uses memtables to absorb writes before flushing them to the on-disk B+ tree.

What is a memtable?

A memtable is an ordered in-memory map.

In LioranDB, each entry contains either:

Value(bytes)

or:

Tombstone

A tombstone represents a deletion.

The memtable also tracks:

Approximate memory usage
Minimum LSN
Maximum LSN
Entry count
Put count
Delete count

The write path

A simplified write path looks like this:

Application write
    ↓
WAL durability
    ↓
Mutable memtable
    ↓
Immutable memtable queue
    ↓
Background flush
    ↓
Disk B+ tree

The active mutable memtable accepts new writes.

When it crosses a size limit, the engine rotates it into an immutable memtable.

That immutable table is no longer modified and can safely be flushed in the background.

Why ordered maps?

LioranDB uses an ordered map for memtable entries.

This helps because the flush process can emit keys in sorted order, which is friendly to the B+ tree and bulk-write paths.

It also simplifies range merging between:

Mutable data
Immutable data
On-disk pages

Backpressure

Background flushing cannot be allowed to fall behind forever.

LioranDB therefore tracks limits such as:

Maximum immutable memtables
Maximum immutable bytes
Partition-wide queue limits
Maximum writer stall duration

If the disk cannot drain the backlog quickly enough, the foreground write path slows down.

That may sound undesirable, but controlled backpressure is much safer than consuming memory until the process dies.

A memtable is not merely a cache.

It is a pressure valve between CPU-speed writes and disk-speed persistence.

Without that valve, the engine would either become slow on every commit or dangerously accumulate unbounded work.

Built by Swaraj Puppalwar under Lioran Group.

Links:

How LioranDB Protects Writes With a WAL

Swaraj Puppalwar — Fri, 24 Jul 2026 18:48:45 +0000

When an application inserts a document, the database cannot immediately rewrite every affected disk page and wait for all of them to be flushed.

That would be painfully slow.

LioranDB therefore uses a Write-Ahead Log, commonly called a WAL.

The core rule

Before a write is acknowledged as durable, its log record must be persisted.

Transaction
    ↓
Append changes to WAL
    ↓
fsync WAL
    ↓
Acknowledge commit
    ↓
Flush data pages later

If the process crashes before the data pages are written, the WAL can replay the committed operation.

Group commit

Calling fsync for every individual transaction would destroy throughput.

LioranDB groups multiple transactions into one durability batch.

A batch can be triggered by:

Enough pending bytes
Enough waiting transactions
A timeout
Strict durability mode
Shutdown
A follow-up durability generation

Conceptually:

Txn A ─┐
Txn B ─┼─→ One WAL append and fsync
Txn C ─┘

All transactions in that durable range can then be released together.

Durability modes

The engine supports multiple WAL profiles:

Strict
Balanced
HighThroughput
UnsafeBenchmark

Balanced is the default.

It groups commits over a very small time window while still syncing before acknowledging them.

UnsafeBenchmark can acknowledge data in the operating system cache without an fsync, but it exists for benchmarking, not production durability.

Why LSNs matter

Every WAL record has a Log Sequence Number, or LSN.

The engine tracks values such as:

Next LSN
Durable LSN
Checkpoint LSN
Pending WAL bytes
Oldest waiter age

These values tell the engine which operations are durable, which pages are behind, and which log segments may eventually be reclaimed.

A WAL turns disk latency from a wall into a queue.

The hard part is not merely appending bytes. It is coordinating batching, fsync, checkpoints, waiters, backpressure, and crash recovery without lying to the caller about durability.

That is where database engineering becomes properly spicy.

LioranDB is created by Swaraj Puppalwar at Lioran Group.

More information:

Why LioranDB Uses a Disk B+ Tree

Swaraj Puppalwar — Fri, 24 Jul 2026 18:47:23 +0000

A database needs more than a data file.

It needs a structure that can quickly locate a key without scanning every document.

LioranDB uses a disk-based B+ tree for this job.

What is a B+ tree?

A B+ tree is a balanced search tree designed for storage systems.

It contains two main page types:

Internal pages
    Store separator keys and child pointers

Leaf pages
    Store actual keys and values

A simplified tree looks like this:

                 [m]
               /     \
       [a ... l]       [m ... z]

Every lookup follows a path from the root to one leaf.

Because every level can contain many keys, the tree remains shallow even when the database contains millions of records.

Pages, not objects

LioranDB stores B+ tree nodes inside fixed-size disk pages.

Each page contains metadata such as:

Page magic
Page version
Page type
Page ID
Page LSN
CRC32 checksum

The checksum helps detect corrupted pages.

The Page LSN records the latest WAL position reflected by that page, which is useful during recovery and checkpointing.

Why not use a hash table?

A hash table is excellent for exact-key lookups, but it does not naturally support ordered scans.

A B+ tree can efficiently execute:

Find user 100
Find every user from 100 to 200
Find every key starting with "email:"
Walk records in sorted order

This is important because database indexes frequently rely on range and prefix scans.

Bulk loading

LioranDB also contains a bulk-load path for building a new B+ tree from sorted input.

Instead of repeatedly traversing and splitting pages, the engine can construct leaf and internal pages in batches.

That can avoid:

Repeated root-to-leaf lookups
Online page splits
Extra page decoding
Unnecessary cache pollution

A B+ tree may sound old-school, but it remains one of the most practical foundations for a disk database.

Good database architecture is often less about inventing a new tree and more about implementing an existing one without setting the disk on fire.

Built by Swaraj Puppalwar at Lioran Group.

Explore: