DEV Community

Cover image for Inside LioranDB: Why the Storage Engine Speaks Bytes, Not JSON
Swaraj Puppalwar
Swaraj Puppalwar

Posted on

Inside LioranDB: Why the Storage Engine Speaks Bytes, Not JSON

Most developers think of LioranDB as a document database.

Internally, however, its storage engine does not understand documents, objects, fields, or JSON.

It understands only:

table + key bytes + value bytes
Enter fullscreen mode Exit fullscreen mode

That separation is intentional.

The architecture

LioranDB is split into two major layers:

Application
    ↓
Document DBMS
    ↓
Transactional key-value engine
    ↓
WAL, memtables, B+ tree, pager and disk
Enter fullscreen mode Exit fullscreen mode

The engine exposes operations such as:

get(table, key)
put(table, key, value)
delete(table, key)
scan(table, range)
Enter fullscreen mode Exit fullscreen mode

The DBMS layer then adds document-oriented features:

  • Collections
  • JSON encoding
  • Queries
  • Updates
  • Secondary indexes
  • Text indexes
  • Transactions

For example, a secondary index can be represented as:

idx:status:active → document_id
Enter fullscreen mode Exit fullscreen mode

A text index can be represented as:

inv:database → posting_list
Enter fullscreen mode Exit fullscreen mode

The storage engine does not need to know what status, active, or database means.

It only stores ordered bytes.

Why this matters

This architecture keeps the core engine small and reusable.

The engine focuses on difficult low-level concerns:

  • Durability
  • Page management
  • Transactions
  • Recovery
  • Ordering
  • Concurrency
  • Range scans

The DBMS focuses on application-level semantics.

This also makes it possible to build different data models over the same engine in the future.

A document database is therefore not one giant component. It is a collection of carefully separated layers.

That separation is one of the most important architectural decisions inside LioranDB.


LioranDB is being developed by Swaraj Puppalwar under Lioran Group.

Learn more:

Top comments (0)