Swaraj Puppalwar

Posted on Jul 24

Why LioranDB Uses LSM-Based Secondary Indexes

#liorandb #database #nosql #indian

A document database usually stores a primary record by document ID.

But applications also ask questions such as:

Find users where status = "active"
Find invoices where customerId = 42
Find posts where category = "rust"

Those queries need secondary indexes.

LioranDB implements secondary indexing through an LSM-style index store.

The mutation path

When a document changes, the DBMS creates index mutations.

These mutations are written to an index mutation log and then applied asynchronously.

Document write
    ↓
Index mutation log
    ↓
Pending mutation queue
    ↓
Index memtable
    ↓
Immutable table
    ↓
Segment file
    ↓
Compaction

This separates the primary document write path from heavier index maintenance.

An LSM design converts many random updates into sequential batches.

Instead of repeatedly rewriting one large index structure, the engine creates immutable sorted segments.

Reads search across those segments and newer in-memory data.

Background compaction later merges segments.

LioranDB defines separate worker groups for:

This matters because secondary and text indexes have very different costs.

A secondary mutation may be a compact key-to-document mapping.

A text mutation may generate many terms and postings.

Treating them as identical would make scheduling less predictable.

Asynchronous indexing introduces index lag.

LioranDB tracks conditions such as:

Pending mutations
Mutation log bytes
Lag in LSNs
Lag in milliseconds
Segment counts
Compaction debt

Soft limits begin rescue work.

Hard limits trigger stronger backpressure.

This prevents the primary writer from outrunning the indexing pipeline forever.

LSM indexes are not free speed.

They exchange immediate random writes for background complexity, read amplification, compaction, and lag management.

Used carefully, that trade is extremely powerful.

Used carelessly, it creates a segment confetti cannon.

Links: