A normal secondary index can answer:
status = "active"
It cannot efficiently answer:
documents containing "distributed database"
LioranDB therefore has a dedicated text-segment architecture.
Tokenization
Text is split on non-alphanumeric characters.
Depending on index options, tokens can be normalized to lowercase and filtered through stopwords.
"Building Distributed Databases"
becomes
["building", "distributed", "databases"]
Segment contents
A LioranDB text segment can contain several files and structures:
Term dictionary
Posting lists
Document map
Document-length norms
Optional term positions
Bloom filter
Segment metadata
A posting connects a term to the local documents containing it.
"database" → [doc 2, doc 8, doc 19]
Positions can record where the term appears inside each document.
That enables more advanced query behaviour and phrase-aware features.
Global and local document IDs
Each segment assigns compact local IDs to its documents.
A separate document map translates them back to global document IDs.
This keeps postings smaller while preserving the external identity of the record.
Bloom filters
Each segment also maintains a Bloom filter for terms.
Before reading a segment's postings, the query path can test whether the term might exist there.
A negative answer is definitive.
A positive answer means the segment may contain the term and should be checked.
Query modes
The text query layer supports modes such as:
AND
OR
It also emits scored documents and metrics including:
- Query time
- Postings read
- Candidate documents
- Segments searched
Full-text search is essentially a specialized database living beside the document database.
Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake.
Built by Swaraj Puppalwar under Lioran Group.
Learn more:
Top comments (0)