DEV Community: Alex Rustic

Jinbase – Multi-model transactional embedded database

Alex Rustic — Sat, 30 Nov 2024 20:41:34 +0000

Hi Dev !

I'm Alex, a tech enthusiast. I'm excited to show you Jinbase, my multi-model transactional embedded database.

Almost a year ago, I introduced Paradict, my take on multi-format streaming serialization. Given its readability, the Paradict text format appears de facto as an interesting data format for config files. But using Paradict to manage config files would end up cluttering its programming interface and making it confusing for users who still have choices of alternative libraries (TOML, INI File, etc.) dedicated to config files. So I used Paradict as a dependency for KvF (Key-value file format), a new project of mine that focuses on config files with sections.

With its compact binary format, I thought Paradict would be an efficient dependency for a new project that would rely on I/O functions (such as Open, Read, Write, Seek, Tell and Close) to implement a minimalistic yet reliable persistence solution. But that was before I learned that "files are hard". SQLite with its transactions, BLOB data type and incremental I/O for BLOBs seemed like the right giant to stand on for my new project.

Jinbase started small as a key-value store and ended up as a multi-model embedded database that pushes the boundaries of what we usually do with SQLite. The first transition to the second data model (the depot) happened when I realized that the key-value store was not well suited for cases where a unique identifier (UID) is supposed to be automatically generated for each new record, saving the user the burden of providing an identifier that could accidentally be subject to a collision and thus overwrite an existing record. After that, I implemented a search capability that accepts UID ranges for the depot store, timespans (records are automatically timestamped) for both the depot and key-value stores and GLOB patterns and number ranges for string and integer keys in the key-value store.

The queue and stack data models emerged as solutions for use cases where records must be consumed in a specific order. A typical record would be retrieved and deleted from the database in a single transaction unit.

Since SQLite is used as the storage engine, Jinbase supports the relational model de facto. For convenience, all tables related to Jinbase internals are prefixed with jinbase_, making Jinbase a useful tool for opening legacy SQLite files to add new data models that will safely coexist with the ad hoc relational model.

All four main data models (key-value, depot, queue, stack) support Paradict-compatible data types, such as dictionaries, strings, binary data, integers, booleans, datetimes, etc. Under the hood, when the user initiates a write operation, Jinbase serializes (except for binary data), chunks, and stores the data iteratively. A record can be accessed not only in bulk, but also with two levels of partial access granularity: the byte-level and the field-level.

While SQLite's incremental I/O for BLOBs is designed to target an individual BLOB column in a row, Jinbase extends this so that for each record, incremental reads cover all chunks as if they were a single unified BLOB. For dictionary records only, Jinbase automatically creates and maintains a lightweight index consisting of pointers to root fields, which then allows extracting from an arbitrary record the contents of a field automatically deserialized before being returned.

The most obvious use cases for Jinbase are storing user preferences, persisting session data before exit, order-based processing of data streams, exposing data for other processes, upgrading legacy SQLite files with new data models and bespoke data persistence solutions.

Jinbase is written in Python, is available on PyPI and you can play with the examples on the README.

Let me know what you think of this project.

Project Link: https://github.com/pyrustic/jinbase

Jesth – Next-level human-readable data serialization format

Alex Rustic — Thu, 18 May 2023 20:43:15 +0000

Hi Dev !

I'm Alex, a tech enthusiast. I'm excited to show you Jesth, a next-level human-readable data serialization format.

This project started out as a markup language for writing the docstrings of functions that would ultimately be consumed by a documentation generator. Basically the idea was to split a docstring into sections like Description and Parameters. Each section would consist of a header in square brackets and a body (lines of text between two headers).

Here's what a docstring for a sum function would look like:

This function takes in two integers a and b and returns their sum.

[parameters]
- a: First integer
- b: Second integer

[return] 
Sum of a and b

The Description section in the example above is actually an anonymous section, i.e., a section with an empty header.

Meanwhile, I was thinking of a way to automate part of my dev workflow by storing in a file commands grouped into tasks such as project creation, build, testing, release, et cetera. Similarly with the markup language for my documentation generator, I would use square brackets to define the tasks. Thus, a task would consist of a header and a body which would be a list of commands to be executed sequentially.

I built this project and named it Backstage. Here is a hypothetical backstage.tasks file:

[release]
& test
& generate_doc
& git_stuff
& build
# upload to PyPI
$ twine upload --skip-existing dist/*

[git_stuff]
$ git add .
$ git commit -m {message}
$ git push origin master

The example above is illustrative only and would not work. It contains 2 sections "release" and "git_stuff". Running the "release" task from the command line is equivalent to sequentially executing the commands in the "release" section.

The documentation generator and the scripting language, despite the obvious similarity in their formats, did not share any parsing code. So, to stop repeating myself, I created a file format and its library named Jesth which stands for Just Extract Sections Then Hack.

The library acts as an incomplete INI file parser that only hands the programmer the sections (as headers and their associated bodies which are lists of strings). No further interpretation of the data is done by the parser, allowing the programmer to unleash their creativity through useful hacks.

In its latest iteration, Jesth has matured and also includes a proper and extensively tested hack to convert a compatible section into a dictionary data structure, making Jesth my de facto preferred format for config files. I find Jesth more readable than TOML, YAML, and JSON.

Here, encoding a dictionary data structure in its own section with another section containing a prompt for ChatGPT:

[prompt]
I want you to act as a detective story writer. I will provide you with
two dictionary data structures representing the profiles of two people.
Your goal is to write a thrilling neo-noir story. My first request is:
"guess who the killer and victim is from the profiles, then build a story
that includes every detail of the profiles".

[profile]
# This section can be converted into a dictionary data structure
name = 'Jane Doe'
birthday = 2000-12-23Z10:17:37Z
photo_jpg = (bin)
    VGhpcyBpcyBub3QgYSBwaG90by4uLiBCdXQgdGhhbmsgeW91
    IGZvciB5b3VyIGludGVsbGVjdHVhbCBjdXJpb3NpdHkgOyk=
    ---
books = (dict)
    romance = (list)
        'Happy Place'
        'Romantic Comedy'
    sci-fi = (list)
        'Dune'
        'Neuromancer'
epitaph = (text)
    According to the law of conservation of energy,
    no a bit of you is gone;
    you are just less orderly.
    ---

[profile]
name = 'John Doe'
birthday = null
books = (list)
    'American Predator'
    'Mindhunter: Inside the FBI's Elite Serial Crime Unit'

You can learn more by reading the project's README and playing with the demo.

Let me know what you think of this project.

Project Link: https://github.com/pyrustic/jesth

Wrap, augment, and override functions and methods

Alex Rustic — Wed, 09 Nov 2022 12:44:54 +0000

HardMediaGroup, CC BY-SA 3.0, via Wikimedia Commons

On decoration

Python has a concept called Decorator which is a function that takes another function and extends the behavior. In the following script, the timeit decorator is used to measure the execution time of the heavy_computation function:

import time
from functools import wraps

def timeit(text):
    def deco(target):
        @wraps(target)
        def wrapper(*args, **kwargs):
            # execute and measure the target run time
            start_time = time.perf_counter()
            result = target(*args, **kwargs)
            total_time = time.perf_counter() - start_time
            # print elapsed time
            print(text.format(total=total_time))
            return result
        return wrapper
    return deco

@timeit(text="Done in {total:.3f} seconds !")
def heavy_computation(a, b):
    time.sleep(2)  # doing some heavy computation !
    return a*b

if __name__ == "__main__":
    result = heavy_computation(6, 9)
    print("Result:", result)

Output:

$ python -m test
Done in 2.001 seconds !
Result: 54

Besides benchmarking, there are many other cool things that can be done with the Python decorator. For example, the Flask and Bottle web frameworks implement routing with decorators.

Hooking

While decorators are cool, it's worth mentioning that using a decorator is much more intuitive than writing its code. The code is entirely different depending on whether the decorator takes arguments or not.

The following code performs the same task as the previous one, except it is more clear and intuitive:

import time
from hooking import on_enter

def timeit(context, *args, **kwargs):
    # execute and measure the target run time
    start_time = time.perf_counter()
    context.result = context.target(*args, **kwargs)
    total_time = time.perf_counter() - start_time
    # print elapsed time
    text = context.config.get("text")  # get 'text' from config data
    print(text.format(total=total_time))
    context.target = None

@on_enter(timeit, text="Done in {total:.3f} seconds !")
def heavy_computation(a, b):
    time.sleep(2)  # doing some heavy computation !
    return a*b

if __name__ == "__main__":
    result = heavy_computation(6, 9)
    print("Result:", result)

Output:

$ python -m test
Done in 2.001 seconds !
Result: 54

The Hooking library used in the code above uses Python decorators to wrap, augment, and override functions and methods. It is a generic hooking mechanism which is perfect for creating a plug-in mechanism for a project, performing benchmarking and debugging, implementing routing in a web framework, et cetera.

Dual paradigm

Also, it is a dual paradigm hooking mechanism since it supports tight and loose coupling. The previous code uses the tight coupling paradigm, that's why the timeit hook is directly tied to the target function.

In loose coupling paradigm, targets functions and methods are tagged using a decorator, and hooks are bound to these tags. So when a target is called, the bound hooks are executed upstream or downstream. This paradigm is served by a class designed for pragmatic access via class methods. This class can be easily subclassed to group tags by theme for example.

Here is an example of the loose coupling paradigm:

import time
from hooking import H

@H.tag
def heavy_computation(a, b):
    print("heavy computation...")
    time.sleep(2)  # doing some heavy computation !
    return a*b

def upstream_hook(context, *args, **kwargs):
    print("upstream hook...")

def downstream_hook(context, *args, **kwargs):
    print("downstream hook...")

# bind upstream_hook and downstream_hook to the "heavy_computation" tag
H.wrap("heavy_computation", upstream_hook, downstream_hook)

if __name__ == "__main__":
    result = heavy_computation(6, 9)
    print("Result:", result)

Output:

$ python -m test
upstream hook...
heavy computation...
downstream hook...
Result: 54

Conclusion

This library is available on PyPI and you can play with the examples which are on the project's README. I would like to know what you think of this project. Your questions, suggestions and criticisms are welcome !

Link to the project: https://github.com/pyrustic/hooking