DEV Community: Johannes Lichtenberger

Hacktoberfest: Contribute to our temporal database system

Johannes Lichtenberger — Fri, 01 Oct 2021 21:34:10 +0000

We are a (very) small team working on a database system in our spare time (https://sirix.io | https://github.com/sirixdb/sirix).

sirixdb / sirix

SirixDB is an an embeddable, bitemporal, append-only database system and event store, storing immutable lightweight snapshots. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach.

An Embeddable, Bitemporal, Append-Only Database System and Event Store

Stores small-sized, immutable snapshots of your data in an append-only manner. It facilitates querying and reconstructing the entire history as well as easy audits.

Download ZIP | Join us on Discord | Community Forum | Documentation | Architecture & Concepts

Working on your first Pull Request? You can learn how from this free series How to Contribute to an Open Source Project on GitHub and another tutorial: How YOU can contribute to OSS, a beginners guide

"Remember that you're lucky, even if you don't think you are because there's always something that you can be thankful for." - Esther Grace Earl (http://tswgo.org)

We want to build the database system together with you. Help us and become a maintainer yourself. Why? You may like the software and want to help us. Furthermore, you'll learn a lot. You may want to…

View on GitHub

It began as a research system at the University of Konstanz and was the main focus of two PhD thesis and several bachelor and master thesis.

Johannes, the current maintainer worked on the system for his bachelor as well as master thesis. Furthermore, he also contributed as a research assistent.

The system first of all builds a trie based index over all currently stored revisions. To efficiently reconstruct a revision the timestamps and the offsets into the log-file, the main storage, are written to a revision file. Second, the main document index is referenced from the revision roots. Furthermore, the system stores a path summary as well as secondary indexes in subtrees of the revision root pages. The tree of in-memory indexes are mapped to a sequential log-file during a commit in a postorder traversal. The parent pages store hashes of their children in references to the child pages. This can be used to check if data has correctly been stored in the future.

Another idea is to version the data leaf pages, thus that not the whole page has to be copied during a write. Instead, a clever sliding snapshot algorithm is used to avoid read- or write-peaks which would occur during full snapshots of a page after increments haved been written.

We'd be very happy to get contributions during Hacktoberfest but also in the long run.

Contributors wanted: Practice your Svelte and TypeScript skills 👩‍💻👨‍💻

Johannes Lichtenberger — Tue, 13 Oct 2020 22:07:59 +0000

👾 An Evolutionary Database System for Modern Hardware

SirixDB is an evolutionary, accumulate-only database system. Each new commit stores more data in a new snapshot, which shares common data with the previous revisions. The system never overwrites data. It supports a single read-write transaction per resource concurrently to N read-only transactions. Furthermore, in stark contrast to other database systems, the persistent, durable tree consists of variable-sized database pages. Versioning takes place at a sub-page level according to a novel sliding.

🐣 Svelte based Frontend

Moshe Uminer and I are looking for contributions to a frontend for the system mentioned above. The frontend is pretty new (the project started a few months ago), thus there's a lot of work to do.

sirixdb / sirix-svelte-frontend

A GUI console for SirixDB, using Svelte/Sapper.

Project Setup - Setting up SirixDB locally

Unfortunately, this is currently a bit messy, but still rather straightforward.

There are several resources required to run SirixDB, and they can all be found in the resource directory of this repository.

SirixDB depends on keycloak for authentication, so we first need to run (in the directory in which you have the above resources):

docker-compose up -d keycloak

It can take about a minute for keycloak to be ready for connections. If you like, you can use the wait.sh script in the above mentioned resources folder to wait until keycloak is ready. The docker-compose file runs a script from the resources folder to initialize keycloak with a user "admin" with the password "admin".

Once keycloak is ready, run:

docker-compose up -d server

The Sirix database is now ready for use.

Developing

Code documentation

A lot of documentation is missing. However you can find…

View on GitHub

XML display #10

mosheduminer posted on Apr 03, 2020

Currently, the /databases route is only capable of displaying JSON. We need to add complete XML functionality.

View on GitHub

Layout customization #249

mosheduminer posted on Sep 30, 2020

Enable resizing the various "sections" in each route,
as well as allow drag-and-drop to rearrange the layout. (a section would be, for example, (in the resource route) the bar showing database/resource names, the history timeline, the resource view. (in the query route) the query box, the query result display, the saved/old queries bar).

I imagine this would require

using something like svelte-dnd-action to allow rearranging the layout,
and using the resize CSS property to allow native element resize.
I imagine the resize property would exist on containers that would each containe a single "region" that is draggable.
There should be listeners to resize the container region when a large section is moved to a small region (and swap the small section to the other region).
There should be listeners to accommodate the resize of a region by resizing other regions.

This would require refactoring the layouts to be compatible with drag and drop.

View on GitHub

The idea is to build a frontend, which can interactively depict the differences between revisions of the data and visualize the result of time-travel queries.

In general, it would be great to provide a similar space-filling approach (modified Sunburst-View) like the one I once developed for my master's thesis:

Sunburst visualization #250

mosheduminer posted on Sep 30, 2020

Create a zoomable multi-layered sunburst visualization, displaying the density (in terms of descendantCount) of various parts of a resource.

This will probably use D3, but I don't have any further ideas than that.

View on GitHub

General visualizations issue #251

mosheduminer posted on Sep 30, 2020

There are many visualizations described in @johanneslichtenberger 's master thesis, these need to be recreated. A PDF of the master thesis is here.

View on GitHub

Furthermore other, related views are imaginable. I'm open to new research ideas 👍

I'm for instance also interested into how best to present the result of time travel queries to a user / analyst.

Create a Rust-Client during Hacktoberfest

Johannes Lichtenberger — Wed, 07 Oct 2020 22:50:35 +0000

Hi all,

you can create a Rust-Client for SirixDB and make a huge contribution during Hacktoberfest. We can split the task into many smaller tasks, of course.

Create a Rust-Client #345

JohannesLichtenberger posted on Oct 07, 2020

We should provide a Rust client in addition to the TypeScript and Python client.

https://sirix.io/rest-api.html#api-description

View on GitHub

We need your help, bringing the project forward :-)

It's an evolutionary, accumulate-only database system, which means that the full history of each of your resources in the database is retained and can be queried to gain further insights, to undo human or machine errors or for audits.

Kind regards
Johannes

Query the full history of your JSON database

Johannes Lichtenberger — Tue, 06 Oct 2020 19:05:38 +0000

Contribute to the Core and the Svelte based Frontend

The following diagram depicts the evolution of states through modification events in SirixDB:

Introduction

First and foremost, you probably don't need a database system to handle small JSON files, ranging to a few megabytes.

However, if you have to manage and query GBs of data, you should use a database system.

Usually, database systems, however, are not designed to keep the full history of your data. Often, the system overwrites data during a change or keeps the data for a short time. The latter usually happens due to transactions, which currently read slightly outdated data. Thus, a garbage collector has to wait until all reading transactions finish. Then, it's able to delete the old data.

Instead, SirixDB makes a huge persistent tree, durable during commits. It only ever appends data. Every revision is indexed, whereas the revisions share unchanged page-fragments. Think of it like Git, but on a sub-file level. Persistent trees are also common in functional languages like Haskell and Closure. A transaction commit, which serializes pages in a postorder traversal, is depicted in the following image:

Usually, mapping these persistent in-memory structures to a durable device adds write amplification, as the full leaf-to-root path has to be adapted. We, however, use a keyed trie for our main document store. Thus no structural changes occur as in B-trees. Furthermore, we developed a new sliding snapshot algorithm to version the data pages and store variable-sized page fragments. Therefore only a few changed records are written to a new page-fragment. Reconstructing an in-memory page involves reading a sliding window of page-fragments from random locations in parallel.

Due to the requirement for fast, random, fine granular reads, modern hardware as Byte Addressable NVM will be essential to good performance.

SirixDB stores the JSON data in a binary format, whereas each transaction is bound to a specific revision. Currently, N read-only transactions on a resource can coexist with one read-write transaction on a resource (represents the JSON data).

The encoding is as follows, except that we lately also introduced a pointer from the parent to the last child node.

Query Your Data

The system uses and extends an XQuery 3.0 processor to process both XML and JSON data.

We're able to import JSON files from a specific directory via:

jn:load('mycol.jn', (), io:ls('/home/johannes/json-data', '\\.json$'))

It'll create a database named mycol.jn and several resources.

We can also store a bunch of JSON-strings within several resources in the database (mycol.jn):

jn:store('mycol.jn',(),('["bla", "blubb"]','{"foo": true}'))

Note, that by when JSON-strings we have to replace true and false with true() and false() as well as null with jn:null().

We can then query the database collection via:

for $doc in jn:collection('mydocs.col') return $doc

Or, if we want to open a specific resource in the database, we can use:

jn:doc('mydocs.col', 'myresource1')

Furthermore, we can update the resource. Let's say we have the following small JSON file:

{
  "foo": [
    "bar",
    null,
    2.33
  ],
  "bar": {
    "hello": "world",
    "helloo": true
  },
  "baz": "hello",
  "tada": [
    {
      "foo": "bar"
    },
    {
      "baz": false
    },
    "boo",
    [],
    {}
  ]
}

We can insert a JSON object with:

insert json { "name": "keyword" } into jn:doc('mydocs.col', 'myresource1')=>tada=>foo

We dereference object field names with the => operator. It even can drill down into arrays and find the value for the field name.

The update-operation will create a new revision with the field name and the added value. For instance

jn:doc('mydocs.col', 'myresource1')=>tada[[0]]

will select the first object in the array value of the field name tada.

{
  "foo": "bar",
  "name": "keyword"
}

We implicitly query revision 2. If we instead query revision 1, we'll still retrieve the old JSON object.

jn:doc('mydocs.col', 'myresource1', 1)=>tada[[0]]

Note that we specified one as the third argument of the function. The output is:

{
  "foo": "bar"
}

Instead of providing a numeric value to the third parameter we can also open a specific revision by a timestamp (here: to check how a specifc resource looked like in April 2018):

jn:open('mydocs.col', 'myresource1', xs:dateTime(\"2018-04-01T05:00:00-00:00\"))

With the function open-revisions we're able to load all revisions of a resource between two points in time:

jn:open('mydocs.col', 'myresource1', xs:dateTime(\"2018-04-01T05:00:00-00:00\", xs:dateTime(\"2019-04-01T05:00:00-00:00\"))

The following query will insert an object into the array value dereferenced by =>foo as the second item:

insert json { "name": "keyword" } into jn:doc('mydocs.col', 'myresource1')=>foo at position 1

Using

jn:doc('mydocs.col', 'myresource1')=>foo

we get the output:

[
  "bar",
  {
    "foo": "bar"
  },
  null,
  2.33
]

Likewise, we can replace a JSON value with

replace json value of jn:doc('mydocs.col', 'myresource1')=>tada[[0]] with "yes"

or delete a value:

delete json jn:doc('mydocs.col', 'myresource1')=>tada[[0]]

If you submit the queries through the REST-API, you first have to be authorized. Thus, a revision, as in Git, also stores the author's name and a UUID.

To get the author name, who committed a specific revision:

jn:author-name(jn:doc(...))

To get the UUID:

jn:author-uuid(jn:doc(...))

You can also project fields:

jn:doc('mydocs.col', 'myresource1')=>bar{hello}

This will output:

{
  "hello": "world"
}

Time travel functions

Besides, we can use a bunch of time travel queries. The following functions retrieve different versions of a JSON item:

jn:future($item as json-item(), $includeSelf as xs:boolean) as json-item()*

Function for selecting the json-item in the future or the future-or-self. The first parameter is the context item. The second parameter denotes if the current item should be included in the result or not.

jn:past($item as json-item(), $includeSelf as xs:boolean) as json-item()*

Function for selecting the json-item in the past or the past-or-self. The first parameter is the context item. Second parameter denotes if the current item should be included in the result or not.

jn:all-times($item as json-item()) as json-item()+

Function for selecting the json-item in all revisions.

jn:first($item as json-item()) as json-item()?

Function for selecting the json-item in the first revision.

jn:last($item as json-item()) as json-item()?

Function for selecting the json-item in the last / most-recent revision.

jn:previous($item as json-item()) as json-item()?

Function for selecting the json-item in the previous revision.

jn:next($item as json-item()) as json-item()?

Function for selecting the json-item in the next revision.

To get an item in all revisions, in which it has been changed (or inserted):

jn:history($item as json-item()) as json-item()?

Diffing

We're of course also able to retrieve diffs:

jn:diff('mydocs.col', 'myresource2', 1, 3)

This function compares revision one and three of a resource called myresource2.

The output format is a JSON-string:

{
  "database": "json-path1",
  "resource": "shredded",
  "old-revision": 1,
  "new-revision": 3,
  "diffs": [
    {
      "insert": {
        "nodeKey": 26,
        "insertPositionNodeKey": 1,
        "insertPosition": "asFirstChild",
        "deweyID": "1.17.9",
        "depth": 2,
        "type": "jsonFragment",
        "data": "{\"tadaaa\":\"todooo\"}"
      }
    },
    {
      "delete": {
        "nodeKey": 13,
        "deweyID": "1.17.49",
        "depth": 2
      }
    },
    {
      "update": {
        "nodeKey": 15,
        "deweyID": "1.17.65",
        "depth": 2,
        "name": "tadaa"
      }
    }
  ]
}

For instance, this format is also used by our GUI, which we currently develop based on Svelte.

It denotes that SirixDB compared revisions 1 and 3. Furthermore, the different diff types are insert, delete, update and replace (the latter not shown). The insertPositionNodeKey is the context node, where an insert occurs, the insertPosition denotes if it's inserted as a first child, as the right sibling, or the left sibling.

We can also diff a subtree and specify two additional parameters. The root node, which is going to be compared, and the depth:

jn:diff('mydocs.col', 'myresource2', 1, 3, 7453, 2)

This compares also revision 1 and 3 of the myresource2 resource in the mydocs.col database. However, this time, the diff starts at the node, denoted by its unique nodeKey 7453. Furthermore the diffing should skip descendants, which are deeper than 2 levels.

In order to get the nodeKey of a specific item, we can use:

sdb:nodekey(...)

Conclusion

SirixDB offers powerful ways to query the full history of your data.

We omitted the creation of index structures, but you can also create secondary index structures with XQuery, which are automatically versioned as well.

Consider the following document (serialized SirixDB resource with two revisions):

{
  "sirix": [{
    "revisionNumber": 1,
    "revision": {
      "foo": ["bar", null, 2.33],
      "bar": {
        "hello": "world",
        "helloo": true
      },
      "baz": "hello",
      "tada": [{
        "foo": "bar"
      }, {
        "baz": false
      }, "boo", {},
        []
      ]
    }
  }, {
    "revisionNumber": 2,
    "revision": {
      "tadaaa": "todooo",
      "foo": ["bar", null, 2.33],
      "bar": {
        "hello": "world",
        "helloo": true
      },
      "baz": "hello",
      "tada": [{
        "foo": "bar"
      }, {
        "baz": false
      }, "boo", {},
        []
      ]
    }
  }]
}

We can create a Content-And-Structure (CAS) index as follows:

let $doc := jn:doc('mycol.jn','mydoc.jn')
let $stats := jn:create-cas-index($doc, 'xs:string', '/sirix/[]/revision/tada//[]/foo/[]/baz') return {"revision": sdb:commit($doc)}

In order to answer the query

let $result := jn:doc('mycol.jn','mydoc.jn')=>sirix[[2]]=>revision=>tada[.=>foo=>baz >= 'baa' and .=>foo=>baz <= 'brr'] return $result

Furthermore, we didn't mention FLOWR expressions, which are the cornerstone of XQuery. A simple example with an implicit join:

for $i in jn:doc('mycol.jn','mydoc.jn')=>paths=>"/consolidated_screening_list/search"=>get
let $j := $i=>parameters=>name
return for $k in $j
       where $k eq 'keyword'
       return { "result": $i, "nodekey": sdb:nodekey($i) }

Contribute to An Evolutionary Database System during #Hacktoberfest (technology stack: Java, Kotlin, Vert.x)

Johannes Lichtenberger — Sun, 04 Oct 2020 22:55:33 +0000

👾 An Evolutionary Database System for Modern Hardware

We have a lot of issues to work on. I'll list some of them for which you don't have to have a database background:

New logo and/or Logo for the Forum #85

JohannesLichtenberger posted on Sep 26, 2019

We are missing a logo in the size 64×64 for the Forum, such that we have a start logo for Android mobile phones...

View on GitHub

Create a Kotlin HTTP-Client #251

JohannesLichtenberger posted on May 19, 2020

Similar to the Python and TypeScript Client we should provide a Kotlin Client.

View on GitHub

Create Java9 and above module-infos. #60

JohannesLichtenberger posted on Dec 29, 2018

In order to use Jigsaw/the module system, we should write module definition files.

View on GitHub

Implement a JsonStreamingShredder based on Vert.x JSON streaming parser #282

JohannesLichtenberger posted on Jul 15, 2020

Thus, we can omit storing first a temp JSON file in the HTTP-Server:

https://vertx.io/docs/apidocs/io/vertx/core/parsetools/JsonParser.html

View on GitHub

Update REST-API to Vert.x 4 Beta 1 #297

JohannesLichtenberger posted on Jul 29, 2020

We should update the REST-API (server-rest-api bundle) to Vert.x 4 Beta 1. Some stuff has been deprecated, though.

View on GitHub

Add a compile time DI framework #343

JohannesLichtenberger posted on Oct 03, 2020

We should add a compile time framework (Dagger2?) to make it easier to mock dependencies of a class.

View on GitHub

Kotlin based wrapper APIs #63

JohannesLichtenberger posted on Dec 30, 2018

Therefore we also could create a nice DSL.

But in essence a basic version for the XdmNode-Transaction layer for instance would involve infix functions trx moveTo 3 and alike, removing builders, just use default parameters...

However, I'd love to discuss this with Kotlin experts :-)

View on GitHub

WebSockets for the REST-API to stream updates/changes into the Browser in real-time #286

JohannesLichtenberger posted on Jul 15, 2020

This feature could be used by visualizations in the front-end for instance.

View on GitHub

Replace the hash function #269

JohannesLichtenberger posted on Jun 04, 2020

We optionally store a rolling hash of all nodes in a resource. Currently, to avoid collisions as much as possible SirixDB uses the first 128 Bits of a 256 Bit SHA256 hash. We probably can use a non-cryptographic fast(er) hash function which produces a low number of hash colissions given 64Bit unique, dense, ascending nodeKeys + values...

View on GitHub

Making the use of Keycloak in the REST-API optional #254

JohannesLichtenberger posted on May 21, 2020

In order to make it easier to setup SirixDB with an HTTP-Server for testing we should make the use of Keycloak optional.

all OAuth2 routes should not be available
the Auth handler can just return the current route
in this case use XmlDBStore instead of XmlSessionDBStore and JsonDBStore instead of JsonSessionDBStore

View on GitHub

Refactoring according to clean code principles #109

JohannesLichtenberger posted on Oct 03, 2019

For instance splitting classes. Maybe also using a dependency injection framework if needed which works at compile time. For instance Dagger2...

View on GitHub

Review REST-API built with Vert.x and Kotlin (Coroutines) as well as for authentication Keycloak #92

JohannesLichtenberger posted on Sep 27, 2019

A code review of one or more classes would be great, as I want to write idiomatic Kotlin code. Furthermore, I'm sure we can abstract some stuff between the CRUD-classes regarding XML and JSON resources, as they are really similar.

View on GitHub

Add documentation and/or tutorials #114

JohannesLichtenberger posted on Oct 06, 2019

If you give SirixDB a try, how about writing a tutorial and/or adding missing stuff to the documentation!?

View on GitHub

Practice your Svelte skills during Hacktoberfest 👩‍💻👨‍💻

Johannes Lichtenberger — Wed, 30 Sep 2020 20:55:08 +0000

👾 An Evolutionary Database System for Modern Hardware

🐣 Svelte based Frontend

Moshe Uminer and I are looking for contributions to a frontend for the system mentioned above. The frontend is pretty new (the project started a few months ago), thus there's a lot of work to do.

sirixdb / sirix-svelte-frontend

A GUI console for SirixDB, using Svelte/Sapper.

Project Setup - Setting up SirixDB locally

Unfortunately, this is currently a bit messy, but still rather straightforward.

There are several resources required to run SirixDB, and they can all be found in the resources directory of this repository.

SirixDB depends on keycloak for authentication, so we first need to run (in the directory in which you have the above resources):

docker-compose up -d keycloak

It can take a minute for keycloak to be ready for connections. If you like, you can use the wait.sh script in the above mentioned resources directory to wait until keycloak is ready. A script from the resources folder initializing keycloak with a user "admin" with the password "admin" is run automatically.

Once keycloak is ready, navigate to http://localhost:8080/admin and login using the credentials admin/admin. Click on Users in the sidebar navigation, then click Add user. Enter the name of the…

View on GitHub

XML display #10

mosheduminer posted on Apr 03, 2020

Currently, the /databases route is only capable of displaying JSON. We need to add complete XML functionality.

View on GitHub

Layout customization #249

mosheduminer posted on Sep 30, 2020

Enable resizing the various "sections" in each route,
as well as allow drag-and-drop to rearrange the layout. (a section would be, for example, (in the resource route) the bar showing database/resource names, the history timeline, the resource view. (in the query route) the query box, the query result display, the saved/old queries bar).

I imagine this would require

using something like svelte-dnd-action to allow rearranging the layout,
and using the resize CSS property to allow native element resize.
I imagine the resize property would exist on containers that would each containe a single "region" that is draggable.
There should be listeners to resize the container region when a large section is moved to a small region (and swap the small section to the other region).
There should be listeners to accommodate the resize of a region by resizing other regions.

This would require refactoring the layouts to be compatible with drag and drop.

View on GitHub

The idea is to build a frontend, which can interactively depict the differences between revisions of the data and visualize the result of time-travel queries.

In general, it would be great to provide a similar space-filling approach (modified Sunburst-View) like the one I once developed for my master's thesis:

Sunburst visualization #250

mosheduminer posted on Sep 30, 2020

Create a zoomable multi-layered sunburst visualization, displaying the density (in terms of descendantCount) of various parts of a resource.

This will probably use D3, but I don't have any further ideas than that.

View on GitHub

General visualizations issue #251

mosheduminer posted on Sep 30, 2020

There are many visualizations described in @johanneslichtenberger's master thesis, these need to be recreated. A PDF of the master thesis is here.

View on GitHub

Furthermore other, related views are imaginable. I'm open to new research ideas 👍

I'm for instance also interested into how best to present the result of time travel queries to a user / analyst.

A Hacktoberfest Project for Java/Kotlin Backend and Svelte/Typescript Frontend developers 👩‍💻👨‍💻

Johannes Lichtenberger — Mon, 28 Sep 2020 17:03:43 +0000

SirixDB - An evolutionary, accumulate only database system (currently storing XML and JSON data) 🦁

Feel free to contribute on Github to the Core and the Svelte based Frontend 💚

Hi all,

there's plenty of possibilities to work on a versioned database system, which preserves every revision of your data in a space-efficient manner through structural sharing and a novel sliding snapshot algorithm. 🐬

SirixDB only stores changes between revisions, but any revision is indexed and can be reconstructed in O(n), whereas n is the number of nodes in the revision.

If you're eager to participate in #Hacktoberfest, there are plenty of opportunities. For instance, working on a frontend, the backend, a non-blocking REST API built with Vert.x, Kotlin and Keykloak, or a query optimizer (for instance, to rewrite the AST based on cost-based rules). Or you might want to look into how to best horizontally scale SirixDB, for example, via a distributed log (Apache Kafka?!) 😃

Furthermore, the sirix.io website needs some love.

🐣 Frontend with Svelte, TypeScript and D3.js

Moshe Uminer currently, besides working on and maintaining a Python and a TypeScript based client for SirixDB, is working on a frontend built with Svelte.

On the one hand, the purpose of the front end is to interact with the database system through queries. On the other hand, to visualize the differences between revisions.

In general, I'd love to provide a similar space-filling approach (modified Sunburst-View) as the one I once developed for my master's thesis:

🚀 Backend written in Java and a non-blocking REST-API built with Vert.x and Kotlin

If you're more into backend stuff, you can either work directly on issues regarding SirixDB or work on Kotlin based stuff with Vert.x.

sirixdb / sirix

SirixDB is an an embeddable, bitemporal, append-only database system and event store, storing immutable lightweight snapshots. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach.

For instance, Web Socket support in the non-blocking HTTP-Server:

Websockets to stream resource changes into the client / browser #98

JohannesLichtenberger posted on Sep 28, 2019

I'd put this into the rest-api module and build it with Vert.x and Kotlin.

View on GitHub

Try to create a native image with the GraalVM (however, Java 14 or 15 might not be supported, though :()

Create native images with the GraalVM #77

JohannesLichtenberger posted on May 08, 2019

For a fast startup time in containers... peek performance however might not be as good.

View on GitHub

A Kotlin-based DSL for SirixDB:

Kotlin based wrapper APIs #63

JohannesLichtenberger posted on Dec 30, 2018

Therefore we also could create a nice DSL.

But in essence a basic version for the XdmNode-Transaction layer for instance would involve infix functions trx moveTo 3 and alike, removing builders, just use default parameters...

However, I'd love to discuss this with Kotlin experts :-)

View on GitHub

Have a look into all the issues :-)

Besides, a Kotlin-Client and a Rust-Client is needed in the long run.

Query optimizations

Or if you would like to work on query optimizations, you can look into how to rewrite the AST produced by Brackit:

JSON with a database storage backend #5

JohannesLichtenberger posted on Jan 24, 2019

Hey Sebastian, sorry to bother you all the time, but maybe you can also give a good tip how you'd implement a binding for a native JSON storage backend. I'm currently finishing the work on a temporal JSON storage layer besides XML for my open source project (http://sirix.io) and would like to also use Brackit. Maybe I could also implement the interfaces you already provide (should be basically the same as for my XDM node transactions) and then implement some functions for opening a JSON resource...

Basically my JSON node layer looks like the one in Marklogic: https://docs.marklogic.com/guide/app-dev/json

View on GitHub

We have plenty of optimization potential here, as we currently don't take any inherent storage optimizations into account. 👍

This will increase the performance of specific queries considerably.

A query looks like this:

let $doc := jn:open('database','resource', xs:dateTime('2019-04-13T16:24:27Z'))
let $statuses := $doc=>statuses
let $foundStatus := for $status in $statuses
  let $dateTimeCreated := xs:dateTime($status=>created_at)
  where $dateTimeCreated > xs:dateTime("2018-02-01T00:00:00")
        and not(exists(jn:previous($status)))
  order by $dateTimeCreated
  return $status
return {"revision": sdb:revision($foundStatus), $foundStatus{text}}

Have a look at Sebastian's thesis Separating Key Concernsin Query Processing

He worked under supervision of Theo Härder, who invented ACID transactions 👌

👾 Think about how to scale SirixDB horizontally

I've thought about it and would love to distribute the transaction log with Apache BookKeeper or Apache Pulsar (which uses Apache BookKeeper). It might have some advantages in comparison to Apache Kafka.

❤️ Happy coding

So, there's plenty of stuff to work on. It's a super diverse field. I deliberately want to invite beginners to work on things as I'm a frontend beginner, for instance, and I have not much knowledge about query optimizers. However, I'm more than eager to learn. 👋

I hope the project is useful for some of you, and I even got a lot of contributions since the last Hacktoberfest after years of working alone; that's simply awesome. Thank you so much 😎

If you like the project, you might share it on twitter and spread the word!? 🙈

Contribute on GitHub SirixDB and GitHub SirixDB Web Frontend 💚

Have a nice day and a great #Hacktoberfest
Johannes

Help to build an evolutionary, accumulate-only Database System

Johannes Lichtenberger — Fri, 18 Sep 2020 21:41:17 +0000

You can help to build an evolutionary, accumulate-only database system for modern hardware called SirixDB.

The core repository (Java and Kotlin):

"We want to research the indexing of data in a huge persistent data-structure, mapped to persistent storage. SirixDB stores data and indexes of database resources in a huge persistent tree of tries. It allows us to retrieve any past revision and the current revision, whereas the revisions itself are indexed. The system supports time travel queries, reconstruction of any past state, and to revert to any past state. It only ever aggregates data over time and never overwrites data."

sirixdb / sirix

SirixDB is a temporal, evolutionary database system, which uses an accumulate only approach. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach called sliding snapshot.

The query processor:

"With our research, we want to develop an extensible, retargetable compilation, and runtime framework for driving scripting-like query languages for structured and semi-structured data. Ideally, we envision to prepare the ground for a tool for query languages similar to what the LLVM compiler framework is for general programming languages: an ecosystem in which proven optimization techniques and algorithms are readily available for implementing custom query processing systems."

sirixdb / brackit

Brackit is the query compiler we are using with temporal enhancements

The frontend:

"We want to explore how to visualize time-travel queries best and how best to depict the differences between two revisions or any subtrees."

sirixdb / sirix-svelte-front-end

A GUI console for SirixDB, using Svelte/Sapper.

Best way to be able to spend more time on an Open Source project?

Johannes Lichtenberger — Tue, 08 Sep 2020 21:01:50 +0000

Hi all,

I wonder what's the best way to get to spend more time on an Open Source project you love. In my case, I'd love to work less for the company I'm currently employed and spend more time on ideas around an Open Source project of mine.

I think a dual license approach, for instance, doesn't make much sense for providing custom solutions on top of the Open Source version. In my opinion, it would be best to spend all the time on the Open Source project itself and make it a better experience for everyone. Meaning, I guess it's all about being "lucky" or more of a genius to work on something that big companies need, and it would not make sense for them to build their solution from scratch, but rather sponsor your project.

Kind regards
Johannes

VLDB 2020 (very large databases conference) for free

Johannes Lichtenberger — Mon, 31 Aug 2020 01:41:11 +0000

Hi all,

don't miss https://vldb2020.org, it's the whole week long and for free :-)

kind regards
Johannes

Source of an Van Emde Boas Tree Layout for Binary Search Trees

Johannes Lichtenberger — Thu, 20 Aug 2020 12:17:44 +0000

Hi all,
where can I find the source code (preferably in Java or Kotlin) for a Van Emde Boas Tree Layout? As it's the best layout for arranging a static binary tree and even for an updatable binary tree (with an array packed structure) I wanted to see a "real world" implementation.

It's also okay to find an implementation in another language.

As it's a cache oblivious method I wonder if the implementation is not practical at all.

kind regards
Johannes

How to prepare for Hacktoberfest as a project owner?

Johannes Lichtenberger — Wed, 19 Aug 2020 20:08:59 +0000

Hi all,

I'm maintaining an Open Source project[1], an embeddable persistent and durable data store. It retains the full history of each resource and follows novel ways to version database pages. It supports time travel queries and the reconstruction of each revision, whereas the revisions are indexed itself. We designed SirixDB, especially for modern hardware such as high-performance SSDs or even Byte Addressable NVM, such as Intel Optane Memory.

I would love to get any tips for improving the README on Github, for instance, and how to make it easier to participate in the project, especially during Hacktoberfest, but also in general. I think the README is way too long and probably also doesn't get to the point.

So, any help regarding how to make it easier to participate in the project would be great.

Furthermore, I'd love to get the project (kind of back) into academia for a bachelor/master/Ph.D. thesis. Still, my former professor just left the university (I finished with an MSc degree about eight years ago). I'm sure there's plenty of opportunities for working on different aspects (new (trie-based? Index structures, working on heuristics for the query compiler, and finally working on cost-based optimizations, distributing the database horizontally, encryption with having versioning in mind...).

Kind regards
Johannes

[1] https://github.com/sirixdb/sirix and https://sirix.io