SirixDB - An evolutionary, accumulate only database system (currently storing XML and JSON data) 🦁
Feel free to contribute on Github to the Core and the Svelte based Frontend 💚
Hi all,
there's plenty of possibilities to work on a versioned database system, which preserves every revision of your data in a space-efficient manner through structural sharing and a novel sliding snapshot algorithm. 🐬
SirixDB only stores changes between revisions, but any revision is indexed and can be reconstructed in O(n), whereas n is the number of nodes in the revision.
If you're eager to participate in #Hacktoberfest, there are plenty of opportunities. For instance, working on a frontend, the backend, a non-blocking REST API built with Vert.x, Kotlin and Keykloak, or a query optimizer (for instance, to rewrite the AST based on cost-based rules). Or you might want to look into how to best horizontally scale SirixDB, for example, via a distributed log (Apache Kafka?!) 😃
Furthermore, the sirix.io website needs some love.
🐣 Frontend with Svelte, TypeScript and D3.js
Moshe Uminer currently, besides working on and maintaining a Python and a TypeScript based client for SirixDB, is working on a frontend built with Svelte.
On the one hand, the purpose of the front end is to interact with the database system through queries. On the other hand, to visualize the differences between revisions.
In general, I'd love to provide a similar space-filling approach (modified Sunburst-View) as the one I once developed for my master's thesis:
🚀 Backend written in Java and a non-blocking REST-API built with Vert.x and Kotlin
If you're more into backend stuff, you can either work directly on issues regarding SirixDB or work on Kotlin based stuff with Vert.x.
sirixdb
/
sirix
SirixDB is a temporal, evolutionary database system, which uses an accumulate only approach. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach called sliding snapshot.
For instance, Web Socket support in the non-blocking HTTP-Server:
Websockets to stream resource changes into the client / browser
#98
I'd put this into the rest-api module and build it with Vert.x and Kotlin.
Try to create a native image with the GraalVM (however, Java 14 or 15 might not be supported, though :()
Create native images with the GraalVM
#77
For a fast startup time in containers... peek performance however might not be as good.
A Kotlin-based DSL for SirixDB:
Kotlin based wrapper APIs
#63
Therefore we also could create a nice DSL.
But in essence a basic version for the XdmNode-Transaction layer for instance would involve infix functions
trx moveTo 3 and alike, removing builders, just use default parameters...
However, I'd love to discuss this with Kotlin experts :-)
Have a look into all the issues :-)
Besides, a Kotlin-Client and a Rust-Client is needed in the long run.
Query optimizations
Or if you would like to work on query optimizations, you can look into how to rewrite the AST produced by Brackit:
JSON with a database storage backend
#5
Hey Sebastian, sorry to bother you all the time, but maybe you can also give a good tip how you'd implement a binding for a native JSON storage backend. I'm currently finishing the work on a temporal JSON storage layer besides XML for my open source project (http://sirix.io) and would like to also use Brackit. Maybe I could also implement the interfaces you already provide (should be basically the same as for my XDM node transactions) and then implement some functions for opening a JSON resource...
Basically my JSON node layer looks like the one in Marklogic: https://docs.marklogic.com/guide/app-dev/json
We have plenty of optimization potential here, as we currently don't take any inherent storage optimizations into account. 👍
This will increase the performance of specific queries considerably.
A query looks like this:
let $doc := jn:open('database','resource', xs:dateTime('2019-04-13T16:24:27Z'))
let $statuses := $doc=>statuses
let $foundStatus := for $status in $statuses
let $dateTimeCreated := xs:dateTime($status=>created_at)
where $dateTimeCreated > xs:dateTime("2018-02-01T00:00:00")
and not(exists(jn:previous($status)))
order by $dateTimeCreated
return $status
return {"revision": sdb:revision($foundStatus), $foundStatus{text}}
Have a look at Sebastian's thesis Separating Key Concernsin Query Processing
He worked under supervision of Theo Härder, who invented ACID transactions 👌
👾 Think about how to scale SirixDB horizontally
I've thought about it and would love to distribute the transaction log with Apache BookKeeper or Apache Pulsar (which uses Apache BookKeeper). It might have some advantages in comparison to Apache Kafka.
❤️ Happy coding
So, there's plenty of stuff to work on. It's a super diverse field. I deliberately want to invite beginners to work on things as I'm a frontend beginner, for instance, and I have not much knowledge about query optimizers. However, I'm more than eager to learn. 👋
I hope the project is useful for some of you, and I even got a lot of contributions since the last Hacktoberfest after years of working alone; that's simply awesome. Thank you so much 😎
If you like the project, you might share it on twitter and spread the word!? 🙈
Contribute on GitHub SirixDB and GitHub SirixDB Web Frontend 💚
Have a nice day and a great #Hacktoberfest
Johannes
Top comments (0)