I am currently looking for a better way to save and retrieve and save data for my favorite side project. I did some research and I thought it might be nice to share what I found out so far.
Requirements
The app is meant to run as pwa as well as electron desktop app and uses rxjs to a wide degree. At the moment there is no backend and if there ever will be one it will probably be optional. As the main focus is the electron app I fortunately don't have to care too much about browser support, though all the libs seem to have support for all modern browsers.
These are the requirements I came up with:
Essential
- Super speed: it should feel as fast as a native desktop app for about 2000 different entities at minimum and up to 10mb of data stored and quick to load combined data and subsets of data
- Relatively easy to implement without too much overhead
- Future proof: should be under active development and supported by a big community
- No UI Blocking should occur even when working with larger datasets
- Framework agnostic
Bonus
- Working typescript support (at least no errors)
- Readable indexddb model => the data should be browseable via chrome dev tools
- Ideally should provide the ability to get a dataset as an observable or it should be easy to implement such thing
- Replication to a backend server: though not yet implemented, it might be nice to have the option
- Bonus bonus: simple file storage replication to google drive, owncloud or dropbox
The competitors
As a starting point I used this list. Please keep in mind that I am by no means an expert for any of the libraries used here, so chances are that I missed something important. If so please let me know in the comments.
These are my picks:
WatermelonDB
Description: Reactive & asynchronous database for powerful React and React Native apps
advantages
- Sync capabilities build in
- Quite fast thanks to LokiJS
- Includes a query language similar to mongodb
- Schema validation
- Database interactions run inside a worker so no risk of ui blocking
- Write sqllite to disk (which might be nice to use with electron)
drawbacks
- Typescript Errors
- Strongly focused on react
- Some issue with fast consecutive writes (but that might have been me using it wrong)
- IndexedDB is just stored as one big string
RxDB
Description: A realtime Database for JavaScript Applications
advantages
- Sync (thanks to PouchDB probably the most advanced)
- Nice Syntax
- Wide framework support and examples
- Schema validation via JSON Schema
drawbacks
- Very slow for larger datasets
- Render blocking occurs when interacting with larger datasets (this might be my fault again though)
- Data is stored separately but is not easy browsable because of all the PouchDB stuff
Dexie
Description: A Minimalistic Wrapper for IndexedDB
advantages
- Schema Validation
- Relatively fast
- Clean Database usage for IndexedDB
- Framework agnostic
drawbacks
- Not clear to me yet if syncing is easy to implement
- No real observable interface (yes there is dexie observable but, but that is just for watching change events rather then getting updated data)
LokiJS
Description: javascript embeddable / in-memory database
advantages
- Fast
- Includes a query language similar to mongodb
- Framework agnostic
- Less overhead than RxDB and WatermelonDB
drawbacks
- No sync capabilities built in
- ? No observable data interface ?
remotestorage
Description: An open protocol for per-user storage on the Web
advantages
- Schema validation
- Own your data: Sync to google drive / dropbox and more
drawbacks
- Out of date dependencies
- No active development in the last couple of months
- Smaller community than the rest
Native IndexedDB
advantages
- no overhead
- relatively fast
- framework agnostic
- typescript should work
drawbacks
- completely custom sync required
- No observable interface
- probably lots of code to write
- not all browsers support all the features
Speed Test
This might be outdated but apart from my own non systematic tests I used this tool to compare the performance.
Premature Conclusion
So far I've only experimented with WatermelonDB, RxDB and a little bit with native IndexedDB. Here are the branches for WatermelonDB and RxDB in case you want to check for yourself:
https://github.com/johannesjo/super-productivity/tree/feat/watermelonDB
https://github.com/johannesjo/super-productivity/tree/feat/rxdb
WatermelonDB put me off because of its strong focus on React and it not working well with typescript atm. It shares one of the "problems" with LokiJS: The data in stored in IndexedDB is just a big string. This might not be a real problem, but it feels wrong to go about persistence like that. I didn't come much further than playing around a little with LokiJS, because I felt that it might be basically WatermelonDB with less issues, but also less of the stuff I need and I was still hoping for something better.
RxDB looked pretty promising and I assume that it really shines when you are using a server. But the bad performance was just not an option for an app that aims at desktop app like performance.
Using native IndexedDB should work fine for the most part. The main disadvantage is that I would have to write a lot of code and I don't want to re-invent the wheel (just quite yet).
Next step is Dexie! The community might not be as big as the ones of it's competitors and the lack of a prebuild performant way to use collections as observables is a little bit disappointing. I am also unsure about the sync stuff, as I didn't find too many good examples and it seemed to as if it was more of a gimmick than something people are using in production, but I hope I am wrong! So far it has been the only library offering a clean IndexedDB to browse while at the same time offering some structure making sure I am not the only going about saving data this way.
I'll update the article once I know more!
Did I get something wrong? Probably! This why I'm very curious to hear your thoughts and experiences on the matter. Please share them in the comments!
Top comments (17)
Hey Johannes,
Radek here - main author and maintainer of WatermelonDB. It's really interesting to hear your thoughts on it because most of your complaints are EXACTLY what I've been working on fixing over the last couple of weeks!
Not anymore! I've completely rewritten LokiJS's IDB adapter to store things in little chunks: github.com/techfort/LokiJS/pull/808 and are working on further performance improvements: github.com/techfort/LokiJS/pull/811 — it's actually not one record per IDB object, because IDB is slow at this. It's more profitable (for mass insert and reading) to store objects in chunks than one-per-record or one-huge-chunk
That's true, there has been some performance regressions, and it was never 100% optimized. This is changing. I've improved fast consecutive writes on web by 6x, and on iOS by a ridiculous 23x. Further 5x on top of the 23x is coming to iOS, and a huge perf upgrade to Android as well.
That's true at the moment, but that's something I hope the open-source community will improve.
WatermelonDB itself is actually 100% React-agnostic at this moment. The part that connects 🍉 to React (withObservables) is by design a separate component. It's what we use, so it's what we documented and published, but in reality, the only thing that's missing to use WatermelonDB in any framework is a small bit of code to connect RxJS Observables to your components. Easy!
That one I have no updates on - TypeScript types are entirely community-led, and I encourage you to contribute to that effort!
Hey Radek,
thanks for your reply! That's really great news! I really like what WatermelonDB hast o offer. I'll definitely give it another go.
I got the same feeling during my experiments. It's a shame that this seems to be the case. Would be cool if IndexedDB would behave more like a real database rather than a fancy version of localstorage.
The changes don't seem to be available in the latest npm version yet. How would you recommend to go about testing them (which of the new pre-releases should I use)?
Agreed. It's a shame that WebSQL got ditched -- it was very powerful, and fast, too.
Try v0.15.0-8 -- and if that doesn't work, v0.15.0-7 — you should be able to insert tens of thousands of records per second with the latest changes -- but even more optimizations are coming.
Thank you! I'll give it another go!
I Really hope that it can support encryption on both mobile platforms and desktop. There's a pull request but it's been inactive for so long.
dev.to/michielnuyts/going-offline-...
I've been using Dexie for the past week. Let me know if I can help with something.
Thank you very much! I setup the very basic stuff and it seems to do the job. Using it in a reactive way (auto stream data updates via an Observable) unfortunately seems to be slow for larger datasets (starting at 100 entries):
github.com/johannesjo/super-produc...
So if I were to use Dexie I probably still would have to go with an in memory store for speed (I probably would keep using the redux stuff already implemented). Or do you maybe know if there is a smarter way do this?
Atm. I am checking out gun which offers a lot of possibilities.
Yes I'm using Overmind for managing my state. I'll have a look at the repo this evening.
That would be really nice! Let me know, if you have any questions! The code is not too pretty atm. just quickly hacked in and the redux stuff isn't probably set up in the most helpful manner as well... The basic idea is to move from the current model structure of loading everything related to a project into the store, to having the possibility to access all the tasks all the time to allow for mixed lists.
Keep in mind that I haven't yet decided which road to go, but nevertheless it would be super cool to be able to discuss these things with someone! I've to admit that even though it might not be the best fit I'm currently intrigued to learn more about gunjs and graph databases. Totally new area to me :)
Are you on twitter or something to chat about it? I'll show you how I'm using Dexie and my in-memory store.
I am: @hugoderhungrige
Though there is also a chat function here. Maybe we can use that?
Sure we just need to follow each other in order to be able to chat :)
how does dexiedb compare to lokijs? dexie has uselivequery dexie.org/docs/dexie-react-hooks/u...
and loki has techfort.github.io/LokiJS/DynamicV... dynamicView. So both have reactivity. Dexie is more closely tied to indexeddb while loki seems to be more tied to nodejs. So maybe dexie is better in the browser while loki is better in the backend. Or maybe loki is better in general, but I dont know because I didnt use it yet. Any thoughts?
Posted twice?
Not on purpose. Just wanted to correct some spelling mistakes. There seems to be a bug with dev.to:
github.com/thepracticaldev/dev.to/...
I archived this post, but it's still showing up. Don't know what else I could do... I could try to unpublish but I'm afraid that this just will create another copy.
Here is the original: dev.to/johannesjo/choosing-the-rig...
Hi! Might be late in the game but the with latest Dexie release 3.2 there's reactivity built in with examples for different frameworks.