Hi,
I'm new to Dev, thanks for checking out my first post.
I'm a data scientist doing data products for actual humans. I want my users to get to wow fast, and I want to develop fast.
You know what gets in my way ? Servers. Writing a server, integrating with a server, authenticating a user on a server and persisting data on a server all stand between me and my goal.
Recently I wanted to let my user upload a csv with a text, give them full text search over that, make some annotations on the text and have the result saved for later.
Back in the day I would have set up a python server, a postgres database, connected them and then tried to get my webapp to talk to the python.
That's annoying because it's a lot of setup for me, and some setup and waiting for my users. It's 2020 no-one has any patience. Let's see how we can get the time to wow down to something a millennial or genexer like me can stomach:
So here's what I did.
- Moved the heavy computation (building the index and search) in a WebWorker .
- Wrapped the worker in a class that gives the main thread a promise when it searches.
- Used react-query to delegate away the request/response state management instead of having a mountain of Redux code.
- Used the Dexie library to store the users data in Indexedb so thay can come back and pick up with they left off.
- And I used ndx which is a really nice in JS full text search implementation.
The end result looks like this, granted the design is rough but a satisfying POC of an inbrowser search engine
This is all going open source soon and I have a lot to share about each of these steps, but I'm not sure if this is interesting and exciting or kind of meh. Let me know what you think and I'll be glad to share more!
Tal
Top comments (4)
Hey Perry nice article and solution. Indexdb is good but its storage capacity changes with browser. Also, there is probability that client clears the indexdb data while playing with browser settings.
Anyway, in my few articles , I advocated that nothing is faster than indexdb storage.
Hi Manish,
Thanks for the comment and I'll definitely check out your articles.
I think you hit the nail on the head. Handling data limits and evictions elegantly is a big challenge here both for engineering and UX.
Haha... But there is a way out. Syncing indexdb with database on server. There are few libraries in GitHub (MIT) to do this. Anyway, will wait to give try to your open-source project
Nice article but not very in detailed. I'm curious about how you use ndx on top of Dexie, could you share more about this?