It's hard to believe that as of 2016, the best method for offline storage in a web app was localStorage, a simple string-only key value store with a 5mb data limit. These kinds of stores are typically meant to store user preferences and basic user information. So if you wanted to build an application that offered end-to-end encryption AND search capability, you couldn't. For this reason, we have had to make compromises in our privacy, at a cost that is everyday becoming more expensive.
But while we were selling our privacy cheaply and letting companies run amuck with our data, a brave few engineers have been building the future of the web, and we have yet to take full notice.
Search and encryption are one another's arch enemies. Simply put, you can't offer convenient search on top of encrypted data. And since most of the heavy lifting and data processing in web apps today are performed server side, servers require read access to all data. But if the server could read your data, then practically any engineer of the company could read your data, and if an engineer could read your data, then so could you know who.
Companies have been clutching on to this excuse with great leisure, and we have capitulated to their seemingly well reasoned response. However, in this great again year of 2017 which we find ourselves in today, that answer no longer holds up.
Enter WebCrypto and IndexedDB.
WebCrypto and IndexedDB are the one-two punch to a new era of encrypted, offline, and privacy centric web applications.
Before WebCrypto, practical applications wanting to perform crypto and encryption methods in the browser relied on libraries like CrpytoJS, a painfully slow library without access to proper system resources and lacking, amongst other things, a cryptographically secure random number generator. WebCrypto on the other hand is, to the most accurate description I could think of, a beast. Whereas the performance of a key derivation function like PBKDF2 in CryptoJS clocked in at around 5,000 hashes in 6 seconds, my simple WebCrypto tests had no problem running 1.5 million hashes in less than a second. This is the crypto library that the world of web apps have been so desperately awaiting.
On the other end is IndexedDB: a full, structured, and queriable database with a data capacity limit of 50% of the user's disk space, available right in the browser. As of 2016, all major browsers support it, including mobile browsers. This means applications like those that center around private user data can now easily encrypt data locally, send encrypted data to the server, and store decrypted data locally in the privacy of the user's own database.
The excuse not to take advantage of these long awaited set of privacy-fulfilling tools that users are desperately craving is no longer technical, but philosophical and economical. Some companies cherish user data far too much to ever forego not being able to read it themselves. You know which companies I'm referring to. They will continue ignoring user demand for privacy at their own peril.
Web developers: the tools for disruption are now at your feet.
We've implemented both WebCrypto and IndexedDB into our own web-based notes app, Standard Notes, for end-to-end encryption and full offline search with surprisingly great ease and little resistance.
More info on WebCrypto and IndexedDB. The docs for IndexedDB state that it "is powerful, but may seem too complicated for simple cases", but this absolutely wasn't the case for us. The API is delightfully high level.
Top comments (6)
The problem with indexedDB is that the api just broken by design. Before I started creating RxDB (github.com/pubkey/rxdb) I persist to solve my needs with indexeddb, but it was just a hurt to the project. Nothing works as expected and when it works on chrome, it off course does not on firefox.
Basically this happens when you sit 10 non-programmers into a room and let them build a consensus on the specs for a complex database.
In my opinion the browsers should provide the simpliest and fastest key-value-store possible and the devs can build next to everything on top of it. No mather if you need a graph-db, mysql-syntax or noSQL, you could just use the right javascript-lib to have it.
This would also make it easy to use the same code for the browser, the desktop, nodejs or mobile apps, since every environement just needs to have the simple key-value-store api. (compareable to leveldb)
I've had very little difficulty building a simple key-value store on top of IndexedDB that works on Chrome, Firefox, Safari, and Edge. Source here: github.com/standardnotes/web/blob/...
IndexedDB had some notable implementation bugs in the past, but they're mostly solved by now. If you don't feel comfortable using it, you can always rely on libraries like PouchDB or Dexie.
"Whereas the performance of a key derivation function like PBKDF2 in CryptoJS clocked in at around 5,000 hashes in 6 seconds, my simple WebCrypto tests had no problem running 1.5 million hashes in less than a second. This is the crypto library that the world of web apps have been so desperately awaiting."
Wait, aren't fast hashing algorithms problematic? Does that not make it easier to brute force the encryption? Is this algorithm asymmetric? How many hashes can you check in one second?
Wow, I was not aware of this. These APIs could fundamentally change the way we architect the flow of data within a web application.
We use Azure local storage for storing any of our logging needs.