DEV Community: krishna404

Challenges with Offline First Framework

krishna404 — Fri, 04 Jun 2021 03:46:20 +0000

Almost all the apps don't work without internet. What are the challenges of making a work with patchy & intermittent internet.

Remember the song 'Cloud Number 9' by Bryan Adams? Well here's the video either ways...

In today's world we practically live in the cloud. Always connected to internet, we are already cyborgs & our devices are our connection to the singularity. We will have to make do with this low bandwidth solution unless a Neuralink solution which directly wires the computers to our brains is developed. Well just think how frustrated we get when our apps don't work for lack of connection. Now I am making an app which would work even in lack of internet. Welcome to world of magic! 🪄

So I am building a project using databases called CouchDB & PouchDB. I have talked them first here where PouchDB was giving me trouble, it turned out be me not knowing my javascript good enough. 🙈

So as I keep going deeper into the project, I am coming up against other issues that wouldcome upp with a Offline-First Implementation. It's a revelation of sorts how it is so difficult to get to the root of the problems by just thinking about them. It's the building that takes you to the trenches & you get face-to-face with what has been hiding in plain sight.

So here are the factors that you need to consider when going for a Offline First implementation.

Now lets talk about the simple architecture when using PouchDB & CouchDB

Stale Data:

In a Online First architecture, the app just shows internet not available & doesn't allow any functionality on the app. The App basically becomes useless. That is not the case in an offline-first app.

In the absence of any internet connection, PouchDB would just return the latest data available with it from the last successful sync. Many times this could pose a significant issue. We often see this problem occuring during shopping online, when the old prices are shown again because of caching issues.

Conflict of Data:

Now taking the above idea further. Think if the data is shared among many users who work in the same company or team. Now if one of the users who has a bad internet connection works on the stale data & updates it. It leads to Conflict in data, which needs to be resolved seperately.

See this image below where i have tried to visualise the concept.

Updating the Users Auth data:

Now this is a very developer specific problem. You always want to route the user-authentication only by the database on server. Now if the user wants to update their password or any of other user auth details, they have to disabled to when the user is offline.

Data Loss when user goes offline:

In an offline first, the user could still input data when offline. Now the above infrastructure allows us to do that. The App just Reads & Writes to the local PouchDB database instance which is always available to itself. It is responsibility of PouchDB to sync this data with the CouchDB database sitting in the server. In the absence of internet, PouchDB patiently keeps trying at intervals to establish connection & sync the data.

Due to multiple reasons this data can get lost from the user side just because the data didn't have the opportunity to connect & sync with the server.

Handling Privacy:

When ever the user Logins, the PouchDB instance pulls a copy of the database to the local machine. Now having this data always on the user machine could pose risks related to security. Other Apps may try to snoop & read into this data. This calls for encryption of data, the most safe is to do it right from the server side. Thankfully CouchDB allows for this.

Handling Logouts:

Now the best practice is to delete all the resident data whenever the user logouts. But is that always a good choice? And what happens if the user has been able to logout for multiple reason & session-cookies have expired? Does the data stay persistently on the users system?

This could lead to many instances of the database clogging up precious space on the users system. After all the space available to PWAs is rationed as per allotted quota.

Background Data Change:

If 2 or more people are working on the same data how do we keep each up-to-date with other. Now just thinking about the way Google Docs & other such collaboration tools would do that kind makes me go dizzy but I can understand they do it in some kind of a shared-state framework.

How does somebody like me who is working with forms implement this?

Now I have figured a neat way this can be done even with forms but I want to talk about it once I have coded this after all there is a lot of minute details you figure only when you are building. I want that to be a complete guide for doing this.

Conclusion:

Now the above is not the exhaustive list of challenges which need to addressed in a Offline-First framework but if you have covered for the above, I would say you have covered more than 90% of the issues.

Now offline first apps are great when there isn't a lot of team collaboration. Or if the teams have siloed work structure. Now lets say a company wants an app for their sales people who are always travelling & often have patchy internet. This would be a great framework, as the chances that 2 sales-person would have different regions, customers, etc... assigned. So a chance of any conflict happening on this end is almost negligible.

But Offline-First framework is not advisable for something like a Forex Trading Platform where the data can't be stale. In the absence of internet not allowing the user to use the App is a feature & not a bug in this case.

So if you are choosing Offline First Framework for your app think about the why deeply as this increases the complexity of your app by a few degrees at least.

Keep Smiling :)
Krishna.

What makes a Impressive Web Developer Portfolio

krishna404 — Mon, 31 May 2021 15:27:53 +0000

Developers either freelancers or job seekers are hired based on their portfolio. But what makes for a good portfolio?

So you are a developer & looking for a new gig or a job & everyone asks you for the portfolio? So what makes a good portfolio?

First of all there is no one solution fits all scenario. The development field is vast & depending on your work area your portfolio needs will vary. This blog is directed towards web-developers but I am sure you can use the concepts discussed here for your own purpose.

The first question that comes on the topic is "What kind of projects are valid?".

The objective of creating these projects is to showcase the diversity of our talents. So our projects need to be from different domains. If we talk about basic CRUD apps, the app should allow all the 4 basic operations - Create, Read, Update & Delete. Throwing in a bit of data visualisation would really make it standout.

Kind of projects:

Social Media - like Facebook, Twitter
e-Commerce - like Amazon, DTC brands
Media Player - Netflix, Youtube
Messaging, Group & Individual Chats - like WhatsApp, Telegram
Productivity - like Asana, Trello
Game, Extra points for Multiplayer games - anything simple like a Maze, Snake
You can get more project ideas from Product Hunt , r/appideas , Dribble , Behance , etc...

FAQs

How many projects?

Ideally it should be 5 projects but the bare minimum is having 3 projects.

Are clones ok?

Yes making clones is ok but try to add / remove a feature, brand name. You can add a disclaimer in the footer that this is just for educational purposes.

Is it necessary to deploy & have a live version?

Yes! It is mandatory to have a live version deployed. You can use anything basic like heroku & netlify to deploy your apps. Extra points for using real world options like AWS, Azure, etc...

Is basic websites ok?

No! Basic websites are not ok as a project. Most companies are not looking for website builders.

Is Backend heavy projects ok?

There are more openings in Front-End rather than Backend today but if you are targeting more for a Full-Stack or Backend heavy role, go for it. Just use the best practices for API & also document your API. The bar for Backend is higher.

Mandatory features:

At least 2 apps should have a Backend & should be connected to it.
At least 2 apps should have a Database & some kind of Database storage.
At least 2 apps should have a Login Auth functionality with user specific features, for eg. Bookmark.
Atleast 2 apps should have some kind of Routing feature used.
All four operations of CRUD.
You may use API mocking providers like miragejs for other projects without back-end.
It should be mobile friendly & preferrably responsive. If you cant make it responsive, limit the view-port to mobile width, so that it looks good even on desktop.

Good to have

Have a few contributions to some Open Source projects & have them merged.
Some Data Analysis & Visualisation.
Contribution in GSoC, GSoD& Hackathons.
A fast-paced video walkthrough of the App with your commentary, highlighting any unique / challenging features. The reviewer doesn't have the time to go through your app in detail, make it easy for him & bonus is you get to highlight unique features of the app.
A Google Document talking about the system architecture of the app. If possible add reasons for choosing the said architecture.
Tests incorporated in the project. The most basic of them are also OK. The idea is to let the reviewer know that you understand how it works.
Optimised performance & lighthouse report.

Good Practices

Add link for the deployed App, Repo, Video & Google Doc in your Resume.
The github code should incorporate the Psuedocode. It makes the job of reviewer really easy.
Your commits in github should have meaningful comments. Avoid comments like "A minor update" rather say "Added the close button".

Good Examples

Check out this portfolio of Rahul Jain. What a pleasure. Though this level is not needed. Something basic is also more than good enough.

I will add more as I come across them

References

This post is mostly inspired from the topics discussed here. Do watch!
Here is another post which goes into really a lot of detail on the topic.

Hit me up with any questions you would have!

Keep Smiling :)
Krishna.

PouchDB.find is not a function

krishna404 — Sun, 30 May 2021 10:21:00 +0000

Debugging the Mango query Find for PouchDB Database for finding new documents.

I am using CouchDB & PouchDB for one of my projects.

Now CouchDB is an amazing database which is

 - specifically designed for offline first applications.
 - Has great replication support. So is insanely scalable.
 - Gives user management out of the box.
 - Individual database for each user.
 - Manage roles & give access based to databases based on roles.
 - Allow custom query-design documents, which pre-indexes the database & which leads to fast read/write times.

Due to above reasons for basic CRUD type apps, you don't even need a backend(with insane security), that is great for both development time & cost optimisation.

There are many more benefits to CouchDB, these are just at top of my mind. There are a few trade-offs too, so be mindful of those & see if this suites your needs. One of the major drawbacks is the community support lacking. Most of the articles are out-of-date & most of the links are stale, this is often frustrating when you need an answer. It took me about 4 months to find the slack channel where the community lives. I believe it will be smooth sail from here. 🤞

While CouchDB sits on the server, PouchDB is the database that sits on the client side. The app is supposed to talk to PouchDB & PouchDB then syncs with CouchDB behind the covers, so the app is not dependent on the server to be available. This neat structure is what allows CouchDB to be an awesome Offline First database. Do note, using PouchDB is not mandatory though.

Note: If you plan to use this or any other similar offline first mechanisms for your app understand about the trade-offs of offline first structure (whether, couchDB or not). Some risks that you will have to plan for

  - Possibility of data loss when offline.
  - Data conflicts between the offline & online version.

There are more, do your research before so that you are not surprised later & see if you can incorporate a contingency plans for any issues. CouchDB has a really good conflict management structure though.

Now one of the most basics of using any database is querying. Other than the get method, CouchDB & PouchDB use Mango queries. Which is very similar to MongoDB queries(the wordplay!!!).

The most common query is PouchDB.find which is actually written like

var res = await this.pouchDB.find(
                              {selector: 
                                    selector
                               }
                 )

Now when I incorporated this in my solution, it kept throwing the error PouchDB.find is not a function. Here is more detail on this...

Issue

Cant using the mango query find. Getting error

PouchDB.find is not a function

Info

Environment: Browser - Reactjs
Platform: Chrome
Adapter: IndexedDB
Server: CouchDB

Reproduce

Now as per the docs here we need to use plugin pouchdb-find to be able to use Mango queries.

But as mentioned here that pouchdb-find is now merged with pouchdb, so should not be needed at all.

So I tried using pouchdb-find plugin with both pouchdb & pouchdb-browser. It just didnt work.

Options tried:

Option1

import PouchDB from 'pouchdb';
import pouchAuth from 'pouchdb-authentication';
PouchDB.plugin(pouchAuth);
PouchDB.plugin(require('pouchdb-find').default);

Option2

import PouchDB from 'pouchdb';
import pouchAuth from 'pouchdb-authentication';
import pouchFind from 'pouchdb-find';
PouchDB.plugin(pouchAuth);
PouchDB.plugin(pouchFind);

Option3

import PouchDB from 'pouchdb-browser';
import pouchAuth from 'pouchdb-authentication';
import pouchFind from 'pouchdb-find';
PouchDB.plugin(pouchAuth);
PouchDB.plugin(pouchFind);

Option4

import PouchDB from 'pouchdb';
import pouchAuth from 'pouchdb-authentication';
PouchDB.plugin(pouchAuth);

Notes:

pouchdb-authentication is need to provide authentication if you are using CouchDBs inbuilt user managment.

Well the glitch wasn't in the plugin at all. It was about how the database was being called. So here is what I was doing

import PouchDB from 'pouchdb-browser';
import pouchAuth from 'pouchdb-authentication';
import pouchFind from 'pouchdb-find';
PouchDB.plugin(pouchAuth);
PouchDB.plugin(pouchFind);

...

const getDB = (dbName) => {

        const localDB = new PouchDB(dbName, {auto_compaction: true});

        return localDB;
}

const findDoc = async (input)=> {

         let db = await getDB(dbName).find({
                         selector: {
                            type: { $eq: props[2]}
                        },
                        limit: 1
                      })
}

What worked for me was

... same as above
const findDoc = async (input)=> {

         let db = await getDB(dbName)
              db = await db.find({
                         selector: {
                            type: { $eq: props[2]}
                         },
                         limit: 1
                      })
}

Thankfully the community helped me figure this out. You can read about detail on the issue here at github .

I don't know what is wrong in the first option. If you know, please do let me know.

Caio till the next time.
Krishna.
Follow me at https://krishna404.com for Tech Blogs & More.