Demangeon Julien

Posted on Sep 25, 2019 • Originally published at marmelab.com

CouchDB, The Open-Source Cloud Firestore Alternative?

#javascript #react #offline #realtime

Note: This post was originally posted on marmelab.com.

During one of our last customer projects, we used Firebase, a backend-as-a-service by Google, as our backend. Althought satisfied by this "all-included" suite as a whole, we remained disappointed by its proprietary aspect.

That's why I took the initiative to look for an open-source alternative to Firebase that could cover all our needs without depending on a third party service.

The first step in this quest is to find a substitute to the Cloud Firestore realtime NoSQL database for the web.

What Do We Need?

Using Firestore rather than a more classic database is not trivial. It often results from the need to quickly develop an application with the following features:

Offline First, client writes to local database that is synchronized with remote one
Realtime, remote changes must be in sync with our local database

Some solutions exist to cover this need, most of them are based on NoSQL databases such as MongoDB, Cassandra, RethinkDB, Gun or others MongoDB based solutions like Minimongo, turtleDB or tortoiseDB.

In our case, we're going to give CouchDB (and PouchDB for the frontend) a try, because it's the more robust and best known solution from our point of view.

CouchDB & PouchDB

CouchDB is an open-source / cross-platform document oriented database software. It is developed on the basis of the concurrency-oriented Erlang language, allowing it to benefit from an high scalability. It uses JSON to store its data, and an HTTP API to expose it.

CouchDB was born in 2005. Since 2008, CouchDB became an Apache Software Foundation project, which allows it to benefit from a lot of support and a large communauty.

Here are the main features of CouchDB:

Multi-Version Concurrency Control (which lets you build offline-first solutions easily)
Distributed Architecture with Replication
Document Storage
HTTP / REST API

Since CouchDB runs on the server, many client library allows to communicate with it thanks to the HTTP interface it offers.

The most known CouchDB client library for the web is called PouchDB. PouchDB is an open-source Javascript database that is designed to run within the browser. This way, it allows to store data locally offline, and sync it with the remote CouchDB server when the user comes back online.

CouchDB & PouchDB in Practice

Enough introduction, let's get practical! In this section, I'll describe the development of a ReactJS application using CouchDB and PouchDB as database system, step by step. Meanwhile, I'll try, as much as I can, to compare the CouchDB implementation to the Firestore one.

Also, I'll present you some of my latest loves in terms on Javascript librairies: Final-Form, ElasticUI and Indicative.

In this project, I'm going to create a beer registry, which allows users to keep track of their beer stocks.

Project Setup

In order to keep this tutorial as simple as possible, I'll create a ReactJS application using create-react-app.

create-react-app reactive-beers && cd reactive-beers

npm install -S pouchdb

The application skeleton looks like the following:

julien@julien-P553UA:~/Projets/marmelab/reactive-beers$ tree -L 1
.
├── node_modules
├── package.json
├── package-lock.json
├── public
├── README.md
└── src

Then, since I don't want to install CouchDB directly on my machine, I'll use Docker. So, the first step is to configure a docker-compose.yml file and the associated Makefile to improve developer experience.

// ./docker-compose.yml

version: "2.1"

services:
  couchdb:
    image: couchdb:2.3.0
    ports:
      - "5984:5984"

  node:
    image: node:10
    command: npm start
    working_dir: "/app"
    volumes:
      - ".:/app"
    ports:
      - "4242:3000"
    depends_on:
      - couchdb

# ./Makefile

USER_ID = $(shell id -u)
GROUP_ID = $(shell id -g)

export UID = $(USER_ID)
export GID = $(GROUP_ID)

DOCKER_COMPOSE_DEV = docker-compose -p reactive-beers

help: ## Display available commands
    @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

install: ## Install docker stack
    $(DOCKER_COMPOSE_DEV) run --rm node bash -ci 'npm install'

start: ## Start all the stack
    $(DOCKER_COMPOSE_DEV) up -d

stop: ## Stop all the containers
    $(DOCKER_COMPOSE_DEV) down

log: ## Show logs
    $(DOCKER_COMPOSE_DEV) logs -f node

So, we're now ready to start our complete stack using make install start.

julien@julien-P553UA:~/Projets/marmelab/reactive-beers$ docker ps
CONTAINER ID        IMAGE            COMMAND                  CREATED       STATUS       PORTS                                        NAMES
6884f92c5341        node:10          "npm start"              3 hours ago   Up 3 hours   0.0.0.0:4242->3000/tcp                       reactive-beers_node_1
21897f166ce4        couchdb:2.3.0    "tini -- /docker-ent…"   3 hours ago   Up 3 hours   4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp   reactive-beers_couchdb_1

Everything is launched. You may have noticed that the 5984 port is exposed in our docker-compose.yml file, it's the CouchDB api. Then, if you open localhost:5984 in the browser, you'll see something similar to the following.

{
    "couchdb": "Welcome",
    "version": "2.3.0",
    "git_sha": "07ea0c7",
    "uuid": "49f4e7520f0e110687dcbc8fbbb5409c",
    "features": ["pluggable-storage-engines", "scheduler"],
    "vendor": {
        "name": "The Apache Software Foundation"
    }
}

Accessing The Document Store

OK, our server is up and running. But, is there an interface to visualise / supervise CouchDB just like Firestore does? The answer is YES! CouchDB already includes an administration interface called Fauxton. We can browse it at http://localhost:5984/_utils/.

Firestore Admin Interface

Fauxton Admin Interface

The Fauxton interface allows to access databases, setup nodes and clusters, configure replication, setup permissions, etc. Although practical, it is still preferable to automate these administration tasks with dedicated scripts.

React Kicks In

Now, we can start to develop our first PouchDB powered interface. Then, here are our main App.js entry point and the Home.js start screen.

// ./src/App.js

import React from 'react';
import { Home } from './screens/Home';

const App = () => <Home />;

export default App;

The App.js file has no interest for the moment. It'll certainly become useful when we need to add more routes and screens in the future.

// ./src/screens/Home.js

import React, { useState, useEffect } from 'react';
import { addBeer, getBeers, onBeersChange } from '../api/beers';

export const Home = () => {
  const [beers, setBeers] = useState([]);

  const refreshBeers = () => getBeers().then(setBeers);

  useEffect(() => {
    // We fetch beers the first time (at mounting)
    refreshBeers();

    // Each change in our beers database will call refreshBeers
    const observer = onBeersChange(refreshBeers);
    return () => {
        // Don't forget to unsubscribe our listener at unmounting
        observer.cancel();
    };
  }, []);

  return (
    <div>
      <button onClick={() => addBeer({ title: 'Beer X' })}>Add a beer</button>
      <ul>
        {/* beer._id is an unique id generated by CouchDB */}
        {beers.map(beer => <li key={beer._id}>{beer.title}</li>)}
      </ul>
    </div>
  );
};

CouchDB Requires More Requests Than Firestore

As you see, in this example we use a combination of a listener (onBeersChange) and a a query (getBeers) to get the initial beers list and refresh it when a change is issued in the database.

This operation is not optimal compared to the one Firestore offers. Indeed, while pouchdb is not able to return both changes and data to us, Firestore is able to do so thanks to a QuerySnapshot system, thereby reducing server trips back and forth. See by yourself with the Firestore example below:

  db.collection("anything")
    .onSnapshot(function(querySnapshot) {
        querySnapshot.forEach(function(doc) {
          // This forEach loop is executed at first execution
          // And executed each time the query result changes
        });
    });

So, if we had used Firestore instead, here's what it would have looked like:

  //...

  const [beers, setBeers] = useState([]);

  useEffect(() => {
    const unsubscribe =  db.collection("beers")
      .onSnapshot(function(querySnapshot) {
          const snapBeers = [];
          querySnapshot.forEach(function(doc) {
              snapBeers.push(doc.data());
          });

          setBeers(snapBeers);
      });

    return () => {
        unsubscribe();
    };
  }, []);

  // ...

It's also possible to use .map on the querySnapshot.docs attribute to retrieve all the documents in a "non-imperative" way. Unfortunately, this functionality is not sufficiently covered by the official documentation.

The Model

Just like in backend development, I like to separate the model logic from the view logic in my frontend apps. So here is the API file for our beers below:

// ./src/api/beers.js

import PouchDB from 'pouchdb';

// We declare a PouchDB instance that is "remote only"
// There's no "offline" capability for the moment, everything is sync
export const beersDatabase = new PouchDB('http://localhost:5984/beers');

// If the beers database does not already exist
// => The database is automatically created when an object is added to it
export const addBeer = beer => beersDatabase.post(beer);

// Here, we list all the documents from our beers database
// A lot of options exists. Eg: we can paginate using "startKey", "endKey" or "limit"
export const getBeers = () =>
  beersDatabase
    .allDocs({
      include_docs: true,
      descending: true,
    })
    .then(doc => doc.rows.map(row => row.doc));

// We listen all the changes that happen since now
// We can also apply a "limit" option to this method
export const onBeersChange = callback => beersDatabase
    .changes({ since: 'now', live: true })
    .on('change', callback);

Here is the result of our first CouchDB application in action. As you can see, everything is in sync between multiple windows.

Offline Sync

Sadly, our actual version only works when the internet access is up and running. In other cases, such as a bloated network or packet loss, beers will never (or slowwwwly...) be added in the beers list because of the "remote only" sync.

The right way to avoid this problem is to keep a local first approach. It means that we must achieve all our database operations on the local database, then synchronize it with the remote one when internet access comes back.

So, the first step is to declare a new PouchDB instance with a database name instead of a remote database url. This way, PouchDB automatically detects that we want to instantiate a local database.

import PouchDB from 'pouchdb';

// Declare local database
const beersDatabase = new PouchDB('beers');

// Declare remote database
const remoteBeersDatabase = new PouchDB(`http://localhost:5984/beers`);

// Keep local and remote databases in sync
PouchDB.sync(beersDatabase, remoteBeersDatabase, {
  live: true, // replicate changes in live
  timeout: false, // disable timeout
  retry: true, // retry sync if fail
});

The PouchDB.sync instruction is the equivalent of a bidirectionnal PouchDB.replicate instruction between local and remote databases.

PouchDB.replicate(beersDatabase, remoteBeersDatabase);
PouchDB.replicate(remoteBeersDatabase, beersDatabase);

By default, PouchDB uses IndexedDB as local database (just like Firestore by the way). So, now that our setup is done, we can take a look at our local database using the Chrome console.

As you can see, we find the complete list of beers we've created. Each one is uniquely identified by a key that is built from _id and _rev CouchDB attributes.

{
  "_id": "0c2738a3-d363-405f-b9bb-0ab6f5ec9655",
  "_rev": "3-b90bd9d62fbe04e36fe262a267efbd42",
  "title": "Beer X"
}

Whereas the _id represents an unique document, the _rev represents the revision identifier of it. In fact, each modification of a document implies a new version of it which then makes it possible to manage conflicts.

Unlike CouchDB, Firestore documents do not have a revision id. So, the only way not to struggle with conflicts using Firestore is to use transactions.

Moreover, since CouchDB records every submitted change, it is possible to return back or resolve conflict in a second time, which is essential in order not to risk losing data.

For more information on conflict management using PouchDB, check the PouchDB Conflict documentation.

Now that we are able to communicate with both local and remote databases, we can focus on the business logic and on the user interface. Moreover, it'll allow us to benefit from optimistic rendering while making our application more flexible in addressing network issues.

Forms & Validation

In this section, we will implement a form to be able to add new beers. To do that, I'm going to use final-form (and react-final-form, an adapter for ReactJS).

npm install -S final-form react-final-form

So, we can create a simple form to handle user input.

// ./src/components/BeerForm.js

import React from 'react';
import { Form, Field } from 'react-final-form';

export const BeerForm = ({ onSubmit }) => (
  <Form
    validate={() => ({})}
    onSubmit={onSubmit}
    render={({
      handleSubmit,
      hasValidationErrors,
      pristine,
      invalid,
      submitErrors,
      submitting,
      form,
    }) => (
        <form onSubmit={handleSubmit}>
         <div>
            <label>Title</label>
            <Field name="title" component="input" />
          </div>
          <div>
            <label>Description</label>
            <Field
              name="description"
              component="textarea"
              rows={2}
              placeholder="Tape your description here..."
            />
          <div/>
          <button type="submit" disabled={pristine || hasValidationErrors || submitting}>
            Submit
          </button>
          {submitErrors && submitErrors.global && (
            <p>{submitErrors.global}</p>
          )}
        </form>
      )
    }
  />
);

Then, we can replace our action button by the form in our home screen.

// ./src/screens/Home.js

import React, { useState, useEffect } from 'react';
import { addBeer, getBeers, onBeersChange } from '../api/beers';

export const Home = () => {
  const [beers, setBeers] = useState([]);

  /* ... */

  return (
    <div>
      <BeerForm onSubmit={beer => queries.addBeer(beer)} />
      <ul>
        {/* beer._id is an unique id generated by CouchDB */}
        {beers.map(beer => <li key={beer._id}>{beer.title}</li>)}
      </ul>
    </div>
  );
};

Data Validation With Indicative

So, we have a form, but there's no data validation for the moment. Users can send anything they want at this time. That's why we're going to set up a data validator using indicative, a library that I've just discovered and that I want to give a try.

npm install -S indicative

The Indicative API is very simple. It consists of a Validator object that uses a set of validation rules and a formatter. Here is a usage example:

import Validator from 'indicative/builds/validator';
import { Vanilla as VanillaFormatter } from 'indicative/builds/formatters';
import { required, email } from 'indicative/builds/validations';

const validator = Validator({ required, email }, VanillaFormatter);

const rules = {
  name: 'required',
  email: 'required|email',
};

const messages = {
  'required': '{{ field }} field is required', // This message works for all required rules
  'email.required': 'You must provide an email!', // This message is specific for required email
  'email.email': 'The email adress is invalid',
};

const values = {
  email: 'bad email',
};

// Validator.validate is async

validator
  .validate(values, rules, messages)
  .then(() => /* everything is ok! */)
  .catch((errors) => {
    /*
      [
        { field: 'name', message: 'name field is required!' },
        { field: 'email', message: 'The email adress is invalid' },
      ]
    */
  });

Here is our custom implementation for BeerForm.js.

// ./src/components/BeerForm.js

import React from 'react';
import { Form, Field } from 'react-final-form';
import { Vanilla } from 'indicative/builds/formatters';
import Validator from 'indicative/builds/validator';
import { required } from 'indicative/builds/validations';

const validator = Validator({ required }, Vanilla);

const rules = {
  title: 'required',
  description: 'required',
};

const messages = {
  'title.required': 'Beer title is required',
  'description.required': 'Beer description is required',
};

const validate = async values =>
  validator
    .validate(values, rules, messages)
    .then(() => ({}))
    .catch(errors => {
      return errors.reduce((acc, error) => {
        acc[error.field] = error.message;
        return acc;
      }, {});
    });

Final Form needs an object as error model, so we format errors in the catch using a reduce. Alternatively, it would have been possible to use a Custom Indicative Formatter.

So, now we have our custom validation function, we can replace our empty validate function.

export const BeerForm = ({ onSubmit }) => (
  <Form
-  validate={() => ({})}
+  validate={validate}

And tadaaa! Our validated form is up and running and we are ready to play with it.

Let's Make It Beautiful!

To summarize, we can display beers, we can add beers, everything works offline and is in sync with a remote server. But right now, it's not very aesthetic, and I wouldn't dare present it to my mother-in-law. So, how about making it a little prettier?

In this section, I'll use the Elastic UI framework (aka eui) that is in use at Elastic, the company that develops ElasticSearch.

I think we all agree that we must remove this despicable list and replace it with a nice grid. Fortunately, Eui allows it easily.

As you can see, we took the opportunity to add editing and deleting beers right from the grid. We'll also put the form in a sliding panel from the right of the page. This way, we can directly add a beer from a "+" button in the navbar, or edit a beer directly from the grid, without changing page.

Handling Image Attachments

I don't know about you, but seeing all these grey beer cans breaks my heart. So it's time to allow image upload in the form.

// ./src/components/BeerForm.js

const handleIllustration = async files => {
  if (files.length === 0) {
    form.change('_image', undefined);
    return;
  }

  const file = files[0];

  form.change('_image', {
    data: file,
    type: file.type,
  });
};

<EuiFormRow label="Beer Illustration">
  <EuiFilePicker onChange={handleIllustration} />
</EuiFormRow>

This custom _image attribute that I just added to the beer object is then handled by our beer api, and considered as a PouchDB attachement.

// ./src/api/queries.js

const saveBeer = async ({ _image, ...beer }) =>
  store
    .collection('beers')
    .post(beer)
    .then(
      ({ id, rev }) =>
        // if an "_image" attribute is present, we put an attachement to the document
        _image &&
        store
          .collection('beers')
          .putAttachment(id, 'image', rev, _image.data, _image.type)
    );

const getBeers = () =>
  store
    .collection('beers')
    .allDocs({
      include_docs: true,
      descending: true,
      attachments: true, // We include images in the output, so we can display them
    })
    .then(doc => doc.rows.map(row => row.doc));
};

In CouchDB, every file can directly be attached to its corresponding document as an attachement. This concept does not exist in Firestore. It is then preferable to use Firebase Storage (Google Cloud Storage) through its bucket system to store files and store paths in Firestore.

Conclusion

The final result of my beer registry application is available on GitHub at the following address: github.com/marmelab/reactive-beers. Feel free to comment and improve!

While I was doubtful about the power of CouchDB at first, I was quickly conquered by its stability and the ease of use of its API.

Since I have not yet deployed this type of application in production, I am not in a position to comment on the ease of maintenance of this type of database. Nevertheless, I would rather recommend using Firestore for POCs, and a third-party service like Couchbase or IBM Cloudant for critical applications in the first place.

Although this experience allowed me to balance the pros and cons of the main features of each database, it was not possible for me to go as far as I had expected.

Indeed, I didn't have time to cover many crucial points such as document access security, rights management, server-side document validation, data pagination or deployment. But no matter what, I am determined to write more articles on these topics.

So, Stay tuned!

Latest comments (6)

Erik Koopmans • Sep 26 '19

Hi, great article! You mention you haven't dived into security and rights yet - I'll warn you and any others that might be interested in CouchDB that this is where the technology is lacking. There is no per-document rights, only per-DB.

A common recommendation around this is to use many (think thousands) of DBs, i.e. DB-per-user, and set up filtered replications to maintain each user's access. Unfortunately this has very poor performance - I found that for a small project (~150 users with a few admin roles) the CPU usage for replication became unwieldy.

CouchDB's cousin, Couchbase, does a much better job of security and rights, though has its own quirks. It requires a secondary layer (called Sync Gateway) to give it a Couch-like API which can connect to PouchDB. This also provides a very useful "sync function" that allows very smooth per-document rights. I'm mostly happy with it. I don't think CouchDB is viable for any project with anything but the most basic rights setup.

I've actually just followed the reverse of your journey, starting with CouchDB and ultimately landing on Firebase as the way to go! One of my main regrets though is the loss of document versioning, which I think is incredibly valuable in CouchDB and Couchbase. That and losing direct control over the data and where it's stored.