DEV Community

Anish Karandikar
Anish Karandikar

Posted on

πš’πš—πšπšŽπšπš›πš’πšπš’ - Firestore referential integrity via triggers

Firestore is amazing, but...

Google Cloud Firestore is a serverless NoSQL document database that scales horizontally - which means it adds/removes nodes to serve your database based on demand automagically. It also does some fancy indexing that allows query times to be proportional to result size instead of total data size. So basically if your query returns 10 records, it will take the same time to run if the total data size is 10, 100, 1000 or squillions of records.

It offers an expressive query language, but does have some limitations which guarantee O(ResultSet) performance. Also, while designing NoSQL database schemas, we have to often "unlearn" data normalization principles we learnt building relational databases.

For example, say you had a database that records comments made by users who have usernames and profile photos. Traditionally you would have stored a foreign key called userId in the comments table, and performed a "join" to get comments together with usernames and profile photos.

But in a NoSQL schema, data is often denormalized - in this case for example, username and photo are repeated in each comment record for ease of retrieval.

The key question then of course is how are updates to username/photo reflected across all comments made by a user? In the case of Firestore, one could write a Cloud Function triggered by updates to any user record which replicates the update to all comment records.

πš’πš—πšπšŽπšπš›πš’πšπš’ can help!

πš’πš—πšπšŽπšπš›πš’πšπš’ is an npm library that offers pre-canned Firestore triggers that help maintain referential and data integrity in some commonly occurring scenarios.

Attribute Replication

Scenario - Continuing the users/comments example above, you could have a schema like this:

  /users/
    userId/
      username
      photoURL

  /comments/
    commentId/
      body
      userId       <-- foreign key
      username     <-- replicated field
      photoURL     <-- replicated field
Enter fullscreen mode Exit fullscreen mode

Solution - To enforce referential integrity on updates of username/photoURL, simply use:

exports.replUserAttrs = integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'users',
  },
  targets: [
    {
      collection: 'comments',
      foreignKey: 'userId',
      attributeMapping: { 
        'username': 'username', 
        'photoURL': 'photoURL', 
      },
    },
  ],
});
Enter fullscreen mode Exit fullscreen mode

Stale Reference Deletion

Scenario - Say you have an articles collection, where each article can have zero or more comments each with an articleId foreign key. And you want to delete all comments automatically if the corresponding article is deleted.

  /articles/
    articleId/
      body
      updatedAt
      isPublished
      ...

  /comments/
    commentId/
      articleId   <-- foreign key
      body
      ...
Enter fullscreen mode Exit fullscreen mode

Solution - To delete all comments corresponding to a deleted article, use:

exports.delArticleRefs = integrify({
  rule: 'DELETE_REFERENCES',
  source: {
    collection: 'articles',
  },
  targets: [
    {
      collection: 'comments',
      foreignKey: 'articleId',
    },
  ],
});

Enter fullscreen mode Exit fullscreen mode

Count Maintainence

Scenario - Say you want to record which users have liked any particular article and also be able to quickly determine how many total likes an article has received.

  /likes/
    likeId/
      userId
      articleId    <-- foreign key

  /articles/
    articleId/
      likesCount   <-- aggregate field
Enter fullscreen mode Exit fullscreen mode

Solution - To maintain a live count of number of likes stored in the corresponding article document, use:

[
  module.exports.incrementLikesCount,
  module.exports.decrementLikesCount,
] = integrify({
  rule: 'MAINTAIN_COUNT',
  source: {
    collection: 'likes',
    foreignKey: 'articleId',
  },
  target: {
    collection: 'articles',
    attribute: 'likesCount',
  },
});
Enter fullscreen mode Exit fullscreen mode

Notice that you get two triggers, one to increment and another to decrement the likesCount attributes for every addition or deletion in the likes collection.

Deploying

πš’πš—πšπšŽπšπš›πš’πšπš’ is meant to be used in conjunction with firebase-functions and firebase-admin. Indeed, they are peerDependencies for πš’πš—πšπšŽπšπš›πš’πšπš’. Typically, your setup would look like:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();
const { integrify } = require('integrify');

integrify({ config: { functions, db } });

// Use integrify here...
Enter fullscreen mode Exit fullscreen mode

Then you would deploy the functions returned by πš’πš—πšπšŽπšπš›πš’πšπš’ like any other Firebase Function:

firebase deploy --only functions
Enter fullscreen mode Exit fullscreen mode

Source Code

Check out the source code, and feel free to open any issues, send out PRs or general comments!

GitHub logo anishkny / integrify

🀝 Enforce referential and data integrity in Cloud Firestore using triggers

πš’πš—πšπšŽπšπš›πš’πšπš’

Build & Test Code Coverage Status Dependency Status Security npm package Mentioned in Awesome Firebase Firebase Open Source

🀝 Enforce referential and data integrity in Cloud Firestore using triggers

Introductory blog post

Usage

// index.js
const { integrify } = require('integrify');

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();

integrify({ config: { functions, db } });

// Automatically replicate attributes from source to target
module.exports.replicateMasterToDetail = integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'master',
  },
  targets: [
    {
      collection: 'detail1',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail1Field1',
        masterField2: 'detail1Field2',
      },
    },
    {
      collection: 'detail2',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail2Field1',
        masterField3: 'detail2Field3',
      }
…
Enter fullscreen mode Exit fullscreen mode

Thanks for reading ✌️✌️✌️

Top comments (6)

Collapse
 
ayyappa99 profile image
Ayyappa • Edited

Regarding "MAINTAIN_COUNT", does it handle for higher parallel updates? Firestore document has a limit of 1 write per sec and the have an extension to handle this drawback.

Can you please let me know internally Integrify uses sharded counters or has 1 write/sec limit?

Overall, this is a serious pain point and glad your package solves this. Great work!

Collapse
 
anishkny profile image
Anish Karandikar

Thanks for the question @ayyappa99

Currently, writes in MAINTAIN_COUNT are not sharded and limited to max 1/sec - but I have opened a enhancement issue to track it for future github.com/anishkny/integrify/issu...

Collapse
 
felixcollins profile image
Felix Collins

Looks useful. Can it do Many to many? How do I delete a stale reference from an array of foreign keys. given:

/parents/
parentId/
mychildren (array of FK childId)

/children/
childId/

if I delete a child, how do I delete from mychildren array?

Or do you think I should use a "joining" document?

Collapse
 
camillo777 profile image
camillo777

Hello very nice!
What about if target is a subcollection of a document?
Like:

users:[
user: {
userID
}
]

comments: [
comment: {
liked: [
userID
]
}
]

Collapse
 
anishkny profile image
Anish Karandikar

Thats a great question @camillo777 .

As of v2.2.0, you can now replicate into and delete references from subcollections by specifying isCollectionGroup: true in the target collection:

integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'master',
  },
  targets: [
    {
      collection: 'detail2',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail2Field1',
        masterField3: 'detail2Field3',
      },

      // Optional:
      isCollectionGroup: true,  // <---- Replicate into group
    },
  ],
});

See README for full details.

Collapse
 
camillo777 profile image
camillo777

Yes thank you I managed to do it.
Works like a charm.

Another question for my example.

Can the foreignKey be the id of the target Firestore doc? What is the exact field name?

And: is it safe to use the same document id in multiple collections?

For example I have a users collection and I want to add "liked" users to a doc as a subcollection. I would use the same id of the user, but is it safe at a Firestore model level?

Thank you