𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢 - Firestore referential integrity via triggers

Firestore is amazing, but...

Google Cloud Firestore is a serverless NoSQL document database that scales horizontally - which means it adds/removes nodes to serve your database based on demand automagically. It also does some fancy indexing that allows query times to be proportional to result size instead of total data size. So basically if your query returns 10 records, it will take the same time to run if the total data size is 10, 100, 1000 or squillions of records.

It offers an expressive query language, but does have some limitations which guarantee O(ResultSet) performance. Also, while designing NoSQL database schemas, we have to often "unlearn" data normalization principles we learnt building relational databases.

For example, say you had a database that records comments made by users who have usernames and profile photos. Traditionally you would have stored a foreign key called userId in the comments table, and performed a "join" to get comments together with usernames and profile photos.

But in a NoSQL schema, data is often denormalized - in this case for example, username and photo are repeated in each comment record for ease of retrieval.

The key question then of course is how are updates to username/photo reflected across all comments made by a user? In the case of Firestore, one could write a Cloud Function triggered by updates to any user record which replicates the update to all comment records.

𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢 can help!

𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢 is an npm library that offers pre-canned Firestore triggers that help maintain referential and data integrity in some commonly occurring scenarios.

Attribute Replication

Scenario - Continuing the users/comments example above, you could have a schema like this:

  /users/
    userId/
      username
      photoURL

  /comments/
    commentId/
      body
      userId       <-- foreign key
      username     <-- replicated field
      photoURL     <-- replicated field

Solution - To enforce referential integrity on updates of username/photoURL, simply use:

exports.replUserAttrs = integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'users',
  },
  targets: [
    {
      collection: 'comments',
      foreignKey: 'userId',
      attributeMapping: { 
        'username': 'username', 
        'photoURL': 'photoURL', 
      },
    },
  ],
});

Stale Reference Deletion

Scenario - Say you have an articles collection, where each article can have zero or more comments each with an articleId foreign key. And you want to delete all comments automatically if the corresponding article is deleted.

  /articles/
    articleId/
      body
      updatedAt
      isPublished
      ...

  /comments/
    commentId/
      articleId   <-- foreign key
      body
      ...

Solution - To delete all comments corresponding to a deleted article, use:

exports.delArticleRefs = integrify({
  rule: 'DELETE_REFERENCES',
  source: {
    collection: 'articles',
  },
  targets: [
    {
      collection: 'comments',
      foreignKey: 'articleId',
    },
  ],
});

Count Maintainence

Scenario - Say you want to record which users have liked any particular article and also be able to quickly determine how many total likes an article has received.

  /likes/
    likeId/
      userId
      articleId    <-- foreign key

  /articles/
    articleId/
      likesCount   <-- aggregate field

Solution - To maintain a live count of number of likes stored in the corresponding article document, use:

[
  module.exports.incrementLikesCount,
  module.exports.decrementLikesCount,
] = integrify({
  rule: 'MAINTAIN_COUNT',
  source: {
    collection: 'likes',
    foreignKey: 'articleId',
  },
  target: {
    collection: 'articles',
    attribute: 'likesCount',
  },
});

Notice that you get two triggers, one to increment and another to decrement the likesCount attributes for every addition or deletion in the likes collection.

Deploying

𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢 is meant to be used in conjunction with firebase-functions and firebase-admin. Indeed, they are peerDependencies for 𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢. Typically, your setup would look like:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();
const { integrify } = require('integrify');

integrify({ config: { functions, db } });

// Use integrify here...

Then you would deploy the functions returned by 𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢 like any other Firebase Function:

firebase deploy --only functions

Source Code

Check out the source code, and feel free to open any issues, send out PRs or general comments!

anishkny / integrify

🤝 Enforce referential and data integrity in Cloud Firestore using triggers

𝚒𝚗𝚝𝚎𝚐𝚛𝚒𝚏𝚢

🤝 Enforce referential and data integrity in Cloud Firestore using triggers

Introductory blog post

Usage

// index.js
const { integrify } = require('integrify');

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();

integrify({ config: { functions, db } });

// Automatically replicate attributes from source to target
module.exports.replicateMasterToDetail = integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'master',
  },
  targets: [
    {
      collection: 'detail1',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail1Field1',
        masterField2: 'detail1Field2',
      },
    },
    {
      collection: 'detail2',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail2Field1',
        masterField3: 'detail2Field3',
      }

…

View on GitHub

Thanks for reading ✌️✌️✌️

Comments

[Ayyappa]

Regarding "MAINTAIN_COUNT", does it handle for higher parallel updates? Firestore document has a limit of 1 write per sec and the have an extension to handle this drawback.

Can you please let me know internally Integrify uses sharded counters or has 1 write/sec limit?

Overall, this is a serious pain point and glad your package solves this. Great work!

[Anish Karandikar]

Thanks for the question @ayyappa99

Currently, writes in MAINTAIN_COUNT are not sharded and limited to max 1/sec - but I have opened a enhancement issue to track it for future github.com/anishkny/integrify/issu...

[Felix Collins]

Looks useful. Can it do Many to many? How do I delete a stale reference from an array of foreign keys. given:

/parents/
parentId/
mychildren (array of FK childId)

/children/
childId/

if I delete a child, how do I delete from mychildren array?

Or do you think I should use a "joining" document?

[camillo777]

Hello very nice!
What about if target is a subcollection of a document?
Like:

users:[
user: {
userID
}
]

comments: [
comment: {
liked: [
userID
]
}
]

[Anish Karandikar]

Thats a great question @camillo777 .

As of v2.2.0, you can now replicate into and delete references from subcollections by specifying isCollectionGroup: true in the target collection:

integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'master',
  },
  targets: [
    {
      collection: 'detail2',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail2Field1',
        masterField3: 'detail2Field3',
      },

      // Optional:
      isCollectionGroup: true,  // <---- Replicate into group
    },
  ],
});

See README for full details.

[camillo777]

Yes thank you I managed to do it.
Works like a charm.

Another question for my example.

Can the foreignKey be the id of the target Firestore doc? What is the exact field name?

And: is it safe to use the same document id in multiple collections?

For example I have a users collection and I want to add "liked" users to a doc as a subcollection. I would use the same id of the user, but is it safe at a Firestore model level?

Thank you