Firestore Many-to-Many: Part 4 - follower feeds

UPDATE 3/7/23

Here is my final article on this subject will all versions:

https://code.build/p/GNWh51AdUxUd3B8vEnSMam/building-a-scalable-follower-feed-with-firestore

---

So, I made a theoretical version of a scalable follower feed here. However, I need to put the theory into practice. Sometimes while putting the pedal to the medal, you realize your theory is wrong. So, I am going to start from simple and work my way to more complex. This post is simply about a working version before I end up writing some complex Firebase Functions.

Let's get something actually working shall we!

Version 1

The only working version I have seen is Fireship.io. I highly recommend you check it out, as all his courses are great.

Here he basically has a user's last 3 posts aggregated on the follower doc in a followers collection. He also has a followers array and uses array-contains to find the relevant userId. He then sorts the results with all the latest 3 posts from his followers. Check out his course for his code, great stuff there.

The problem with this version is that you can only see aggregated posts (3 in his example) from a user. They are not necessarily sorted, and a user is limited by the number of followers. What if my top 10 posts are from 1 user?

Version 2

So, without using any backend indexing, let's try a different approach. This approach will limit the number of people you can subscribe to, but a user can have as many subscribers as possible. This approach also displays sorted posts, always up-to-date, and requires no backend. However, it will do a lot of reads and sorting on the front end. This can work for some situations, but I am posting it to be thorough.

Note: There are limits to other social media's number of people you can follow:

• Twitter - 5,000
• Instagram - 7,500
• Facebook - 5,000
• Snapchat - 5,000
• TikTok - 2,000

I googled these answers, so I apologize if I am not perfect on the numbers. You get the gist. There are also daily follow limits.

So here is the basic model

userDoc

users/{userId} --> {
  displayName: 'Jon',
  email: '[e-mail]',
  following: [
    32321a2,
    f2232k3,
    fkelses,
    ...
  ]
}

Keep the users you are following in your user doc. All of them are readily available in an array by reading one doc. Let's give a 500 limit for the sake of argument. You could possibly store more than 10k, but that is not why we have the limit as you shall see.

postDoc

posts/{postId} --> {
  title: 'some posts',
  content: 'some stuff here',
  creatorId: 'user id of creator'
}

This could obviously be snap, or tweet, or whatever.

Now let's use the tactics from Part 3 of this series.

Query

First get the followers from the user doc...

const followers = (await this.afs.doc(`users/${userId}`)
.ref.get() as any).data().followers;

If this were a where AND clause, no problem. However, since this is an OR clause, which Firestore does not support, we are stuck with frontend indexing...

const result = combineLatest(followers.map(
  (student: string) => this.afs.collection('posts',
    (ref: any) => ref.where('creatorId', '==', follower)
  ).valueChanges({ idField: 'id' })
)).pipe(
  // combine results
  map((a: any[]) => a.reduce(
    (acc: any[], cur: any) => acc.concat(cur)
    // filter duplicates
  ).filter(
    (b: any, n: number, a: any[]) => a.findIndex(
      (v: any) => v.id === b.id) === n
    // sort by createdAt
  ).sort((a: any, b: any) => {
    const f = 'createdAt';
    if (a[f] < b[f]) { return -1; }
    if (b[f] < a[f]) { return 1; }
    return 0;
  }))
);

But we have severe limitations here as well. We are basically grabbing all posts, and filtering and sorting them on the front end. If this were an AND query, it would be no problem. But Firestore does not support any kind of OR queries except array-contains, which is limited to 10.

Back to the drawing board...

Version 3

So I had another brainstorm here where we index a feed on the frontend by date, then just read the feed.

I have been thinking about this problem for a long, long, time. 😩

Let's see if this version can work (theoretical up to this point).

Model

users/{userId} => {
  displayName: 'Jon',
  following: [
    3929293ssks,
    jeons202swpeo,
    ...
  ],
  ...
}
posts/{postId} => {
  createdBy: 2l2l2l32
  ...
}
users/{userId}/feed

There is another step here similar to Version 2. The difference is we only have to aggregate a user's feed so we don't copy the same posts over and over.

Get Docs to Aggregate

// how many days old
const x = 100;
const date = new Date(Date.now() - x * 24 * 60 * 60 * 1000);

const feed = (await combineLatest(authors.map(
  (author: string) => this.afs.collection('posts',
    (ref: any) => ref
      .where('authorId', '==', author)
      .where('createdAt', '<', date)
  ).valueChanges({ idField: 'id' })
)).pipe(
  take(1),
  // combine results
  map((a: any[]) => a.reduce(
    (acc: any[], cur: any) => acc.concat(cur)
    // filter duplicates
  ).filter(
    (b: any, n: number, a: any[]) => a.findIndex(
      (v: any) => v.id === b.id) === n
  )),
).toPromise() as any[])
// all we need is id and createdAt, user might update their post
  .map((r: any) => {
    return {
      id: r.id,
      createdAt: r.createdAt
    }
  });

Save last update date

// get userId
const userId = (await this.afa.user.pipe(take(1))
.toPromise())?.uid;

// save now date
this.afs.doc(`users/${userId}`)
.set({ 
  lastFeedUpdate: firebase.firestore.FieldValue.serverTimestamp() 
});

You should probably have a button called "Update Feed" that does all this.

To make the above code more concise...

New Date or Use lastFeedUpdate

// get user
const userId = (await this.afa.user.pipe(take(1))
.toPromise())?.uid;
// get last update time
const lastFeedUpdate = (await this.afs.doc(`users/${userId}`)
.valueChanges().pipe(take(1)).toPromise() as any).lastFeedUpdate;
// use new date if never created
const date = lastFeedUpdate 
? lastFeedUpdate 
: new Date(Date.now() - x * 24 * 60 * 60 * 1000);

Save Data and Create Feed

const batch = this.afs.firestore.batch();

feed.map((data: any) => {
  // save each doc in feed
  const newDoc = this.afs.doc(`users/${userId}/feed/${data.id}`).ref;
  batch.set(newDoc, data);
});

await batch.commit();

You could save the whole doc here and just read the feed as any collection, but then the posts may not be up-to-date.

Read the Feed

const read = this.afs.collection(`users/${userId}/feed`,
ref => ref.orderBy('createdAt')
).valueChanges({ idField: 'id' }).pipe(
// map each id to a post doc
switchMap((r: any[]) => combineLatest(r.map(
  (d: any) => this.afs.doc(`posts/${d.id}`).valueChanges()
))),
);

You might want to keep it an observable so it is always up-to-date, or at the very least re-run the get promise if there is an update.

If you get a result of 'undefined', you may want to display a blank doc, or 'user has removed their post' UX.

So I proved 2 types of follower feeds are possible.

But what about a scalable follower feed!!!!???!!!

So, I am going to work on my theoretical feed using Firebase Functions. I am going to try and simplify everything I can, and I have to start with just one feature at a time.

TO BE CONTINUED...

J

Comments

[Devin Smaldore]

Hi Jonathan, I've enjoyed reading this article. How are you doing with creating the scalable version of this? I am interesting in hearing what you've learned.

[Jonathan Gamble]

I will see if I can post the last section this weekend.

[Atlas-1510]

Just wanted to let you know I am super keen to see the rest of this series. I stumbled on this follower/following modelling issue with firebase and am really interested in your solution.

[Jonathan Gamble]

Should have next part posted this weekend. I finished up a way to handle infinite arrays, which is kind of step one to the big picture...