CurateBot Devlog 8: Listing Queued Tweets using Firebase compound queries adn custom indexes

Now that we have curated tweets enqueued in the database, we should list them out, so users can see them, and potentially decide whether to delete or send them back to curation should they feel unsure about it. The code is here, and most of the changes are in the new Tweets.vue file

The View Tweets template

The template makes use of a powerful Vue feature: rendering lists using a v-for:

<template>
  <v-container>
    <Section v-for="(item, idx) in tweets" :key="item.id">
      <template v-slot:title>
        Queued Tweet
        <v-spacer></v-spacer>
        <v-chip outlined dark>{{ idx + 1 }}</v-chip>
        <v-menu bottom left>
          <template v-slot:activator="{ on, attrs }">
            <v-btn dark icon v-bind="attrs" v-on="on">
              <v-icon>mdi-dots-vertical</v-icon>
            </v-btn>
          </template>

          <v-list>
            <v-list-item @click="deleteAction(idx)">
              <v-list-item-title>Delete</v-list-item-title>
            </v-list-item>
            <v-list-item  @click="curateAction(idx)">
              <v-list-item-title>Send to Curate</v-list-item-title>
            </v-list-item>
          </v-list>
        </v-menu>
      </template>

      <v-textarea
        :value="item.tweet"
        counter
        auto-grow
        rows=2
        :readonly="currentlyEditing !== item.id"
      >
      </v-textarea>
    </Section>

    <v-container class="mx-auto text-center" max-width="860">
      <v-progress-linear v-if="loading" indeterminate></v-progress-linear>
      <v-btn v-else-if="!listComplete" color="primary" @click="loadMore()">Load more</v-btn>
      <v-btn v-else disabled>No more to load</v-btn>
    </v-container>
  </v-container>
</template>

The <Section v-for="(item, idx) in tweets" :key="item.id"> will cause this to be rendered multiple times, once for each member of the data structure tweets, which are loaded from Firestore on mount.

Loading

I actually structured the design of this page around how Firebase charges for billing - Firebase charges you per database transaction when using Firestore, which means I actually want to discourage bulk-viewing tweets unnecessarily. Furthermore, as a nosql database, I don't have that many options when it comes to counting how many entries there are (without incurring more charges), and it's hard to keep track of the exact number as subsequent data loading could overwrite earlier ones, and I don't know when this happens without having to do more data reads!

Therefore the simplest option that optimizes on cost, is simply have a "view more" button the user has to press to load the next batch, until all batches are exhausted.

The load code looks like this, including both initial load on startup, and every time "view more" is pressed:

  loadTweetsFromQuery(query: firebase.firestore.QuerySnapshot) {
    if (query.size < pagesize) {
      this.listComplete = true;
    }
    query.forEach(doc => {
      this.tweets.push({
        tweet: doc.get("tweet"),
        id: doc.id,
        doc: doc
      })
    })
  }
  loadMore() {
    this.loading = true;
    const lastDoc = this.tweets[this.tweets.length - 1].doc;
    return firestore.collection('users').doc(this.uid).collection('tweets')
    .where('queued', '==', true).orderBy('added').startAfter(lastDoc).limit(pagesize).get()
    .then(this.loadTweetsFromQuery)
    .catch(err => {
      console.error(err);
      this.showError("Something went wrong, could not load tweets");
    })
    .finally(() => {
      this.loading = false;
    })
  }
  mounted() {
    this.loading = true;
    return firestore.collection('users').doc(this.uid).collection('tweets')
    .where('queued', '==', true).orderBy('added').limit(pagesize).get()
    .then(this.loadTweetsFromQuery)
    .catch(err => {
      console.error(err);
      this.showError("Something went wrong, could not load tweets");
    })
    .finally(() => {
      this.loading = false;
    })
  }

The main thing happening here besides the composite query .where('queued', '==', true).orderBy('added'), is we do a startAfter() method when loading more, which uses Firestore's document pagination method to fetch more data.

Firestore index

As it turns out, this composite query requires us to define a custom index. We can do this via the web UI, but it's also possible to do it via the firestore/firestore.indexes.json file we set up earlier:

{
  "indexes": [
    {
      "collectionGroup": "tweets",
      "queryScope": "COLLECTION",
      "fields": [{
          "fieldPath": "queued",
          "order": "ASCENDING"
        }, {
          "fieldPath": "added",
          "order": "ASCENDING"
        }
      ]
    }
  ],
  "fieldOverrides": []
}

This sets up an index that includes queued and added fields so that we can query by .where('queued', '==', true).orderBy('added')

Dropdown menu

As can be seen in the template above, there's an extra dropdown menu for each item, which lets us put the tweet back into the curate list, or to delete it entirely

The code simply either deletes the tweet, or sets queued to false:

deleteAction(idx: number) {
    return this.tweets[idx].doc.ref.delete()
    .then(() => {
      this.tweets.splice(idx, 1);
      this.showWarning("Tweet Deleted");
    })
    .catch(err => {
      console.error(err);
      this.showError("Something went wrong, could not delete tweet");
    })
  }
  curateAction(idx: number) {
    return this.tweets[idx].doc.ref.update({
      queued: false
    })
    .then(() => {
      return firestore.collection('users').doc(this.uid).update({
        newCount: firebase.firestore.FieldValue.increment(1)
      });
    })
    .then(() => {
      this.tweets.splice(idx, 1);
      this.showInfo("Tweet moved back to Curate");
    })
    .catch(err => {
      console.error(err);
      this.showError("Something went wrong, could not delete tweet");
    })
  }

That's all for now!