Firebase + Algolia: How to set up transform functions

Firebase is an incredible service, especially their Firestore database which I use to handle data in Mylo.

Recently I updated the website with a page to show all public workouts. Implementing that is just a read operation but then I thought to add search capability and things quickly got more involved.

Firestore has no native full-text search support and trying to make your own on top of Firebase is:

• expensive
• impractical
• not what I set out to do
• leaves me with more software to maintain

Luckily Firebase is aware that this is a common use case, so their own docs recommend using a dedicated third-party search service. I decided to go with Algolia because they were the service I had heard of the most.

The official guides do a great job of walking you through the setup process so I won't parrot them here. Instead this article goes over transform function and how to set them up. I ran into a few pitfalls doing this and decided to write the guide I wish I had.

• Full-text Search with Algolia - Firebase Docs
• Firestore search extension - Algolia

Heads up: you can access nested fields with dot notation "data.book.name".

What is a transform function

When you install the Algolia extension and point it to a collection to search through, all the documents there and the fields you listed get used to create a search index. A search index is essentially a list that Algolia formats and organizes in a way that is better for searching.

Each uploaded document is referred to as a record and a good practice is keep these small. The record data is also what is returned with search results.

This leads to a balancing act, you want small records that are faster to search through but you also want records to have all the data you need to display results (otherwise you have to request each document again for the rest of the data).

As a concrete example, I wanted to show the total number of exercises within each workout. To do this with the Firestore documents I do something like this:

const numberOfExercises = (workout) => {
  return Object.keys(workout.exercises).length;
}

This is fine since I've already requested and received the whole document. But the exercises object is actually quite large and I definitely don't want to have it in a record.

That's where the transform function comes in. Before sending a document to be indexed, it will go through the transform function where I can do what I need to reduce the size or shape of the object.

const transformWorkout = (workout) => ({
  // ...
  exercises: Object.keys(workout.exercises),
  // ...
});

How to add a transform function to Algolia Firebase

After (or during) the setup of the Algolia extension, you can add the name of your transform function (same as the actual function) in the Transform Function Name (Experimental) (Optional) configuration field.

Firestore Algolia Search extension - Github

Note that you can access nested fields with dot notation "data.exercise.name". That's also the key you have to use to access the data in your transform function and your search results (records). You can rename them by returning an object with the key name you want instead (name: req.body.data["data.exercise.name"]).

Creating

First, set up Firebase functions and the tools needed as per Google's own guide:
Get Started - Firebase Functions Docs

Then create your function in the functions/index.js folder that was made during the init process.

Now, contrary to what the extension docs say (at the time of writing), you should make your function callable via HTTP as described here and not one callable from your app.

This means using functions.https.onRequest instead of functions.https.onCall. This is because the extension calls your function by its URL and not with the Firebase SDK.

• Where the function is invoked - Extension Repo
• How to Invoke an HTTP Function - Firebase Docs

Another important thing is the geographical location of your function. It needs to be in the same region as the extension to work. So if your extension is located in europe-west1 make sure your function is too.

Cloud Functions Location - Google Cloud

// index.js

const functions = require("firebase-functions");

exports.transformPayloadForSearch = functions
  // this needs to be the same location as the extension
  .region("europe-west1")
  .https.onRequest((req, res) => {
  try {
    // extract the actual data payload
    const payload = req.body.data;

    // the returned object MUST be wrapped within "result"
    res.send({
      result: {
        // the `objectID` property is required, don't forget it!
        objectID: payload.objectID,

        // the object keys are the ones you put in the comma separated list when setting up the extension
        name: payload["data.exercise.name"],

        ...transformPayload(payload),
      },
    });
  } catch (e) {
    // You can send logs this way but be aware of quotas or you'll get a surprise in your bill
    functions.logger.log("There was an error", { message: e.message });

    // Make sure to call `.send()`, `.end()`, or `.redirect()` on `res`
    // so that your function doesn't run longer than needed
    res.status(500).end();
  }
})

About nested fields

When configuring the extension, you can optionally pass a comma separated list of Indexable Fields. This is the key the extension will use to access the fields of your Firestore documents and is also the name under which they will appear within your Algolia index.

You can access nested fields by using dot notation, for example "data.exercise.name". The annoying thing is this name will also trickle down to the front-end as you get your search results. You will have to access everything with these long String keys.

Luckily, the keys you return in your transform function is what Algolia will use. So a transform function can also be used to avoid the key hassle.

Re-indexing

Now you can deploy your function with this command (replace transformPayloadForSearch with the name of your function):

firebase deploy --only functions:transformPayloadForSearch

To make Algolia update its index, run the script with the proper env variables set, you can find it in your Firebase console for the project where you installed Algolia.

Extensions > Search with Algolia > How this extension works

Note that you might need to create a service account and generate a key. Make sure to place that key in a safe place but not in version control (git).

You can paste the script into your terminal to run it but if you end up running it a lot (like I did), create a bash script to help you out (also keep it out of git).

#!/bin/bash
# Re-index the Algolia index
# ----------------------------
export LOCATION=<add-yours>
export PROJECT_ID=<add-yours>
export ALGOLIA_APP_ID=<add-yours>
export ALGOLIA_API_KEY=<add-yours>
export ALGOLIA_INDEX_NAME=<add-yours>
export COLLECTION_PATH=<add-yours>
export FIELDS=<add-yours>
export TRANSFORM_FUNCTION=<add-yours>
export GOOGLE_APPLICATION_CREDENTIALS=<add-yours>

npx firestore-algolia-search

Then run this command to make your script executable:

chmod +x reindexAlgolia.sh

And finally use this while in the same directory to run it

./reindexAlgolia.sh

Conclusion

That about covers the process. I myself ran into quite a few roadblocks trying to set it up and couldn't find more information on the topic. I hope this can help you out and save you some headaches!

Feel free to share any feedback or ask any questions 😁

You can reach me on Twitter @MaxMonteil

Comments

[Charles Fries]

Where does transformPayload() come from

[ViktorL]

I believe @maxmonteil meant the transformWorkout function that he defined outside of the cloud function and does the transformation work. transformPayload might just be a typo.

[Honestwork]

Good article. How can I secure the connection between algolia extension cloud funcion and our transform cloud function? Because it sets a allusers cloud function invoker access to the transform function.