Building a custom GCP architecture to give TTL ( Time to Live ) to a firebase document or document fields

Rajdeep Sengupta — Sat, 12 Aug 2023 11:20:02 +0000

Building a custom GCP architecture to give TTL ( Time to Live ) to a firebase document or document fields

Let’s get started .

First things first why do you need a custom GCP architecture to give TTL to a firebase document when there you can set custom TTL policies from GCP Data store ??

The TTL Policies have a lot of limitations as of now.

You can’t really do complex things like giving custom times according to the other fields of the documents.
You can not trigger deletion of a document if some changes in another document happens.
If you want to delete some specific field of the document not the whole document that is also not possible with inbuilt TTL
Deletions using TTL are not done transactionally. Documents with the same expiration time are not necessarily deleted at the same time. If you require this behavior, perform the deletions using a client library.
Only one field per collection group can be marked as a TTL field. With this two you can’t to use two or more fields combined to decide the TTL.
A total of 200 field level configurations are allowed. One field configuration can contain multiple configurations for the same field. For example, a single-field indexing exemption and a TTL policy on the same field count as one field configuration towards the limit.
Deleting a document through TTL does not delete sub-collections under that document.

There are more but these are the main cons you may want to setup a custom architecture to give TTL to firebase documents, with this you will get more control over when and how you want to delete things.

Pre-requisites: 👇

A basic understating of Typescipt.
A billing account in GCP as we are going to use Cloud Functions and tasks which will be needed a billing account to work.

I will be assuming you already have a GCP billing account, firestore database setup.

The Architecture : ☁️

Let’s see the architecture and go through it step by step to understand how we are gonna make it. Then we will jump into coding part of it.

Let’s go through the architecture really quick: ⚡

Step 1 : The application creates a new document in the firestore database. This automatically triggers a cloud function using onDocumentCreate method. Read more about it here :

Step 2 : The cloud function performs things as per your instruction. You can do almost anything in that cloud function. At the end the cloud function will add a task in the cloud task to delete the document.

Step 3 : The cloud task will run the task and delete the document or any specific field of the document as you will program it. ⭐

Get started with real stuff

First of all we need to make the cloud task first for that first head towards GCP console and got Cloud Tasks and click on **Create push queue**

Choose a queue name and region and press create. Make sure to choose a region near your users and also make sure top have the cloud functions and the queue in same region for faster performance.

Once the Queue is created make a note of the name and the region the queue. For this blog I am using

QUEUE NAME : test-queue
REGION : asia-south1 (Mumbai)

Writing the cloud functions 💭

We need two cloud function one to schedule the task and one to perform the delete action of the document.

Setup environment :

We will use the firebase CLI to get the local environment ready in seconds.

If you don’t have firebase CLI installed you need to install it first. You can use npm for this.

npm install -g firebase-tools

Or you can visit official docs for more ways of installation. Also you need to use firebase login and choose the current project.

Once this is done. Run this script to make the local development environment 👇

firebase init functions

You will get a folder structure like this.

We will be directly start writing in the index.ts file as this is small project and only consists two cloud functions.

Writing the delete function :

Here is a quick example of writing the delete function. We are making it a http function which will be called by the cloud task. It just expects the id of the document which needs deletion.

async function deleteDoc(req: Request, res: Response) {
  const {id} = req.body;
  try {
      await admin.firestore().collection("expire").doc(id).delete();
      res.status(200).send("Expired");
  } catch (e) {
    functions.logger.error("Error :", e);
  }
}

export const expireLink = functions.https.onRequest(expireSharedLink);

Scheduling a task Cloud function:

Once a new document is added to expire/{id} this path. This firestore trigger will be automatically get triggered and schedule a deletion for this.

export const expitySystem = onDocumentCreated("expire/{id}", (event) => {
const id = event.id;
  // Scheduling the cloud function - Cloud tasks
  const project = "test-project-019";
  const queue = "test-queue";
  const location = "asia-south1";
  const url =
    "https://asia-south1-test-project-019.cloudfunctions.net/deleteDoc";
  const payload = {id}; // passing the payload - the id of the document
  const inSeconds = ttl; // setting the TTL you can use any calculation to make this

  const tasksClient = new CloudTasksClient();
  const queuePath: string = tasksClient.queuePath(project, location, queue);

  const task = { // Creating Task and what task it need to perform
    httpRequest: {
      httpMethod: "POST",
      url, // Sending the delete function url
      headers: {
        "Content-Type": "application/json",
      },
      body: Buffer.from(JSON.stringify(payload)).toString("base64"),
    },
    scheduleTime: {
      seconds: Date.now()/1000 + inSeconds,
    },
};

// Creating the task with the previous parameters me made
await tasksClient.createTask({parent: queuePath, task});

})

Full code : `index.ts`

import * as functions from "firebase-functions";
import * as admin from "firebase-admin";
import {Request, Response} from "firebase-functions";
const {CloudTasksClient} = require("@google-cloud/tasks");

export const expitySystem = onDocumentCreated("expire/{id}", (event) => {
const id = event.id;
  // Scheduling the cloud function - Cloud tasks
  const project = "test-project-019";
  const queue = "test-queue";
  const location = "asia-south1";
  const url =
    "https://asia-south1-test-project-019.cloudfunctions.net/deleteDoc";
  const payload = {id}; // passing the payload - the id of the document
  const inSeconds = ttl; // setting the TTL you can use any calculation to make this

  const tasksClient = new CloudTasksClient();
  const queuePath: string = tasksClient.queuePath(project, location, queue);

  const task = { // Creating Task and what task it need to perform
    httpRequest: {
      httpMethod: "POST",
      url, // Sending the delete function url
      headers: {
        "Content-Type": "application/json",
      },
      body: Buffer.from(JSON.stringify(payload)).toString("base64"),
    },
    scheduleTime: {
      seconds: Date.now()/1000 + inSeconds,
    },
};

// Creating the task with the previous parameters me made
await tasksClient.createTask({parent: queuePath, task});

})

async function deleteDoc(req: Request, res: Response) {
  const {id} = req.body;
  try {
      await admin.firestore().collection("expire").doc(id).delete();
      res.status(200).send("Expired");
  } catch (e) {
    functions.logger.error("Error :", e);
  }
}

export const expireLink = functions.https.onRequest(expireSharedLink);

Deploy the Cloud Functions :

Make sure you are in the project root directory and then run this script 👇

npm run deploy

This will deploy all the functions and you are good to go.

Extras :

You can modify the and calculate the TTL as you want and as well as you can modify the deleteDoc cloud function as you want. So you have complete control over the process.
If you are using Golang or any other language the onDocumentCreate is not available there. So you need to do that part manually.

This is for this blog if you find it helpful make sure to follow me on Twitter as I post a lot of content cloud and DevOps there. Till then Byeeeeeee.

DEV Community: Rajdeep Sengupta

Building a custom GCP architecture to give TTL ( Time to Live ) to a firebase document or document fields

Building a custom GCP architecture to give TTL ( Time to Live ) to a firebase document or document fields

Let’s get started .

The TTL Policies have a lot of limitations as of now.

Pre-requisites: 👇

The Architecture : ☁️

Let’s go through the architecture really quick: ⚡

Get started with real stuff

Writing the cloud functions 💭

Setup environment :

Writing the delete function :

Scheduling a task Cloud function:

Full code : index.ts

Deploy the Cloud Functions :

Extras :

Full code : `index.ts`