Firebase function retries with pubsub

Ever tried to use exponential backoff for pubsub triggered firebase functions? It’s not straight forward, but I’ll walk you through.

Why?

Decoupling cloud/firebase functions from the trigger/invocation via pubsub message topics is a best practice to improve reliability and resilience. Especially for the latter you’d want to have a sensible retry setup, so that transient errors are retried. This retry can be configured in the pubsub subscription that triggers the function.

What is the problem?

Consider this function:

export const myFunction = functions.region('europe-west1')
    .runWith({failurePolicy: {retry: {}}})
    .pubsub.topic('my-topic')
    .onPublish(...)

When deploying the function using the firebase cli, the subscription is recreated with standard retry settings. Using the firebase cli, it is currently not possible to define exponential backoff for the retry.

Note that the default settings can have a severe impact on your cloud bill. When retries are enabled (failurePolicy: {retry: {}}), the standard configuration retries immediately and for 7 days! So for non-transient errors or problems with a longer recovery time, you’ll pay for many invitations..

What is the solution?

After each function deployment, you need to update the subscription accordingly. The following configures retries with exponential backoff for at most 2 hours and delays between 10 seconds and 10 minutes:

# regular deployment:
firebase deploy --only functions --force
# update pubsub subscription for retries with exponential backoff:
gcloud pubsub subscriptions update \
 projects/$GCP_PROJECT_ID/subscriptions/gcf-myFunction-europe-west1-my-topic \
 --min-retry-delay=10s \
 --max-retry-delay=10m \
 --message-retention-duration=2h

Note that --force is needed in CI setup to deploy functions with retries.

What are the alternatives?

You could as well

• manually update the subscription (not recommend)
• move from firebase cli to a direct deployment on GCP (eg via terraform)

If I missed something or you have other solutions, please comment!

Comments

[Reid Beels]

Thanks! This pointed me in the right direction.

I found that by using the built-in $GCLOUD_PROJECT env var, you can add this as a postdeploy hook in firebase.json to ensure it runs with every deploy, like so:

{
  "functions": {
    "postdeploy": [
      "gcloud pubsub subscriptions update projects/$GCLOUD_PROJECT/subscriptions/gcf-myFunction-europe-west1-my-topic --min-retry-delay=10s --max-retry-delay=10m --message-retention-duration=2h"
    ]
  }
}

[Luis Pais]

This was spot on my friend, thank you very much for the article!

Now all that's left is a good way to setup DLQs and we are golden 👌

[Daniel Schreiber]

This should be possible with some more gcloud commands - would really be interested in a small write up! :-)

[Luis Pais]

Hahaha nice baton pass!
I'll see what I can come up with, it might not be as straight forward since we will be creating a new topic and subscription (+ permissions and all that fun stuff), but it should work.
Don't expect it any time soon though xP