DEV Community

Daniel Schreiber
Daniel Schreiber

Posted on

3

Firebase function retries with pubsub

Ever tried to use exponential backoff for pubsub triggered firebase functions? It’s not straight forward, but I’ll walk you through.

Why?

Decoupling cloud/firebase functions from the trigger/invocation via pubsub message topics is a best practice to improve reliability and resilience. Especially for the latter you’d want to have a sensible retry setup, so that transient errors are retried. This retry can be configured in the pubsub subscription that triggers the function.

What is the problem?

Consider this function:

export const myFunction = functions.region('europe-west1')
    .runWith({failurePolicy: {retry: {}}})
    .pubsub.topic('my-topic')
    .onPublish(...)
Enter fullscreen mode Exit fullscreen mode

When deploying the function using the firebase cli, the subscription is recreated with standard retry settings. Using the firebase cli, it is currently not possible to define exponential backoff for the retry.

Note that the default settings can have a severe impact on your cloud bill. When retries are enabled (failurePolicy: {retry: {}}), the standard configuration retries immediately and for 7 days! So for non-transient errors or problems with a longer recovery time, you’ll pay for many invitations..

What is the solution?

After each function deployment, you need to update the subscription accordingly. The following configures retries with exponential backoff for at most 2 hours and delays between 10 seconds and 10 minutes:

# regular deployment:
firebase deploy --only functions --force
# update pubsub subscription for retries with exponential backoff:
gcloud pubsub subscriptions update \
 projects/$GCP_PROJECT_ID/subscriptions/gcf-myFunction-europe-west1-my-topic \
 --min-retry-delay=10s \
 --max-retry-delay=10m \
 --message-retention-duration=2h
Enter fullscreen mode Exit fullscreen mode

Note that --force is needed in CI setup to deploy functions with retries.

What are the alternatives?

You could as well

  • manually update the subscription (not recommend)
  • move from firebase cli to a direct deployment on GCP (eg via terraform)

If I missed something or you have other solutions, please comment!

Heroku

Build apps, not infrastructure.

Dealing with servers, hardware, and infrastructure can take up your valuable time. Discover the benefits of Heroku, the PaaS of choice for developers since 2007.

Visit Site

Top comments (4)

Collapse
 
reidab profile image
Reid Beels

Thanks! This pointed me in the right direction.

I found that by using the built-in $GCLOUD_PROJECT env var, you can add this as a postdeploy hook in firebase.json to ensure it runs with every deploy, like so:

{
  "functions": {
    "postdeploy": [
      "gcloud pubsub subscriptions update projects/$GCLOUD_PROJECT/subscriptions/gcf-myFunction-europe-west1-my-topic --min-retry-delay=10s --max-retry-delay=10m --message-retention-duration=2h"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
ranguna profile image
Luis Pais

This was spot on my friend, thank you very much for the article!

Now all that's left is a good way to setup DLQs and we are golden 👌

Collapse
 
danielsc profile image
Daniel Schreiber

This should be possible with some more gcloud commands - would really be interested in a small write up! :-)

Collapse
 
ranguna profile image
Luis Pais

Hahaha nice baton pass!
I'll see what I can come up with, it might not be as straight forward since we will be creating a new topic and subscription (+ permissions and all that fun stuff), but it should work.
Don't expect it any time soon though xP

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay