orliesaurus

Posted on Aug 8, 2023 • Updated on Aug 28, 2023

Debugging Typeform webhooks

#typeform #debugging #dashcam #webhooks

I love using Typeform to capture data from users, especially because it provides me with an EASY and truly SMOOTH user experience to capture data from users anywhere and everywhere.

When a new submission is recorded in Typeform, I realized that the best way to poll to get the results was to use webhooks and not hammering the API every 15 minutes 🔨!

However during the process of implementing a small webhook backend, I encountered quite some issues with the "verification" of the signature header.

In this post, I will go over what I did to fix it!

Why webhooks?

Now you might wondering - "but orlie why not just use an API"?!

APIs are not the right choice for this because they require you to manually prompt them.

They need to be invoked every time you want to check for data.

On the other hand webhooks automatically send me data in response to a specific event.

After learning that webhooks were natively supported by Typeform, I decided to dig right in!

Typeform and webhooks

You can use Typeform webhooks to alert your application, or to connect it to an integration platform like Zapier: the basic idea is that your application wants to get notified when certain event happens in Typeform without having to ask consistently.

Here's what it would look like if I implemented it with an API pull strategy (N.B. read the graph starting from the bottom)

        Nope.
  ┌───────────────────────────┐
  │                           │
  │         Still nothing     │
  │ ┌───────────────────────┐ │
  │ │                       │ │
  │ │         No            │ │
  │ │ ┌───────────────────┐ │ │
  │ │ │                   │ │ │
┌─┴─┴─┴────┐         ┌────▼─▼─▼─┐
│ Typeform │         │My Backend│
└▲─▲──▲────┘         └───┬───┬─┬┘
 │ │  │                  │   │ │
 │ │  └──────────────────┘   │ │
 │ │Is there a new response? │ │
 │ │                         │ │
 │ │                         │ │
 │ └─────────────────────────┘ │
 │  Is there a new response?   │
 │                             │
 │                             │
 └─────────────────────────────┘
    Is there a new response?

Instead of your app constantly polling Typeform to check if anything has changed, Typeform can use a webhook to push new data to your application when it is available.

          Here's a response
        ┌───────────────────┐
        │                   │
  ┌─────┴────┐         ┌────▼─────┐
  │ Typeform │         │My Backend│
  └────▲─────┘         └────┬─────┘
       │                    │
       └────────────────────┘
              thanks

This becomes specifically awesome because it allows you to:

Work in real-time:
Webhooks allow data to be sent immediately when an event happens, without almost any delay.
Work more efficiently:
No need to write polling algorithms. This saves you time and resources.
Stay flexible:
Webhooks can be used for many different types of applications and events.

Here are some examples of using webhooks with Typeform:

When a new Typeform response is registered → send a webhook to my backend to retrieve that data and insert it into a queue
When a new Typeform response is registered → send a webhook to Zapier to perform further automation with the data

Webhooks provide a simple webhook mechanism for different applications to communicate real-time data in an efficient way. Whether it's your backend or a middleware platform that enables you to run automation of any sort, webhooks are truly an integral part of your API plumbing toolkit, which is why they are commonly used for notifications, automation, and integrating modern services.

What can go wrong?

Sometimes Webhooks fail to be delivered, or the payload content changes and thus it becomes difficult to parse the content.

Using tools like Dashcam and ngrok you can understand when a webhook delivery has failed when working on your local development environment, or during experimentation (like during a hackathon).

Specifically Dashcam, a screen recorder for debugging makes it simple to find and fix bugs in your locally running apps. When you encounter an error, Dashcam will play back your screen in sync with terminal, logs, and network requests in a format called a "Dash."

This form of debugging is known as time-travel debugging. Time Travel Debugging (TTD) can help you debug issues easier by letting you "rewind" your desktop, instead of having to reproduce the issue until you find the bug.

This is important because as you develop your platform you want to add enough flexibility to catch errors in webhooks so that the rest of your code doesn't error out.

If you want to learn how to setup Dashcam to create cool videos and catch bugs here's a mini guide

How to catch Webhooks errors

Let's fire up our code editor and write some code to catch a webhook!

We're gonna use Fastify as our library, so let's install it with npm i fastify

Then let's create index.js as our entry point

const fastify = require('fastify')({
  //this enables the logger, good to have enabled while we develop locally
  logger: true
})

// Declare a route called /response to which we send a response when we parse the request
fastify.post('/response', function (request, reply) {
  const payload = request.body;
  reply.code(200).send({ status: 'ok' });
})

// Run the server on port 5544!
fastify.listen({ port: 5544 }, function (err, address) {
  if (err) {
    fastify.log.error(err)
    process.exit(1)
  } 
})

Let's fire up our server!

I'm using ngrok to expose my localhost and port to the internet and I am logging any error to ngrok.log in the current directory where my code lives

ngrok http 5544 --log=stdout > ngrok.log

Then once that's running I take the URL that ngrok gives you!

Open up ngrok.log and see what that address looks like, for me it was:

https://dd6e2439d50e.ngrok.app/response

Finally run

node index.js 2>&1 | tee -a typeform.log

This will write all errors to a log file called typeform.log

Setting up Typeform's webhooks

Navigate into Typeform and add that address as the Webhook configuration endpoint

If you click the Send Test Request button, you should see an example request being sent to your backend

Now if you go back to your terminal, you should see your code run and accept the example webhook payload!

What happens when you hit a 404 early on?

Many times when you get started with Webhooks you configure it a little wrong…

When I first ran the above code, I realized I forgot to add the name of the endpoint ( defined on line 7 in index.js). The name of the endpoint is /response.

Luckily I was running Dashcam so I caught the error on video and created a Dash below!

Watch on Dashcam

To solve this problem, I had to change the webhook address to actually be

https://dd6e2439d50e.ngrok.app/response

In the video, you will note the extra s at the end of my path

Debug your Webhooks further

In this other example, I will add a security layer to the backend so that it will only accept webhooks if they're signed!

To learn how to set up a signed webhook follow the instructions here, specifically step 9

This is secret key I used in my example:

mysupersecret

This means that if another person sends data to my backend endpoint, without using my secret key as the hashing key, the backend will not process the data because it's not coming from Typeform!

To do this, I change my backend to look like this:

const fastify = require('fastify')({
  //this enables the logger, good to have enabled while we develop locally
  logger: true
})
//require the crypto module to verify the signature
const crypto = require('crypto');

// Declare a route called /response to which we send a response when we parse the request
fastify.post('/response',
function (request, reply) {
  const payload = request.body;
  // This is where the magic happens, we take the whole body
  // turn it into a string and format it so it can be verified
  const signature = request.headers['typeform-signature'];

  if (verifySignature(signature, JSON.stringify(payload))) {
    reply.code(200).send({ status: 'ok' });
    return
  }
  else {
    reply.code(401).send({ status: 'unauthorized' });
    return
  }
})

// Run the server on port 5544!
fastify.listen({ port: 5544 }, function (err, address) {
  if (err) {
    fastify.log.error(err)
    process.exit(1)
  } 
})

// This function verifies that it's a valid  & signed request
const verifySignature = function(sign, payload){
    const hash = crypto
      .createHmac('sha256', process.env.SECRET)
      .update(payload)
      .digest('base64')
    return sign === `sha256=${hash}`
  }

I launch the app passing the secret as an environment with the following command

SECRET=mysupersecret node index.js dev 2>&1 | tee -a typeform.log

and the following happens:

Watch on Dashcam

If we inspect line 34 to 40 we can see that we're not doing anything strange….

BUT HERE's THE CATCH

When we want to verify the signature and the hashed payload we need to add a new line to the request's body.

To do this let's modify the code to add a new line so the hashed body and the signature will match

check out line 16 of the code below to see what has changed

const fastify = require('fastify')({
  //this enables the logger, good to have enabled while we develop locally
  logger: true
})
//require the crypto module to verify the signature
const crypto = require('crypto');

// Declare a route called /response to which we send a response when we parse the request
fastify.post('/response',
function (request, reply) {
  const payload = request.body;
  // This is where the magic happens, we take the whole body
  // turn it into a string and format it so it can be verified
  const signature = request.headers['typeform-signature'];

  if (verifySignature(signature,`${JSON.stringify(request.body)}\n`)) {
    reply.code(200).send({ status: 'ok' });
    return
  }
  else {
    reply.code(401).send({ status: 'unauthorized' });
    return
  }
})

// Run the server on port 5544!
fastify.listen({ port: 5544 }, function (err, address) {
  if (err) {
    fastify.log.error(err)
    process.exit(1)
  } 
})

// This function verifies that it's a valid  & signed request
const verifySignature = function(sign, payload){
    const hash = crypto
      .createHmac('sha256', process.env.SECRET)
      .update(payload)
      .digest('base64')
    return sign === `sha256=${hash}`
  }