How to Integrate Your App With Webhooks Using Amazon SNS

Have you ever been in a meeting talking about a new product build, and you get to the integration requirements of products X, Y, and Z?

You don't really want to build direct integrations with these systems, you'd prefer to build an integration that is more generic and could be used by anybody. Why build custom when you could build for everyone?

In event-driven architectures, like a serverless application, you will have integration points at many different places in your application. These integration points are known as events and represent exactly what you'd think - something happened.

You know how you get a notification on your phone whenever your security camera detects motion? That's an event. The detect-motion event from your smart camera was published as soon as it saw that cat run across your driveway.

But what do you do with events? You listen to them with a webhook.

Photo by Jake Young on Unsplash

What is a Webhook?

A webhook is an API intended to receive a message whenever an event happens in a system. Typically, an event-driven application will POST a predefined message to all listening, or subscribing, webhooks when a specific event occurs.

The system that publishes the event is then done with all responsibilities in this exchange. It let other systems know that something happened, now it can continue on its merry way.

It is the responsibility of the subscriber to do something with that message. Whether that means to use the APIs of the originating application to load more data, or maybe call another application to do something else, the real work is done in the webhook.

Amazon SNS

Amazon provides a managed Simple Notification Service to publish events. Honestly it does a lot more than that, but today we're going to focus on how it publishes HTTPS events so external systems can get notified as the application runs.

In SNS, an event is known as a topic and is used to organize and classify different actions in your application.

When something happens that causes the application to do something, like the cat running across the driveway for our security camera application, SNS will publish to the motion-detected topic and notify all the systems that care.

Photo by Markus Spiske on Unsplash

Publishing an Event

While SNS might be a managed service, it still requires a little bit of coding to get it working properly.

In the code that handles an action in your application, you must build a message and make a call to SNS to publish it. The message details are entirely up to you, and there are no rules around what you can and can't add in there. But some specific things to keep in mind when building your message:

• By default, SNS messages are unencrypted, so avoid adding things like PII (Personally Identifiable Information).
• Once you decide on a message structure, don't change it! You might break your integrators.
• Integrators can always call your API to get more information, give them only what is necessary.

Broadcast

If you want to publish an event to everybody, you can send a broadcast message and it will go to all subscribers of your event. If your application is single tenant/multi-instance, or if you have an event that applies to everybody (like notifying users of a major system outage), this type of message applies to you.

Here's an example of the motion-detected event for a single tenant application in a Node.js lambda function:

function publishMotionDetectedEvent = async (cameraId, coords, duration) => {
  const message = {
    eventType: 'motion-detected',
    data: {
      cameraId: cameraId,
      boundingBox: {
        x: coords.x,
        y: coords.y,
        width: coords.width,
        length: coords.length
      },
      durationMs: duration
    }
  };

  const params = {
    Message: JSON.stringify(message),
    TopicArn: process.env.MOTION_DETECTED_TOPIC_ARN
  };

  const sns = new SNS();
  await sns.publish(params).promise();
}

In our message, we have only the required information about the event being published. Things like cameraId, building the boundingBox details, and the durationMs of the motion.

This information will get sent to everyone who is subscribed to the event. If they need more information, they could use the API into our application to enrich their data.

Filtered

In multi-tenant applications, you do not want to broadcast messages to everyone when an event happens. This would be very bad for user privacy.

Instead, what you would do is add a message filter to the subscription and add details onto the event so only specific subscribers would pick it up.

Here is an example of publishing a filtered message, this time doing it natively via a step function:

{
  "StartAt": "Publish motion-detected Event",
  "States": {
    "Publish motion-detected Event": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "${MOTION_DETECTED_TOPIC_ARN}",
        "Message": {
          "eventType": "motion-detected",
          "data": {
            "cameraId.$": "$.cameraId",
            "boundingBox": {
              "x.$": "$.coords.x",
              "y.$": "$.coords.y",
              "width.$": "$.coords.width",
              "length.$": "$.coords.length",
            },
            "durationMs.$": "$.duration"
          }
        },
        "MessageAttributes": {
          "tenant": {
            "DataType": "String",
            "StringValue.$": "$.tenant"
          }
        }
      },
      "ResultPath": null,
      "Next": "Notified Subscribers Successfully"
    },
    "Notified Subscribers Successfully": {
      "Comment": "The motion-detected event was sent to all subscribers for the tenant.",
      "Type": "Succeed"
    }
  }
}

The message looks pretty much the same except now we added the MessageAttributes property with the tenant information. This tells SNS to perform message filtering and only send it to select subscribers.

The other difference in the message here of course is our use of a Step Function to do the publishing. This is nice because in the event where you need orchestration, you don't have to build a lambda specifically to do your publishing, it can be done directly in the step function definition.

The values passed into the message are defined in the execution context, and are accessed via JSON Path.

Subscribing to an Event

Now that we know how to publish an event via SNS, let's figure out how we can set up subscribers. We want people to be able to add their API endpoints into our system to subscribe for events they care about.

Subscriptions are a little different than publishing, as they require a two step process:

• Adding the subscription (us)
• Confirming the subscription (them)

Adding the Subscription

Our half of the deal is to write the code and make sure things are configured appropriately. Here is an example of adding a subscription to the motion-detected event for a multi-tenant application using a Node.js lambda function:

function subscribeToMotionDetectedEvent = async (endpoint, tenant) => {
  const params = {
    Protocol: 'https',
    TopicArn: process.env.MOTION_DETECTED_TOPIC_ARN,
    Endpoint: endpoint,
    Attributes: {
      FilterPolicy: {
        tenant: [ tenant ]
      }
    }
  };

  const sns = new SNS();
  await sns.subscribe(params).promise();
}

There are many rules you can add for your Filter Policy if you need complex or conditional logic. But for now, we'll just stick with tenant filtering.

Confirming the Subscription

After our code runs, the endpoint we are subscribing is going to receive a message with a confirmation link. This is to make sure the endpoint has the capability to receive POST messages and we aren't just sending data to a dead link.

No events will be published to the endpoint until the subscription has been confirmed. This will likely result in the subscriber having to code for the confirmation, so be sure to make that known in your documentation.

Once the subscription has been confirmed, you're live! Events will be published to the subscriber based on the FilterPolicy and Attributes applied to the messages you're sending.

Try It!

Events and webhooks are an easy way to make robust yet generic integrations for your application. Amazon SNS makes it super easy to get started, so why not give it a shot and add the snippets from above into a lambda or step function?

If you use CloudFormation to build and deploy your microservices (which you really should, if you're not), a topic is dirt simple to add:

MotionDetectedSNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: motion-detected

Add that to your current template, deploy, and start publishing!