DEV Community: Brandin Chiu

Extensibilty through Webhooks Part 3: Google Cloud

Brandin Chiu — Mon, 08 Jun 2020 12:56:02 +0000

Now that we've had an opportunity to see howe we can implement webhooks using Amazon Web Services, we're going to replicate the process with Google Cloud. Much of the process will be very similar -- only using Google's version of the same or similar services to what we used with AWS.

Note: just like with AWS, the features we will be working with may require a credit card on file with Google Cloud to activate. However, our testing should not reach a level that would invoke actual charges on your account.

How it works

The Google Cloud products we will be using for this demonstration will be: Cloud Functions, and Cloud Pub/Sub.

Cloud Functions is Google's version of Lambda, and allows for us to execute serverless code running on Google Cloud's hardware. We will also be using node for our implementation, but Cloud Functions supports Go, Java, and Python as well.

We will be connecting our Cloud Function to a Cloud Pub/Sub topic, which will be how we invoke our function.

Cloud Pub/Sub is a messaging service for Google Cloud, and functions in a similar way to Amazon's Simple Queue Service, only with less focus on the queueing functionality (which we will not be needing anyway).

Step One: Create a Project

Google Cloud manages resources differently from Amazon Web Services. In AWS, resources are defined globally on the account and segmented by regions.

In Google Cloud, resources are managed in projects. A project is a separate entity within your Google Cloud account, and any resources you use (compute, networking, storage) are only available within that project.

That means, in order to start working with Cloud Pub/Sub and Cloud Functions, we first need to a project.

If you've never worked with Google Cloud before, you will be prompted to create a project when you start. If you already have an account, you can create a new one by clicking the dropdown menu next to the page title and selecting "New Project" in the top-right corner of the dialog.

Give your project a name, and then click "Create". Once your project finishes being initialized, you'll be greeted with your new project's dashboard.

We're now ready to begin!

Step Two: Create the Pub/Sub Topic

In order to trigger our webhooks to fire, we'll be making use of Cloud Pub/Sub as a messaging bus. Messages sent to Pub/Sub are organized using "topics".

Open up the side navigation in Google Cloud and launch Pub/Sub (located under the "Big Data" header).

next, select "Create Topic" from the Pub/Sub landing page to create our webhooks topic.

Give your topic a name, and leave the encryption options as the default.

Your topic will take a few seconds to be generated, and then you'll see the topic dashboard.

Note: it's so common for Pub/Sub to be connected to a Cloud Function that there are helpful shortcuts built into the page near the top!

Step Three: Configuring our Cloud Function

Now that our topic has been created, and we have a way to dispatch messages through our Google Cloud infrastructure, we need to build our message consumer so we can actually do something with those messages.

Open our side menu back up and navigate to Cloud Functions (located under the "Compute" header).

You'll be dropped on the empty list page for your project's Cloud Functions, so let's create one now.

Start by giving your function a name, and then leave the "Memory Allocated" as the default value of 256 MB.

Next, change the value for "Trigger" from "HTTP" to "Cloud Pub/Sub", and then select the topic we created earlier.

Leave the rest of the values as their defaults, and click "Create" at the bottom.

Your function will take a few minutes to prepare while Google Cloud initializes the environment. Once it's done, open up the function to review it's dashboard.

At this point, our function is still using the default "Hello Pub/Sub" function it was created with. We'll need to update the function to send our webhook instead.

Just like with AWS, we'll use a very simple node script to push our requests to our webhook endpoints.

Our first step will be to modify the package.json file. The end result should look like this:

{
  "name": "dispatch-webhooks",
  "version": "0.0.1",
  "dependencies": {
    "@google-cloud/pubsub": "^0.18.0",
    "node-fetch": "^2.6.0"
  }
}

We'll leave the Pub/Sub dependency that's already there, and add our "node-fetch" package.

Next, we'll update our source code using the following simple script:

const fetch = require('node-fetch');

exports.dispatchWebhook = (event, context) => {
  const endpoint = event.attributes.endpoint;
  const eventName = event.attributes.event_name;
  const data = JSON.parse(Buffer.from(event.data, 'base64').toString());

  const headers = {
    "Content-Type": "application/json"
  };

  fetch(endpoint, { method: 'POST', headers: headers, body: JSON.stringify(data) })
    .then((response) => {
        console.log("Dispatched webhook for [" + eventName + "]: " + response.status);
    })
  ;
};

This script doesn't actually do very much. It pulls some event attributes for us from the Pub/Sub message (stored as the event object), and deserializes the body which is encoded.

Then, it pushes the data to the endpoint from the attribute and logs the eventName attribute.

Now that we know what our code will be doing, we can click the "Edit" button in the function toolbar, and then update our function. We'll start with the package.json, and then the index.js file.

Note: Remember to update the "Function to Execute" field with the name of our exported node function.

Once you're done, click the "Deploy" button to update our function.

Note: you may have noticed that unlike in our AWS example, Google Cloud manages our "node_modules" dependencies for us when using node as our runtime. This means we don't need to upload those dependencies in a zip like we had to with Lambda.

Step Four: Test our Solution

We're going to use the same tool to test our Google Cloud backed webhooks as we did with AWS: webhooks.site.

Head there now to generate a new test suite, and copy the endpoint URL from the page.

Next, we'll jump back to Cloud Pub/Sub and open up our topic.

In the top header bar for our topic, click the "Publish Message" button to test our solution the same way we did with SQS on AWS.

In the "Message Body", add whatever data you want to test with. In my case, I'm testing a simulated JSON webhook.

Next, add an attribute called "endpoint", which we've told our function to look out for, and will contain the destination URL for our webhook event.

Click "Publish" to send our message, which will trigger our function and submit the request to our webhook.site endpoint.

After a few seconds, we should see our request in our webhook.site dashboard, confirming our backend functioning as expected.

Finally, we can review the results of our function's execution by navigating back to our function, and clicking "View Logs" from the top bar, which will bring us to Google Cloud's log viewer.

Note: Notice anything? Our function was successfully logged, but our log line says "Dispatched webhook for [Undefined]". This is because we did not test our Pub/Sub message with an "event_name" parameter, which our function uses to log. Oops!

Conclusion

Similar to AWS, our final step is to connect our own application to this infrastructure we've just built.

Google provides a number of SDKs in popular languages to support the seamless integration of Google Cloud with your products. Simply download the one that makes sense for you, connect it to your Google Cloud account, and start pushing messages to Cloud Pub/Sub.

Whether you prefer AWS or Google Cloud (or even Azure, Digital Ocean, or someone else), setting up your infrastructure to support webhooks is simple.

Extensibility should be considered an essential component in any software solution. The needs of your users will often evolve in ways that you won't anticipate, and trying to plan for everything is good way to end up overwhelmed and exhausted.

In less than half an hour you can set up a quick implementation to start adding webhooks to your application's lifecycle and immediately start working with external solutions to your users' needs.

Interested in learning more about my thoughts on extensibility and my other pillars of software design? Check out my take on Pragmatic Software Design!

Extensibilty through Webhooks Part 2: Amazon Web Services

Brandin Chiu — Mon, 08 Jun 2020 12:55:11 +0000

In the previous post we briefly discussed what webhooks are and why they might be helpful. In this post, we will look at a step-by-step implementation of a cloud-hosted solution for dispatching webhooks for our application.

Note: In order to proceed you will need an AWS account. Luckily, all of the features within AWS we will be using are free within the limits we will be exploring.

How it Works

We will be making use of two AWS products in order to build our webhooks solution: Simple Queue Service (SQS), and Lambda.

Lambda is a serverless execution platform for running code remotely. This will be the logic component of our solution, written in node, and will be ultimately responsible for dispatching the webhook. Our function will live inside Lambda, and it will listen for events within the AWS ecosystem and then execute that function using parameters from that event.

Our Lambda function will be listening for an SQS event, which will fire every time a new item is added to our SQS queue.

Simple Queue Service is essentially a message bus for AWS and third party services. We push items (called messages) into SQS, which are then read by consumers who act on those messages.

In our case, we will configure our Lambda function to be our consumer, who will automatically read any message added to the queue and use it to submit our webhook request.

Step One: Configure the Queue

Our first step is to create the queue where we will submit our webhook events.

From your AWS homepage, find and open the SQS service.

Note: you can either search the page for the service by name, or use the search function, which supports common acronyms for AWS services, such as "SQS".

Assuming you have never worked with SQS before, you should see the following screen:

Click the "Get Started Now" button to be prompted to create your first queue, or create a new queue from the queue list.

SQS supports two different kinds of queues: standard and FIFO. The SQS service provides high-availability and redundancy for your messages by mirroring them across hundreds of AWS servers. This can sometimes mean that whenever you query AWS for a new set of messages, you may possibly receive the same message twice. Normally, it's up to your application to correctly handle duplicates. Additionally, in standard queues, messages are not handled in-order.

For these reasons, we will be using Amazon's "FIFO" (or first-in-first-out) queues. Amazon will limit these queues to process the messages sent to them in order. This also prevents duplicates from being entered into the queue. You can learn more about the different queue types here.

Click the "Quick-Create Queue" button to create our FIFO queue.

There are a number of configuration parameters that can be defined for how our queue should behave. These are outside the scope of what we need to worry about at this point, but I encourage you to read more about them here.

You should see the queue list screen with our new queue!

Step Two: Configuring our Lambda Function

Now that our queue is configured and ready to receive messages, we need to set up our Lambda function and hook it up to the queue to act as our consumer.

Return to the AWS homepage or click on the "Services" header and search for "Lambda". You should see the Lambda screen like below:

Click the orange "Create Function" button in the top-right corner to get started creating a new function.

Give your function a name and choose node as your runtime.

Under "Permissions" select "Create a new role from AWS policy templates". Give your new role a name, and then under "Policy templates" start typing "sqs" until you see the "Amazon SQS poller permissions" policy.

Note: Amazon and most other cloud providers use a system called IAM to restrict permissions to invoke their services. These rules are created as roles which are then attached to users and other services. In our case, to allow our Lambda function to talk to SQS, we need to create a new IAM role (or use an existing one) with the correct permissions and attach it to it.

Click "Create function" in the bottom right when ready.

Next, click the "+ Add Trigger" button to link our Lambda to our SQS queue.

Set "SQS" as the service, select the queue we created in step one, and set the batch size to 1.

Click "Add" when done.

Now, select our Lambda functions name from the center of the diagram and scroll down to the "Function code" section.

Our code is going to submit the webhook request using the node-fetch package, which means we'll need to add it as a dependency. In order to properly bundle the needed node_modules, I've included a zip file below for you to upload to Lambda by "Code entry type" to "Upload a zip file".

https://misfitpixel-public.s3.amazonaws.com/lambda.zip

The zip file includes the necessary dependencies and a single JS file, index.js, which includes our function code:

const fetch = require('node-fetch');
const { name, version } = require('./package.json');

exports.handler = async event => {
  const promises = event.Records.map(record => {
    const url = record.messageAttributes.url;
    const method = record.messageAttributes.method;

    if (url == null || method == null) {
      return Promise.resolve();
    }

    return fetch(url.stringValue, {
      method: method.stringValue,
      body: record.body,
      headers: { 'Content-Type': 'application/json' }
    }).catch(console.error);
  });

  await Promise.all(promises);
};

Lambda provides data from our incoming SQS messages from an event object it passes to our function which we then use to submit the request. Ultimately, all this function does is read data from the incoming SQS message, and submit a new request to a provided URL from that data. It submits those requests as a promise which we do not bother resolving, since we don't really care how the target endpoint responds to our request.

Note: in a production example, we would probably want to do some level of error handling in order to provide some kind of retry service for failed hooks or better error logging to the client.

Once your code is uploaded and you're ready, click the orange "Save" button at the top right.

Step Three: Test our Solution

Good news for us, the SQS dashboard provides a nice, clean way to push messages directly to our queue without needing to install or build any other software. This gives us a simple mechanism to simulate our backend infrastructure.

We're going to need a destination endpoint to test that our webhook was delivered, so we'll use the service provided by https://webhook.site. This will give us a temporary endpoint to play with, and an interface to verify our requests.

Copy your unique URL from the page, and then navigate back to the SQS dashboard inside AWS. If you right-click on the name of your queue, a context menu will pop up and give you the option to "Send a message".

SQS messages consist of a combination of a string body, and key-value pairs called "attributes".

Our solution in particular makes use of both. Our HTTP request body is stored in the message body, and then we store the HTTP method name, and the URL itself as an attribute.

Note: we've included the HTTP method as an attribute as an example. However, your webhook requests should really always dispatched as a POST request.

Start by adding a message to your queue with whatever body you want.

You'll also notice two other fields underneath our body. These requirements are specific to FIFO queues. For our tests, you can really enter any text data into these fields.

Note: the Message Group ID is an optional sub-collection within your FIFO queue. Messages with the same Group ID will always be managed in FIFO order. If you have messages where the processing order does not matter, you can give them different (or random) Group IDs.

Note: the Message Deduplication ID is a unique identifier for a particular message. No two items in a FIFO queue can share a deduplication ID. If you push a message to the queue with the same ID as one already in it, your new upload will be ignored. It's common practice to use an MD5 hash of your message contents as the deduplication ID to unique identify messages (which SQS can do for you automatically through your queue configuration).

Next, add the attributes we discussed above: your HTTP method, which will be POST, and your url, which will be your unique webhook.site endpoint.

Simply enter the attribute name, followed by it's value, and then click the "Add Attribute" button to add it to the message.

Once you're done, click "Send Message", then close the confirmation dialog.

Your message will be pushed to the queue, but you may notice that after you return to the queue dashboard, AWS will list there being no messages pending in your queue.

This is due to the connection of our Lambda to our queue. As soon as our message hits the queue, AWS sends an event to Lambda, who then consumes and deletes the message with our Lambda function.

If we navigate back to our webhook.site dashboard, we should see our request with the message body:

Our solution works! We can do one last step of confirmation by checking the Lambda logs.

Navigate back to your Lambda function, and then click the "Monitoring" tab. At the top right of that page, you'll find a "View logs in CloudWatch" button.

Clicking on that button will open the cloud logs where you can view the output of your Lambda function's execution.

Note: our logs here only contain default logging information from Lambda. If we were to add our own console.log() statements to our function, those outputs would be captured here as well.

Conclusion

We've completed the backend cloud architecture of our webhook solution and verified that it is working. We've established a queue to hold our outgoing webhooks, and linked it to a serverless function via Lambda to consume those messages and dispatch the webhook.

At this point, the last step is to connect it directly to our own application.

Amazon provides SDKs for most major languages, which you can quickly add to your application to start interacting with AWS services. Doing this requires creating a "programmatic" user account account ID and secret to authenticate with the SDK, and then using it to push messages to SQS, which will then automatically be picked up and handled by Lambda.

Ultimately, your application will also require the ability to register an endpoint with an event in your application.

Hopefully this tutorial gave you a quick glimpse of how you can use Amazon Web Services to quickly set up a backend to handle dispatching webhooks for your application!

The next article will look to do the same thing over again, but using Google Cloud instead!

Extensibilty through Webhooks Part 1: What are webhooks?

Brandin Chiu — Mon, 08 Jun 2020 12:54:33 +0000

At the center of my discussion on Pragmatic Software Design I discussed the four pillars of what I consider to be successful software solutions.

This post revolves around one of those pillars in particular: extensibility, and how we can make our solution as easy to work with as possible for other developers and/or services.

What are webhooks?

A webhook is an automated submission of data over HTTP as a response to some lifecycle event of importance. Users (typically third-party developers) can register to "listen" to these events by providing a URL, which the service will use to submit a request whenever one of those events fires.

Now, why is this useful? Before webhooks, retrieving data out of another system often required some kind of polling. However, polling is inherently inefficient as it generates significantly more network traffic than needed as you constantly ask the service for data.

Webhooks give us the ability to restrict network usage to only the bare minimum needed to obtain the data we're looking for.

As an example, imagine you are looking to integrate with a service that manages users, with another that sends them emails. Service A is responsible for registering and maintaining your primary source of users. Service B maintains a separate list of users that have opted-in to receive marketing, and periodically sends them deals on new products.

If Service A supports webhooks for new user registrations, then it can automatically notify Service B each time a new user registers. Service B can then ingest that data in near real-time, confirm that the user has given permission to receive marketing emails, and then add them to the list of users in Service B.

The true power of webhooks, however, isn't any one specific use-case, but is instead the flexibility for services to integrate with each other in a variety of ways that the initial developers of that service do not expect. It allows for more efficient use of a team's resources by allowing them to focus on their own service, and instead provide a flexible bridge for others to extend the functionality of that service naturally.

Adding Webhooks to your service

Now that we have a handle on what webhooks are and why they might be valuable for us to implement, we're going to discuss a bare-bones and quick way to add webhooks to our own application using Amazon Web Services or Google Cloud.

We want to use a cloud service as the backend for our webhooks because we want to decouple the delivery and execution of our events from our actual application. We want the execution of our webhooks to occur independently from our application -- we're registering endpoint URLs that we do not control and therefore we want any potential errors or issues arising from our webhook requests to be processed externally.

If you're ready, proceed ahead to follow along with the tutorial.

Pragmatic Software Design

Brandin Chiu — Fri, 22 May 2020 11:26:41 +0000

As developers, it's easy to get lost at sea with all the patterns, techniques, tools, services, and paradigms we're expected to somehow know.

This is especially difficult when needing to design software systems from scratch. Which language should I use? What patterns should I follow? Which dependencies should I rely on? The list of questions never ends, and it can be paralyzing to think about how a perceived wrong decision from the start could come back to haunt you later.

I've experienced these fears at one time or another for most of my career, and it's really only been recently that I've adopted a change in perspective that has sincerely helped reshape my attitude and approach to software design.

This idea, distilled to its most basic foundation is pragmatism.

Focusing on Solutions

At its heart, programming and development is about solving problems. I've quickly discovered that the fundamental characteristics of these problems are almost always the same, regardless of the original scale or scope of the project.

For a business, the most important thing is presenting a solution to your users. In the real world, that solution doesn't need to be perfect, and it likely never will be.

We're both fortunate and unfortunate as programmers that there are often many different ways to resolve a particular problem. Being pragmatic means picking a solution and moving forward with it, without worrying about whether that solution will be viable or successful forever.

At the end of the day, it's the solution that matters. Your users and your stakeholders are very rarely going to care if you've reached the optimal resource usage or if you've solved for Big O Notation.

Planning Ahead

Now, that isn't to say that efficiency isn't important, and being pragmatic is not a blank-cheque to write sloppy code.

But it is important to keep in mind how our solutions are actually going to run in the wild. For most of us, the efficiency ceiling of our solutions will be defined by our product and not by how our users actually use that product.

When we're designing software, we should be planning at a higher level for how our solution might grow, and to ultimately make it as flexible as possible. At the end of the day, our biggest issue shouldn't be whether our solution will live on Apache, nginx, nodejs, or maybe something else -- it should be how it might be adaptable to work on any of them later.

The Pillars of Software Design

Good software design requires maintaining a balance between reasonable planning and actually getting things done. A more pragmatic and flexible approach not only helps keep us from being boxed into corners with specific toolings, but also makes us more Agile.

Thus, instead of worrying about granular-level problems like hosting, caching, etc, we take a step back and plan from a higher level. For me, that means breaking down each of my solutions in a way that helps them meet the following core objectives, or "pillars":

Maintainability
It goes without saying that for most of our careers we will be spent working on software someone else has written. The number of unique opportunities each of us will have to design a solution from nothing will be few and far between.

This is why designing that solution in a way that is maintainable is so important. Our goal here is to make it as simple as possible for developers to understand our code. There a number of easy and simple ways to do this that many of you will likely already do:

descriptive and useful inline documentation
consistent application of style guidelines and linting
common-sense naming rules for variables, methods, and classes

The next step is to actually look at the code we write, and how it could be improved to make it naturally easier to understand. Avoiding the "Code Golf" trap is definitely valuable and contributes a lot to making our code easier to read.

And once we've mastered these simple things to look out for, we can dive into some more complex ones like better managing our application's Cyclomatic Complexity so that it's easier to follow our execution path from the code alone.

Scalability
How our solution scales should always be on our mind. How does our solution hold up with 1000 users? What about 100000? These are the kinds of questions we should be asking ourselves as we design our products.

The great thing about thinking about these questions early is that it means we can plan for how we might solve these problems later, without actually needing to implement things immediately.

The more important of these internal conversations is not what tool we will use to implement our scaling, but instead is and understanding where our potential scaling vulnerabilities are.

Understanding which aspects of your application scale vertically vs horizontally, or which processes will benefit from caching, is significantly more valuable that the implementation of these steps.

Stability
Building a stable solution is always a goal for a developer. However, in my experience, the most stable solutions are the ones that are proactive rather than reactive.

Stability can often be achieved as a byproduct of already building maintainable and scalable solution. And this makes sense since stability is really just a function of how long it takes to fix a problem (maintainability), and how well it handles traffic (scalability) over time.

Extensibility
The most successful solutions that we hear about in recent years have been those that tend to be developer-friendly. These huge enterprises often have dedicated developer resources where they invite others to build custom solutions for the problems their users face.

This is the difference between being a product, and being a platform.

And while not every solution, app, or service will have need to mobilize a network of developers to extend their platform, it should always be considered as a way to broaden your solution's reach.

Common and easy ways to promote extensibility include:

making either your entire or a portion of your API public
implementing webhooks for specific important lifecycle events
authenticating through OAuth

Whenever possible, at least some level of extensibility should be built into your solution in order to give you room to organically grow with your users. It will never be possible to fully anticipate every need of every user -- leaving room for external developers to work with you to solve some of those unique problems allows you to focus on larger-scale goals while still responding quickly to the ever-changing requirements of your users.

Conclusion

Many otherwise good (or even great) ideas have died on the page because they were never given the opportunity to live. Getting stuck on the minutia of things like "should I use a factory pattern?", or "will this be deployed on Kubernetes/containers?" can often derail our progress.

If we adopt a more pragmatic approach that focuses more on the solution itself, and not the pieces underneath. Instead, we take a much higher-level view of the goals we must accomplish in order to design an effective solution. In the end, it must be:

maintainable
scalable
stable
extensible

Once we've addressed these goals, we can look to fill in the blanks as we go, rather than front-loading all that responsibility at the start.

Inherently, this mindset should allow us to develop more flexible software, giving us the freedom to make adjustments quickly and efficiently to pivot as needed to keep our solution in scope of our pillars.

I'd love to hear from you! Have you ever let a personal project die because you got overwhelmed by having to design the perfect piece of software the first time? What strategies do you use when trying to solve this problem?

Let me know!

image courtesy of Unsplash