DEV Community: Alexandre Bouchard

Working with Webhooks: Security

Alexandre Bouchard — Wed, 05 May 2021 15:31:00 +0000

The series "Working With Webhooks" explore the most important concepts to consider when receiving incoming webhooks.

Working with webhooks exposes an HTTP endpoint that can be called from any actor on your server. Without appropriate measures, this could be extremely unsafe. However, there are now well-understood strategies that ensure your webhook endpoints are secured.

There are 3 main vectors of attacks that you need to watch out and for and protect yourself against.

1) Man-in-the-middle

A man-in-the-middle attack is a vulnerability where a third party obtains access to your webhook data by capturing and reading the request. It's essential that you only work with HTTPS URLs (using SSL) when working with sensitive data. Some providers such as Shopify will enforce this restriction, but many platforms will let you input unencrypted URLs. By using HTTPS, the content of the request is encrypted and can't be read or used by anyone that intercepts the requests.

While technically not a man-in-the-middle attack, it's also possible to provide a URL that you do not own. The burden is on you to make sure the URL that you provide as your Webhook URL points to your own server that this server is also secure. Some platforms like Otka will perform a one-time verification to validate that you are indeed the owner of the URL and will not send any webhooks until that verification is performed. Otherwise, audit your server logs and configured webhook URLs to ensure they are pointing to the correct address.

2) Forged requests

A forged request is a request made to your webhook endpoint that acts as if it was sent from the original source (ex: Shopify), but contains forged data. It's critical to ensure that bad actors are not causing side effects in your systems by sending forged requests to your webhook endpoints. There are multiple strategies to verify that the request isn't coming from an impostor, signature verification being the most popular and secure approach.

Signature verification

Platforms that sign their webhooks can be verified by comparing a value found in the headers (ex: Shopify includes an X-Shopify-Hmac-Sha256 header) with a computed HMAC made up of a secret key and the content of your webhook payload. In generating a Hash-based message authentication code (HMAC), you are required to use your payload content along with the secret key that the platform provides you when you create a webhook subscription. If the header and your computed signature match, then you can be certain the message was originally sent from the expected provider.

This works because only you and the platform have access to that secret key and can compute an identical hash. By including the body of the payload within the hash, you can also guarantee that it has not been tempered. In event that this secret is leaked, you should replace it as soon as possible.

Different providers will use different hashing algorithms (SHA256, SHA128, MD5, etc.), and your implementation should use the same algorithm. Many platforms will also provide SDKs that include the verification logic to spare you most of the trouble.

Example of adding webhook verification for Shopify in Javascript

To get started with adding webhook verification to the server, run the following command:

$ npm install raw-body

The raw-body module we just installed would help us parse our incoming request body in a way that can be used to generate the hash. The default body-parser module that comes with Express could also be used in place of the raw-body package.

Next, Import the raw-body & crypto packages into the project by adding the following lines of code:

const getRawBody = require("raw-body");
const crypto = require("crypto");

The crypto module would be used to generate the hash from our request body and the secret key provided to us by Shopify.

Replace the app.post request handler created above with this

app.post("/webhook", async (req, res) => {
//Extract X-Shopify-Hmac-Sha256 Header from the requestconst hmacHeader = req.get("X-Shopify-Hmac-Sha256");

//Parse the request Bodyconst body = await getRawBody(req);
//Create a hash based on the parsed bodyconst hash = crypto
    .createHmac("sha256", secret)
    .update(body, "utf8", "hex")
    .digest("base64");

// Compare the created hash with the value of the X-Shopify-Hmac-Sha256 Headerif (hash === hmacHeader) {
    console.log("Notification Requested from Shopify received");
    res.sendStatus(200);
  } else {
    console.log("There is something wrong with this webhook");
    res.sendStatus(403);
  }
});

In the code above, we extract the X-Shopify-Hmac-SHA256 HTTP header from the request, create a hash based on the Hmac-SHA256 algorithm from the request body then compare both hashes.

Lastly, go ahead and create a constant called secret which would hold the value of the secret Shopify returned to you when created a new webhook connection. You would want to store that as an environmental variable to ensure it is safe.

Secrets

Alternatively, you might find platforms with the option to add custom headers to the webhook requests, or a basic authentication header. In that case, you should generate a secret key on your own that you would set as a header and would verify on your end. This is similar to how most API authentication works and is considered secured when used over HTTPS. However, absolutely do not rely on this strategy when using standard HTTP, because any man-in-the-middle could steal your secret and forge authenticated requests

IP whitelisting

If the platform that's publishing webhooks provides a list of IPs that they send requests from, you can set up the receiving endpoint to only accept requests from those IPs. This is generally straightforward, but it depends on the platform that you're hosting on. While IP whitelisting is effective, it can also get cumbersome because you have to maintain the IP whitelist. If the provider adds an IP address, and you fail to update your list, you run the risk of potentially refusing payloads. For that reason, it should be used as a complement to other strategies listed above. Whitelisting IPs is not a requirement to protect yourself against forged requests.

3) Replay attacks

Even with the previous two vulnerabilities addressed, you are still susceptible to replay attacks. A bad actor could, in theory, intercept an encrypted request with signature verification and simply replay or make identical requests. While he wouldn't be able to see any of the data, nor alter it, he could still cause unintended side-effects such as creating multiple orders after a "Payment Captured" webhooks. This limits the possible consequences of this kind of attack, but nonetheless there are practical solutions to protect yourself against it.

Timestamped signatures

Some webhook providers, like Stripe, will include a timestamp within the signature. The timestamp used at the time of signing will be contained within the header and can be checked against a time window that you determine. Stripe SDK has a default tolerance of 5 minutes. That same timestamp must be appended to the signature before it's hashed.

Idempotency

Idempotency is a core concept when working with webhooks: if you have already ensured that your endpoints are idempotent, you won't be impacted by a replay attack. By making your endpoint idempotent you ensure that any webhooks are only processed once, even if received multiple times. That's an important case to handle regardless of replay attacks, since most (if not all) providers operate on an at least once delivery strategy. Timestamped signatures are unnecessary if idempotency is correctly implemented - idempotency is the only solution to protect yourself against attacks if your provider does not support timestamped signatures (and most don't).

Learn how to implement idempotency

Conclusion

Webhooks, when used correctly, are very secure. As with most things, the burden often falls on you to make sure you are properly verifying signatures, using HTTPS and implementing idempotency. Luckily, you're not alone, and the information is available!

Hookdeck is a feature-complete webhooks handling platform that handles resilience and monitoring for you. It enables you to decide how to route webhooks requests, debug and visualize your payloads, offer insight into errors and trends, retry on delivery failure, hook into your development process with a local proxy, and much more. Try Hookdeck now for free by creating an account here.

Working With Webhooks: Delayed Processing

Alexandre Bouchard — Thu, 29 Apr 2021 02:28:58 +0000

The series “Working With Webhooks” explores the most important concepts to consider when receiving incoming webhooks.

As a developer, ensuring your back-end works smoothly is key to delivering reliable services. If you don’t ingest and process your webhooks properly, you risk poor performance and server outages, which can negatively impact your product, your users, and your team.

In this language-agnostic reference guide, you’ll learn to take your webhook game to the next level by implementing delayed processing, one of the most important concepts when building reliable webhooks. Along the way, we’ll cover the basics of ingestion, queueing, processing, retries, and alerts.

After reading this guide, you’ll have a solid understanding of what it takes to build a robust and well-designed webhook infrastructure.

The bare-bones approach to webhooks

Webhooks are considered asynchronous by nature. This is because no specific response is expected from your server, besides a receipt confirmation.

Developers who are just getting started with webhooks will often set up their handlers in a very simple way:

Receive the request.
Perform some method.
Return once the method is completed.

While this might seem like a solid approach to handling webhooks, it has several weaknesses.

You don't control the rate at which you receive webhooks.The platform sending webhooks to your server might deliver a burst of requests all at once. This could be due to bulk actions taken on your end (like uploading a new email list to an email service), or it could be because the platform has accumulated a backlog of webhooks to send after an outage. Whatever the cause, you can see how a large volume of requests in a short span of time might overwhelm a system built using a bare-bones approach.
It doesn’t account for errors.Your webhook handling method might run into an error because it's running out of resources, depends on a third party that is returning an error, or is simply a bad deploy.
It doesn’t account for outages.Outages happen, and your webhook infrastructure needs to be prepared for them. You can’t assume your HTTP endpoint will be listening when you receive the webhook. In the case of an outage, you still need to be able to deliver consistent results.
It doesn’t account for timeout windows.Many platforms offer small timeout windows on the request. If your server fails to perform the work within that window, the connection will be terminated. In the overly simplistic approach outlined above, the method you perform upon receiving the inbound webhook must complete before that window closes, or the requesting platform may report a failure – even if the method eventually succeeds.

You’ve decided that the bare-bones approach isn’t for you. Good instincts. So, how should you design your webhook infrastructure? This is where delayed processing for webhooks comes in.

What is delayed webhook processing?

Most of the time, webhooks can be handled easily. But if your server receives too many requests at once, it can become overloaded. Add all the other unrelated jobs your server handles to that ballooning volume of requests, and you may have a service outage on your hands.

A well-designed webhook system will handle these volume spikes efficiently and process all inbound requests, regardless of volume. To that end, your server should perform the bare minimum amount of work upon receiving a webhook. We call this delayed processing.

A server that follows the principles of delayed processing will first return an HTTP 200 response, then queue the event to be processed asynchronously by your system.

Utilizing delayed processing has several advantages:

It limits the number of resources necessary to process a request.
It gives you more capacity to handle multiple concurrent requests.
It allows you the freedom to employ processing-intensive rules and logic on inbound webhooks, without affecting the reliability of your infrastructure.

How to implement delayed processing

A webhook infrastructure with delayed processing must successfully implement three key steps: ingestion, queuing and processing.

Ingestion

Our only goal here is to expose a simple HTTP POST endpoint. This endpoint must:

Receive webhooks
Store the request headers & body in a queue
Return an HTTP 200 status code

The endpoint should be built to utilize as few resources as possible, which will ensure it remains scalable.

Serverless services like AWS Lamda, Google Cloud Functions, and Google Cloud Run are a great fit for ingestion, since they scale very easily based on the volume of concurrent requests.

Queuing

Once we’ve successfully received the request, we need to keep track of it to process at a later time. The best solution here is to implement a queue.

However you design your queue, make sure it won't crumble under the load. Most queues are limited by the number of concurrent connections, or throughput in GB/s – you want to make sure that your queueing system can keep pace with any unexpected spikes in volume. Some queues like AWS SQS and PubSub are virtually limitless, making them great candidates for this use case.

One good strategy to save on costs is to store the webhook headers and bodies in a file on AWS S3 or GCP Cloud Storage and queue a reference to the file instead of the actual contents.

Depending on the nature of your use case, you might have to have multiple queues for different webhook topics or types. Some request types will naturally be more important than others. By introducing multiple queues, you can assign different priorities to each one.

Processing

Once you have your webhooks in a queue, you can safely process them. Just be sure to do so at a rate that is reasonable for your infrastructure and use case. Different queues take different approaches to processing – but, generally speaking, you will want a set of worker services, each pulling from the queue at its own pace.

How to handle retries

A webhook queueing system is a great way to ingest and process webhook requests. But even the best queueing systems can drop requests when faced with server overload or other network issues. When this happens, your system won’t complete the underlying transaction or event notification, which can lead to a poor user experience. To handle such cases, we need to implement a retry feature that triggers on two scenarios: ingestion failure and processing failure.

Retry on ingestion failure

If you fail to ingest a webhook request, you'll need to rely on your provider retry policy. Most platforms have some form of retry strategy. If the platform doesn't offer retries, or you fail to receive all retries, you will need to reconcile your data by pulling the provider API. Since this is not always possible, the reliability of your ingestion service is critical.

Retry on processing failure

It's virtually guaranteed that, at one point or another, you will run into errors when processing your queue. Thankfully you can leverage your queue to perform retries by properly acknowledging (or not acknowledging) the messages.

Don't forget to also include a dead letter policy to set aside messages that fail to process multiple times in a row. If you don’t, you may end up jamming the queue with events that are unprocessable.

How to handle alerts

You need to know when things aren't going as planned in your production environment. When it comes to delayed processing of webhooks, there are two places where alerts become critical.

When ingesting

If your ingestion service responds with an error, this means you are actively missing webhooks. You will want to monitor HTTP responses from your endpoint, and set up an alert if the status isn't a 2xx. Some API providers will also offer some form of alerting if your endpoint doesn’t return a successful response.

When sending to a dead letter queue

If you fail to process a message multiple times in a row, you'll want to inspect it so you can determine the root cause of the failure. You will want to monitor the depth of your dead letter queue to alert you when new messages appear.

Get started with delayed webhook processing in Hookdeck

If you’re looking for a robust platform to handle all the above complexity for you, Hookdeck has your back. You can easily set up and monitor your webhooks without worrying about ingestion, queuing, or troubleshooting processes in your workflow. And it’s free!

To test-drive these features and more, sign up today!

Working With Webhooks: Implementing Idempotency

Alexandre Bouchard — Mon, 26 Apr 2021 16:35:22 +0000

Most webhook providers operate on an "at least once" delivery guarantee. The key phrase here is "at least" — you will eventually get the same webhook multiple times. Your application needs to be built to handle those scenarios

What is idempotency?

In computing, when repeating the same action results in the same outcome, we call it idempotent. One common example you have probably encountered is the HTTP PUT vs the HTTP POST methods.

The distinction between the two is that PUT denotes that the action is idempotent. Updating an inventory count, a profile first name, or assigning an order to a customer can be done multiple times in a row without the use of new or extra resources.

A POST, however, implies side effects. If you create a new order for entry, every time you call the endpoint, a new entry will be created, even if it contains the same properties.

Because webhooks are standardized around HTTP POST calls, it's up to you to figure out what is idempotent by nature, versus what needs to be built to be idempotent. In most cases, the burden will fall on you.

When to build for idempotency

Generally, events that either create a new resource or cause side-effects in other systems are the trickiest to handle. You wouldn't want to create the same order multiple times because you got the same webhook from Shopify twice. You could also be causing side effects, like sending an email when a product runs out of stock, which no one wants to do multiple times.

Those are cases where you would need to carefully audit your code to look for any areas where idempotency issues could arise, and then build strategies to make those webhook events idempotent.

Enforcing a unique constraint inherited from the event data

In many cases, you'll have some unique ID that you can leverage to know if you've already performed the action for any given webhook request. For example, if you are indexing orders from a Shopify store on the orders/created webhook topic, you can use the order_id from Shopify as a unique property in your database.

In SQL, you might do something like:

 CREATE TABLE orders (
    id text PRIMARY KEY,
    shopify_order_id text UNIQUE NOT NULL,
        [...]
);

That is the simplest solution if your database supports unique constraints. If, say, you also want to send an email to the customer, you would perform the INSERT before you send the email. Since you are using that unique constraint to check for idempotency, you will want to perform side effects afterwards. Lastly, make sure to handle the error and return a 2XX status code that corresponds to the unique constraint violation.

Tracking webhook history and handling status

In some cases the first strategy won't be available to you, which could be because you aren't storing any of the data. Nonetheless, every provider will include some identifier for the webhook itself. In Shopify, the request contains X-Shopify-Webhook-Id in the headers. You can leverage that ID to track the status of the webhooks you are receiving.

Repeated requests for the same webhook will have the same webhook identifier.

To handle those scenarios, you will want to create a processed_webhooks table with a unique constraint on the ID.

CREATE TABLE processed_webhooks (
    id text PRIMARY KEY,
        [...]
);

The first thing to do when you receive the request is to store it in the table using the webhook unique ID. Once you have successfully completed your method, you can then update the row to a status of COMPLETED. In the event that you fail to successfully handle it, just remove the row, and allow next attempts.

You can wrap your webhook calls with a generic method to verify for idempotency. Here's an example using Postgres, Express and NodeJS:

const processWebhook = async (req, handler) => {
// Extract the unique ID, using Shopify for this exampleconst unique_id = req.headers["X-Shopify-Webhook-Id"];
// Create a new entry for that webhook idawait client
    .query("INSERT INTO processed_webhooks (id) VALUES $1", [unique_id])
    .catch((e) => {
// PostgreSQL code for unique violationif (e.code == "23505") {
// We are already processing or processed this webhook, return silentlyreturn true;
      }
      throw e;
    });
  try {
// Call you methodawait handler(req.body);
    return true;
  } catch {
// Delete the entry on error to make sure the next one isn't ignoredawait client.query("DELETE FROM processed_webhooks WHERE id = $1", [
      unique_id,
    ]);
  }
};

app.post("/webhooks/order-created", (req, res) => {
// Wrap your doSomething method to handle your webhookreturn processWebhook(req, doSomething).catch(() => res.sendStatus(500));
});

Conclusion

You want to build your application to avoid taking action on the same webhooks received multiple times. To implement idempotency, you can either enforce a unique constraint inherited from the event data or track webhook history and the handling status.

Dealing With Webhooks Sucks But There’s Something You Can Do About It

Alexandre Bouchard — Fri, 17 Apr 2020 00:02:29 +0000

I’ve been working in e-commerce for the last three years dealing with millions of fairly mission-critical webhook events. My key takeaway is that it sucks to deal with multiple APIs incoming webhooks such as Shopify, Stripe & Intercom, and I hate not having an alternative.

It’s April 2020, we are stuck inside self quarantining, productivity has been high, and it just strikes me as the perfect time to publish my first post.

What’s the problem?

Mistakes are human, and so are bugs. Every once in a while, you will introduce errors in your webhook handling methods. Even with the best integration tests, you are probably not prepared for unexpected payload changes (perhaps a stealth API version upgrade) or a platform downtime. You probably have server logs or even better, something like Sentry to catch unexpected exceptions in your code. If these are mission-critical webhooks, are you confident you can find and take action on every single webhook you missed? Maybe, but most likely, you’ll spend the day on it.

“Queue first, take action later”, they said

I know what you’ll tell me. You aren’t supposed to take action directly on a webhook! That’s right, the “proper way” of dealing with webhooks is to postpone any meaningful processing through the use of queues and to handle errors & retries on your own. Clubhouse.io engineering team wrote a great article on this topic. They are right; you should use queues.

However, to handle a single webhook safely, you would need (as per Clubhouse) 4 new services (SQS, S3, a Publisher(s) and a Consumer(s)). It depends on your current architecture, but in short, that’s a lot of new tools, a lot of new code, and especially a lot of time. Time that I could better spend building products for our customers.

If it feels a lot like reinventing the wheel, well, it’s because it is. Solving this particular problem isn’t specific to your business or your application.

Instead, we rely on webhook providers

Let’s be honest, why does almost every platform out there offer an ‘automatic retry’ feature for their webhooks? Because developers generally DO NOT postpone taking action on their webhooks and really, nobody can blame them. The time required to set up queues and related retry logic makes it so that developers, such as myself, are instead relying on the platform to do the work for them.

Each platform has its own rules and logic. Shopify will retry 19 times over 48 hours and completely delete your webhook subscription once all attempts are exhausted as opposed to Stripe that will attempt to deliver your webhooks for up to three days with exponential backoff. This becomes increasingly complex to deal with if you are using 3, 5, or 10 API’s webhooks, that each have their own rules on when they will retry, when they will alert you and when they will disable your webhooks.

Credit where credit is due, Stripe does a lot of things right. They provide a useful UI (but no API) to view your events and retry them. That’s the kind of visibility we need, as developers. I’ve saved many hours of troubleshooting because of that feature alone.

Stripe’s UI to view failed webhooks.

The thing is that you are out of luck if you are using any other platform, Stripe is the only one that I’ve used that provides this level of tooling for developers. Have you encountered other platforms with a similar level of tooling?

An alternative?

For my webhook monitoring, I was looking for a solution that can:

Provide full visibility over all my webhooks and their status;
Allow me to take actions on the failures that occurred;
Will alert me on failures that need looking in into;
Work the same way, regardless of the platforms in my stack.

The bad news is that I couldn’t find any.

The good news is that I built it.

Hookdeck solves these problems by sitting between the API provider and your systems to monitor, distribute and retry all your webhooks. With it, you can trace every webhook event, view the full request and response payloads. You can also standardize the retry and alerting rules across all your webhooks regardless of the provider. This will guarantee that your webhooks behave the same way and be configured to your own use cases. It allows you to take action on any issues or errors that occur with confidence.

Hookdeck — The wheel you won’t have to reinvent

If you are wondering how Hookdeck works, here’s a look at the inner workings of our architecture.

A platform-agnostic solution is possible. Why should it be the developers or the responsibility of the platforms to have those tools? I like to think it’s the missing piece of the puzzle.

If you would like to spare yourself the hassle I went through, Hookdeck now available.

What are your thoughts on handling a large volume of mission-critical webhooks?

Godspeed,
Alex

Maker of Hookdeck & CTO at Rachel