DEV Community: Nick Abbene

Improving Developer Experience through Local Webhook Routing

Nick Abbene — Sat, 15 Apr 2023 11:51:15 +0000

What is developer experience?

In software engineering, developer experience (DX) is a term used to refer to the entire process of developing, implementing and maintaining software and/or web apps. It covers everything from the initial concept and design of a solution, through coding and testing it, to long-term maintenance and troubleshooting.

Developer experience is an important factor when it comes to creating successful applications, as a smooth and positive developer experience can lead to faster development times, improved code quality, and higher productivity. Even if you provide your developers with all the tools they need, it’s still possible to have a poor experience.

While your primary user when it comes to developer experience is your engineers, ultimately, poor developer experience tends to show in the product that your external customers use and consume.

API Integrations

As part of building any system, we spend a lot of time developing APIs for other parts of the system to call. If the system is not large, it may be possible to spin up an entire environment for the entire team.

How do we develop against integrations with external platforms?

The most common route that is taken is creating mocks and triggers for APIs and webhooks that we don’t control. Another alternative is developing off of a contract, creating a Postman collection and integration tests to verify things work as we expect. This works, but is not ideal and can lead to errors popping up later in the development lifecycle due to mismatches between the mocks and the actual functionality.

There is a third, superior alternative that is available to us most of the time, that I haven’t seen commonly used in the wild. Each developer connects directly to the external party’s development environment, and all webhook calls are routed back to their own local machine.

A huge advantage of this (amongst others) is it enables new developers to begin testing your product locally almost immediately.

We’ll explore this below.

State of the Union

When we integrate with external parties, they typically only have one staging/development environment available for us to use, so here’s what most firms settle on.

Before we arrive at our better solution, let’s consider the capabilities of the provider, which will influence our implementation.

Easiest Implementation – Tell me where you want it (Callback URL)

You can define a callback url in the outgoing payload where you expect to receive an incoming webhook. In this implementation, you tend not to set a global webhook URL in an admin panel. Our solution in this case, will look a bit like this. Some examples of APIs that do this well are Google’s Push Notifications as well as Plaid.

More Difficult Implementation – Tell you where to route (Route Yourself)

You can define a pass-through metadata object in the outgoing request which will be present in an incoming webhook (preferably a header, but anything works). With this type of provider, you tend to set global webhook URLs in an admin panel. With this, we’ll need something internally to direct the webhook back to the proper environment.

The level of complexity here is largely dependent on how much risk you’re willing to take. You can implement this by introducing an internal router. You can also implement this by allowing the Development API to redirect to each machine.

Most Difficult Implementation – WTF? (Lookup and Route Yourself)

You can not define any metadata in the payload that will be present in an incoming webhook. This type of provider is similar to Route Yourself, where you tend to set a global webhook in an admin panel. This implementation, similar to the above, requires some internal code that you would not expect to run in higher environments.

The difference here is that you’ll need a lookup table within the internal router, telling it where to route the request based on some properties of the request (usually, an “id” field). It’s important to pick an identifier that does not clash across different developer machines here (such as a uuid).

Lookup and Route Yourself Implementation

You can imagine how this would be problematic if we selected an auto-incrementing userId as our routing identifier, and both My Machine, and Joe’s Machine wrote to the routing table with a userId of 2 like so:

id	resource	identifier (userId)	target
1	user-event	2	https://mymachine.foo/user-event
2	user-event	2	https://joesmachine.foo/user-event

Some Considerations

In larger companies, you may want to run the Lookup and Route Yourself implementation alone, by itself, even if the Callback URL or Route Yourself solutions are available for some providers. This would be done in order to stick to only one paradigm for internal webhooks, reducing the complexity of having multiple different patterns used at once across different integrations.

The tradeoff here is you will introduce unnecessary steps for webhooks from external platforms that could simply integrate with a Callback URL.

A sticking point for implementing this in most organizations is exposing the development machines to the internet (understandably so). This can be mitigated by working behind a VPN (most larger organizations operate this way to begin with in lower environments).

Additionally, you can roll a bit of your own implementation and keep the ingress setup in-house, with some investment and work. A list of open source solutions that bootstrap some of the networking can be found here.

How it works

Let’s scope things out:

You’ll need to expose your local API via a unique url (ngrok is a great option)
For the Callback URL option, we need to pass the webhook URL to the external API (easy enough).
- The external platform will automatically send the webhook back to your machine.
For the Route Yourself option, you’ll need to pass the target (the webhook URL set up through ngrok as your local machine) to the external API in their “metadata” json field.
- All traffic can hit the internal router as illustrated above
- The internal router reads the url in the metadata field you created, and calls the API at that url with the payload in the incoming webhook.
For Lookup and Route Yourself, we’ll need to maintain an extra mapping table for routing. This table will not be used in production.
- When your machine makes a call to the external API, you’ll need to write an identifier that you expect to be present in the webhook on the way back in, to a table alongside a target url (the URL set up through ngrok as your local machine). This is usually a userId or ID field.
- All traffic can hit the internal router as illustrated above.
- The internal router looks up the target url based on the unique identifier we wrote to the table before.
- The internal router calls the target with the payload in the incoming webhook.

There is a caveat here that your local API will have to be up. If there’s some sort of delay in the call and you don’t want to eat errors, you’ll may want to implement a queuing and retry system with something like Kafka or RabbitMQ.

Whenever an event occurs and the API is down, you can queue it up for when the local environment comes back up. You can also just ignore the failing call if it’s not a big deal for your testing use case.

For Route Yourself and Lookup and Route Yourself, we can use either the nonproduction/dev API as the router, or add an extra component in path that the request takes, and use something like a reverse proxy to route. The choice is yours.

Conclusion

In this article, we spoke a bit about what good DX (developer experience) is, why it’s important to have a positive developer experience, and a targeted way that we can improve it – through internal webhook routing for our local development environments.

We spend a ton of time interacting with APIs and webhooks, so we should be able to implement directly against them whenever possible, removing barriers for engineers. Other developers and other teams in your organization will take notice of the improved experience, leading to the development of better products that your users trust.

If you enjoyed this article, you might also enjoy my walkthrough on a typical implementation of idempotent APIs.

Preventing Duplicates with an Idempotence Middleware in AdonisJS

Nick Abbene — Sun, 12 Mar 2023 18:24:39 +0000

This is a practical code walkthrough of Idempotency, including Dockerfiles you can pull down to run and test your code on.

Do note, this article is intended to be illustrative - not production ready code, and we will be taking some shortcuts for the sake of brevity. With that said, let’s get started.

In this post, we’ll be focused on preventing duplicates and fault tolerance through a mechanism called Idempotency. We'll demonstrate this through implementing a REST API, making multiple calls to the same endpoint an idempotent operation.

What is Idempotency?

Idempotence loosely means that no matter how many times we attempt to execute an operation, it will only be processed once when there are multiple identical requests. The same result should be returned to the client.

Idempotence thus allows REST API calls (or any API calls for that matter) to be retried safely. Networks are unreliable and fail. When they do fail, information is lost. An API call being duplicated or not recorded correctly is a nuisance and it leads to poor customer experience at best, and regulatory action at its worst.

An Example

Let’s examine a scenario and ground it in a real life use case.

John tries to make a $5 purchase at his local fast food restaurant. The restaurant has spotty internet.

Scenario 1: The initial POST request fails as the point of sale system tries to connect to a server
Scenario 2: The connection fails while the server is fulfilling the request.
Scenario 3: The call succeeds, but the point of sale system never receives the response from the server.
Scenario 4: The cashier, frustrated with the point of sale system for being so slow, hits the send button twice.

When the first request fails, imagine what the multiple requests might look like:

HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

Without an implementation for idempotent requests, it’s hard to tell what happened with the checkout. Did it fail? Did it succeed? Did it create a duplicated transaction?

While I’ve demonstrated the issue here with a toy example, there are many applications for a solution to this problem. One such example is when you need to update a bank account balance.

I also want to note that this issue compounds itself in distributed systems. Failures are common, and you will need a strategy for handling dropped network connections.

The How

Okay, let’s outline a high level algorithm for how we can handle the issue we noted above and turn this into an idempotent operation.

Before trying to make any external calls that can fail, generate a key - X-Idempotency-Key that will be used for all requests to charge John for his food.

Send a request to the payment processor with X-Idempotency-Key as a header.

If the call fails, retry again with the same key, knowing that the processor will only process any given X-Idempotency-Key once.

The server takes an X-Idempotency-Key and registers an IdempotentRequest

The server processes the transaction as requested.

The server stores the response and response code for that IdempotentRequest

Any subsequent calls with that X-Idempotency-Key (the same idempotency key) will receive the same result as the first call.

The HTTP Requests examined earlier will now look like this instead:

HTTP Headers: 
{ 
  "X-Idempotency-Key": "dbcfc06c-...-0f13e7a987ca"
}
HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

HTTP Headers: 
{ 
  "X-Idempotency-Key": "dbcfc06c-...-0f13e7a987ca"
}
HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

Ah, the server now knows it’s the same transaction because the X-Idempotency-Key header is the same! These are now idempotent operations.

With this said, let’s dive into implementation. If you’re interested in following along or implementing yourself, you can checkout the repo here.

To implement yourself, checkout this commit hash before starting: cde73d369ecf2060d5fca60a89fc7b172229cb11

I'll be linking the GitHub commits as we move along as well.

Implementation

I’m using AdonisJs as my framework of choice, Dockerized with Postgres, but feel free to implement in a language and framework of your choosing.

First let’s get up and running. This will install the dependencies and start our docker containers.

cd api && npm install
cd .. && docker compose up

Next, let’s create our model and migration for the database that will drive this functionality.

npm run ace make:model IdempotentRequest -m

Now that we have our model and migration existing, let’s add the columns we’ll need.

We'll need the responseBody and responseStatusCode to respond directly to the client when we have already handled a specific key. resourcePath is not strictly needed, but you'll likely find it handy.

export default class IdempotentRequest extends BaseModel {
  /// Existing code here ...

  @column()
  public idempotencyKey: string

  @column()
  public resourcePath: string

  @column()
  public responseBody: string

  @column()
  public responseStatusCode: string
}

export default class extends BaseSchema {
  protected tableName = 'idempotent_requests'
  public async up () {
    this.schema.createTable(this.tableName, (table) => {
      // Existing code here ...

      table.string('idempotency_key')
      table.string('resource_path')
      table.jsonb('response_body')
      table.string('response_status_code')
    })
  }

Next up is creating our middleware, and wiring it to Adonis so the framework knows to always check for an Idempotency Key. This code will apply the middleware to every incoming HTTP Request.

node ace make:middleware Idempotency

Then the wiring, registering with Adonis:

Server.middleware.register([
  () => import('@ioc:Adonis/Core/BodyParser'),
  () => import('App/Middleware/Idempotency'),
])

We actually haven't implemented any logic yet, we've just set ourselves up for success. Now... it's time.

Let's grab the X-Idempotency-Key header, as well as the resourcePath. If we don't see an Idempotency Key, we can skip the idempotency logic altogether and still allow the requests to flow through to our controllers (this may not always be desirable behavior if you do not want to allow non idempotent operations).

export default class Idempotency {
  public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {
    const idempotencyKey = request.header('X-Idempotency-Key')
    const resourcePath = request.url(false)

    if (!idempotencyKey) {
      await next()
      return
    }
  }
}

Okay - let's check for an existing IdempotentRequest with the same key, or create one if it doesn't exist yet (because this is the first request!)

public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {
    // Code referenced above...

    const idempotentRequest = await IdempotentRequest.firstOrCreate(
      { idempotencyKey: idempotencyKey },
      {
        idempotencyKey,
        resourcePath: resourcePath,
      }
    )
  }

Not quite at the finish line yet, we need to either save the response body and status code, or return the cached response (if it's not the first request with the X-Idempotency-Key)

public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {
    // Code referenced above...

    // This is a property from the Lucid ORM - $isLocal is true if it was just created with the firstOrCreate.  
    if (idempotentRequest.$isLocal) {
      response.response.on('finish', () => {
        idempotentRequest.responseBody = JSON.stringify(response.getBody())
        idempotentRequest.responseStatusCode = response.getStatus()
        idempotentRequest.save()
      })
      await next()
    } else {
      return response
        .status(idempotentRequest.responseStatusCode)
        .send(idempotentRequest.responseBody)
    }
   }

There's also some housekeeping in this commit, like changing the responseStatusCode type to a number, telling our test runner to migrate our database before the tests run, and writing our first test to make sure the logic works. For the sake of brevity, I'll point you to the commit here.

Alright, now that we’ve written the code, let’s try it out for real.

docker compose up (If you haven't already)
npm run ace migration:run (Behind the scenes, this is just a alias to node ace ... from the AdonisJS documentation, you can see this in the package.json)

In your console, let's test this by making multiple identical requests to make sure that the result is the same, and this is now an idempotent operation.

curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-Idempotency-Key: 333b2841-e854-4ba7-892f-e01787333049" \
   http://127.0.0.1:3333/authorizations

There's also a handy script in the package.json if you'd like to get into the database to see the idempotent_requests table: npm run docker-db.

I also would really encourage you to add some more tests! I have a few more commits to clean things up and flesh out the tests.

Edge Cases

Remember when I said we'll be taking some shortcuts? Yeah... about that.

Let's force some processing time here, and make a second request before the first one completes.

export default class AuthorizationsController {
  public async process({}: HttpContextContract) {
    await this.sleep(5000)
    return {
      id: uuid(),
      status: 'success',
    }
  }
  private async sleep(ms) {
    return new Promise((resolve) => setTimeout(resolve, ms))
  }
}

Run the curl command twice in quick succession (remember to use a new key!):

curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-Idempotency-Key: a-new-idempotency-key" \
   http://127.0.0.1:3333/authorizations

{
    "message": "Invalid status code: null",
    "stack": "RangeError: Invalid status code: null\n    at new NodeError (node:internal/errors:371:5)\n    at ServerResponse.writeHead (node:_http_server:274:11)\n    at Response.flushHeaders (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:422:23)\n    at Response.endResponse (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:113:14)\n    at Response.writeBody (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:193:18)\n    at Response.finish (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:837:32)\n    at Server.handleRequest (/home/node/app/node_modules/@adonisjs/http-server/build/src/Server/index.js:125:26)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
    "code": "ERR_HTTP_INVALID_STATUS_CODE"
}

Now, this is not really the end of the world. This is happening because the second request is seeing the database object before the first request gets a chance to save it's response. If you retry again after our first request succeeds, you'll see all works as expected, in an idempotent manner.

Ideally, in a production system you'll have some sort of retry policy to adhere to, and the client should only be retrying with a duplicate request after a reasonable amount of time (when we'd expect the first request to be complete).

There's some ways around this, depending on the path you want to go down.

Hold on to the second request, looping until the response body and response code is present. This is somewhat resource intensive (as it holds an extra connection open) and dangerous. If your first request never completes, this request (and every request after) will be held until theres a timeout.
Return 429 - Too Many Requests to the client. This is also a bit dangerous, for the same reason as above. If the first request never completes, every request will return a 429.
Return 429 and add a locked_at column. After a set amount of time, we assume the first request failed, and allow the second request to process the request as if it was the first. This is a bit better, but in a real system you'd need to make sure all your external calls (if there are any) are also idempotent-aware.

Let's implement the 3rd option.

npm run ace make:migration alter_idempotent_requests_add_locked_at

export default class extends BaseSchema {
  protected tableName = 'idempotent_requests'

  public async up() {
    this.schema.alterTable(this.tableName, (table) => {
      table.timestamp('locked_at', { useTz: true })
    })
  }

  public async down() {
    this.schema.dropTable(this.tableName)
  }
}

npm run ace migration:run

Then let's modify our logic to accommodate.

public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {

    /** Code above **/

    if (idempotentRequest.$isLocal) {
      response.response.on('finish', () => {
        idempotentRequest.responseBody = JSON.stringify(response.getBody())
        idempotentRequest.responseStatusCode = response.getStatus()
        idempotentRequest.save()
      })
      await next()

      // There is an existing idempotent request.
    } else {
      const thirtySecondsAfterOriginalRequest = idempotentRequest.lockedAt.plus({ seconds: 30 })
      const now = DateTime.fromJSDate(new Date())

      // The first request is not done processing yet!
      if (!idempotentRequest.responseBody && thirtySecondsAfterOriginalRequest > now) {
        return response.status(429)
      } else if (!idempotentRequest.responseBody) {
        response.response.on('finish', () => {
          idempotentRequest.responseBody = JSON.stringify(response.getBody())
          idempotentRequest.responseStatusCode = response.getStatus()
          idempotentRequest.save()
        })
        await next()
        return
      }

      /** Code below **/
  }

Boom! We're done here.

Build Upon It

Some ideas if you’d like to play with this implementation further and build upon it.

Require the same request to be passed with the same key.
Do not allow failures to be retried, store them so each key truly is always the same operation, and each call has the same effect.
Consider implementing webhooks.
Improve your developer experience around webhooks.