What is developer experience?
In software engineering, developer experience (DX) is a term used to refer to the entire process of developing, implementing and maintaining software and/or web apps. It covers everything from the initial concept and design of a solution, through coding and testing it, to long-term maintenance and troubleshooting.
Developer experience is an important factor when it comes to creating successful applications, as a smooth and positive developer experience can lead to faster development times, improved code quality, and higher productivity. Even if you provide your developers with all the tools they need, it’s still possible to have a poor experience.
While your primary user when it comes to developer experience is your engineers, ultimately, poor developer experience tends to show in the product that your external customers use and consume.
API Integrations
As part of building any system, we spend a lot of time developing APIs for other parts of the system to call. If the system is not large, it may be possible to spin up an entire environment for the entire team.
How do we develop against integrations with external platforms?
The most common route that is taken is creating mocks and triggers for APIs and webhooks that we don’t control. Another alternative is developing off of a contract, creating a Postman collection and integration tests to verify things work as we expect. This works, but is not ideal and can lead to errors popping up later in the development lifecycle due to mismatches between the mocks and the actual functionality.
There is a third, superior alternative that is available to us most of the time, that I haven’t seen commonly used in the wild. Each developer connects directly to the external party’s development environment, and all webhook calls are routed back to their own local machine.
A huge advantage of this (amongst others) is it enables new developers to begin testing your product locally almost immediately.
We’ll explore this below.
State of the Union
When we integrate with external parties, they typically only have one staging/development environment available for us to use, so here’s what most firms settle on.
Before we arrive at our better solution, let’s consider the capabilities of the provider, which will influence our implementation.
Easiest Implementation – Tell me where you want it (Callback URL)
You can define a callback url in the outgoing payload where you expect to receive an incoming webhook. In this implementation, you tend not to set a global webhook URL in an admin panel. Our solution in this case, will look a bit like this. Some examples of APIs that do this well are Google’s Push Notifications as well as Plaid.
More Difficult Implementation – Tell you where to route (Route Yourself)
You can define a pass-through metadata object in the outgoing request which will be present in an incoming webhook (preferably a header, but anything works). With this type of provider, you tend to set global webhook URLs in an admin panel. With this, we’ll need something internally to direct the webhook back to the proper environment.
The level of complexity here is largely dependent on how much risk you’re willing to take. You can implement this by introducing an internal router. You can also implement this by allowing the Development API to redirect to each machine.
Most Difficult Implementation – WTF? (Lookup and Route Yourself)
You can not define any metadata in the payload that will be present in an incoming webhook. This type of provider is similar to Route Yourself, where you tend to set a global webhook in an admin panel. This implementation, similar to the above, requires some internal code that you would not expect to run in higher environments.
The difference here is that you’ll need a lookup table within the internal router, telling it where to route the request based on some properties of the request (usually, an “id” field). It’s important to pick an identifier that does not clash across different developer machines here (such as a uuid).
Lookup and Route Yourself Implementation
You can imagine how this would be problematic if we selected an auto-incrementing userId as our routing identifier, and both My Machine, and Joe’s Machine wrote to the routing table with a userId of 2 like so:
| id | resource | identifier (userId) | target |
|---|---|---|---|
| 1 | user-event | 2 | https://mymachine.foo/user-event |
| 2 | user-event | 2 | https://joesmachine.foo/user-event |
Some Considerations
In larger companies, you may want to run the Lookup and Route Yourself implementation alone, by itself, even if the Callback URL or Route Yourself solutions are available for some providers. This would be done in order to stick to only one paradigm for internal webhooks, reducing the complexity of having multiple different patterns used at once across different integrations.
The tradeoff here is you will introduce unnecessary steps for webhooks from external platforms that could simply integrate with a Callback URL.
A sticking point for implementing this in most organizations is exposing the development machines to the internet (understandably so). This can be mitigated by working behind a VPN (most larger organizations operate this way to begin with in lower environments).
Additionally, you can roll a bit of your own implementation and keep the ingress setup in-house, with some investment and work. A list of open source solutions that bootstrap some of the networking can be found here.
How it works
Let’s scope things out:
- You’ll need to expose your local API via a unique url (ngrok is a great option)
- For the Callback URL option, we need to pass the webhook URL to the external API (easy enough).
- The external platform will automatically send the webhook back to your machine.
- For the Route Yourself option, you’ll need to pass the target (the webhook URL set up through ngrok as your local machine) to the external API in their “metadata” json field.
- All traffic can hit the internal router as illustrated above
- The internal router reads the url in the metadata field you created, and calls the API at that url with the payload in the incoming webhook.
- For Lookup and Route Yourself, we’ll need to maintain an extra mapping table for routing. This table will not be used in production.
- When your machine makes a call to the external API, you’ll need to write an identifier that you expect to be present in the webhook on the way back in, to a table alongside a target url (the URL set up through ngrok as your local machine). This is usually a userId or ID field.
- All traffic can hit the internal router as illustrated above.
- The internal router looks up the target url based on the unique identifier we wrote to the table before.
- The internal router calls the target with the payload in the incoming webhook.
There is a caveat here that your local API will have to be up. If there’s some sort of delay in the call and you don’t want to eat errors, you’ll may want to implement a queuing and retry system with something like Kafka or RabbitMQ.
Whenever an event occurs and the API is down, you can queue it up for when the local environment comes back up. You can also just ignore the failing call if it’s not a big deal for your testing use case.
For Route Yourself and Lookup and Route Yourself, we can use either the nonproduction/dev API as the router, or add an extra component in path that the request takes, and use something like a reverse proxy to route. The choice is yours.
Conclusion
In this article, we spoke a bit about what good DX (developer experience) is, why it’s important to have a positive developer experience, and a targeted way that we can improve it – through internal webhook routing for our local development environments.
We spend a ton of time interacting with APIs and webhooks, so we should be able to implement directly against them whenever possible, removing barriers for engineers. Other developers and other teams in your organization will take notice of the improved experience, leading to the development of better products that your users trust.
If you enjoyed this article, you might also enjoy my walkthrough on a typical implementation of idempotent APIs.
Top comments (0)