One fine fintech startup wanted to build a webhook payment handler. This article will walk through how my NodeJS code was able to handle webhook deliveries like a piece of cake.
Context
This client lives and breaths Java and it was also related to payment. So he insisted on using Java, multithreading, Thread pools and all of that shit which I did not understand. And he cared too much about getting it done asap! In fact thats why he came to me cause I had built it already with Javascript. But how am I going to make my single threaded Javascript code handle huge traffic?
Objectives
- The webhook payload needs to be validated, transformed and stored into a database.
- The webhook was delivered by a 3rd party, so a success response need to be sent back immediately after the delivery. We can't let the webhook get clogged by traffic!
- It was a startup, we did not want to spend too much time thinking about scaling and load balancing
Brainstorming
The API needs to be super responsive, so my first thought was to process it asynchronously.
We were on AWS, so we could use EC2! but how are we going to handle scaling? I didn't want to get into that mess. Even with ELB(Elastic load balancer), still need to run multiple containers to process them parallelly.
It hit me after a while, that we are actually fine if we have some delay in the processing. We just want to respond back to the webhook immediately and do not want the webhook server to become unresponsive.
So instead of thinking parallel processing, I started to think how to optimise the response TAT. And we were totally fine with processing the requests with a delay if there is more.
Finally the price, we already had an EC2 running for the APIs. So we need to use the same instance but again cant let the webhook bottleneck API. So our only go to is a lambda which is bad for most cases. But thats the best we could do right now.
The final spec
We have the API gateway connected to the Lambda function. The lambda function was on autoscale so it took care of traffic out of the box. The whole role of the function is to accept the request, push the message to SQS and respond back. This took just 100ms once the lambda is warm.
The same EC2 used for APIs run another docker container that polls the queue every 30 seconds for new messages. If it finds any, it dequeues the message and processes it one by one. Any failure happening while processing wont affect the response sent to the 3rd party as its strictly decoupled. The response has already been sent!
Conclusion
I am not sure if this is the best way to build a webhook handler. But this was the quickest, cheapest solution I could come up with. Feel free to criticise and suggest better ideas.
Follow me on Twitter: twitter.com/HarishTeens
Follow me on GitHub: github.com/HarishTeens
Top comments (0)