Scalable CRON job executor for AWS Lambda

Chris More — Thu, 30 Dec 2021 14:41:02 +0000

Running scheduled jobs at scale in the cloud is an interesting problem to solve. In this post I will describe an architecture we came up with at Fusebit that allows us to execute a large number of arbitrarily scheduled CRON jobs in AWS using Lambda, SQS, and CloudWatch.

The problem

Say you are building a large-scale, multi-tenant system where tenants define programmatic jobs to be executed in the cloud on a given schedule. You have a large number of such jobs, each with a potentially different execution schedule.

In the context of Fusebit, we needed a solution to this problem to enable customers of our integration platform to implement and run integration logic on a given schedule. For example, to export new leads from HubSpot to MailChimp every night, or to send a status report based on Jira tickets to Slack every hour.
The non-scalable CRON solution

We have decided to use AWS Lambda to run the customer-provided logic. While Lambda is not an ideal compute layer for all types of integration workloads, in our case it satisfied the functional requirements while providing a convenient way of scaling and isolating workloads of multiple tenants.

Once the infrastructure was in place to run the customer code, the next step was to trigger it on a customer-defined schedule. There are many ways to trigger execution of a Lambda function in AWS. One of them uses scheduled CloudWatch Events. On the surface, CloudWatch Events meet all the requirements. You can define an arbitrary CRON schedule following which a CloudWatch Event will be triggered, and attach a specific Lambda function to execute in response.

Non-scalable CRON executor for AWS Lambda

The problem with using CloudWatch Events in a highly scalable system is the limits. Per AWS region and account, one can only support up to 100 scheduled events (as of this writing), which was not sufficient to support the needs of a highly scalable multi-tenant system. We clearly needed a different approach.
The scalable CRON solution

After looking around for the right building blocks, a lesser known feature of Amazon SQS drew our attention: SQS delay queues. The feature enables a message published to an SQS queue to remain invisible to consumers for a configurable time, up to 15 minutes.

Equipped with this new tool, we have split our CRON processing pipeline by adding SQS delay queues in the middle, and using two additional Lambda functions: a scheduler and an executor.

First, we defined a single scheduled CloudWatch Event that triggered the scheduler Lambda function every 10 minutes, starting at minute 8 of an hour. So the scheduler Lambda was running at minute 8, 18, 28, 38 etc. of every hour.

Scalable CRON executor for AWS Lambda

The purpose of the scheduler Lambda was to consider all scheduled jobs in the system (possibly millions), and select those that were due for execution in the subsequent whole-10-minute interval. For example, if the scheduler Lambda was running at 3:18, it would determine all scheduled job executions that need to occur in the 3:20-3:30 time span. The scheduler Lambda would then enqueue those job definitions to the SQS queue, setting the delayed delivery for each to correspond to the exact intended moment of execution of that job. For example, if a job was to run at 3:24, the scheduler Lambda running at 3:18 would enqueue that job to SQS setting the delayed execution to 6 minutes. The granularity of delayed execution allows the execution time to be set with 1 second precision.

Lastly, the executor Lambda would consume messages from SQS as they are released following their delayed delivery settings. The executor Lambda would then invoke the appropriate customer-defined Lambda function to execute the intended logic.

With this architecture, we were able to use a single CloudWatch Event to schedule a very large number of jobs and support our scheduled integration needs.

Shameless plug

This is just one of many interesting technical problems we are continuously solving when building the integration platform at Fusebit. If you enjoy working on developer-centric products and cracking this type of technical problems on a daily basis and in a good company, we are hiring. Also, if you need code-first integrations and connectors to 3rd-party SaaS platforms, check out our integrations.

Make Git Your API

Chris More — Sat, 23 Oct 2021 03:09:46 +0000

When designing HTTP APIs for your application, it sometimes makes sense to embrace git as a part of the protocol.

If your application operates on data that your users naturally manage using a source control system like git, adding first-class git support to your APIs may be a good idea.

If you choose to add first-class git support to your own APIs, the fusebit/cloud-git project may come in handy. It provides a lightweight, pure JavaScript implementation of the git protocol that enables you to add git endpoints to your Node.js application.

Check our cloud-git and the open source github repo

Fusetunnel - A Fast, Scalable, and Free Tunnel

Chris More — Sat, 23 Oct 2021 03:03:07 +0000

There are many times where we want to share our localhost app with the world, either for testing purposes, to show progress, demo websites or run personal cloud services from your home. Fusetunnel helps you to do this easily! No need to mess with DNS or deploy to have others test out your changes.

Fusetunnel is great for working with browser testing tools like Browserling or external API callback services like Twilio, which require a public URL for callbacks.

Fusetunnel is Fusebit’s version of localtunnel.

The main difference with localtunnel (and other tunnels) is that Fusetunnel is end-to-end encrypted, giving the user greater security, promoting data protection, and preventing unauthorized access to data.