CRONs are an essential part of web applications, from automatic invoicing, triggering weekly notifications, or triggering search reindex or data backups.
This article covers the CRONs pitfalls (windowing, monitoring, alerting) and an extensive list of deployment and hosting solutions.
Working with CRONs
CRON, by running in the background, comes with multiple challenges.
First, alerting and monitoring tools set up for the API do not apply to CRONs executions that run on a separate process for a long time and can fail silently.
Sentry recently released Sentry Crons that help track if a CRON is running as expected and if any errors occurred; however, we'll see that it requires some setup and custom error handling, like many “self-hosted setups” (ex: push to Slack when catching an exception — then, how to handle OOM errors?).
Finally, deploying and running CRONs comes with many technical challenges:
Scheduling accuracy
Your CRON needs to run at a specific time, but can you tolerate a few seconds of delay?
Also, can you tolerate that some of your CRON occurrences are missed?
Your answer to those questions will determine which managed solution best fits your use case.
Windowed/Fixed executions and execution stacking issues
CRONs such as data export or weekly notifications tend to run slow in time, proportionally to the processed data or sent notifications. How do you ensure that multiple execution occurrences do not overlap with each other? Should you CRON run on a fixed schedule or relative to other occurrences?
We'll see mechanisms to leverage and how some managed solutions prevent execution stacking.
CRON frequency limits
How often does your CRON need to run?
Most solutions will allow a minimum occurrence of 1-10min; however, very few allow a recurrence below 1min.
Long-running CRONs
The longer a CRON runs, the harder it is to manage, with the rise of resource issues (ex: memory), previously covered execution stacking issues, and monitoring issues (ex: do you have live logs access to identify a long-running vs. stuck execution?).
We will cover the numerous approach to Node.js CRONs deployment by evaluating them on the above technical and tooling criteria.
Setup and deploying Node.js CRONs
Self-host an open-source library
NPM hosts some great open-source Node.js CRON libraries: node-cron, with an in-memory scheduler; BullMQ, with a Redis-based scheduler; and Agenda.js, with a MongoDB-based scheduler.
In a nutshell, Agenda offers a more resilient approach by persisting the jobs to disk (through MongoDB). On the other hand, node-cron is the simplest to set up, with no database/storage required.
All three require creating a dedicated entry point to your project to get the workers started. Also, they provide the software but leave you with a hosting choice.
A dedicated dyno on Heroku or process on Fly would be the easiest solution to deploy node-cron, BullMQ, or Agenda for Node.js CRONs in under a few minutes.
Pros & Cons
Using a free open-source library deployed on Heroku or Fly.io's free tier is cost-effective.
Also, BullMQ offers windowed-CRONs that cannot overlap even in case of long-running executions, and Agenda.js features the finest scheduling frequency on the market, with a 5 secs minimum CRON frequency (depending on your MongoDB setup and number of jobs active).
However, you'll need to invest time to handle errors manually (ex: with Sentry Crons), set up proper alerting, and deploy the dashboards offered by BullMQ and Agenda (which lack support for live logs for ongoing executions).
Finally, you might have to manage frequent issues related to self-hosting: memory issues linked to long-running CRONs, database reconnections and connections pooling strategy, graceful shutdown, and CRON versioning.
Conclusion
The self-hosted open-source library approach is relevant for non-business essential use cases (e.g. any failure has no financial impact) or DevOps-capable software engineers.
Let's now take a look at managed Node.js CRONs solutions.
Managed Node.js CRONs configured from a UI
Many solutions offer to configure CRONs directly from a UI:
Render and Heroku will enable you to configure a command to run at a given frequency from a dashboard and guarantee that only one occurrence runs at any given time.
Note that the Heroku scheduler is eventually reliable and only support predefined scheduling values (every 10min, hourly, daily).
Pros & Cons
Heroku Scheduler and Render Cron Jobs make it easy to quickly set up a CRON on an existing script without any deployment or code change.
Both require a paid account ($1/mo minimum on Render, $5/mo minimum on Heroku) and will need some time investment to set up proper error handling, alerting, and monitoring if used for more critical use cases (ex: deactivating expired free trial accounts).
Conclusion
Heroku Scheduler and Render Cron Jobs are great solutions to quickly set up CRONs to automate non-critical existing quick scripts requiring low monitoring (ex: posting a report on Slack daily).
Let's now cover two approaches to managed Node.js CRONs solutions configured from the code.
Serverless Node.js CRONs (Vercel, Netlify)
Netlify released Scheduled Functions in 2022, and Vercel recently released Vercel Cron Jobs, allowing developers to define Cron Jobs directly from within their application's code:
{
"crons": [
{
"path": "/api/cron",
"schedule": "0 5 * * *"
}
]
}
Pros & Cons
Vercel and Netlify's approach brings a better Developer Experience than the previously covered solutions by allowing the CRON function to live within the application and benefiting from CRONs being versioned in code.
However, Serverless constraints are applied to your CRONs: max execution time of 1min to 15 min (on paid plans), connection pool issues for high concurrency scenarios, and lack of support for native dependencies (ex: necessary for PDF document generation).
Conclusion
While being both in beta, Vercel and Netlify CRON offerings are great solutions for lightweight operational or product automation (ex: Updating Stripe subscription quantities, ending email and Slack notifications) that are not compute-intensive and do not require native dependencies.
It definitely offers the best unified-developer experience on Node.js CRONs for Serverless applications.
Managed Node.js CRONs
Inspired by the Framework-defined infrastructure principles of Vercel, Defer offers a managed background jobs platform (CRONs included) for all types of Node.js applications, from serverless to monolith, from express to Next.js:
import { defer } from "@defer/client"
const weeklyBrief = async () => {
// ...
}
export default defer.cron(
weeklyBrief,
"5 0 * * *"
)
Pros & Cons
You can benefit from the Developer Experience of platforms such as Vercel on your monolith Express, RedwoodJS, or Remix app, with long-running executions and native dependencies support to power all your CRON use cases, from lightweight to critical (ex: document or image processing, data-intensive tasks).
Also, you can scale your CRON by transforming them into workflows and leveraging the Defer Console's metrics, logs, and alerting.
Of course, using Defer means dealing with 2 PaaS, one for your application (ex: Vercel or Heroku) and one for your background operations. Still, it can be eased by leveraging the Doppler integration.
Conclusion
As a managed Node.js CRON solution, Defer is a great match for developers and indie hackers looking for a fast time-to-market while choosing a reliable solution that will power all their CRON use cases, even critical.
Node.js CRONs cheatsheet
Lightweight CRON use cases can be set up and deployed using the open-source and self-hosted combo for developers willing to manage their own servers.
Another way to handle Lightweight CRONs, especially on existing scripts, is to leverage UI CRONs solutions such as Render Cron Jobs and Heroku Scheduler.
On the other hand, developers willing to keep a unified experience on Vercel or Netlify might want to use their CRON products.
However, to benefit from both the best developer experience and managed solutions on all stacks, Defer is the fastest way to write resilient Node.js CRON. with multiple challenges.
Top comments (0)