Build a Job Queue with Rust Using Aide-De-Camp (Part 4)

This is the fourth and final installment in our series. We continue to run you through creating durable job queues using Zero Assumption's open source solution, Aide-De-Camp, which is built using Rust.

In this post, we'll build cron job support into Aide-De-Camp.

Aide-De-Camp Code

Code from this series is available in our open source Aide-De-Camp GitHub repository.

The Crate is published to crates.io for everyone to use.

Challenges of Making a Cron Job Scheduler

Making a persistent and distributed cron job scheduler is easier said than done. There are many cases to account for that aren't immediately apparent. For example, in a distributed environment, there can only be one scheduler that keeps track of what has and hasn't been scheduled. Another example is the case for smart decision making. If you want to run a scheduler at midnight and the scheduler is off at midnight.

It's straightforward when you have a cluster with a leader with distributed storage. It's different when you're working with just a job queue and job processor.

Let's break down how a cron job scheduler functions.

From a high level, a cron job scheduler performs the following in a continuous loop.

1. Check if there is a job to run
2. If there is a job to run, run that job
3. Sleep for a designated period of time

Leader Election

If you have a single leader instance, this instance is always a leader because that's the only possible outcome of any election. What if you have two or more? Well, if instances don't coordinate between each other then your jobs will get scheduled N times every time, where N is number of schedulers.

One solution can be to have a single scheduler. However, the goal of Aide-De-Camp is to be "serverless" in that there isn't a special server running somewhere.

Aide-De-Camp won't handle the leader election process. This will be done by the crate consumer.

To Aide-De-Camp, the scheduler will take watch::Receiver<IsLeader> and check if the current instance is a leader before proceeding.

Cron Job Storage for Scheduling

We use cron job storage to keep track of what jobs we've scheduled and hold the following.

• When a new job should be scheduled
• When the last job was scheduled
• Whether or not a job is already schedule during the requested job time

Storage enables us to time our cron jobs in a most sophisticated way than checking the schedule on a +/- 1 second cadence, or something similar. Scheduling a short cadenced check would saturate CPU if many tasks were running.

Let's look at in-memory storage for scheduling, which has some downsides.

• With more than one scheduler, all schedulers must agree on what schedules exist
• We can lose some ticks:
  • Imagine there is a single scheduler
  • There is a schedule that should fire at T
  • Instance goes offline at T - x
  • Instance comes back at T + y
  • Scheduler calculated a new T that is the next tick
  • This is fine if this task is scheduled often, but for another task, this will create issues

For Aide-De-Camp, we're going to implement in-memory store and SQLite store for now.

Another reason to have storage is to keep track what we've already scheduled. This is a bit tricker because we want to support multiple strategies to handle overlapping jobs:

• Schedule anyway
• Wait for current job to finish
• Skip

Up Next

This post wraps up our first Aide-De-Camp series!

Follow us here on DEV, Twitter, and LinkedIn for updates on Aide-De-Camp.

Aide-De-Camp is a product of Zero Assumptions.