How to Run Cloud-based Cron Jobs with Semaphore

cron is a Unix job scheduler that runs commands periodically according to a schedule given by the user, known as the crontab. Scheduled jobs can be used to automate routine, repetitive tasks and receive notifications or tasks that you don't want to run for every push.

This tutorial shows how to use Semaphore's workflow-scheduler/cron feature. The step-by-step example creates a project with a super-simple pipeline that runs a bash script to check that a website is up (returning a 200 HTTP response) every 5 minutes.

So that it's clear just how quick and easy Semaphore is to use, the example project in this tutorial is very, maybe excessively, simple. But don't be fooled: Semaphore is widely used to implement Continuous Integration (CI) and Continuous Deployment (CD) flows for enterprise-grade software systems. For example, Semaphore builds Docker images 7x faster than Docker Hub.

Prerequisites

To follow this tutorial, you will need:

• git, a GitHub account and a working knowledge of forking, branches, committing and pushing with git.
• A Semaphore account. You can get one for free at semaphoreci.com.

Let's get started!

Create your Website Healthcheck Repository in GitHub

To get started, create a new repository in GitHub and add just one bash script, run_healthcheck.sh given below, to the repo. The script works as follows:

• Send an HTTP request to an ENDPOINT (Google in the example) using curl.
• Store the HTTP response code in status_code.
• Check the response code and set the script's exit code as follows:
  • 1 if the response code is not 200 (signifying a status of OK); or
  • 0 if the response code is 200.

The script's exit code is arguably the most important detail, as this is what Semaphore will check to determine the success or failure of a job that runs the script.

#!/bin/bash

# run_healthcheck.sh

ENDPOINT=https://www.google.com

status_code=$(curl --write-out %{http_code} --silent --output /dev/null $ENDPOINT)

if [[ "$status_code" -ne 200 ]] ; then
  echo "$ENDPOINT status $status_code"
  exit 1
else
  echo "$ENDPOINT OK"
  exit 0
fi

Create your Semaphore Project and Choose a Starter Workflow

• Log in to Semaphore.
• Use the + New Project button on the top toolbar to add a new project to your account.
• Select your repository from the list.

• Wait for Semaphore to set up the project and connect it to GitHub.
• When presented with the Invite People page, click Continue to workflow setup.

• Choose Single job as your starter workflow.

• In the workflow configuration pane, choose to Customize it.

Customize your Semaphore Workflow

Here's a very quick overview of the main modelling concepts used in Semaphore workflows:

• Ultimately, things happen when Semaphore runs your commands.
• Commands are grouped into sequences within jobs.
• Jobs are collected into blocks, where the jobs can run in parallel.
• Finally, blocks are arranged into a pipeline, which sequences and triggers interdependent blocks.

The Semaphore docs give more detail on the concepts Semaphore uses.

For this tutorial, we want the pipeline to run a command that executes our run_healthcheck.sh script. For that, we need to create a single job in a single block.

• Add a command to execute our script to the existing job.
  • Leave the existing checkout command in place. It is Semaphore's pre-installed script for downloading your code from your Git repository.
  • Append ./run_healthcheck.sh to Job #1.

• Optionally, rename the job to something more meaningful.

• You'll see that, as you rename the job, the pipeline diagram is instantly updated to help you understand the entirety of the configuration.

Run the Workflow and Check the Output

• Click Start first workflow. All workflows execute from version-controlled configuration stored in your repository, so you can always reproduce the result of a workflow. The dialog that pops up allows you customise the branch and commit message that will be pushed to your repository.

• Clicking Looks good, Start will cause Semaphore to push the new .semaphore/semaphore.yml file to the named branch, which it will create if necessary. The pipeline will run and in a few seconds the result will be displayed.

The pipeline, with it's single job, passed! Maybe you, like me, have had the experience where when something seems to work first time, nothing was happening at all? We can check that the job passed for the expected reason by checking the log.

• Simply click on the Healthcheck Job to see the log.

We can see, at line 73, the command that executes the run_healthcheck.sh script. Use the arrow to expand the output and it's clear that the script ran successfully.

Merge the Semaphore Configuration Topic Branch to Master

Semaphore created a topic branch with the YAML configuration files. This allowed us to test the configuration before unleashing it on the master branch. Now that we've verified the pipeline behaviour, we can adopt the pipeline on our master branch, so:

• Use GitHub or git in your shell to merge the new branch (named set-up-semaphore by default) to master.

Schedule Your Pipeline to Run Every 5 Minutes

The configuration we've done so far has created a pipeline that will be run (under Semaphore's default settings) whenever there is a push to the repository. The next stage is for us to schedule the running of the pipeline every 5 minutes.

• Return to your new project's main page by clicking the project name in the Jump to project... drop-down.

• On the project's page, click Schedulers.

• Click New Scheduler.
• Fill in the Name of the Schedule field.
• Set the Branch to master.
• Set the Pipeline file to run field to be the file we created on the branch in the earlier steps (.semaphore/semaphore.yml by default).

• Type a schedule into the When field for every 5 minutes using cron syntax: */5 * * * *. Help with the syntax can be found by following the Crontab Guru link.

• Click Create!

That's it! You're done! To see the result of the scheduled run, return to your new project's main page.

You can see the history of results for a branch by clicking on the branch name.

Conclusion

You've seen how quick and simple it is to set up a Semaphore workflow from scratch and have it run according to a schedule.

Currently, the results are visible from within your Semaphore account. An even more useful setup can be achieved by having Semaphore notify you should the workflow ever detect a failure. To find out more, take a look at this Introduction to Slack Integration and the Slack Notifications docs

Thanks for reading this tutorial! If you'd like to find out more about what Semaphore is being used for, head over to the Continuous Delivery Blog!