Running scheduled jobs / tasks using GitLab CI/CD

Moti Lavian — Sun, 14 Mar 2021 11:40:29 +0000

TL;DR

I use GitLab's CI/CD free feature to automatically run code in the cloud.

Background

Every now and then I come across the need to automate a task (written in code) to be performed periodically, so that it'll run by itself, without my intervention.
Such tasks can be submitting an online form, scraping data from a website, running some script that updates something else online, etc.
I obviously don't want this code to run on my computer, as it is not on and ready all the time, nor do I want to setup and plug in a Raspberry Pi device or alike just for that.

Now, running it periodically sounded like a very simple and straight-forward task, but it turned out to be a bit tricky! After some research (not too much, I admit), I was not able to find a free and easy-to-use tool / service that would allow me to have some piece of code run periodically in the cloud. I did not want to pull out the bigger guns such as AWS or GCP just in order to setup a tiny serverless function either.

Solution

My solution was to turn to GitLab's awesome CI/CD-as-a-service feature - which I know very well already and have used in many other personal / work project.
This feature, in a nutshell, allows you to define a .yml file with your CI/CD pipeline configuration in a repo, and have set it to run in any automatic schedule (or with a trigger). A pipeline is composed of jobs, which run any script you need in any container.

So in that case, all we need to do is:

Create a project in GitLab.
Code the task that we want to perform (in any language / framework).
Set up the GitLab's project's CI/CD so that it will run the correct job with your desired settings (schedule, webhook, manually, etc.).
Move on with your (easier) life.

Create a GitLab project

Sign up if you haven't already, and go to https://gitlab.com/projects/new#blank_project in order to create a new project.
Fill in the relevant details and create the project.

Add the task itself

For this example, let's create this run-me.js file:



console.log('Hi there, I ran and made your life easier!', new Date());

You can either clone the repo that you've just created and commit-push the task's file from your computer, or easily enough add it straight into gitlab.com using their nice "Web IDE":

Commit the changes, and now you have the task's code in the repo - this code will be available inside the container that will run the pipeline.

Set up CI/CD to run your code as you wish

The same way you did in the previous step, add another file called .gitlab-ci.yml.
GitLab offer some useful templates, but in our case, we'll just go ahead with a blank file, and in it add your definition for container image, and what to run in it (including the task you wrote):



image: node:alpine
run-task:
 only:
   - schedules
 script:
   - node ./run-me.js

Notes:

Here I chose node:alpine as the container. Why? node because my code is in Node.js, and :alpine because it's light and fast (quicker to launch). Why not? You may not work with Node.js, and in Alpine Linux you might need to install additional tools / programs that do not come out-of-the-box.
The only: part makes sure that our job only runs in scheduled runs, and not in any push to master (which is the default).
Whatever is under script: are commands that you run in a bash (in Alpine it's sh), already inside the directory of the cloned repo. This is already a script by itself, so if you're a bash expert, go ahead and do what you need directly there.

For testing, you can go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipelines/new and (provided that you commit to master) start a new pipeline manually.

After running, you can go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipelines, and see a new line with the pipeline that has started:

The pipeline will fetch and run the node:alpine image, and then execute our code from run-me.js inside it 💪.

This is already awesome, as you can now run any code inside a container in the cloud!

In order to schedule it, go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipeline_schedules/new, and create a new schedule for this repo's pipeline.
In this example, let's say we want it to be Sunday to Friday at 7:30 AM, which means setting a custom interval pattern as 30 7 * * 0–5. If you need help with cron syntax, check https://crontab.guru.

Save the pipeline schedule and you can expect it to run surely and independently ☁️.

Notes

GitLab's free plan includes 400 CI/CD minutes per month, which should be more than enough for simple tasks running on a lean container (such as Alpine). For example the code in this example takes 22 seconds to run, which means it has ~1,200 runs per month before exceeding the 400 minutes quota. If you need more, you can either upgrade your plan, or setup your own CI/CD runner anywhere you want.
You may want to expose the failure / success of your code run by exiting with a code. So for example, if your code exits with anything that's not 0, it will be pronounced a failure, and from there you can monitor and even get notified about.
Suspending the schedule can be easily done by deactivating the pipeline's schedule.
GitLab CI/CD supports any publicly available docker image, and uses Docker Hub by default. So either use one from there (choose carefully), or even publish your own.
You can have more than one task in one repo - you'll need to play around with different branches, or configure your .gitlab-ci.yml properly.
Refer to GitLab CI/CD's extensive documentation - you can achieve really awesome things with it (especially for bigger serious projects).

DEV Community: Moti Lavian