Running scheduled jobs / tasks using GitLab CI/CD

TL;DR

I use GitLab's CI/CD free feature to automatically run code in the cloud.

---

Background

Every now and then I come across the need to automate a task (written in code) to be performed periodically, so that it'll run by itself, without my intervention.
Such tasks can be submitting an online form, scraping data from a website, running some script that updates something else online, etc.
I obviously don't want this code to run on my computer, as it is not on and ready all the time, nor do I want to setup and plug in a Raspberry Pi device or alike just for that.

Now, running it periodically sounded like a very simple and straight-forward task, but it turned out to be a bit tricky! After some research (not too much, I admit), I was not able to find a free and easy-to-use tool / service that would allow me to have some piece of code run periodically in the cloud. I did not want to pull out the bigger guns such as AWS or GCP just in order to setup a tiny serverless function either.

---

Solution

My solution was to turn to GitLab's awesome CI/CD-as-a-service feature - which I know very well already and have used in many other personal / work project.
This feature, in a nutshell, allows you to define a .yml file with your CI/CD pipeline configuration in a repo, and have set it to run in any automatic schedule (or with a trigger). A pipeline is composed of jobs, which run any script you need in any container.

So in that case, all we need to do is:

1. Create a project in GitLab.
2. Code the task that we want to perform (in any language / framework).
3. Set up the GitLab's project's CI/CD so that it will run the correct job with your desired settings (schedule, webhook, manually, etc.).
4. Move on with your (easier) life.

Create a GitLab project

1. Sign up if you haven't already, and go to https://gitlab.com/projects/new#blank_project in order to create a new project.
2. Fill in the relevant details and create the project.

Add the task itself

For this example, let's create this run-me.js file:

console.log('Hi there, I ran and made your life easier!', new Date());

You can either clone the repo that you've just created and commit-push the task's file from your computer, or easily enough add it straight into gitlab.com using their nice "Web IDE":

Commit the changes, and now you have the task's code in the repo - this code will be available inside the container that will run the pipeline.

Set up CI/CD to run your code as you wish

The same way you did in the previous step, add another file called .gitlab-ci.yml.
GitLab offer some useful templates, but in our case, we'll just go ahead with a blank file, and in it add your definition for container image, and what to run in it (including the task you wrote):

image: node:alpine
run-task:
 only:
   - schedules
 script:
   - node ./run-me.js

Notes:

• Here I chose node:alpine as the container. Why? node because my code is in Node.js, and :alpine because it's light and fast (quicker to launch). Why not? You may not work with Node.js, and in Alpine Linux you might need to install additional tools / programs that do not come out-of-the-box.
• The only: part makes sure that our job only runs in scheduled runs, and not in any push to master (which is the default).
• Whatever is under script: are commands that you run in a bash (in Alpine it's sh), already inside the directory of the cloned repo. This is already a script by itself, so if you're a bash expert, go ahead and do what you need directly there.

For testing, you can go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipelines/new and (provided that you commit to master) start a new pipeline manually.

After running, you can go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipelines, and see a new line with the pipeline that has started:

The pipeline will fetch and run the node:alpine image, and then execute our code from run-me.js inside it 💪.

This is already awesome, as you can now run any code inside a container in the cloud!

In order to schedule it, go to https://gitlab.com/YOUR_GITLAB_NAME/YOUR_PROJECT/-/pipeline_schedules/new, and create a new schedule for this repo's pipeline.
In this example, let's say we want it to be Sunday to Friday at 7:30 AM, which means setting a custom interval pattern as 30 7 * * 0–5. If you need help with cron syntax, check https://crontab.guru.

Save the pipeline schedule and you can expect it to run surely and independently ☁️.

---

Notes

• GitLab's free plan includes 400 CI/CD minutes per month, which should be more than enough for simple tasks running on a lean container (such as Alpine). For example the code in this example takes 22 seconds to run, which means it has ~1,200 runs per month before exceeding the 400 minutes quota. If you need more, you can either upgrade your plan, or setup your own CI/CD runner anywhere you want.
• You may want to expose the failure / success of your code run by exiting with a code. So for example, if your code exits with anything that's not 0, it will be pronounced a failure, and from there you can monitor and even get notified about.
• Suspending the schedule can be easily done by deactivating the pipeline's schedule.
• GitLab CI/CD supports any publicly available docker image, and uses Docker Hub by default. So either use one from there (choose carefully), or even publish your own.
• You can have more than one task in one repo - you'll need to play around with different branches, or configure your .gitlab-ci.yml properly.
• Refer to GitLab CI/CD's extensive documentation - you can achieve really awesome things with it (especially for bigger serious projects).

Comments

[Solomon Ekrebe]

This is helpful.